Laptop Disaster Recovery

Waffle Background

This is what it's like getting Windows to play "nice" with Linux some times (and vice-versa)

And I don't mean the Linux sub-systems for Windows, but actual dual-boot scenarios where the Microsoft device is sulking a little.

I've been a Surface Pro user for some years and still haven't found anything that combines the flexible form factor & performance as well. I'm a little tempted by the Surface Book range but as I'm pretty tired of the limitations of Windows I'm still a little reluctant to shell out £3k+. The SP network adaptor is very limited so if you do work on network security you'll need that USB Alfa device. 

In my cynical way of thinking I've always thought these laptops were Microsoft's way of forcing the device manufacturers to get off their lazy backsides, spurring them to try something other than the clam shell form.

It's good to see HP and Lenovo making inroads to that effect, however none of the main device manufacturers seem to have come up with anything other than copies. Even now Microsoft seem to be the only vendor still pushing for innovation.

I won't dwell too much on Apple devices - they make great devices but terrible, bland software. However it is based on a variant of OpenBSD (albeit some time distant) and the hardware is fairly good.

The SP4 seems to have maxed out the potential and over the last ten years I've changed from a staunch Windows user & dev to multiplatform. I don't really have faith in Windows for anything other than Xbox, Netflix and Visio - everything else is generally better on Linux. They're both frustrating OS's at times but for altogether different reasons.

Of course the problem is that should you, for example, go on hols to Portugal and drop your SP on the hard stone floor, the screen might shatter and crack along it's entire ... er... surface? I bought my SP3 from John Lewis through my business but later found out that they don't have an instant replacement mechanism. You'll be without the laptop for up to 8 weeks whilst they review and - in all likelihood - replace your Surface. Microsoft are careful to tell you that if you have anything on your device when you pass it to them... it's as good as gone.

The only major problem I've seen with SP - and the one that's driven the need for this blog post - is that these form factors are even more sealed than the legacy clam-shell form. You just can't open them up and replace bits without some knowledge and a lot of faith in your skills.

Instead of JLP, I'd originally got my SP4 direct from the Microsoft store, and just logged in to my MS account to select the device for a support request gets the ball rolling. This cost to me £399 inc. VAT as this device was out-of-warranty. There were problems: the self-print return envelope didn't get sent, I had to find a UPS Collection point where there are seemingly none in my local area. Not earth-shatteringly problematic.

In fact the only delivery delays were at my end - my company uses a postal service to manage inbound mail and there was a complication with the replacement devices' onward travel.

Fresh new SP4 newly unpacked on arrival

Total time including cock-ups? About 3 weeks, would have been a little over 1 with smooth sailing.

Equipment Used

So as I mentioned I'm using a Surface Pro 4, which is dual booted using Ubuntu 16 over Jake Day's superb Linux Kernel build; and Windows 10 Pro.

If you're only working with Windows I'd really recommend using the Windows backup features as they will be far easier to manage. The complexities that follow are due to the dual-boot nature of my devices.

However if you're using another encryption solution (non-MS) and need to ensure Windows doesn't try to be too clever, I'd recommend using a more assured process like the one I'm about to describe.

I was lucky though - although the screen was busted the device actually worked. Mouse & keyboard or Type cover got me around the worst. If the screen is totalled, you can try using a docking station or the mini DisplayPort out to a separate monitor - just hope that you'd already enabled USB boot otherwise you won't be able to get into the UEFI config.

Because of this I've accumulated the following useful trinkets as time has passed:

• a 256Gb - 1Tb SDD - depending on compression and device disk size
• a USB 3.1 external SDD case / caddy
• a USB SD card caddy / reader + something like a 4Gb class 10 SD card; or
• a 4 GB USB data stick
• a USB hub
• a type-C to type-A USB cable for the SDD caddy (likely will be different depending on which caddy you choose)
• a large bag of patience or an appreciation of progress bars
• a spare laptop (if you need to work whilst your primary is in repair); or
• your desktop (if your SP4 screen is busted o
• Surface Type keyboard (best) or USB keyboard (make sure it works with Linux and Windows OotB)

USB SD card reader, SD card, USB hub, type-C cable and Win10 Pro on USB

I of course would like to point out that all software and hardware is fully licensed and legally purchased.

Because I do a lot of things with Raspberry Pi's I tend to have a number of SD cards around and a lack of USB sticks - the SP has an SD card slot and only one USB so you tend to adjust your thinking somewhat. Replace that SD card & reader with a USB stick as suits you.

Taking The Backups

All partitions except the original Windows 10 partition are encrypted, and there's nothing on the Win partitions that provides a poacher with anything useful, so I didn't encrypt the external drive first.

However if you're working on a mobile device without disk encryption - especially for work or where there's anything sensitive on it - you should really protect yourself. If your SP is shafted now though and you need to take a backup before sending it back to the shop, just encrypt the removable SDD.

You will need to ensure that the appropriate boot options are set in your UEFI config. I also disable secure boot as there's a number of security packages that don't work with it on Linux, and it doesn't provide a huge amount more than a deterrent on a Microsoft-made device.

Enable USB, disable PX network & IPv6, finally relock the configuration and ensure a UEFI config password is set 

Please note that all the commands from here on in are based on bash (Linux).

One SDD - I'm going to re-use this in my desktop at some point so went upmarket

I use the Inateck SDD cases - they work well, are robust and nicely designed

If you need to encrypt your external device due to the reasons I've mentioned above, and example would be:
sudo cryptsetup luksFormat --cipher serpent-xts-plain64 --hash whirlpool --key-size 256 --use-random /dev/sdd2

Where:

• use cryptsetup benchmark to list all the ciphers and hashes your machine supports
• I've selected serpent-xts as I don't entirely believe AES is best placed under guidance of NIST
• I've selected whirlpool as a hash algorithm simply to demonstrate you have the option
• /dev/sdd2 just happened to be the device name I was using 

You can add a key file if you like, but just remember you'll have to make that file available on the SD card on the live installer we'll be creating. It may not provide the security on the key file itself that we should employ - a decent horse battery staple will do. Grown-up explanation with big words here if you want it.

This is just one route as LUKS is common to a number of distros. Veracrypt and others are also excellent as well as x-platform. You'll just need to ensure the packages or executables are available in your live installation distro-of-choice.

You'll need another OS to boot into from a USB (or SD if you can make that work) device: I opted for Tails as Ubuntu 16 didn't live boot very well. Rather than tread old ground you can read up on that here.

USB hub holding the USB SD card reader with Tails on it; and the other cable goes to the portable SDD

Actually in that photo is a Surface Bluetooth Keyboard: great keyboard action and layout but needs a re-sync / pair when booting between OS's, and won't pair during boot for things like FDE password entry. Such a shame that it's basically only usable once you've booted into your OS. Avoid. The MS Sculpt Keyboard is USB-dongle and great to use. They've been practising making keyboards for thirty years so no matter what you think of their software they still make arguably the best finger peripherals.

Assuming you've now got Tails on a USB-able device and have prepped a partition on your USB SDD, plug it all in and boot into the Tails loader, making sure you've enabled root.

You can run the live installer with an encrypted file store, but you still need to change the defaults every boot

Once in, open up a root terminal and figure out what name Tails has given to your USB SDD. Among many ways of doing this you can try lsblk. In my case there was only the SP NVME device and the external SDD, so were both obviously distinct.

lsblk example output

You'll need to mount the SDD to allow you to to use dd to capture backups. Create a new folder with a mkdir /media/backup, mount your SDD with something like mount /dev/<name> /media/backup then CD into this folder.

You can run the following command for each partition you have, or just the whole device. I took backups of individual partitions "just in case" too.

WARNING: the dd command is sometimes referred to as the "disk destroyer". It doesn't validate much before executing your request so make absolutely sure you have the right sources and targets!

dd if=/dev/nvme0n1 bs=1024 conv=sync,noerror | gzip nvme0n1.gzip

You can use pipe view in there like this too:

dd if=/dev/nvme0n1 bs=1024 conv=sync,noerror | pv | gzip nvme0n1.gzip

That'll give you an indication of progress if you're so inclined. Once that's complete I'd recommend you wipe the disk with zeros and then randoms. There's documentation on the ArchLinux wiki so I won't duplicate here.

You'll need to do some maths to work out the block size of your actual disk and you don't need to do this if you were using FDE on all your partitions on this disk. I did it anyway as I was sending the device back to vendor and wanted to leave no trace of the types of opsec I employ.

You just don't know who's handling the device once it's returned to vendor.

Restoration

Again, ensure you enable root in the Tails boot config and apply whatever locals you need.

Because we're using a live installer to manage the disk restore, as we did in the backup, we can't guarantee that the device name will be the same. Use lsblk again and take a note of the correct device (use size or number of partitions if you're not sure).

open root terminal and script it like so:

# make a folder to neatly mount the external drive to
mkdir /media/restore
# mount the device again
mount /dev/<device> /media/restore
# change the working directory to reduce typing in the next command
cd /media/restore

I've already got my entire disk image gzipped, so I'm going to decompress that image and write it straight to the disk. I just don't care what the default SP4 install state is and I have a Win10 Pro USB key plus device license if anything goes wrong.

So I used:
gzip -dc nvme0n1.gzip | pv | dd of=/dev/nvme0n1 bs=4096 conv=sync,noerror

The first time I rebooted I nearly panicked... I got nasty error messages and returned to the UEFI config. Removing the boot order lock setting and allowing the device to get used to itself again sorted that out.

So happy to see the GRUB screen after restore. Now for all the updates since the backup was taken...

War Report

So for a device that cost roughly £2k and was out of warranty, and assuming I'd bought all those parts especially for the job; the total damage would have been around £700. As it was, I had all the parts already from various projects over the years.

If you have these devices in a corporate network I'd recommend taking a base image and ensuring your directory services force a network sync for all your user files. Anything else is the responsibility of the users and providing them a device with FDE enabled mitigates most other things. There's tools out there that do this across multiple OS's.

Costs to my company: £399 for the Microsoft part, around £250 for the SDD, and about £50 for the cables, cases and SD card. That's an expensive repair in comparison to some case opening and SDD swapping with a desktop or Dell laptop. It also cost me about an hour to sort out Tails, encrypt my drive and do the backup and about 30 mins restore.

If I'd had the base image and a replacement device that would figure to about 45 mins (including updates and file syncs post system restore), which isn't too bad along with a reduction in hardware & therefore cost.

Glad I'm operational again but this is the Achilles heel of the SP - and devices like it - cost of ownership is very high, as is maintenance and repair.

Took Some Finding

I've had some of my servers report that something has been running updates off-schedule, and it's taken me a good while to figure it out.

Some flavours of Debian - including Raspbian - have no unattended-upgrades service but do apply a cron job which triggers silent package updates.

I run all updates on a specific schedule so I can easily tell the difference in logs & reporting between a breach and an actual update so this isn't appropriate for our use at work. So the first advice I'd supply before using the configuration below is that ensure you have a valid and automated update mechanism to ensure your servers are kept up-to-date.

For example, I often use a custom script which not only does the update but then sends an encrypted message containing information about the update (or other types of jobs).

So with that in mind - and rather than altering package deployed cron scripts - I'd suggest changing (or creating) the /etc/apt/apt.conf.d/10periodic config to add or modify the Periodic apt setting to "disabled like this:

 APT::Periodic::Enable "0";

I suppose I could have put this on Stack Overflow but it's not really a question.

Running VMWare Player on an Ubuntu 16 SP4

I had some minor headaches trying to get a decent hypervisor working on Linux desktop, and figured out a more manageable approach whilst retaining secure boot & UEFI.

One of the initial challenges is actually getting to the download for the VMW Player - rather than the full (paid for) Workstation etc. but can be found here at the time of writing. VMWare seem to have made it far simpler to access than when I first dug it out. NB this is only for non-commercial use, otherwise you'll need the paid-for Pro version.

Had to use Chromium as FF didn't want to play with vmware.com

Of concern for me was the lack of checksum or PGP verification for the download, something VMWare need to work on. The other major annoyance is that every time the kernel is updated this process needs to be repeated.

High-Level Views

The reason you may need to do this is that you've tried to use VMWare Player / Worktation but the networking does not work. After digging into your logs you'll see that the drivers couldn't be loaded at boot time.

Unlike VirtualBox et al VMWare seems a lot more stable on my SP4 i7 16GB, and can run multiple VMs without the need to have their UI windows open. It also seems to handle host-guest device management (e.g. USB) far better.

Personally, I was tired of VB being flakey and am used to VMware and Hyper-V.

Step-by-Step

1. Download the VMWare bundle from the link listed above
2. Apply executable permissions via sudo chmod ug+x <vmw.bundle> 
3. Run the .bundle (it's just a shell script) via sudo ./<vmw.bundle
4. Once the installer has completed you may need a reboot - if you do you'll see systemd errors relating to failed service starts for the vmware.service due to the unsigned network drivers vmmon / vmnet 
5. You'll need to run the kernel module updater - either via GUI or via sudo vmware-modconfig --console --install-all - this ensures that the modules VMWare needs to operate it's core networking capability are available

GUI version of the installer is invoked if you try and run the player at this point

This step should produce a script output ending something like this:
Starting VMware services:
   Virtual machine monitor                                            failed
   Virtual machine communication interface                             done
   VM communication interface socket family                            done
   Blocking file system                                                done
   Virtual ethernet                                                   failed
   VMware Authentication Daemon                                        done
Unable to start services 

Checking the status of system services should show vmware loaded but unable to run.

This indicates that everything is ready for signing now the modules are ready. After step #5 download or clone a copy of this signer script and follow the instructions. You will be asked to create a password during generation, which is then requested during MOK install after you reboot.

Expanding on That

The last item on that list is a bit abrupt but there's a couple of things you must do. Firstly you need to adapt the certificate definition to your own needs.

Change the subject of each of the certificates from "/C=CountyCode/ST=OfficeState/L=OfficeCity/O=Dept/CN=local.yourdomain.ext" on line 9 as appropriate to your specific needs. Ensure that these details are not accessible by anyone other than yourself.

Two certificates are generated - one for each driver. You can simplify to one certificate if you prefer.

Problems

1. Errors during step 5 could mean issues with VMware version and the Linux version. I upgraded to Ubuntu 16.10 which upgraded the kernel. To solve issues in error messages with the VMW kernel module updater download the latest version of the VMware player
2. I found that a reboot was needed between dist-upgrade of Ubuntu and VMware re-sign, otherwise something would get itself tied up in knots and have no effect on the player.