Correcting VirtualBox raw disk VMDK file after resizing a partition

Question

I have a Windows 8.1 installation on second partition of my second HDD (/dev/sdb2 in Ubuntu) created using the command

VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2

Everything worked just fine - Windows installation was runable from VirtualBox and even bootable normally from GRUB. Last time when installing some software in Windows (PC booted up directly into Windows), I discovered there's not enough space on the system partition (/dev/sdb2) and enlarged it by 15 GB that were left spare on the HDD.

These changes made the Windows installation unusable in VirtualBox of course - it fails to boot offering some repair options. The first thing which I realized that is needed to do was enlarging the partition in the VMDK file, so I backed up the old sdb2.vmdk and sdb2-pt.vmdk files and recreated them with the same command as before.

This, however, made no change, because sdb2-pt.vmdk seems to be storing the boot record (MBR in my case, currently with GRUB) and some more stuff needed for Windows to work properly. My next attempt was replacing the new sdb2-pt.vmdk with the old one (with Windows bootloader and perhaps the old partition table) - this didn't work either.

How to update the VMDK files with the new partition size to make the enlarged Windows 8.1 installation bootable from VirtualBox again?

Answer 1

On github there is a tool that will do that automatically (and re-running with same options will update its vmdk and auxiliary files, so you can change partitions too later) https://github.com/vasi/vmdk-raw-parts

Answer 2

I have finally found the solution myself. Since the VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2 command produces two valid files based on the current disk structure, the only change needed was to recover the Windows boot loader from the old sdb2-pt.vmdk file which is a rather straightforward process. If you only wish to learn the recovery steps, you can skip the following theoretical part.

Some background information on the VMDK file format

VMWare Disk Format (VMDK) consists of two files - a descriptor file (sdb2.vmdk in the original question) and an extent file (sdb2-pt.vmdk). Their internal structure is well defined in the specification from VMWare. I'll sum up the most important parts:

The descriptor file (sdb2.vmdk) contains a section annotated # Extent description which can look something like this:

# Extent description
RW 63 FLAT "sdb2-pt.vmdk" 0
RW 41943040 ZERO 
RW 83886080 FLAT "/dev/sdb" 58722304
RW 2 FLAT "sdb2-pt.vmdk" 63
RW 1191843568 ZERO 

One extent description (a row from those above) has the following structure:

Access  Size in sectors  Type of extent  Filename  (Offset)

The offset parameter (specified only for FLAT type extents) specifies the offset (in sectors) of the given extent within the file Filename. Notice that file sdb2-pt.vmdk consists of two extents, the first 63 sectors long and the second only 2 sectors long.

The FLAT extent file sdb2-pt.vmdk is a raw data binary file identical to one you would obtain e.g. using the dd command on Unix-like systems. Since the sector size was 512 bytes in my case (I don't know if this is a general rule), the sdb2-pt.vmdk file (based on the new disk partitioning described in the extent description above) was (63+2)*512 bytes long.

Now to the second extent (the one with only 2 sectors in size). This is a padding extent which arose in my new partition table after enlarging the Windows partition (third extent in the description table). Since my previous partition table did not contain any such padding, the old sdb2-pt.vmdk file only contained the first 63 sectors long extent and thus was 1 024 bytes smaller than the new one generated by the VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2 command. This obviously rendered the old extent file and the new one incompatible.

The recovery process

Please be aware that the following steps apply to the old MBR disk structure only!

You surely want to keep the new partition structure and to propagate any changes made in the partition table to the VMDK file. Proceed with these steps:

• Backup your old description file (sdb2.vmdk) and extent file (sdb2-pt.vmdk). In the following steps, you will only need the second one but you never know what else could happen.

• Generate new descriptor and extent files issuing the command:

  VBoxManage internalcommands createrawvmdk -filename sdb2.vmdk -rawdisk /dev/sdb -partitions 2

• Now, the first extent entry in your new description file (sdb2.vmdk) should look like this:

  RW ## FLAT "sdb2-pt.vmdk" 0

  With the knowledge that you want to keep the new partition table (with everything what follows) and only restore the Windows boot loader stored in the backed up extent file (old sdb2-pt.vmdk), you have to copy the first 440 bytes (boot loader) from the old extent file to the new one. This can either be done with a hex editor (copy all values starting from address 0x0 up to 0x1B8 exclusive) or on a Unix-like system using the command:

  dd if=old-sdb2-pt.vmdk of=sdb2-pt.vmdk bs=1 count=440

• Violà.