How do I create VMware or VirtualBox virtual machine from Balena app image?

Question

Given a Balena app that can be downloaded as an image and flashed to a device using a USB stick, how can it be used to create a virtual machine? The idea is that it could be distributed to others as an alternative to shipping a physical device.

Answer 1

There are two main parts to achieving this:

1. Download Balena device image (Intel NUC) and optionally pre-load it with an existing app
2. Build a virtual machine image based on that image

Download an Intel-NUC based Balena device image

• Create an app if you haven't already on https://dashboard.balena-cloud.com/
• Download the production version of the image (about 600mb)

Install a very specific version of Docker that still supports AUFS

• Windows: https://docs.docker.com/docker-for-windows/release-notes/#docker-community-edition-18061-ce-win73-2018-08-29

• Mac: https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18061-ce-mac73-2018-08-29

• Open the Docker Desktop settings panel and tick the checkbox Expose daemon on tcp://localhost:2375 without TLS. Also go to the Shared Drives settings and be sure to share the C drive else you'll get a drive not shared error when running balena preload.

Pre-load the Balena image with your app (optional)

• Install the Balena cli: https://github.com/balena-io/balena-cli

Docker must be running properly for the next step to work.

• Run balena preload <img name>.img --app <app #> --splash-image logo.png

Convert the .IMG file to .VMDK

The image needs to be mounted in the virtual machine, similar to plugging in a USB drive, however you can't mount the .img file directly. Unfortunately VMware doesn't have a utility for this (please comment if so) but VirtualBox does.

It must be converted to a .vmdk (virtual disk) using the VBoxManage convertdd command.

• Run VBoxManage convertdd <img name>.img <img name>.vmdk --format VMDK

Create a VMware virtual machine image

Note: This appears to require VMware Workstation 15 for VMDK compatibility, however there could be ways to get it working with earlier versions if you're ambitious.. perhaps changing the VMDK file type?

You have to modify the VMDK file type from 5 (streamOptimized) to 2 (monolithicFlat). It doesn't matter here, but types 0, 1, and 3 would also work. This is not needed for VirtualBox.

• Run vmware-vdiskmanager -r <source>.vmdk -t 2 <target>.vmdk

This creates a -flat.vmdk file that's 4x the size of the original. Note: We're going to attach the 1kb .vmdk file that references that -flat file, not the -flat file itself.

Open VMware Workstation (you'll need a license or trial version) and create a new Virtual Machine. Use Custom (advanced) otherwise you won't get the option to specify disk controller type or select an existing disk.

• Install the operating system later
• Linux / Debian 6 (64-bit)
• 4GB Disk, store virtual disk as a single file
• Use default settings until you get to Select a Disk Type - choose SATA as SCSI doesn't support AHCI and it will not boot properly.
• At Select a Disk - choose 'Use an existing virtual disk'
• Choose the 1kb VMDK file (non-flat) that was created earlier. It may prompt to convert it to the latest version - go ahead, it doesn't matter unless you plan on using it in previous versions of VMWare.

Click finish, then edit the virtual machine settings. You need to add a Click Add on the Hardware tab and select Hard Disk.

• Choose SATA > Create a new virtual disk
• 4 GB, Store virtual disk as a single file

Note: It is critical that the original Balena VMDK is set to SATA 0:0 otherwise it will not boot properly. It should be already configured as such, but you can click the Advanced button to double check.

If the virtual machine does not boot (ie. gets stuck at a network boot prompt) go to the Settings of the virtual machine > Options tab > Advanced > Firmware type and make sure it is set for UEFI (not BIOS). This should fix the problem.

Click the Options tab and go to Advanced - change the Firmware type from BIOS to UEFI. If this is not done it will not boot to the Balena VMDK.

Skip the next section.

Create a VirtualBox virtual machine image

• Open VirtualBox and create a new Virtual Machine
• Type: Linux
• Version: Debian (64-bit)
• Memory: at least 2GB
• Hard disk: Create a virtual hard disk:
  • VDI
  • Dynamically allocated
  • 4GB in size

You should now have a virtual machine created. Edit the settings:

• System:
  • Uncheck Enable I/O APIC (will not boot otherwise)
  • Check Enable EFI
• Storage:
  • You will see the virtual disk you created earlier under Controller: SATA
  • Click Controller: SATA and add a new hard disk
  • Choose existing disk
  • Click Add (you will end up with 2 different disks attached)
  • Select the pre-loaded .vmdk file that was created earlier - this is the disk that the virtual machine will boot from and flash the other disk with the Balena image
  • Click Choose and you should now see the two disks attached to Controller: SATA
• Network:
  • This should default to NAT which is what you want

Provision the virtual machine with the Balena image

Now you should have the original VMDK attached for boot and a new virtual disk to provision with the Balena image. Start the virtual machine and open up the Balena dashboard to see it come online (https://dashboard.balena-cloud.com) Once it appears, you should name it something meaningful.

Note: If you see Network PXE as the bootloader and it just hangs, it means you likely haven't changed BIOS setting to UEFI.

The device will enter a Post-Provisioning state. Do not interrupt this process - it will take a few minutes and the virtual machine will power off. Be sure to let it power off on it's own. If it does not turn off, or if you accidentally turn it off, delete the device from Balena and start the virtual machine again - it should provision a new device.

Important: Once it is powered off, go to the virtual machine settings and remove the Balena VMDK disk, otherwise you will re-flash a new image every time you boot. This is the equivalent of removing the USB stick from the device. You should only have one virtual disk attached to Controller: SATA.

At this point you have a virtual machine image that is the equivalent of a physical device. Now the image can be packaged as a virtual appliance so that it can be easily distributed (and users can just double click on and load into VirtualBox or VMware).

Create a virtual appliance from a Balena image

Open VirtualBox or VMware (the process is the same):

• Click File > Export
• Be sure to create an .ova file, as this combines everything into a single file (for VMware, you can manually type in the extension in the filename save field). An .ovf file will separate out the .vmdk so would require a .zip file to redistribute easily.

Click Export and that's it!

One thing to note: If you send this appliance to 10 people, there will still only be 1 device in Balena, so you may want to create 10 separate appliances instead, which then provision 10 separate devices in Balena.