Building and Running Zephyr with ACRN

Zephyr’s is capable of running as a guest under the x86 ACRN hypervisor (see https://projectacrn.org/). The process for getting this to work is somewhat involved, however.

Build your Zephyr App

First, build the Zephyr application you want to run in ACRN as you normally would, selecting an appropriate board:

west build -b acrn_ehl_crb samples/hello_world

Note the kconfig output in build/zephyr/.config, you will need to reference that to configure ACRN later.

The Zephyr build artifact you will need is build/zephyr/zephyr.bin, which is a raw memory image. Unlike other x86 targets, you do not want to use zephyr.elf!

Configure and build ACRN

First you need the source code, clone from:

git clone https://github.com/projectacrn/acrn-hypervisor

Like Zephyr, ACRN favors build-time configuration management instead of runtime probing or control. Unlike Zephyr, ACRN has single large configuration files instead of small easily-merged configuration elements like kconfig defconfig files or devicetree includes. You have to edit a big XML file to match your Zephyr configuration. Choose an ACRN host config that matches your hardware (“ehl-crb-b” in this case). Then find the relavent file in misc/config_tools/data/<platform>/hybrid.xml.

First, find the list of <vm> declarations. Each has an id= attribute. For testing Zephyr, you will want to make sure that the Zephyr image is ID zero. This allows you to launch ACRN with just one VM image and avoids the need to needlessly copy large Linux blobs into the boot filesystem. Under currently tested configurations, Zephyr will always have a “vm_type” tag of “SAFETY_VM”.

Configure Zephyr Memory Layout

Next, locate the load address of the Zephyr image and its entry point address. These have to be configured manually in ACRN. Traditionally Zephyr distributes itself as an ELF image where these addresses can be automatically extracted, but ACRN does not know how to do that, it only knows how to load a single contiguous region of data into memory and jump to a specific address.

Find the “<vm id=”0”>…<os_config>” tag that will look something like this:

<os_config>
    <name>Zephyr</name>
    <kern_type>KERNEL_ZEPHYR</kern_type>
    <kern_mod>Zephyr_RawImage</kern_mod>
    <ramdisk_mod/>
    <bootargs></bootargs>
    <kern_load_addr>0x1000</kern_load_addr>
    <kern_entry_addr>0x1000</kern_entry_addr>
</os_config>

The kern_load_addr tag must match the Zephyr LOCORE_BASE symbol found in include/arch/x86/memory.ld. This is currently 0x1000 and matches the default ACRN config.

The kern_entry_addr tag must match the entry point in the built zephyr.elf file. You can find this with binutils, for example:

$ objdump -f build/zephyr/zephyr.elf

build/zephyr/zephyr.elf:     file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000012:
EXEC_P, HAS_SYMS
start address 0x0000000000001000

By default this entry address is the same, at 0x1000. This has not always been true of all configurations, however, and will likely change in the future.

Configure Zephyr CPUs

Now you need to configure the CPU environment ACRN presents to the guest. By default Zephyr builds in SMP mode, but ACRN’s default configuration gives it only one CPU. Find the value of CONFIG_MP_NUM_CPUS in the Zephyr .config file give the guest that many CPUs in the <cpu_affinity> tag. For example:

<cpu_affinity>
    <pcpu_id>0</pcpu_id>
    <pcpu_id>1</pcpu_id>
</cpu_affinity>

Note that these indexes are physical CPUs on the host. When configuring multiple guests, you probably don’t want to overlap these assignments with other guests. But for testing Zephyr simply using CPUs 0 and 1 works fine. (Note that ehl-crb-b has four physical CPUs, so configuring all of 0-3 will work fine too, but leave no space for other guests to have dedicated CPUs).

Build ACRN

Once configuration is complete, ACRN builds fairly cleanly:

$ make -j BOARD=ehl-crb-b SCENARIO=hybrid

The only build artifact you need is the ACRN multiboot image in build/hypervisor/acrn.bin

Assemble EFI Boot Media

ACRN will boot on the hardware via the GNU GRUB bootloader, which is itself launched from the EFI firmware. These need to be configured correctly.

Locate GRUB

First, you will need a GRUB EFI binary that corresponds to your hardware. In many cases, a simple upstream build from source or a copy from a friendly Linux distribution will work. In some cases it will not, however, and GRUB will need to be specially patched for specific hardware. Contact your hardware support team (pause for laughter) for clear instructions for how to build a working GRUB. In practice you may just need to ask around and copy a binary from the last test that worked for someone.

Create EFI Boot Filesystem

Now attach your boot media (e.g. a USB stick on /dev/sdb, your hardware may differ!) to a Linux system and create an EFI boot partition (type code 0xEF) large enough to store your boot artifacts. This command feeds the relevant commands to fdisk directly, but you can type them yourself if you like:

# for i in n p 1 "" "" t ef w; do echo $i; done | fdisk /dev/sdb
...
<lots of fdisk output>

Now create a FAT filesystem in the new partition and mount it:

# mkfs.vfat -n ACRN_ZEPHYR /dev/sdb1
# mkdir -p /mnt/acrn
# mount /dev/sdb1 /mnt/acrn

Copy Images and Configure GRUB

ACRN does not have access to a runtime filesystem of its own. It receives its guest VMs (i.e. zephyr.bin) as GRUB “multiboot” modules. This means that we must rely on GRUB’s filesystem driver. The three files (GRUB, ACRN and Zephyr) all need to be copied into the “/efi/boot” directory of the boot media. Note that GRUB must be named “bootx64.efi” for the firmware to recognize it as the bootloader:

# mkdir -p /mnt/acrn/efi/boot
# cp $PATH_TO_GRUB_BINARY /mnt/acrn/efi/boot/bootx64.efi
# cp $ZEPHYR_BASE/build/zephyr/zephyr.bin /mnt/acrn/efi/boot/
# cp $PATH_TO_ACRN/build/hypervisor/acrn.bin /mnt/acrn/efi/boot/

At boot, GRUB will load a “efi/boot/grub.cfg” file for its runtime configuration instructions (a feature, ironically, that both ACRN and Zephyr lack!). This needs to load acrn.bin as the boot target and pass it the zephyr.bin file as its first module (because Zephyr was configured as <vm id="0"> above). This minimal configuration will work fine for all but the weirdest hardware (i.e. “hd0” is virtually always the boot filesystem from which grub loaded), no need to fiddle with GRUB plugins or menus or timeouts:

# cat > /mnt/acrn/efi/boot/grub.cfg<<EOF
set root='hd0,msdos1'
multiboot2 /efi/boot/acrn.bin
module2 /efi/boot/zephyr.bin Zephyr_RawImage
boot
EOF

Now the filesystem should be complete

# umount /dev/sdb1
# sync

Boot ACRN

If all goes well, booting your EFI media on the hardware will result in a running ACRN, a running Zephyr (because by default Zephyr is configured as a “prelaunched” VM), and a working ACRN command line on the console.

You can see the Zephyr (vm 0) console output with the “vm_console” command:

ACRN:\>vm_console 0

----- Entering VM 0 Shell -----
*** Booting Zephyr OS build v2.6.0-rc1-324-g1a03783861ad  ***
Hello World! acrn