.. _zephyr-audio-dsp-development-on-chromebooks: Zephyr Audio DSP Development on Chromebooks ########################################### The Audio DSP on Intel Chromebooks is configured to use the SOF "Community" key for firmware signing, and can therefore accept arbitrary user-developed firmware like Zephyr applications (of which SOF is one), including the Zephyr samples and test suite. Initial TGL Chromebook Setup **************************** (These instructions were written specifically to the Asus Flip CX5 device code named "delbin". But they should be reasonably applicable to any recent Intel device.) Power the device on and connect it to a wireless network. It will likely want to download a firmware update (mine did). Let this finish first, to ensure you have two working OS images. Enable Developer Mode ===================== Power the device off (menu in lower right, or hold the power button on the side) Hold Esc + Refresh (the arrow-in-a-circle "reload" key above "3") and hit the power key to enter recovery mode. Note: the touchscreen and pad don't work in recovery mode, use the arrow keys to navigate. Select "Advanced Options", then "Enable Developer Mode" and confirm that you really mean it. Select "Boot from Internal Storage" at the bootloader screen. You will see this screen every time the machine boots now, telling you that the boot is unverified. Wait while the device does the required data wipe. My device takes about 15 minutes to completely write the stateful partition. On reboot, select "Boot from Internal Storage" again and set it up (again) with Google account. Make a Recovery Drive ===================== You will at some point wreck your device and need a recovery stick. Install the Chromebook Recovery Utility from the Google Web Store and make one. You can actually do this on any machine (and any OS) with Chrome installed, but it's easiest on the Chromebook because it knows its device ID (for example "DELBIN-XHVI D4B-H4D-G4G-Q9A-A9P" for the Asus Tiger Lake board). Note that recovery, when it happens, will not affect developer mode or firmware settings but it **will wipe out the root filesystem and /usr/local customizations you have made**. So plan on a strategy that can tolerate data loss on the device you're messing with! Make the root filesystem writable ================================= For security, ChromeOS signs and cryptographically verifies (using Linux's dm-verity feature) all access to the read-only root filesystem. Mucking with the rootfs (for example, to install modules for a custom kernel) requires that the dm-verity layer be turned off: First open a terminal with Ctrl-Alt-T. Then at the "crosh> " prompt issue the "shell" command to get a shell running as the "chronos" user. Finally (in developer mode) a simple "sudo su -" will get you a root prompt. .. code-block:: console crosh> shell chronos@localhost / $ sudo su - localhost ~ # Now you need to turn of signature verification in the bootloader (because obviously we'll be breaking whatever signature existed). Note that signature verification is something done by the ROM bootloader, not the OS, and this setting is a (developer-mode-only) directive to that code: .. code-block:: console cros# crossystem dev_boot_signed_only=0 (*Note: for clarity, commands in this document entered at the ChromeOS core shell will be prefixed with a hostname of cros.*) Next you disable the validation step: .. code-block:: console cros# /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification **THIS COMMAND WILL FAIL**, give you an error that you are changing the setting for the entire running system, and suggest an alternative "--partitions X" argument to use that modifies only the currently used partition. Run that modified command, then reboot. After rebooting, you will notice that your chromebook boots with the raw storage device (e.g. /dev/nvme0n1p5) mounted as root and not the "dm-0" verity device, and that the rootfs is read-write. Note: What this command actually does is modify the command line of the installed kernel image (it saves a backup in /mnt/stateful_partition/cros_sign_backups) so that it specifies "root=" and not "root=dm-0". It does seem to leave the other verity configuration in place though, it just doesn't try to mount the resulting (now-invalid!) partition. Metanote: The astute will note that we're probably going to throw this kernel out, and that we could probably have just edited the command line of the new kernel instead of flashing and rebooting into this modified one. But that's too many balls to juggle at once for me. Enable ChromeOS SSH =================== Once you are booted with a writable partition, you can turn on the built-in ssh server with: .. code-block:: console cros# /usr/libexec/debugd/helpers/dev_features_ssh By default neither the "chronos" user nor root accounts have passwords, so unless you want to type a ssh key in by hand, you probably want to set a password for the first login (before you run ssh-copy-id, of course): .. code-block:: console cros# passwd Now ssh into the chromebook and add your key to ``.ssh/authorized_keys`` as you do for any Linux system. Install Crouton *************** The Zephyr integration tools require a proper Linux environment and won't run on ChromeOS's minimal distro. So we need to install a Linux personality. **DO NOT** bother installing the "Linux Development Environment" (Crostini) from the ChromeOS Developer settings. This personality runs inside a VM, where our tools need access to the real kernel running on the real hardware. Instead install Crouton (https://github.com/dnschneid/crouton), which is a community chroot-based personality that preserves access to the real hardware sysfs and /dev filesystem. These instructions install the "cli-extra" package list, there are X11-enabled ones available too if you prefer to work on the device screen directly. See the project page, etc... At a root shell, grab the installer and run it (note: /usr/local is the only writable filesystem without noexec, you must place the binary there for it to run!): .. code-block:: console cros# mkdir -p /usr/local/bin cros# curl -L https://github.com/dnschneid/crouton/raw/master/installer/crouton \ > /usr/local/bin/crouton cros# chmod 755 /usr/local/bin/crouton cros# crouton -r focal -t cli-extra Start the Crouton chroot environment: .. code-block:: console cros# startcli Now you are typing commands into the Ubuntu environment. Enable inbound ssh on Crouton, but on a port other than 22 (which is used for the native ChromeOS ssh server). I'm using 222 here (which is easy to remember, and not a registered port in /etc/services): .. code-block:: console crouton# apt install iptables openssh-server crouton# echo "Port 222" >> /etc/ssh/sshd_config crouton# mkdir /run/sshd crouton# iptables -I INPUT -p tcp --dport 222 -j ACCEPT crouton# /usr/sbin/sshd (*As above: note that we have introduced a hostname of "crouton" to refer to the separate Linux personality.*) NOTE: the mkdir, iptables and sshd commands need to be run every time the chroot is restarted. You can put them in /etc/rc.local for convenience. Crouton doesn't run systemd (because it can't -- it doesn't own the system!) so Ubuntu services like openssh-server don't know how to start themselves. Building and Installing a Custom Kernel *************************************** On your build host, grab a copy of the ChromeOS kernel tree. The shipping device is using a 5.4 kernel, but the 5.10 tree works for me and seems to have been backporting upstream drivers such that its main hardware is all quite recent (5-6 weeks behind mainline or so). We place it in the home directory here for simplicity: .. code-block:: console dev$ cd $HOME dev$ git clone https://chromium.googlesource.com/chromiumos/third_party/kernel dev$ cd kernel dev$ git checkout chromeos-5.10 (*Once again, we are typing into a different shell. We introduce the hostname "dev" here to represent the development machine on which you are building kernels and Zephyr apps. It is possible to do this on the chromebook directly, but not advisable. Remember the discussion above about requiring a drive wipe on system recovery!*) Note: you probably have an existing Linux tree somewhere already. If you do it's much faster to add this as a remote there and just fetch the deltas -- ChromeOS tracks upstream closely. Now you need a .config file. The Chromebook kernel ships with the "configs" module built which exposes this in the running kernel. You just have to load the module and read the file. .. code-block:: console dev$ cd /path/to/kernel dev$ ssh root@cros modprobe configs dev$ ssh root@cros zcat /proc/config.gz > .config You will need to set some custom configuration variables differently from ChromeOS defaults (you can edit .config directly, or use menuconfig, etc...): + ``CONFIG_HUGETLBFS=y`` - The Zephyr loader tool requires this + ``CONFIG_EXTRA_FIRMWARE_DIR=n`` - This refers to a build directory in Google's build environment that we will not have. + ``CONFIG_SECURITY_LOADPIN=n`` - Pins modules such that they will only load from one filesystem. Annoying restriction for custom kernels. + ``CONFIG_MODVERSIONS=n`` - Allow modules to be built and installed from modified "dirty" build trees. Now build your kernel just as you would any other: .. code-block:: console dev$ make olddefconfig # Or otherwise update .config dev$ make bzImage modules # Probably want -j for parallel build The modules you can copy directly to the (now writable) rootfs on the device. Note that this filesystem has very limited space (it's intended to be read only), so the INSTALL_MOD_STRIP=1 is absolutely required, and you may find you need to regularly prune modules from older kernels to make space: .. code-block:: console dev$ make INSTALL_MOD_PATH=mods INSTALL_MOD_STRIP=1 modules_install dev$ (cd mods/lib/modules; tar cf - .) | ssh root@cros '(cd /lib/modules; tar xfv -)' Pack and Install ChromeOS Kernel Image ====================================== The kernel bzImage file itself needs to be signed and packaged into a ChromeOS vboot package and written directly to the kernel partition. Thankfully the tools to do this are shipped in Debian/Ubuntu repositories already: .. code-block:: console $ sudo apt install vboot-utils vboot-kernel-utils Find the current kernel partition on the device. You can get this by comparing the "kernel_guid" command line parameter (passed by the bootloader) with the partition table of the boot drive, for example: .. code-block:: console dev$ KPART=`ssh root@cros 'fdisk -l -o UUID,Device /dev/nvme0n1 | \ grep -i $(sed "s/.*kern_guid=//" /proc/cmdline \ | sed "s/ .*//") \ | sed "s/.* //"'` dev$ echo $KPART /dev/nvme0n1p4 Extract the command line from that image into a local file: .. code-block:: console dev$ ssh root@cros vbutil_kernel --verify /dev/$KPART | tail -1 > cmdline.txt Now you can pack a new kernel image using the vboot tooling. Most of these arguments are boilerplate and always the same. The keys are there because the boot requires a valid signature, even though as configured it won't use it. Note the cannot-actually-be-empty dummy file passed as a "bootloader", which is a holdover from previous ROM variants which needed an EFI stub. .. code-block:: console dev$ echo dummy > dummy.efi dev$ vbutil_kernel --pack kernel.img --config cmdline.txt \ --vmlinuz arch/x86_64/boot/bzImage \ --keyblock /usr/share/vboot/devkeys/kernel.keyblock \ --signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk \ --version 1 --bootloader dummy.efi --arch x86_64 You can verify this image if you like with "vbutil_kernel --verify". Now just copy up the file and write it to the partition on the device: .. code-block:: console $ scp kernel.img root@cros:/tmp $ ssh root@cros dd if=/tmp/kernel.img of=/dev/nvme0n1p4 Now reboot, and if all goes well you will find yourself running in your new kernel. Wifi Firmware Fixup =================== On the Tiger Lake Chromebook, the /lib/firmware tree is a bit stale relative to the current 5.10 kernel. The iwlwifi driver requests a firmware file that doesn't exist, leading to a device with no network. It's a simple problem, but a catastrophic drawback if uncorrected. It seems to be sufficient just to link the older version to the new name. (It would probably be better to copy the proper version from /lib/firmware from a recent kernel.org checkout.): .. code-block:: console cros# cd /lib/firmware cros# ln -s iwlwifi-QuZ-a0-hr-b0-62.ucode iwlwifi-QuZ-a0-hr-b0-64.ucode Build and Run a Zephyr Application ********************************** Finally, with your new kernel booted, you are ready to run Zephyr code. Build rimage Signing Tool ========================= First download and build a copy of the Sound Open Firmware "rimage" tool (these instructions put it in your home directory for clarity, but anywhere is acceptable): .. code-block:: console dev$ cd $HOME dev$ git clone https://github.com/thesofproject/rimage dev$ cd rimage/ dev$ git submodule init dev$ git submodule update dev$ cmake . dev$ make Copy Integration Scripting to Chromebook ======================================== There is a python scripts needed on the device, to be run inside the Crouton environment installed above. Copy them: .. code-block:: console dev$ scp boards/xtensa/intel_adsp_cavs15/tools/cavstool.py user@crouton: Then start the service in the Crouton environment: .. code-block:: console crouton$ sudo ./cavstool.py user@crouton: Build and Sign Zephyr App ========================= Zephyr applications build conventionally for this platform, and are signed with "west flash" with just a few extra arguments. Note that the key in use for the Tiger Lake DSP is the "3k" key from SOF, not the original that is used with older hardware. The output artifact is a "zephyr.ri" file to be copied to the device. .. code-block:: console dev$ west build -b intel_adsp_cavs25 samples/hello_world dev$ west sign --tool-data=~/rimage/config -t ~/rimage/rimage -- \ -k $ZEPHYR_BASE/../modules/audio/sof/keys/otc_private_key_3k.pem Run it! ======= The loader script takes the signed rimage file as its argument. Once it reports success, the application begins running immediately and its console output (in the SOF shared memory trace buffer) can be read by the logging script. .. code-block:: console dev$ west flash --remote-host crouton Hello World! intel_adsp_cavs25 Misc References *************** Upstream documentation from which these instructions were drawn: This page has the best reference for the boot process: http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format This is great too, with an eye toward booting things other than ChromeOS: https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/custom-firmware