Building a Linux distribution for the Raspberry Pi 3 using Yocto

What is Yocto?

The Yocto Project is an open source collaboration project that provides templates, tools and methods to help you create custom Linux-based systems for embedded products regardless of the hardware architecture. It was founded in 2010 as a collaboration among many hardware manufacturers, open-source operating systems vendors, and electronics companies to bring some order to the chaos of embedded Linux development.

The citation above is taken from the Yocto project's website and the gist of the project is to provide a system for building an entire Linux distribution for an embedded system. It builds everything from the kernel to the user space.

Why use Yocto?

What is so great about Yocto? Why not install an existing distribution for the Raspberry Pi?

The point of Yocto is taking control of exactly what is included in the system, down to every single system utility, if desirable. Something that can be vital on a system that has stricter hardware constraints than the comparably powerful Raspberry Pi 3 card.

Another strength is that since everything is built from source, Yocto provides simple mechanisms for applying patches and configuring the existing code. When developing highly integrated software on custom hardware, it is, for example, often necessary to modify the configuration of the Linux kernel, patching the kernel and integrating custom drivers.

If the system is shipped to someone else that wants to develop programs for it, Yocto supports building an SDK for the system. This allows easily exporting all the tools needed by the third party and the SDK provides everything from libraries to a cross compilation environment.

When a high level of control is wanted for a system that is tailored to a specific task, that is when Yocto shines. In other cases, when the system does not need as much tailoring, it is probably a better idea to install a normal distribution, using the package manager to get the software needed.

Structure of a Yocto project

A Yocto project is organised into layers, with each layer providing recipes that describe how the software is to be configured, built and installed. These recipes are written in a combination of Python and syntax that approximates shell scripts.

poky

Yocto itself has all its base functionality in a layer called "poky". It provides scripts for configuring the build environment, base classes for recipes to inherit from and generally defines the Yocto build environment.

meta-openembedded

The meta-openembedded layer is a core layer that provides a common ground for systems built using Yocto. It provides basic recipes for software that is common among a lot of systems, ex. the Linux kernel.

You can read more about openembedded at the Yocto website.

BSP layer

BSP is short for Board Support Layer and it is the layer that adapts a Yocto build to a specific hardware. It provides vendor specific configuration of the kernel, device drivers and other hardware specific software needed to make the system run.

Raspberry Pi 3 BSP

The Yocto project provides a BSP layer for the Raspberry Pi 3 that allows quickly building a bootable image for the card.

The BSP layer is freely available on the Yocto project's git from where it can be cloned. It's dependencies are listed on the same page and, at the time of writing this, it is only dependent on poky and meta-openembedded.

Labs

This series of labs aims to give an introduction to using Yocto for building embedded systems. An image will be built for the Raspberry Pi 3, modified to include a driver for Wi-Fi and modified to add a custom recipe.

These labs are meant to be done on Lysator's workstations; thus, all dependencies needed to build the software should already be installed. If Yocto refuses to build due to dependency issues, contact a root or Andreas Kempe <kempe@lysator.liu.se>, who is responsible for Inbyggda hack, to get help installing the missing software.

Getting the source

The first task, to get started, is to acquire the sources needed to build our system. To build the system for the Raspberry Pi 3, the meta-raspberrypi layer as well as its dependencies are needed. Start by creating the following directory structure to contain the code:

mkdir -p ~/yocto/sources
# Yocto creates a whole lot of files. At least on Lysators
# systems it's a good idea to exclude them from the backup.
touch ~/yocto/sources/.nobackup

After the command above has been run, there should be a folder in the home directory named Yocto, with one subfolder named sources. The sources folder is where all the different meta-layers will live. They are put in a subfolder because it is nice to keep them separate if automation scripts for building are created at the top-level.

As the storage folder has been created, it is now time to clone the required source code. The Yocto core layer, i.e. poky, can be found at the Yocto download page and the lab will be using the version with tag yocto-2.4. As a reminder, poky contains the core part of Yocto and provides much of the framework used for writing recipes.

Execute the following commands to get poky:

cd ~/yocto/sources
git clone git://git.yoctoproject.org/poky.git
cd poky
git checkout yocto-2.4

By looking at the meta-raspberrypi page we see that, in addition addition to poky, we also need meta-openembedded; the other standard Yocto component. meta-openembedded contains many core recipes that BSP layers are built on. Check out the rocko branch for compatibility with the Yocto version chosen.

Execute the following commands to get meta-openembedded:

cd ~/yocto/sources
git clone git://git.openembedded.org/meta-openembedded
cd meta-openembedded
git checkout rocko

Last, but not least, is getting the meta-raspberrypi BSP layer. It contains configuration and software needed to actually build an image for the Raspberry Pi 3. Check out the rocko branch for compatibility with the chosen Yocto version.

Execute the following commands to get the meta-raspberrypi BSP layer:

cd ~/yocto/sources
git clone git://git.yoctoproject.org/meta-raspberrypi
cd meta-raspberrypi
git checkout rocko

If everything has gone well, the directory structure for ~/yocto should look something like the following:

yocto
| - sources
    | - poky
    | - meta-openembedded
    | - meta-raspberrypi

That should be all the code needed to build an image. In the next lab, the configuration will be altered to tie the layers together and an actual image will be built.

Configuring Yocto and building a basic image

Configuring the meta layers

The first thing to configure is to add the relevant layers to the Yocto layer configuration file. It can be found under ~/yocto/sources/poky/meta-poky/conf/bblayers.conf.sample. This file will later be copied automatically to your build folder by Yocto. Change the file to look like below, where the environment variable PROJ_DIR will be set at a later stage. BBLAYERS is the variable that contains the paths to all layers that should be parsed for a Yocto build.

~/yocto/sources/poky/meta-poky/conf/bblayers.conf.sample:

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ${PROJ_DIR}/sources/poky/meta \
  ${PROJ_DIR}/sources/poky/meta-poky \
  ${PROJ_DIR}/sources/poky/meta-yocto-bsp \
  ${PROJ_DIR}/sources/meta-openembedded/meta-oe \
  ${PROJ_DIR}/sources/meta-openembedded/meta-multimedia \
  ${PROJ_DIR}/sources/meta-openembedded/meta-networking \
  ${PROJ_DIR}/sources/meta-openembedded/meta-python \
  ${PROJ_DIR}/sources/meta-raspberrypi \
  "

Note that meta-openembedded has several meta layers that all need to be included separately.

The configuration above simply tells Yocto where it can expect to find all of its layers. A build script will later be created that sets the relevant PROJ_DIR variable to the top folder in in the directory structure created in the previous section.

Configuring the machine

Yocto can handle building for several different machines from the same meta layer. In the case of meta-raspberrypi, it supports a number of different machines (See the wiki page for the layer for a list.). To build a 64-bit version of the Raspberry Pi 3 image, change the MACHINE setting in ~/yocto/sources/poky/meta-poky/conf/local.conf.sample to "raspberrypi3-64". It should look something like below.

Section from ~/yocto/sources/poky/meta-poky/conf/local.conf.sample:

#MACHINE ?= "beaglebone"
#MACHINE ?= "genericx86"
#MACHINE ?= "genericx86-64"
#MACHINE ?= "mpc8315e-rdb"
#MACHINE ?= "edgerouter"
#
# This sets the default machine to be qemux86 if no other machine is
# selected:
MACHINE ??= "raspberrypi3-64"

In the example above, "qemux86" has been changed to "raspberrypi3-64".

Create a build script

This is the last stage before building. A build script will be created to make it easier to trigger a build without having to manually set PROJ_DIR, nor remember the target name.

~/yocto/build.sh

#!/bin/bash

export PROJ_DIR="$(pwd)"
source sources/poky/oe-init-build-env $1
export BB_ENV_EXTRAWHITE="PROJ_DIR"
bitbake rpi-hwup-image

First, the script exports the current working directory, then it calls source to have Yocto set up the build environment at the folder given as the first argument to the script. After the environment has been set up, the BB_ENV_EXTRAWHITE variable is exported to allow bitbake, the build system, to read PROJ_DIR. Last, but not least, bitbake is called and asked to build rpi-hwup-image, which is the standard system image.

Bitbake is a build system much like Make. It is the system used by by Yocto to parse and execute the recipes describing how to build the software for the system.

rpi-hwup-image is the system image for the Raspberry Pi 3 and is defined like any other recipe. Its recipe file can be found at ~/yocto/sources/meta-raspberrypi/recipes-core/images/rpi-hwup-image.bb. The fact that the image is built like any other part of the system means that Bitbake can be invoked for other recipes by simply changing rpi-hwup-image in the script for their name. Thus, any .bb file you find in the meta layers hierarchy can be built directly.

The last step in this part of the lab is to make the build script executable. Run the following command to set the executable bit on the script:

chmod +x ~/yocto/build.sh

Building

Yocto doesn't like when the temporary folder of the build directory is on an NFS mount. In fact, it will refuse to build if it detects that to be the case. It is also prudent to build using the local disk as it is a large project that takes considerable time to build the first time around. Consecutive times, the build is sped up considerably as Yocto caches build files for later use and only rebuilds those parts that have changed.

Finally, SSH to a computer with quick disks, a lot of RAM and a good processor. Claptrap in ~ is such a computer. Execute the following command sequence:

ssh claptrap
cd ~/yocto
./build /tmp/$USER

The command sequence above should start a Yocto build for the Raspberry Pi 3 in /tmp/$USER, where $USER is your Lysator username.

Now is a good time to get a cup of coffee and be social in the sofa in ~ as the compilation will take quite a while. Bitbake should show a progress bar detailing the status of the compilation.

If something goes wrong with the build and changes to settings in either bblayers.conf or local.conf are needed, be aware that these files are copied to the build directory and will not be overwritten by calls to build. To change the settings, either change them directly in the build folder's conf directory or change them in the meta-layer and delete the copies in the build folder.

Flashing the image and booting the card

When everything has been built, the image should be available under /tmp/$USER/tmp/deploy/images/raspberrypi3-64. The image file should be named rpi-hwup-image-raspberrypi3-64.rpi-sdimg, which is a symlink to the newest build of the image.

To get it running on the Raspberry, all that is required is to flash it onto a memory card and plug it in.

Insert the memory card into a computer with a card reader. Claptrap for example. It should pop up as a device under /dev. If you execute the dmesg command, a line similar to the following should appear:

[333100.780806] sd 9:0:0:2: [sdd] 31223808 512-byte logical blocks: (16.0 GB/14.9 GiB)

The line above means that the memory card is present at /dev/sdd. To flash the image to the memory card above, the following command can be used:

dd if=/tmp/$USER/tmp/deploy/images/raspberrypi3-64/rpi-hwup-image-raspberrypi3-64.rpi-sdimg \
   of=/dev/sdd bs=1M

NOTE: The computer claptrap in ~ should be configured with special permissions to allow a user to copy the image to the SD card without administrative rights. Please use that computer for the dd command and notify Andreas Kempe <kempe@lysator.liu.se> if it does not work!

The if flag specifies the input file, the of flag specifies the output target and bs=1M means that it should read and write 1 MiB data at a time, this to speed up the operation.

Once the dd has finished, plug the memory card into the Raspberry Pi 3, connect a keyboard to it and plug it into a screen. The default log in is:

Username: root

With no password needed. If all has gone well, a terminal on the system should be available for use.

Adding a WiFi driver to the system

In this lab, drivers for the wireless network card will be added to the image, along with wpa-supplicant, which handles authentication against the network.

Distro features

Yocto has a concept called distro features. Distro features are global settings for the system that allow packages to modify their configurations according to the settings. One distro feature is the wifi feature. It tells all included packages to enable their wifi settings if they have any.

Adding the wifi distro feature is done in the local.conf(.sample) file in the meta-poky layer. Add the following line to the end of the file:

DISTRO_FEATURES_append = " wifi "

Adding firmware and wpa-supplicant

To make the wireless network card work, firmware is required, and to facilitate authentication to the network wpa-supplicant also needs to be added. Adding extra software to a system image can be done through the image file's recipe in a variable called IMAGE_INSTALL.

Modify the image recipe at rpi-hwup-image.bb to look like the following:

~/yocto/sources/meta-raspberrypi/recipes-core/images/rpi-hwup-image.bb:

# Base this image on core-image-minimal
include recipes-core/images/core-image-minimal.bb

# Include modules in rootfs
IMAGE_INSTALL += " \
    kernel-modules \
    linux-firmware \
    wpa-supplicant \
"

Rebuild the image before moving on to the next step.

Configuring and connecting

Flash the new image to the memory card and boot. Once the system has booted, log in as root. Start by modifying the wpa-supplicant file under /etc/wpa_supplicant.conf. Change the config to look like below, but replace the identity and password with your own. You can find your lysator password at the Lysator admin page.

network={
    ssid="eduroam"
    key_mgmt=WPA-EAP
    eap=PEAP
    phase1="peaplabel=0"
    phase2="auth=MSCHAPV2"
    identity="user@lysator.liu.se"
    password="asupersecretpassword"
}

Now, it is time to start wpa-supplicant and the DHCP client. Start wpa-supplicant with the following command:

wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf -B

The -i flag specifies the interface, -c specifies the configuration file and -B instructs wpa-supplicant to run in the background. Now start the DHCP client with the following command:

udhcpc -i wlan0 -b

Again, the -i flag is the interface and the -b flag instructs the client to start in the backgroud.

Assuming everything has gone alright, a connection to eduroam should now be established. Try downloading a webpage using wget:

cd /tmp
wget lysator.liu.se
cat index.html

The command sequence above should download Lysator's front page to the /tmp folder and display the HTML in the terminal.

WARNING: Make sure to remove your username and password from the wpa_supplicant.conf file when you are done with the lab, or request a new password to overwrite the old one at the admin page!

Creating a custom layer and recipe

This lab will show how to create a custom meta-layer and add an existing piece of software to the image. The software chosen for this task is the game Brogue, a roguelike that can be played in the terminal, and is available at the Brogue website.

Creating a new meta-layer

Poky comes with a helper script that aids in the creation of a new meta-layer. Use the script by executing the following commands:

cd ~/yocto/sources
./poky/scripts/yocto-layer create meta-lysator

The commands above will initiate the creation of a new layer called meta-lysator. Accept the suggested default priority, say yes to creating an example recipe and name the recipe something. It doesn't matter what the name of the example recipe is as it will be changed later. When asked about creating a bbappend file, answer no.

A bbappend file is a file that modifies an existing recipe. It is named the same as another recipe and any change made in the bbappend file will overwrite the settings in the original recipe.

Now, add the new meta-layer to the bblayers.conf file just like what was done previously for the other meta-layers.

Arranging files

An example recipe should have been created at ~/yocto/sources/meta-lysator/recipes-example/example. Change the name of the recipe to brogue_1.7.4.bb. Now, remove the folder with the same name as the original example recipe and create a new folder called "files". When done, the folder structure should look something like the following:

recipes-example
| -> example
     | -> brogue_1.7.4.bb
     | -> files

A recipe is expected to be named $name_$version.bb, where the name and version parts will be available as ${PN} and ${PV}, respectively, in the recipe.

When building something using Yocto, a few things are needed of the Makefile. The tools used in the build system cannot have their paths hard-coded since Yocto needs to overwrite them to use the cross compilation tools. When linking software the ${LDFLAGS} variable also has to be included, since Yocto uses symbols in the final executable to make sure that the correct tools were used.

In the case of Brogue, a bit of patching is required to make the Makefiles compatible. It needs to have compiler variables changed, files need to be moved around a bit and linker flags added. This lab will not go into more detail than that regarding the patches, but feel free to read the patch files and try to understand the changes.

Now, download the brogue_make.patch and the tcod_make.patch files and place them in the files folder created earlier, located at ~/yocto/sources/meta-lysator/recipes-example/example/files.

Modifying the recipe

Now, it is time to modify the recipe to download, patch and compile the Brogue source code. After which the binaries will be installed.

The recipe will be modified to look like below and it is recommended to read the explanation, provided under the recipe, of each part while changing it:

~/yocto/sources/meta-lysator/recipes-example/example/brogue_1.7.4.bb

#
# This file was derived from the 'Hello World!' example recipe in the
# Yocto Project Development Manual.
#

SUMMARY = "The game brogue"
LICENSE = "AGPLv3"
LIC_FILES_CHKSUM = "file://${WORKDIR}/${PN}-${PV}/agpl.txt;md5=08657324f94cfae679f345f428c68f9b"

DEPENDS = "libsdl pkgconfig-native"
RDEPENDS_${PN} = "ncurses"

SRC_URI = "https://sites.google.com/site/broguegame/${PN}-${PV}-linux-amd64.tbz2 \
           file://tcod_make.patch \
           file://brogue_make.patch "
SRC_URI[md5sum] = "b2883ee83528f703dbc20b2fb30d6105"

S = "${WORKDIR}/${PN}-${PV}"

do_compile() {
    cd ${S}/src/libtcod-1.5.2
    oe_runmake release

    cd ${S}
    oe_runmake curses
}

do_install() {
         install -d ${D}${bindir}
         install -m 0755 ${S}/bin/brogue ${D}${bindir}

         install -d ${D}${libdir}
         install -m 0755 ${S}/bin/libtcod.so ${D}${libdir}/libtcod.so.1
}

Finally, there are two functions defined; do_compile and do_install. There are many more functions called by default and they can be listed by sourcing the oe-init-build-env script and executing the command "bitbake brogue -c listtasks". An individual task can be called by using the command "bitbake brogue -c $taskname", where taskname is the name of the tasks to perform for the specific recipe. Note that when listing tasks, they are all prefixed with "do_", but when calling them using "bitbake -c", the do prefix should be skipped. Sometimes, it can be useful to be able to execute a specific task for a specific recipe. An example is the task cleanall. It cleans the build directory, the download directory and the cache for the recipe. It can, for example, be useful when debugging recipes, to make sure that all steps of the recipe are run.

Example to list available tasks for the brogue recipe:

export PROJ_DIR="$(pwd)"
source sources/poky/oe-init-build-env /tmp/$USER
export BB_ENV_EXTRAWHITE="PROJ_DIR"
bitbake brogue -c listtasks

do_compile() is the function where the software is expected to be compiled. The first line enters the folder of the library tcod, which is bundled with brogue and needed by it. Then oe_runmake is called, it is just like a normal make command, with the only difference that it automatically supplies the flags needed to make Yocto build the software correctly. Normally oe_make should always be used instead of the normal make command. The next two commands do the same thing, but for the game itself.

In the last function, do_install(), the files that have been created by the do_compile() step are installed to the system image. The "install -d" commands are used to create a dir if it doesn't exist, while the "install -m" commands install specific files to the image file system with the permission mask specified as the argument to the -m flag. Yocto provides a few standard paths, with ${D} representing the image destination and ${libdir} and ${bindir} being the default library and binary directories on the system.

TIP: All variables set by Yocto are available in the file ~/yocto/sources/poky/meta/conf/bitbake.conf.

Hopefully, the recipe should now work and the only thing remaining is adding the new software to the system image and then rebuilding and flashing it to the memory card.

Building the image

Start by adding brogue to the IMAGE_INSTALL variable for the rpi-hwup-image, just like what was done for wpa-supplicant and the firmware earlier.

Before starting the build, remove the bblayers.conf file in /tmp/$USER/conf since it has been modified with the new meta-lysator layer and Yocto doesn't overwrite the copy in the build folder.

NOTE: If Yocto thinks that nothing needs to be rebuilt, the recipe is either missing in the IMAGE_INSTALL variable for the image, the meta layer is missing in the bblayers.conf file or the /tmp/$USER/conf/bblayers.conf file has not been deleted!

Using the build script by executing "./build.sh /tmp/$USER" should at this point build the system with the new software included.

Once the new image has been flashed, start the game on the system by giving the command brogue. Starting a new game can then be accomplished by pushing the n button. Have fun playing!

Last updated Mon Oct 15 19:38:24 CEST 2018