/ 21.06.2024
The rapid growth of IoT devices today comes as no surprise to anyone, but with it comes a growing need for efficient and reliable software. To achieve this, it is essential to have a streamlined development process to get the product to market faster and with high quality. How can this be achieved? The answer is the Yocto Project and containerization in Docker.
Linux is a popular choice for embedded systems due to its high reliability and availability of source code. However, developing such a system is proving to be a real challenge due to limited resources, optimization requirements, and a difficult entry threshold.
Experienced developers working within embedded systems development must not only possess an in-depth understanding of the Linux kernel, device drivers, and low-level programming, but also have knowledge of hardware architecture and real-time operating systems. This comprehensive knowledge allows them to tackle the challenges of developing software for resource-constrained devices.
Table of Contents
Why is software development for embedded systems a challenge?
One of the key aspects of streamlining development processes is the efficient use of available resources. As embedded systems often have limited memory processing power, developers must carefully manage memory, CPU usage, and power consumption. This involves optimizing code, using efficient algorithms and hardware acceleration techniques. By streamlining these processes, developers can significantly increase the performance of their software while minimizing resource use.
Another important aspect of streamlining developer processes is using version control systems and collaboration tools appropriately. These tools enable developers to collaborate seamlessly, track changes, and resolve conflicts effectively. By adopting version control and collaboration best practices, development teams can ensure seamless coordination and minimize the risk of errors or code conflicts.
In this article, we will discuss one way to streamline embedded system development using YP and Docker containers.
What is the Yocto Project?
According to its definition, Yocto Project is an open-source collaborative project that provides templates, tools, and methods to create custom Linux distributions for embedded systems such as IoT devices, smartphones, tablets, touchscreen-based devices, etc., regardless of hardware architecture. You can create a distribution for your Raspberry Pi, even for an Arduino UNO.
A recent survey conducted by the Eclipse Foundation shows that Yocto is used by 71% of respondents who use it to build custom distributions for embedded systems.
Thanks to the Yocto Project, developers gain access to recipes, allowing them to install software quickly and easily in their created distribution. Moreover, writing recipes is so trivial that adding new software is effortless, enabling them to create dedicated Linux distributions optimized for the target hardware. Whether it is a small IoT device or a complex industrial control system, Yocto’s design provides the flexibility and scalability needed to meet the unique requirements of each embedded system.
Benefits of using Yocto in embedded Linux systems
Yocto simplifies the process of creating and maintaining Linux distributions tailored specifically for embedded systems.
Yocto’s main advantages include several factors:
1. Yocto enables developers to create a standardized and repeatable compilation environment, ensuring consistent results across different hardware platforms
Yocto uses the OpenEmbedded framework as its cross-compilation system.
It is based on BitBake, which functions like a package manager, i.e., it reads prescriptions and manages package installation. It allows developers to define the software components and configurations required for their embedded Linux distribution. Furthermore, it automatically downloads the necessary source code, applies patches, and compiles the software, resulting in a complete Linux image ready for deployment on the target hardware. This is particularly important in the field of embedded systems, where hardware configurations can vary widely.
2. Yocto provides an architecture based on layers (meta layers), allowing easy customization and integration of software components.
Developers can add or remove layers to enable or disable specific features, making it straightforward to tailor the Linux distribution to the requirements of the target system. This flexibility ensures that the resulting system is optimized.
3. Yocto offers a wide range of ready-made recipes, significantly reducing the time and effort required for software integration.
The recipes, which are recipes for package installations, are maintained by the YP project community and cover many software components commonly used in embedded systems. These tools include a package manager for managing software components, a software development kit (SDK) for cross-compiling applications, and a runtime environment for testing and debugging the system. Developers can use these resources to quickly add functionality to their Linux distribution, saving valuable development time.
What are Docker containers?
Containerization is a technique that allows developers to pack an application and its dependencies into a single container, ensuring that the application works the same way in any environment it is deployed. Docker technology is unique in that it focuses on the requirements of developers and system operators to separate the application’s dependencies from the infrastructure. This enables consistent and predictable deployment across platforms and environments, reducing the risk of errors and downtime.
Docker containers offer several advantages for software development:
1. Provide a consistent and repeatable environment.
Docker containers eliminate the ‘it only works on my hardware’ problem. Developers can create a container image that contains the entire development environment, including libraries, tools, and dependencies. This image can be shared with other team members, ensuring consistency throughout the development cycle.
2. Facilitate collaboration between team members.
Developers can share container images via the container registry, enabling others to replicate the development environment quickly. This promotes a DevOps culture where development, testing, and deployment happen seamlessly.
Combining Yocto and Docker – why does it work great?
There are many benefits to using Docker to build your own distributions using YP. Developers can ensure reproducibility across different machines and operating systems by encapsulating the compilation environment in a Docker container. This eliminates the need to configure the compilation environment manually and reduces the risk of inconsistencies and errors.
How does Docker solve the problems of the traditional approach?
Instead of an installation document, we can give the docker image to our colleagues. By building the root file system, the so-called rotfs (root file system), we run the docker container (docker-composer), and in it, we only run bitbake. The docker container guarantees that the compilation of our distribution using YP will be executed by everyone in the same execution environment. YP has reproducibility tests designed to create completely reproducible versions. This means if you run a compilation today and then run the same configuration X times in the future, the binaries you get from the compilation should be binary identical. By running upstream to maintain reproducibility after each change of these tests, no matter what patches you add to the BitBake code, the OpenEmbedded layer, or other minor layers like meta-python, you will get the same compilation result every time.
Thanks to the use of YP, the situation will not change even in a few years. When we have newer versions of Linux on our computers or when the compilation experts leave the team, we can recreate the same Linux image we built three or five years earlier. This is possible because Docker allows you to revert to previously frozen code. If the code no longer compiles on new library versions, containerization allows us to compile the code using older ones. Docker makes the maintenance of long-lived products much easier than the traditional approach, as it significantly speeds up the resolution of compilation errors.
7 Best practices used to manage Yocto Project & Docker compilations effectively
Following best practices and being vigilant about versioning and debugging in container environments is crucial. It is best to use a version control system and thorough testing.
Below, we discuss some helpful tips that will influence the effectiveness of YP & Docker compilation management:
1. Apply the main principles of containerization
Start with official and minimal base images to reduce unnecessary overhead. Also, use multi-stage Docker compilation to keep the final image size to a minimum.
2. Use Yocto Project best practices
- Organize Yocto layers effectively and update layers regularly to keep up to date with the latest enhancements and bug fixes.
Each layer in YP has an equivalent for particular version of Yocto. For example, if working in YP version tag: yocto-4.0.13, tag: 2022-04.13-kirkstone, tag: 2022-04.13, it means the kirkstone version 4.0.13 is running on the docker image. Every layer, from the most important one, i.e., openembedded-core, to minor ones, such as meta-python, etc., must be in the kirkstone version. The bitbake, on the other hand, must be in the version associated with the Yocto release.
- Good practice in layer management is to use .bbappend files. If there is an error in a layer from outside, you do not need to insert changes in the layer in question but use bbappend.
3. Cache Docker images
Use the Docker layer caching mechanism to speed up subsequent compilations by reusing intermediate layers (Leverage Docker Layer Caching). Each dependency should be cached separately to avoid unnecessary re-downloading and rebuilding during iterative development.
4. Optimize Yocto
Adjust the number of parallel compilation tasks (e.g., make -jX) based on available system resources to speed up Yocto compilations.
SSTATE Cache: Configure and use the Shared State Cache (SSTATE) to store intermediate compilation artifacts, thus accelerating the process significantly.
5. Create images selectively
When developing for a specific purpose, selectively build only those necessary images and components to save time and resources. Use the BitBake environment variables to control which packages will be installed on our distribution image.
6. Take care to monitor and record
Regular profiling and compile-time monitoring are essential to identify bottlenecks and optimize resource-intensive tasks. Detailed logging during compilation is also important to help debug and identify problems.
7. Use version control integration
If you use Yocto layers from Git repositories, consider employing Git submodules to manage dependencies more efficiently. Also, store your Yocto environment configurations in your version control system to ensure reproducibility across different development environments.
Potential challenges when compiling Yocto with Docker
While the combination of Yocto and Docker offers significant benefits, there are also challenges that developers may face. One typical challenge is the size of the resulting Docker image, which can be quite large due to the compilation process itself already causing a huge overhead. For example, a small program whose code takes 10MiB may create a transient code for the linker, the so-called object file, etc., which will take 500MB to make the executable file take 1MB. However, this can be remedied by using multi-stage compilation and image optimization techniques to reduce the final size of the Docker image.
Other challenges include:
- Conflicts between formulations can cause compilation failure.
- Ensuring that the Dockerfile is optimized for performance and resource use.
- Managing dependencies and layer integration in Docker containers when working with Yocto can be challenging.
- Ensuring the reproducibility and consistency of Yocto compilation in different environments.
To overcome these challenges, choosing a compatible base image, securely installing the required packages, adding Yocto layers, and carefully using parallel and incremental compilation are important. In addition, it is important to follow best practices for dependency management and layer integration in Docker containers during development.
Summary
It is best to use a fully containerized development environment for embedded Linux development to ensure repeatable compilation and configuration. This allows you to work in parallel on multiple projects with different configuration requirements on the same machine. Furthermore, dockerizing the development environment helps maintain the system clean and not burden it with project-specific packages.
If you would like to discuss the experience of the specialists at VM.PL in embedded systems projects, please contact us. We will be happy to advise you on choosing the right solutions.
Sources:
- https://embeddeduse.com/2019/02/11/using-docker-containers-for-yocto-builds/
- https://subscription.packtpub.com/book/iot-and-hardware/9781788399210/1/ch01lvl1sec21/using-docker-as-a-yocto-build-system-container
- https://wiki.yoctoproject.org/wiki/Releases
