Ricardo Pardini 90e0fe0ba2 extensions framework + UEFI aarch64/x86 + rpi4b + core changes/fixes (#3300)
* extensions framework (née "fragments")

- this should actually change nothing at this point, just add capabilities
- the framework is implemented in lib/extensions.sh
- the "if function x exists then call x" replaced with call_extension_method()
  - +inline documentation
  - +compatibility names

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions framework; meta-extensions: auto-docs and sample extension gen

- 2 extensions dealing with extensibility itself
  - detect-unused-extensions: shows which extensions are enabled, but never called.
  - gen-sample-extension-docs: generates a sample empty extension & Markdown documentation for extensions

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* new extension methods and features via config variables in core Armbian

- `SKIP_EXTERNAL_TOOLCHAINS=yes` - does not download or use any linaro toolchains, only build host-installed ones
- `SKIP_BOOTSPLASH=yes` - does not patch kernel for splash file
- `EXTRA_BSP_NAME=xyz` - allows for BSP variants, useful for when extensions modify the BSP
- `EXTRA_ROOTFS_MIB_SIZE=x` - add x mib's to rootfs size, for use with very small images
- `KERNEL_EXTRA_TARGETS` - what extra targets to make kernel for, default to "modules dtbs"
- `BOOTCONFIG=none` - does not build nor install u-boot; also doesn't handle bootscripts et al
- `unset KERNELSOURCE` - does not build nor install kernel, nor build initrd, nor build nor install firmware
- `ARMHF_ARCH=skip` - does not add armhf to apt/dpkg, thus pure arm64
- `SKIP_ARMBIAN_REPO=yes` - results in armbian.list.disabled in the final image
- define `APT_EXTRA_DIST_PARAMS` with apt-cacher-ng options and use it for `PACKAGE_LIST_INSTALL/REMOVE` et al
- initial support for targeting x86/amd64 UEFI and BIOS
    - some do's/don'ts for x86/amd64, like a different `UBUNTU_MIRROR` default
    - GPT/EFI(ESP) partitions (fat, `UEFISIZE=256` to enable, mount `UEFI_MOUNT_POINT=/boot/efi`, first on disk but ends
      up at `$uefipart`=15)
    - GPT/BIOS partitions (fat, `BIOSSIZE=1` to enable, second on disk but ends up at partition 14)
    - `UEFI_FS_LABEL="armbiefi"` - to set the FAT label for the EFI partition, visible in Win/Mac
    - hard-requires gdisk package host-side
- add add_host_dependencies() extension method; fill `EXTRA_BUILD_DEPS="pkg pkg2"` to install to host before toolchains
  download
- add pre_prepare_partitions() extension method, for custom partition size calculations
- add create_partition_table() extension method, used to do full-custom partitioning if `USE_HOOK_FOR_PARTITION=yes`
- add post_create_partitions() extension method, mostly for easy debugging
- add post_write_sdcard() extension method, where you can also set `SKIP_VERIFY=yes` to skip sdcard verification
- add post_install_kernel_debs() extension method.
- multiple fixes to bsp to avoid spurious errors when files are not where it expects
- v4: detect `update-initramfs` failure and abort build with useful message if it does
- v4: show useful stacktrace in `exit_with_error`
  - if `ERROR_DEBUG_SHELL=yes`, drop into a shell before unmounting/deleting everything, so we can inspect what went wrong
- v4: display a message before `apt-get remove PACKAGE_LIST_BOARD_REMOVE` packages, so any errors while removing are easy to understand
- v4: preserve kernel .config's dates when copying

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions framework; refactor tool fetching/building into extensions

- a few examples of core refactoring using extensions
- sunxi-tools extension, enabled by 2 different sunxi family includes ("reuse" example)
- marvel-tools extension, enabled by 2 different mvebu family includes
- rkbin-tools extension, enabled by rockship64_common family include
- amlogic-fip/c2-blobs stuff refactored directly into meson64_common.inc ("single-use" example)
- removed the 'testings' fetch_from_repo completely since not used anywhere.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* .wip's for UEFI arm64 and UEFI/BIOS x86 via new GRUB extension

- v3: added `growroot`-awareness to `armbian-resize-filesystem`
    - the partition-growing part of `armbian-resize-filesystem` does not deal correctly with the UEFI layout
    - `growroot` is installed on UEFI images by default, that handles growing partition during initramfs
    - now `armbian-resize-filesystem` handles `resize2fs` only, and works.
- v4: reworked UEFI board/family/include structure:
    - use Distro's `linux-generic` kernel only for `current`
    - `edge` now builds it's own pure-mainline `5.15.y` kernel, for both x86 and arm64
        - `.config` taken from Ubuntu, probably needs tuning for EXTRAWIFI=yes et al
- v4: introduce `SKIP_KERNEL_SYMLINK=yes`, tested in `builddeb`
    - to avoid symlinking kernel; u-boot likes it, but grub and flash-kernel hates it
- v5: many fixes
- v7: more small fixes.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* .wip for the RaspberryPi 4B via new flash-kernel extension

- this does not build it's own kernel "yet", but uses default linux-raspi kernel from Ubuntu
- flash-kernel is not really a bootloader
    - it just prepares kernel et al a FAT partition for booting by the RPi4b bootloader
    - flash-kernel is standard Debian package, but has only been tested on Ubuntu releases
- it is really only known-working since Hirsute release.
  - Debian's rpi kernel is armhf only, so out of scope here, at least until we add source-built kernels.
- v3: fixed focal rootfs build. untested.
- v3: better variable names, preparing for source-built kernel.
- v5: new edge build with pure mainline kernel.
- v6: many fixes and some hacks for packaging and layout, also firmware (using Ubuntu's)

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Added first patch to edge x86 related to wifi drivers

* extensions: leave hostapd alone; remove hackish ext; block reentrancy

- package-list-utils does not belong in this PR
- grub or bcm2711 is not the place to remove hostapd
- block recursive enable_extension() calls, for now.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* gen-sample-extension-docs: fix: avoid counter in generated sample

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: dependencies: enable_extension() in extensions with a stack

- and better stacktraces, I hope

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Remove code from package list since we don't have it in repository
Adjust kernel config to disable driver that needs further polishing.

* Allow amd64 to build the same desktops as aarch64. We only have this limit for armhf, where some desktops don't work

* amd64: allow building amd64 on aarch64 with system toolchain

- conditionally add gcc-x86-64-linux-gnu to hostdeps

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* add libelf-dev directly to hostdeps (and Dockerfile), remove extension

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* packaging: remove SKIP_KERNEL_SYMLINK hack, fix the root cause

- which was the missing $image_name for non-arm64 & non-arm, so: x86 for example

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* grub: really obliterate u-boot stuff from BSP

- for now. soon we'll refactor u-boot so not have to do this

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* flash-kernel: really obliterate u-boot stuff from BSP

- for now. soon we'll refactor u-boot so not have to do this

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: add host_dependencies_ready() hook

- this passes FINAL_HOST_DEPS containing all hostdeps for the run after they're installed

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Add verification functions for correct selection.

* If UEFI Skip symlink creation

* Do not create dtb package for amd64

* Skip scripts folder cleaning if build process native.

Skip creating  postinst prerm scripts  for headers.

* Skip applying headers-debian-byteshift.patch if build native

* Fix architecture syntax as x86_64

* Revert "amd64: allow building amd64 on aarch64 with system toolchain"

This reverts commit 0c5ee20bb1.

* Compare architectures before starting compilation.



Signed-off-by: The-going <48602507+The-going@users.noreply.github.com>

* extensions: cleanups after fixes by the-Going

- packaging:
  - there is _no need_ anymore for the symlink hack, CONFIG_EFI or no. But check is great, see below
  - it's not `amd64` that has no DTB's, it's all UEFI, thus: `is_enabled CONFIG_EFI`, thanks!
- Explicitly disallow "reverse cross compile" in amd64.conf.
- whitespace-only-deletions: revert. we shall shellfmt the whole thing one day, but not today.
- fix a few syntax warnings in newly introduced code (floating `$ARCH` vs `"${ARCH}`) - blame shellcheck

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* packaging: fix: turns out a lot of boards have CONFIG_EFI=y, can't use that for dtb/no-dtb decision.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* grub: remove debug

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* firmware: allow installing `armbian-firmware-full`; make it really full

- can now use `BOARD_FIRMWARE_INSTALL="-full"` to install full firmware for the board. enable for UEFI.
- don't rely on KERNELSOURCE for firmware-related decisions. introduce `INSTALL_ARMBIAN_FIRMWARE` which defaults to `yes`
- rpi4b/flash-kernel: disable Armbian firmware; we need linux-firmware-raspi2, which conflicts.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: log to /${LOG_SUBPATH}/ instead of fixed /debug/

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: introduce cleanup_extension_manager() called by build-all-ng's unset_all()

- to reset/unset everything done by the the initializer, so build can run again

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: remove 'global' logging, for use with build_all_ng

- enable_extensions() will have to live on without logging to file. it's just too early.
- now init EXTENSION_MANAGER_TMP_DIR in initialize_extension_manager()
- now init EXTENSION_MANAGER_LOG_FILE in initialize_extension_manager()

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: build-all-ng.sh bugfix due to extension's debug to stdout

- extensions (among other things) can produce output to stdout when activated
- fix: check_hash() produced "idential" (sic, now changed to IDENTICAL) to stdout as a trigger
- debugging output got mixed with "idential", rendering hash cache void for families that used extensions
  - eg: sunxi, others
- fix is to send stdout to the bitbucket when sourcing the board & arch config files
- proper fix would be stop using stdout in this case and use return code for check_hash()
  - one day soon

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Add CI build targets

Co-authored-by: Igor Pecovnik <igor.pecovnik@gmail.com>
Co-authored-by: The-going <48602507+The-going@users.noreply.github.com>
2021-12-06 09:49:49 +01:00
2021-12-05 17:39:26 +01:00
2017-08-01 12:30:34 +03:00
2021-02-24 18:19:09 +01:00
2014-10-08 10:14:29 +02:00
2021-10-21 22:30:51 +02:00
2021-09-03 08:31:01 +02:00

Armbian logo
build tools

 

GitHub last commit (branch) GitHub Workflow Status Twitter Follow Become a patron

Table of contents

 

What this project does?

  • builds custom Debian based Linux system optimized for supported single board computers,
  • covers root filesystem generation, kernel image compilation and bootloader compilation,
  • maintains low-level control software for a selection of hardware,
  • provides a consistent user experience by keeping system standards across different SBC platforms.

 

What do you need to get started?

  • x64 machine with at least 2GB of memory and ~35GB of disk space for a VM, container or native OS,
  • Ubuntu Hirsute 21.04 x64 for native building or any Docker capable x64 Linux for containerised,
    • Hirsute is required for newer non-LTS releases.. ex: Bullseye, Sid, Groovy, Hirsute
    • If building for LTS releases.. ex: Focal, Bionic, Buster, it is possible to use Ubuntu 20.04 Focal, but it is not supported
  • superuser rights (configured sudo or root access).

How to build an image or a kernel?

apt-get -y install git
git clone https://github.com/armbian/build
cd build
./compile.sh

Armbian logo

  • The script will take care about preparing the workspace like installing necessary dependencies and downloading sources and tools
  • It guides through the process until either a kernel package set or a ready-to-use image for a SDcard is created

Build parameter examples

Show work in progress areas in interactive mode:

./compile.sh EXPERT="yes"

Run build tools inside Docker container:

./compile.sh docker

Build minimal CLI Armbian Focal image for Orangepi Zero. Use modern kernel and write image to the SD card:

./compile.sh \
BOARD=orangepizero \
BRANCH=current \
RELEASE=focal \
BUILD_MINIMAL=yes \
BUILD_DESKTOP=no \
KERNEL_ONLY=no \
KERNEL_CONFIGURE=no \
CARD_DEVICE="/dev/sda"

Build parameters, advanced build options, user defined configuration, build with Docker?

Compare with industry standards

Check similarity, advantages and disadvantages compared with leading industry standard build software.

Function Armbian Yocto Buildroot
Target general purpose embedded embedded / IOT
U-boot and kernel compiled from sources compiled from sources compiled from sources
Board support maintenance   complete outside outside
Root file system Debian or Ubuntu based custom custom
Package manager APT any none
Configurability limited large large
Initramfs support yes yes yes
Getting started quick very slow slow
Cross compilation yes yes yes

Where to download prebuilt images?

https://www.armbian.com/download/

Armbian releases quarterly at the end of February, May, August, November. You are welcome to propose changes to our default images build list.

Additional information

Build tools overview

├── cache                                Work / cache directory
│   ├── rootfs                           Compressed vanilla Debian and Ubuntu rootfilesystem cache
│   ├── sources                          Kernel, u-boot and various drivers sources. Mainly C code
│   ├── toolchains                       External cross compilers from Linaro™ or ARM™
├── config                               Packages repository configurations
│   ├── targets.conf                     Board build target configuration
│   ├── boards                           Board configurations
│   ├── bootenv                          Initial boot loaders environments per family
│   ├── bootscripts                      Initial Boot loaders scripts per family
│   ├── cli                              CLI packages configurations per distribution
│   ├── desktop                          Desktop packages configurations per distribution
│   ├── distributions                    Distributions settings
│   ├── kernel                           Kernel build configurations per family
│   ├── sources                          Kernel and u-boot sources locations and scripts
│   ├── templates                        User configuration templates which populate userpatches
│   └── torrents                         External compiler and rootfs cache torrents
├── lib                                  Main build tools libraries
├── output                               Build artifact
│   └── deb                              Deb packages
│   └── images                           Bootable images - RAW or compressed
│   └── debug                            Patch and build logs
│   └── config                           Kernel configuration export location
│   └── patch                            Created patches location
├── packages                             Support scripts, binary blobs, packages
│   ├── blobs                            Wallpapers, various configs, closed source bootloaders
│   ├── bsp-cli                          Automatically added to armbian-bsp-cli package 
│   ├── bsp-desktop                      Automatically added to armbian-bsp-desktopo package
│   ├── bsp                              Scripts and configs overlay for rootfs
│   └── extras-buildpkgs                 Optional compilation and packaging engine
├── patch                                Collection of patches
│   ├── atf                              ARM trusted firmware
│   ├── kernel                           Linux kernel patches
|   |   └── family-branch                Per kernel family and branch
│   ├── misc                             Linux kernel packaging patches
│   └── u-boot                           Universal boot loader patches
|       ├── u-boot-board                 For specific board
|       └── u-boot-family                For entire kernel family
└── userpatches                          User: configuration patching area
    ├── lib.config                       User: tools common config/override file
    ├── config-default.conf              User: default user config file
    ├── customize-image.sh               User: script will execute just before closing the image
    ├── atf                              User: ARM trusted firmware
    ├── kernel                           User: Linux kernel per kernel family
    ├── misc                             User: various
    └── u-boot                           User: universal boot loader patches

Support

Contribute

Social

Credits

Sponsors

Most of the project is sponsored with a work done by volunteer collaborators, while some part of the project costs are being covered by the industry. We would not be able to get this far without their help.

Do you want to see yourself below?

Languages
Shell 68.7%
Python 9.5%
GSC 7.7%
JavaScript 5%
Batchfile 3.7%
Other 5.4%