Files
build/extensions/gen-sample-extension-docs.sh
Ricardo Pardini 90e0fe0ba2 extensions framework + UEFI aarch64/x86 + rpi4b + core changes/fixes (#3300)
* extensions framework (née "fragments")

- this should actually change nothing at this point, just add capabilities
- the framework is implemented in lib/extensions.sh
- the "if function x exists then call x" replaced with call_extension_method()
  - +inline documentation
  - +compatibility names

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions framework; meta-extensions: auto-docs and sample extension gen

- 2 extensions dealing with extensibility itself
  - detect-unused-extensions: shows which extensions are enabled, but never called.
  - gen-sample-extension-docs: generates a sample empty extension & Markdown documentation for extensions

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* new extension methods and features via config variables in core Armbian

- `SKIP_EXTERNAL_TOOLCHAINS=yes` - does not download or use any linaro toolchains, only build host-installed ones
- `SKIP_BOOTSPLASH=yes` - does not patch kernel for splash file
- `EXTRA_BSP_NAME=xyz` - allows for BSP variants, useful for when extensions modify the BSP
- `EXTRA_ROOTFS_MIB_SIZE=x` - add x mib's to rootfs size, for use with very small images
- `KERNEL_EXTRA_TARGETS` - what extra targets to make kernel for, default to "modules dtbs"
- `BOOTCONFIG=none` - does not build nor install u-boot; also doesn't handle bootscripts et al
- `unset KERNELSOURCE` - does not build nor install kernel, nor build initrd, nor build nor install firmware
- `ARMHF_ARCH=skip` - does not add armhf to apt/dpkg, thus pure arm64
- `SKIP_ARMBIAN_REPO=yes` - results in armbian.list.disabled in the final image
- define `APT_EXTRA_DIST_PARAMS` with apt-cacher-ng options and use it for `PACKAGE_LIST_INSTALL/REMOVE` et al
- initial support for targeting x86/amd64 UEFI and BIOS
    - some do's/don'ts for x86/amd64, like a different `UBUNTU_MIRROR` default
    - GPT/EFI(ESP) partitions (fat, `UEFISIZE=256` to enable, mount `UEFI_MOUNT_POINT=/boot/efi`, first on disk but ends
      up at `$uefipart`=15)
    - GPT/BIOS partitions (fat, `BIOSSIZE=1` to enable, second on disk but ends up at partition 14)
    - `UEFI_FS_LABEL="armbiefi"` - to set the FAT label for the EFI partition, visible in Win/Mac
    - hard-requires gdisk package host-side
- add add_host_dependencies() extension method; fill `EXTRA_BUILD_DEPS="pkg pkg2"` to install to host before toolchains
  download
- add pre_prepare_partitions() extension method, for custom partition size calculations
- add create_partition_table() extension method, used to do full-custom partitioning if `USE_HOOK_FOR_PARTITION=yes`
- add post_create_partitions() extension method, mostly for easy debugging
- add post_write_sdcard() extension method, where you can also set `SKIP_VERIFY=yes` to skip sdcard verification
- add post_install_kernel_debs() extension method.
- multiple fixes to bsp to avoid spurious errors when files are not where it expects
- v4: detect `update-initramfs` failure and abort build with useful message if it does
- v4: show useful stacktrace in `exit_with_error`
  - if `ERROR_DEBUG_SHELL=yes`, drop into a shell before unmounting/deleting everything, so we can inspect what went wrong
- v4: display a message before `apt-get remove PACKAGE_LIST_BOARD_REMOVE` packages, so any errors while removing are easy to understand
- v4: preserve kernel .config's dates when copying

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions framework; refactor tool fetching/building into extensions

- a few examples of core refactoring using extensions
- sunxi-tools extension, enabled by 2 different sunxi family includes ("reuse" example)
- marvel-tools extension, enabled by 2 different mvebu family includes
- rkbin-tools extension, enabled by rockship64_common family include
- amlogic-fip/c2-blobs stuff refactored directly into meson64_common.inc ("single-use" example)
- removed the 'testings' fetch_from_repo completely since not used anywhere.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* .wip's for UEFI arm64 and UEFI/BIOS x86 via new GRUB extension

- v3: added `growroot`-awareness to `armbian-resize-filesystem`
    - the partition-growing part of `armbian-resize-filesystem` does not deal correctly with the UEFI layout
    - `growroot` is installed on UEFI images by default, that handles growing partition during initramfs
    - now `armbian-resize-filesystem` handles `resize2fs` only, and works.
- v4: reworked UEFI board/family/include structure:
    - use Distro's `linux-generic` kernel only for `current`
    - `edge` now builds it's own pure-mainline `5.15.y` kernel, for both x86 and arm64
        - `.config` taken from Ubuntu, probably needs tuning for EXTRAWIFI=yes et al
- v4: introduce `SKIP_KERNEL_SYMLINK=yes`, tested in `builddeb`
    - to avoid symlinking kernel; u-boot likes it, but grub and flash-kernel hates it
- v5: many fixes
- v7: more small fixes.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* .wip for the RaspberryPi 4B via new flash-kernel extension

- this does not build it's own kernel "yet", but uses default linux-raspi kernel from Ubuntu
- flash-kernel is not really a bootloader
    - it just prepares kernel et al a FAT partition for booting by the RPi4b bootloader
    - flash-kernel is standard Debian package, but has only been tested on Ubuntu releases
- it is really only known-working since Hirsute release.
  - Debian's rpi kernel is armhf only, so out of scope here, at least until we add source-built kernels.
- v3: fixed focal rootfs build. untested.
- v3: better variable names, preparing for source-built kernel.
- v5: new edge build with pure mainline kernel.
- v6: many fixes and some hacks for packaging and layout, also firmware (using Ubuntu's)

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Added first patch to edge x86 related to wifi drivers

* extensions: leave hostapd alone; remove hackish ext; block reentrancy

- package-list-utils does not belong in this PR
- grub or bcm2711 is not the place to remove hostapd
- block recursive enable_extension() calls, for now.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* gen-sample-extension-docs: fix: avoid counter in generated sample

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: dependencies: enable_extension() in extensions with a stack

- and better stacktraces, I hope

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Remove code from package list since we don't have it in repository
Adjust kernel config to disable driver that needs further polishing.

* Allow amd64 to build the same desktops as aarch64. We only have this limit for armhf, where some desktops don't work

* amd64: allow building amd64 on aarch64 with system toolchain

- conditionally add gcc-x86-64-linux-gnu to hostdeps

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* add libelf-dev directly to hostdeps (and Dockerfile), remove extension

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* packaging: remove SKIP_KERNEL_SYMLINK hack, fix the root cause

- which was the missing $image_name for non-arm64 & non-arm, so: x86 for example

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* grub: really obliterate u-boot stuff from BSP

- for now. soon we'll refactor u-boot so not have to do this

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* flash-kernel: really obliterate u-boot stuff from BSP

- for now. soon we'll refactor u-boot so not have to do this

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: add host_dependencies_ready() hook

- this passes FINAL_HOST_DEPS containing all hostdeps for the run after they're installed

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Add verification functions for correct selection.

* If UEFI Skip symlink creation

* Do not create dtb package for amd64

* Skip scripts folder cleaning if build process native.

Skip creating  postinst prerm scripts  for headers.

* Skip applying headers-debian-byteshift.patch if build native

* Fix architecture syntax as x86_64

* Revert "amd64: allow building amd64 on aarch64 with system toolchain"

This reverts commit 0c5ee20bb1.

* Compare architectures before starting compilation.



Signed-off-by: The-going <48602507+The-going@users.noreply.github.com>

* extensions: cleanups after fixes by the-Going

- packaging:
  - there is _no need_ anymore for the symlink hack, CONFIG_EFI or no. But check is great, see below
  - it's not `amd64` that has no DTB's, it's all UEFI, thus: `is_enabled CONFIG_EFI`, thanks!
- Explicitly disallow "reverse cross compile" in amd64.conf.
- whitespace-only-deletions: revert. we shall shellfmt the whole thing one day, but not today.
- fix a few syntax warnings in newly introduced code (floating `$ARCH` vs `"${ARCH}`) - blame shellcheck

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* packaging: fix: turns out a lot of boards have CONFIG_EFI=y, can't use that for dtb/no-dtb decision.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* grub: remove debug

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* firmware: allow installing `armbian-firmware-full`; make it really full

- can now use `BOARD_FIRMWARE_INSTALL="-full"` to install full firmware for the board. enable for UEFI.
- don't rely on KERNELSOURCE for firmware-related decisions. introduce `INSTALL_ARMBIAN_FIRMWARE` which defaults to `yes`
- rpi4b/flash-kernel: disable Armbian firmware; we need linux-firmware-raspi2, which conflicts.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: log to /${LOG_SUBPATH}/ instead of fixed /debug/

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: introduce cleanup_extension_manager() called by build-all-ng's unset_all()

- to reset/unset everything done by the the initializer, so build can run again

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: remove 'global' logging, for use with build_all_ng

- enable_extensions() will have to live on without logging to file. it's just too early.
- now init EXTENSION_MANAGER_TMP_DIR in initialize_extension_manager()
- now init EXTENSION_MANAGER_LOG_FILE in initialize_extension_manager()

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* extensions: build-all-ng.sh bugfix due to extension's debug to stdout

- extensions (among other things) can produce output to stdout when activated
- fix: check_hash() produced "idential" (sic, now changed to IDENTICAL) to stdout as a trigger
- debugging output got mixed with "idential", rendering hash cache void for families that used extensions
  - eg: sunxi, others
- fix is to send stdout to the bitbucket when sourcing the board & arch config files
- proper fix would be stop using stdout in this case and use return code for check_hash()
  - one day soon

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>

* Add CI build targets

Co-authored-by: Igor Pecovnik <igor.pecovnik@gmail.com>
Co-authored-by: The-going <48602507+The-going@users.noreply.github.com>
2021-12-06 09:49:49 +01:00

127 lines
4.5 KiB
Bash

## Hooks
function extension_metadata_ready__499_display_docs_generation_start_info() {
display_alert "Generating hook documentation and sample extension"
}
function extension_metadata_ready__docs_markdown() {
generate_markdown_docs_to_stdout >"${DEST}/"${LOG_SUBPATH}"/hooks.auto.docs.md"
}
function extension_metadata_ready__docs_sample_extension() {
mkdir -p "${SRC}/userpatches/extensions"
generate_sample_extension_to_stdout >"${SRC}/userpatches/extensions/sample-extension.sh"
}
## Internal functions
### Common stuff
function read_common_data() {
export HOOK_POINT_CALLS_COUNT=$(wc -l <"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt")
export HOOK_POINT_CALLS_UNIQUE_COUNT=$(sort <"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt" | uniq | wc -l)
export HOOK_POINTS_WITH_MULTIPLE_CALLS=""
# Read the hook_points (main, official names) from the hook point ordering file.
export ALL_HOOK_POINT_CALLS=$(xargs echo -n <"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt")
}
function loop_over_hook_points_and_call() {
local callback="$1"
HOOK_POINT_COUNTER=0
for one_hook_point in ${ALL_HOOK_POINT_CALLS}; do
export HOOK_POINT_COUNTER=$((HOOK_POINT_COUNTER + 1))
export HOOK_POINT="${one_hook_point}"
export MARKDOWN_HEAD="$(head -1 "${EXTENSION_MANAGER_TMP_DIR}/${one_hook_point}.orig.md")"
export MARKDOWN_BODY="$(tail -n +2 "${EXTENSION_MANAGER_TMP_DIR}/${one_hook_point}.orig.md")"
export COMPATIBILITY_NAMES="$(xargs echo -n <"${EXTENSION_MANAGER_TMP_DIR}/${one_hook_point}.compat")"
${callback}
done
}
## Markdown stuff
function generate_markdown_docs_to_stdout() {
read_common_data
cat <<MASTER_HEADER
# Armbian build system extensibility documentation
- This documentation is auto-generated.
MASTER_HEADER
[[ $HOOK_POINT_CALLS_COUNT -gt $HOOK_POINT_CALLS_UNIQUE_COUNT ]] && {
# Some hook points were called multiple times, determine which.
HOOK_POINTS_WITH_MULTIPLE_CALLS=$(comm -13 <(sort <"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt" | uniq) <(sort <"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt") | sort | uniq | xargs echo -n)
cat <<MULTIPLE_CALLS_WARNING
- *Important:* The following hook points where called multiple times during the documentation generation. This can be indicative of a bug in the build system. Please check the sources for the invocation of the following hooks: \`${HOOK_POINTS_WITH_MULTIPLE_CALLS}\`.
MULTIPLE_CALLS_WARNING
}
cat <<PRE_HOOKS_HEADER
## Hooks
- Hooks are listed in the order they are called.
PRE_HOOKS_HEADER
loop_over_hook_points_and_call "generate_markdown_one_hook_point_to_stdout"
cat <<MASTER_FOOTER
------------------------------------------------------------------------------------------
MASTER_FOOTER
}
function generate_markdown_one_hook_point_to_stdout() {
# Hook name in 3rd level title, first line of description in a blockquote.
# The rest in a normal block.
cat <<HOOK_DOCS
### \`${one_hook_point}\`
> ${MARKDOWN_HEAD}
${MARKDOWN_BODY}
HOOK_DOCS
[[ "${COMPATIBILITY_NAMES}" != "" ]] && {
echo -e "\n\nAlso known as (for backwards compatibility only):"
for old_name in ${COMPATIBILITY_NAMES}; do
echo "- \`${old_name}\`"
done
}
echo ""
}
## Bash sample extension stuff
generate_sample_extension_to_stdout() {
read_common_data
cat <<HEADER
# Sample Armbian build system extension with all extension methods.
# This file is auto-generated from and by the build system itself.
# Please, always use the latest version of this file as a starting point for your own extensions.
# Generation date: $(date)
# Read more about the build system at https://docs.armbian.com/Developer-Guide_Build-Preparation/
HEADER
loop_over_hook_points_and_call "generate_bash_sample_for_hook_point"
}
generate_bash_sample_for_hook_point() {
# Include the markdown documentation as a comment.
# Right now clean it up naively (remove backticks, mostly) but we could pipe through stuff to get better plaintext. (pandoc is a 155mb binary FYI)
local COMMENT_HEAD="#### $(echo "${MARKDOWN_HEAD}" | tr '`' '"')"
# shellcheck disable=SC2001
local COMMENT_BODY="$(echo "${MARKDOWN_BODY}" | tr '`' '"' | sed -e 's/^/### /')"
local bonus=""
[[ "${HOOK_POINT_COUNTER}" == "1" ]] && bonus="$(echo -e "\n\texport PROGRESS_DISPLAY=verysilent # Example: export a variable. This one silences the built.")"
cat <<SAMPLE_BASH_CODE
${COMMENT_HEAD}
${COMMENT_BODY}
function ${HOOK_POINT}__be_more_awesome() {
# @TODO: Please rename this function to reflect what it does, but preserve the "${HOOK_POINT}__" prefix.
display_alert "Being awesome at \${HOOK_POINT}" "\${EXTENSION}" "info"${bonus}
}
SAMPLE_BASH_CODE
}