mirror of
https://github.com/armbian/build
synced 2025-09-24 19:47:06 +07:00
extensions framework + UEFI aarch64/x86 + rpi4b + core changes/fixes (#3300)
* extensions framework (née "fragments")
- this should actually change nothing at this point, just add capabilities
- the framework is implemented in lib/extensions.sh
- the "if function x exists then call x" replaced with call_extension_method()
- +inline documentation
- +compatibility names
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions framework; meta-extensions: auto-docs and sample extension gen
- 2 extensions dealing with extensibility itself
- detect-unused-extensions: shows which extensions are enabled, but never called.
- gen-sample-extension-docs: generates a sample empty extension & Markdown documentation for extensions
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* new extension methods and features via config variables in core Armbian
- `SKIP_EXTERNAL_TOOLCHAINS=yes` - does not download or use any linaro toolchains, only build host-installed ones
- `SKIP_BOOTSPLASH=yes` - does not patch kernel for splash file
- `EXTRA_BSP_NAME=xyz` - allows for BSP variants, useful for when extensions modify the BSP
- `EXTRA_ROOTFS_MIB_SIZE=x` - add x mib's to rootfs size, for use with very small images
- `KERNEL_EXTRA_TARGETS` - what extra targets to make kernel for, default to "modules dtbs"
- `BOOTCONFIG=none` - does not build nor install u-boot; also doesn't handle bootscripts et al
- `unset KERNELSOURCE` - does not build nor install kernel, nor build initrd, nor build nor install firmware
- `ARMHF_ARCH=skip` - does not add armhf to apt/dpkg, thus pure arm64
- `SKIP_ARMBIAN_REPO=yes` - results in armbian.list.disabled in the final image
- define `APT_EXTRA_DIST_PARAMS` with apt-cacher-ng options and use it for `PACKAGE_LIST_INSTALL/REMOVE` et al
- initial support for targeting x86/amd64 UEFI and BIOS
- some do's/don'ts for x86/amd64, like a different `UBUNTU_MIRROR` default
- GPT/EFI(ESP) partitions (fat, `UEFISIZE=256` to enable, mount `UEFI_MOUNT_POINT=/boot/efi`, first on disk but ends
up at `$uefipart`=15)
- GPT/BIOS partitions (fat, `BIOSSIZE=1` to enable, second on disk but ends up at partition 14)
- `UEFI_FS_LABEL="armbiefi"` - to set the FAT label for the EFI partition, visible in Win/Mac
- hard-requires gdisk package host-side
- add add_host_dependencies() extension method; fill `EXTRA_BUILD_DEPS="pkg pkg2"` to install to host before toolchains
download
- add pre_prepare_partitions() extension method, for custom partition size calculations
- add create_partition_table() extension method, used to do full-custom partitioning if `USE_HOOK_FOR_PARTITION=yes`
- add post_create_partitions() extension method, mostly for easy debugging
- add post_write_sdcard() extension method, where you can also set `SKIP_VERIFY=yes` to skip sdcard verification
- add post_install_kernel_debs() extension method.
- multiple fixes to bsp to avoid spurious errors when files are not where it expects
- v4: detect `update-initramfs` failure and abort build with useful message if it does
- v4: show useful stacktrace in `exit_with_error`
- if `ERROR_DEBUG_SHELL=yes`, drop into a shell before unmounting/deleting everything, so we can inspect what went wrong
- v4: display a message before `apt-get remove PACKAGE_LIST_BOARD_REMOVE` packages, so any errors while removing are easy to understand
- v4: preserve kernel .config's dates when copying
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions framework; refactor tool fetching/building into extensions
- a few examples of core refactoring using extensions
- sunxi-tools extension, enabled by 2 different sunxi family includes ("reuse" example)
- marvel-tools extension, enabled by 2 different mvebu family includes
- rkbin-tools extension, enabled by rockship64_common family include
- amlogic-fip/c2-blobs stuff refactored directly into meson64_common.inc ("single-use" example)
- removed the 'testings' fetch_from_repo completely since not used anywhere.
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* .wip's for UEFI arm64 and UEFI/BIOS x86 via new GRUB extension
- v3: added `growroot`-awareness to `armbian-resize-filesystem`
- the partition-growing part of `armbian-resize-filesystem` does not deal correctly with the UEFI layout
- `growroot` is installed on UEFI images by default, that handles growing partition during initramfs
- now `armbian-resize-filesystem` handles `resize2fs` only, and works.
- v4: reworked UEFI board/family/include structure:
- use Distro's `linux-generic` kernel only for `current`
- `edge` now builds it's own pure-mainline `5.15.y` kernel, for both x86 and arm64
- `.config` taken from Ubuntu, probably needs tuning for EXTRAWIFI=yes et al
- v4: introduce `SKIP_KERNEL_SYMLINK=yes`, tested in `builddeb`
- to avoid symlinking kernel; u-boot likes it, but grub and flash-kernel hates it
- v5: many fixes
- v7: more small fixes.
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* .wip for the RaspberryPi 4B via new flash-kernel extension
- this does not build it's own kernel "yet", but uses default linux-raspi kernel from Ubuntu
- flash-kernel is not really a bootloader
- it just prepares kernel et al a FAT partition for booting by the RPi4b bootloader
- flash-kernel is standard Debian package, but has only been tested on Ubuntu releases
- it is really only known-working since Hirsute release.
- Debian's rpi kernel is armhf only, so out of scope here, at least until we add source-built kernels.
- v3: fixed focal rootfs build. untested.
- v3: better variable names, preparing for source-built kernel.
- v5: new edge build with pure mainline kernel.
- v6: many fixes and some hacks for packaging and layout, also firmware (using Ubuntu's)
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* Added first patch to edge x86 related to wifi drivers
* extensions: leave hostapd alone; remove hackish ext; block reentrancy
- package-list-utils does not belong in this PR
- grub or bcm2711 is not the place to remove hostapd
- block recursive enable_extension() calls, for now.
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* gen-sample-extension-docs: fix: avoid counter in generated sample
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions: dependencies: enable_extension() in extensions with a stack
- and better stacktraces, I hope
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* Remove code from package list since we don't have it in repository
Adjust kernel config to disable driver that needs further polishing.
* Allow amd64 to build the same desktops as aarch64. We only have this limit for armhf, where some desktops don't work
* amd64: allow building amd64 on aarch64 with system toolchain
- conditionally add gcc-x86-64-linux-gnu to hostdeps
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* add libelf-dev directly to hostdeps (and Dockerfile), remove extension
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* packaging: remove SKIP_KERNEL_SYMLINK hack, fix the root cause
- which was the missing $image_name for non-arm64 & non-arm, so: x86 for example
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* grub: really obliterate u-boot stuff from BSP
- for now. soon we'll refactor u-boot so not have to do this
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* flash-kernel: really obliterate u-boot stuff from BSP
- for now. soon we'll refactor u-boot so not have to do this
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions: add host_dependencies_ready() hook
- this passes FINAL_HOST_DEPS containing all hostdeps for the run after they're installed
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* Add verification functions for correct selection.
* If UEFI Skip symlink creation
* Do not create dtb package for amd64
* Skip scripts folder cleaning if build process native.
Skip creating postinst prerm scripts for headers.
* Skip applying headers-debian-byteshift.patch if build native
* Fix architecture syntax as x86_64
* Revert "amd64: allow building amd64 on aarch64 with system toolchain"
This reverts commit 0c5ee20bb1.
* Compare architectures before starting compilation.
Signed-off-by: The-going <48602507+The-going@users.noreply.github.com>
* extensions: cleanups after fixes by the-Going
- packaging:
- there is _no need_ anymore for the symlink hack, CONFIG_EFI or no. But check is great, see below
- it's not `amd64` that has no DTB's, it's all UEFI, thus: `is_enabled CONFIG_EFI`, thanks!
- Explicitly disallow "reverse cross compile" in amd64.conf.
- whitespace-only-deletions: revert. we shall shellfmt the whole thing one day, but not today.
- fix a few syntax warnings in newly introduced code (floating `$ARCH` vs `"${ARCH}`) - blame shellcheck
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* packaging: fix: turns out a lot of boards have CONFIG_EFI=y, can't use that for dtb/no-dtb decision.
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* grub: remove debug
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* firmware: allow installing `armbian-firmware-full`; make it really full
- can now use `BOARD_FIRMWARE_INSTALL="-full"` to install full firmware for the board. enable for UEFI.
- don't rely on KERNELSOURCE for firmware-related decisions. introduce `INSTALL_ARMBIAN_FIRMWARE` which defaults to `yes`
- rpi4b/flash-kernel: disable Armbian firmware; we need linux-firmware-raspi2, which conflicts.
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions: log to /${LOG_SUBPATH}/ instead of fixed /debug/
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions: introduce cleanup_extension_manager() called by build-all-ng's unset_all()
- to reset/unset everything done by the the initializer, so build can run again
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions: remove 'global' logging, for use with build_all_ng
- enable_extensions() will have to live on without logging to file. it's just too early.
- now init EXTENSION_MANAGER_TMP_DIR in initialize_extension_manager()
- now init EXTENSION_MANAGER_LOG_FILE in initialize_extension_manager()
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* extensions: build-all-ng.sh bugfix due to extension's debug to stdout
- extensions (among other things) can produce output to stdout when activated
- fix: check_hash() produced "idential" (sic, now changed to IDENTICAL) to stdout as a trigger
- debugging output got mixed with "idential", rendering hash cache void for families that used extensions
- eg: sunxi, others
- fix is to send stdout to the bitbucket when sourcing the board & arch config files
- proper fix would be stop using stdout in this case and use return code for check_hash()
- one day soon
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
* Add CI build targets
Co-authored-by: Igor Pecovnik <igor.pecovnik@gmail.com>
Co-authored-by: The-going <48602507+The-going@users.noreply.github.com>
This commit is contained in:
Ricardo Pardini
447
lib/extensions.sh
Normal file
447
lib/extensions.sh
Normal file
@@ -0,0 +1,447 @@
|
||||
# global variables managing the state of the extension manager. treat as private.
|
||||
declare -A extension_function_info # maps a function name to a string with KEY=VALUEs information about the defining extension
|
||||
declare -i initialize_extension_manager_counter=0 # how many times has the extension manager initialized?
|
||||
declare -A defined_hook_point_functions # keeps a map of hook point functions that were defined and their extension info
|
||||
declare -A hook_point_function_trace_sources # keeps a map of hook point functions that were actually called and their source
|
||||
declare -A hook_point_function_trace_lines # keeps a map of hook point functions that were actually called and their source
|
||||
declare fragment_manager_cleanup_file # this is a file used to cleanup the manager's produced functions, for build_all_ng
|
||||
# configuration.
|
||||
export DEBUG_EXTENSION_CALLS=no # set to yes to log every hook function called to the main build log
|
||||
export LOG_ENABLE_EXTENSION=yes # colorful logs with stacktrace when enable_extension is called.
|
||||
|
||||
# This is a helper function for calling hooks.
|
||||
# It follows the pattern long used in the codebase for hook-like behaviour:
|
||||
# [[ $(type -t name_of_hook_function) == function ]] && name_of_hook_function
|
||||
# but with the following added behaviors:
|
||||
# 1) it allows for many arguments, and will treat each as a hook point.
|
||||
# this allows for easily kept backwards compatibility when renaming hooks, for example.
|
||||
# 2) it will read the stdin and assume it's (Markdown) documentation for the hook point.
|
||||
# combined with heredoc in the call site, it allows for "inline" documentation about the hook
|
||||
# notice: this is not involved in how the hook functions came to be. read below for that.
|
||||
call_extension_method() {
|
||||
# First, consume the stdin and write metadata about the call.
|
||||
write_hook_point_metadata "$@" || true
|
||||
|
||||
# @TODO: hack to handle stdin again, possibly with '< /dev/tty'
|
||||
|
||||
# Then a sanity check, hook points should only be invoked after the manager has initialized.
|
||||
if [[ ${initialize_extension_manager_counter} -lt 1 ]]; then
|
||||
display_alert "Extension problem" "Call to call_extension_method() (in ${BASH_SOURCE[1]- $(get_extension_hook_stracktrace "${BASH_SOURCE[*]}" "${BASH_LINENO[*]}")}) before extension manager is initialized." "err"
|
||||
fi
|
||||
|
||||
# With DEBUG_EXTENSION_CALLS, log the hook call. Users might be wondering what/when is a good hook point to use, and this is visual aid.
|
||||
[[ "${DEBUG_EXTENSION_CALLS}" == "yes" ]] &&
|
||||
display_alert "--> Extension Method '${1}' being called from" "$(get_extension_hook_stracktrace "${BASH_SOURCE[*]}" "${BASH_LINENO[*]}")" ""
|
||||
|
||||
# Then call the hooks, if they are defined.
|
||||
for hook_name in "$@"; do
|
||||
echo "-- Extension Method being called: ${hook_name}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
# shellcheck disable=SC2086
|
||||
[[ $(type -t ${hook_name}) == function ]] && { ${hook_name}; }
|
||||
done
|
||||
}
|
||||
|
||||
# what this does is a lot of bash mumbo-jumbo to find all board-,family-,config- or user-defined hook points.
|
||||
# the meat of this is 'compgen -A function', which is bash builtin that lists all defined functions.
|
||||
# it will then compose a full hook point (function) that calls all the implementing hooks.
|
||||
# this centralized function will then be called by the regular Armbian build system, which is oblivious to how
|
||||
# it came to be. (although it is encouraged to call hook points via call_extension_method() above)
|
||||
# to avoid hard coding the list of hook-points (eg: user_config, image_tweaks_pre_customize, etc) we use
|
||||
# a marker in the function names, namely "__" (two underscores) to determine the hook point.
|
||||
initialize_extension_manager() {
|
||||
# before starting, auto-add extensions specified (eg, on the command-line) via the ENABLE_EXTENSIONS env var. Do it only once.
|
||||
[[ ${initialize_extension_manager_counter} -lt 1 ]] && [[ "${ENABLE_EXTENSIONS}" != "" ]] && {
|
||||
local auto_extension
|
||||
for auto_extension in $(echo "${ENABLE_EXTENSIONS}" | tr "," " "); do
|
||||
ENABLE_EXTENSION_TRACE_HINT="ENABLE_EXTENSIONS -> " enable_extension "${auto_extension}"
|
||||
done
|
||||
}
|
||||
|
||||
# This marks the manager as initialized, no more extensions are allowed to load after this.
|
||||
export initialize_extension_manager_counter=$((initialize_extension_manager_counter + 1))
|
||||
|
||||
# Have a unique temporary dir, even if being built concurrently by build_all_ng.
|
||||
export EXTENSION_MANAGER_TMP_DIR="${SRC}/.tmp/.extensions/${LOG_SUBPATH}"
|
||||
mkdir -p "${EXTENSION_MANAGER_TMP_DIR}"
|
||||
|
||||
# Log destination.
|
||||
export EXTENSION_MANAGER_LOG_FILE="${EXTENSION_MANAGER_TMP_DIR}/extensions.log"
|
||||
echo -n "" >"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt"
|
||||
|
||||
# globally initialize the extensions log.
|
||||
echo "-- lib/extensions.sh included. logs will be below, followed by the debug generated by the initialize_extension_manager() function." >"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
# log whats happening.
|
||||
echo "-- initialize_extension_manager() called." >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
# this is the all-important separator.
|
||||
local hook_extension_delimiter="__"
|
||||
|
||||
# list all defined functions. filter only the ones that have the delimiter. get only the part before the delimiter.
|
||||
# sort them, and make them unique. the sorting is required for uniq to work, and does not affect the ordering of execution.
|
||||
# get them on a single line, space separated.
|
||||
local all_hook_points
|
||||
all_hook_points="$(compgen -A function | grep "${hook_extension_delimiter}" | awk -F "${hook_extension_delimiter}" '{print $1}' | sort | uniq | xargs echo -n)"
|
||||
|
||||
declare -i hook_points_counter=0 hook_functions_counter=0 hook_point_functions_counter=0
|
||||
|
||||
# initialize the cleanups file.
|
||||
fragment_manager_cleanup_file="${SRC}"/.tmp/extension_function_cleanup.sh
|
||||
echo "# cleanups: " >"${fragment_manager_cleanup_file}"
|
||||
|
||||
local FUNCTION_SORT_OPTIONS="--general-numeric-sort --ignore-case" # --random-sort could be used to introduce chaos
|
||||
local hook_point=""
|
||||
# now loop over the hook_points.
|
||||
for hook_point in ${all_hook_points}; do
|
||||
echo "-- hook_point ${hook_point}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
# check if the hook point is already defined as a function.
|
||||
# that can happen for example with user_config(), that can be implemented itself directly by a userpatches config.
|
||||
# for now, just warn, but we could devise a way to actually integrate it in the call list.
|
||||
# or: advise the user to rename their user_config() function to something like user_config__make_it_awesome()
|
||||
local existing_hook_point_function
|
||||
existing_hook_point_function="$(compgen -A function | grep "^${hook_point}\$")"
|
||||
if [[ "${existing_hook_point_function}" == "${hook_point}" ]]; then
|
||||
echo "--- hook_point_functions (final sorted realnames): ${hook_point_functions}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
display_alert "Extension conflict" "function ${hook_point} already defined! ignoring functions: $(compgen -A function | grep "^${hook_point}${hook_extension_delimiter}")" "wrn"
|
||||
continue
|
||||
fi
|
||||
|
||||
# for each hook_point, obtain the list of implementing functions.
|
||||
# the sort order here is (very) relevant, since it determines final execution order.
|
||||
# so the name of the functions actually determine the ordering.
|
||||
local hook_point_functions hook_point_functions_pre_sort hook_point_functions_sorted_by_sort_id
|
||||
|
||||
# Sorting. Multiple extensions (or even the same extension twice) can implement the same hook point
|
||||
# as long as they have different function names (the part after the double underscore __).
|
||||
# the order those will be called depends on the name; eg:
|
||||
# 'hook_point__033_be_awesome()' would be caller sooner than 'hook_point__799_be_even_more_awesome()'
|
||||
# independent from where they were defined or in which order the extensions containing them were added.
|
||||
# since requiring specific ordering could hamper portability, we reward extension authors who
|
||||
# don't mind ordering for writing just: 'hook_point__be_just_awesome()' which is automatically rewritten
|
||||
# as 'hook_point__500_be_just_awesome()'.
|
||||
# extension authors who care about ordering can use the 3-digit number, and use the context variables
|
||||
# HOOK_ORDER and HOOK_POINT_TOTAL_FUNCS to confirm in which order they're being run.
|
||||
|
||||
# gather the real names of the functions (after the delimiter).
|
||||
hook_point_functions_pre_sort="$(compgen -A function | grep "^${hook_point}${hook_extension_delimiter}" | awk -F "${hook_extension_delimiter}" '{print $2}' | xargs echo -n)"
|
||||
echo "--- hook_point_functions_pre_sort: ${hook_point_functions_pre_sort}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
# add "500_" to the names of function that do NOT start with a number.
|
||||
# keep a reference from the new names to the old names (we'll sort on the new, but invoke the old)
|
||||
declare -A hook_point_functions_sortname_to_realname
|
||||
declare -A hook_point_functions_realname_to_sortname
|
||||
for hook_point_function_realname in ${hook_point_functions_pre_sort}; do
|
||||
local sort_id="${hook_point_function_realname}"
|
||||
[[ ! $sort_id =~ ^[0-9] ]] && sort_id="500_${sort_id}"
|
||||
hook_point_functions_sortname_to_realname[${sort_id}]="${hook_point_function_realname}"
|
||||
hook_point_functions_realname_to_sortname[${hook_point_function_realname}]="${sort_id}"
|
||||
done
|
||||
|
||||
# actually sort the sort_id's...
|
||||
# shellcheck disable=SC2086
|
||||
hook_point_functions_sorted_by_sort_id="$(echo "${hook_point_functions_realname_to_sortname[*]}" | tr " " "\n" | LC_ALL=C sort ${FUNCTION_SORT_OPTIONS} | xargs echo -n)"
|
||||
echo "--- hook_point_functions_sorted_by_sort_id: ${hook_point_functions_sorted_by_sort_id}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
# then map back to the real names, keeping the order..
|
||||
hook_point_functions=""
|
||||
for hook_point_function_sortname in ${hook_point_functions_sorted_by_sort_id}; do
|
||||
hook_point_functions="${hook_point_functions} ${hook_point_functions_sortname_to_realname[${hook_point_function_sortname}]}"
|
||||
done
|
||||
# shellcheck disable=SC2086
|
||||
hook_point_functions="$(echo -n ${hook_point_functions})"
|
||||
echo "--- hook_point_functions (final sorted realnames): ${hook_point_functions}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
hook_point_functions_counter=0
|
||||
hook_points_counter=$((hook_points_counter + 1))
|
||||
|
||||
# determine the variables we'll pass to the hook function during execution.
|
||||
# this helps the extension author create extensions that are portable between userpatches and official Armbian.
|
||||
# shellcheck disable=SC2089
|
||||
local common_function_vars="HOOK_POINT=\"${hook_point}\""
|
||||
|
||||
# loop over the functions for this hook_point (keep a total for the hook point and a grand running total)
|
||||
for hook_point_function in ${hook_point_functions}; do
|
||||
hook_point_functions_counter=$((hook_point_functions_counter + 1))
|
||||
hook_functions_counter=$((hook_functions_counter + 1))
|
||||
done
|
||||
common_function_vars="${common_function_vars} HOOK_POINT_TOTAL_FUNCS=\"${hook_point_functions_counter}\""
|
||||
|
||||
echo "-- hook_point: ${hook_point} will run ${hook_point_functions_counter} functions: ${hook_point_functions}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
local temp_source_file_for_hook_point="${EXTENSION_MANAGER_TMP_DIR}/extension_function_definition.sh"
|
||||
|
||||
hook_point_functions_loop_counter=0
|
||||
|
||||
# prepare the cleanup for the function, so we can remove our mess at the end of the build.
|
||||
cat <<-FUNCTION_CLEANUP_FOR_HOOK_POINT >>"${fragment_manager_cleanup_file}"
|
||||
unset ${hook_point}
|
||||
FUNCTION_CLEANUP_FOR_HOOK_POINT
|
||||
|
||||
# now compose a function definition. notice the heredoc. it will be written to tmp file, logged, then sourced.
|
||||
# theres a lot of opportunities here, but for now I keep it simple:
|
||||
# - execute functions in the order defined by ${hook_point_functions} above
|
||||
# - define call-specific environment variables, to help extension authors to write portable extensions (eg: EXTENSION_DIR)
|
||||
cat <<-FUNCTION_DEFINITION_HEADER >"${temp_source_file_for_hook_point}"
|
||||
${hook_point}() {
|
||||
echo "*** Extension-managed hook starting '${hook_point}': will run ${hook_point_functions_counter} functions: '${hook_point_functions}'" >>"\${EXTENSION_MANAGER_LOG_FILE}"
|
||||
FUNCTION_DEFINITION_HEADER
|
||||
|
||||
for hook_point_function in ${hook_point_functions}; do
|
||||
hook_point_functions_loop_counter=$((hook_point_functions_loop_counter + 1))
|
||||
|
||||
# store the full name in a hash, so we can track which were actually called later.
|
||||
defined_hook_point_functions["${hook_point}${hook_extension_delimiter}${hook_point_function}"]="DEFINED=yes ${extension_function_info["${hook_point}${hook_extension_delimiter}${hook_point_function}"]}"
|
||||
|
||||
# prepare the call context
|
||||
local hook_point_function_variables="${common_function_vars}" # start with common vars... (eg: HOOK_POINT_TOTAL_FUNCS)
|
||||
# add the contextual extension info for the function (eg, EXTENSION_DIR)
|
||||
hook_point_function_variables="${hook_point_function_variables} ${extension_function_info["${hook_point}${hook_extension_delimiter}${hook_point_function}"]}"
|
||||
# add the current execution counter, so the extension author can know in which order it is being actually called
|
||||
hook_point_function_variables="${hook_point_function_variables} HOOK_ORDER=\"${hook_point_functions_loop_counter}\""
|
||||
|
||||
# add it to our (not the call site!) environment. if we export those in the call site, the stack is corrupted.
|
||||
eval "${hook_point_function_variables}"
|
||||
|
||||
# output the call, passing arguments, and also logging the output to the extensions log.
|
||||
# attention: don't pipe here (eg, capture output), otherwise hook function cant modify the environment (which is mostly the point)
|
||||
# @TODO: better error handling. we have a good opportunity to 'set -e' here, and 'set +e' after, so that extension authors are encouraged to write error-free handling code
|
||||
cat <<-FUNCTION_DEFINITION_CALLSITE >>"${temp_source_file_for_hook_point}"
|
||||
hook_point_function_trace_sources["${hook_point}${hook_extension_delimiter}${hook_point_function}"]="\${BASH_SOURCE[*]}"
|
||||
hook_point_function_trace_lines["${hook_point}${hook_extension_delimiter}${hook_point_function}"]="\${BASH_LINENO[*]}"
|
||||
[[ "\${DEBUG_EXTENSION_CALLS}" == "yes" ]] && display_alert "---> Extension Method ${hook_point}" "${hook_point_functions_loop_counter}/${hook_point_functions_counter} (ext:${EXTENSION:-built-in}) ${hook_point_function}" ""
|
||||
echo "*** *** Extension-managed hook starting ${hook_point_functions_loop_counter}/${hook_point_functions_counter} '${hook_point}${hook_extension_delimiter}${hook_point_function}':" >>"\${EXTENSION_MANAGER_LOG_FILE}"
|
||||
${hook_point_function_variables} ${hook_point}${hook_extension_delimiter}${hook_point_function} "\$@"
|
||||
echo "*** *** Extension-managed hook finished ${hook_point_functions_loop_counter}/${hook_point_functions_counter} '${hook_point}${hook_extension_delimiter}${hook_point_function}':" >>"\${EXTENSION_MANAGER_LOG_FILE}"
|
||||
FUNCTION_DEFINITION_CALLSITE
|
||||
|
||||
# output the cleanup for the implementation as well.
|
||||
cat <<-FUNCTION_CLEANUP_FOR_HOOK_POINT_IMPLEMENTATION >>"${fragment_manager_cleanup_file}"
|
||||
unset ${hook_point}${hook_extension_delimiter}${hook_point_function}
|
||||
FUNCTION_CLEANUP_FOR_HOOK_POINT_IMPLEMENTATION
|
||||
|
||||
# unset extension vars for the next loop.
|
||||
unset EXTENSION EXTENSION_DIR EXTENSION_FILE EXTENSION_ADDED_BY
|
||||
done
|
||||
|
||||
cat <<-FUNCTION_DEFINITION_FOOTER >>"${temp_source_file_for_hook_point}"
|
||||
echo "*** Extension-managed hook ending '${hook_point}': completed." >>"\${EXTENSION_MANAGER_LOG_FILE}"
|
||||
} # end ${hook_point}() function
|
||||
FUNCTION_DEFINITION_FOOTER
|
||||
|
||||
# unsets, lest the next loop inherits them
|
||||
unset hook_point_functions hook_point_functions_sortname_to_realname hook_point_functions_realname_to_sortname
|
||||
|
||||
# log what was produced in our own debug logfile
|
||||
cat "${temp_source_file_for_hook_point}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
cat "${fragment_manager_cleanup_file}" >>"${EXTENSION_MANAGER_LOG_FILE}"
|
||||
|
||||
# source the generated function.
|
||||
# shellcheck disable=SC1090
|
||||
source "${temp_source_file_for_hook_point}"
|
||||
|
||||
rm -f "${temp_source_file_for_hook_point}"
|
||||
done
|
||||
|
||||
# Dont show any output until we have more than 1 hook function (we implement one already, below)
|
||||
[[ ${hook_functions_counter} -gt 0 ]] &&
|
||||
display_alert "Extension manager" "processed ${hook_points_counter} Extension Methods calls and ${hook_functions_counter} Extension Method implementations" "info" | tee -a "${EXTENSION_MANAGER_LOG_FILE}"
|
||||
}
|
||||
|
||||
cleanup_extension_manager() {
|
||||
if [[ -f "${fragment_manager_cleanup_file}" ]]; then
|
||||
display_alert "Cleaning up" "extension manager" "info"
|
||||
# this will unset all the functions.
|
||||
# shellcheck disable=SC1090 # dynamic source, thanks, shellcheck
|
||||
source "${fragment_manager_cleanup_file}"
|
||||
fi
|
||||
# reset/unset the variables used
|
||||
initialize_extension_manager_counter=0
|
||||
unset extension_function_info defined_hook_point_functions hook_point_function_trace_sources hook_point_function_trace_lines fragment_manager_cleanup_file
|
||||
}
|
||||
|
||||
# why not eat our own dog food?
|
||||
# process everything that happened during extension related activities
|
||||
# and write it to the log. also, move the log from the .tmp dir to its
|
||||
# final location. this will make run_after_build() "hot" (eg, emit warnings)
|
||||
run_after_build__999_finish_extension_manager() {
|
||||
# export these maps, so the hook can access them and produce useful stuff.
|
||||
export defined_hook_point_functions hook_point_function_trace_sources
|
||||
|
||||
# eat our own dog food, pt2.
|
||||
call_extension_method "extension_metadata_ready" <<'EXTENSION_METADATA_READY'
|
||||
*meta-Meta time!*
|
||||
Implement this hook to work with/on the meta-data made available by the extension manager.
|
||||
Interesting stuff to process:
|
||||
- `"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt"` contains a list of all hook points called, in order.
|
||||
- For each hook_point in the list, more files will have metadata about that hook point.
|
||||
- `${EXTENSION_MANAGER_TMP_DIR}/hook_point.orig.md` contains the hook documentation at the call site (inline docs), hopefully in Markdown format.
|
||||
- `${EXTENSION_MANAGER_TMP_DIR}/hook_point.compat` contains the compatibility names for the hooks.
|
||||
- `${EXTENSION_MANAGER_TMP_DIR}/hook_point.exports` contains _exported_ environment variables.
|
||||
- `${EXTENSION_MANAGER_TMP_DIR}/hook_point.vars` contains _all_ environment variables.
|
||||
- `${defined_hook_point_functions}` is a map of _all_ the defined hook point functions and their extension information.
|
||||
- `${hook_point_function_trace_sources}` is a map of all the hook point functions _that were really called during the build_ and their BASH_SOURCE information.
|
||||
- `${hook_point_function_trace_lines}` is the same, but BASH_LINENO info.
|
||||
After this hook is done, the `${EXTENSION_MANAGER_TMP_DIR}` will be removed.
|
||||
EXTENSION_METADATA_READY
|
||||
|
||||
# Move temporary log file over to final destination, and start writing to it instead (although 999 is pretty late in the game)
|
||||
mv "${EXTENSION_MANAGER_LOG_FILE}" "${DEST}/${LOG_SUBPATH:-debug}/extensions.log"
|
||||
export EXTENSION_MANAGER_LOG_FILE="${DEST}/${LOG_SUBPATH:-debug}/extensions.log"
|
||||
|
||||
# Cleanup. Leave no trace...
|
||||
[[ -d "${EXTENSION_MANAGER_TMP_DIR}" ]] && rm -rf "${EXTENSION_MANAGER_TMP_DIR}"
|
||||
}
|
||||
|
||||
# This is called by call_extension_method(). To say the truth, this should be in an extension. But then it gets too meta for anyone's head.
|
||||
write_hook_point_metadata() {
|
||||
local main_hook_point_name="$1"
|
||||
|
||||
[[ ! -d "${EXTENSION_MANAGER_TMP_DIR}" ]] && mkdir -p "${EXTENSION_MANAGER_TMP_DIR}"
|
||||
cat - >"${EXTENSION_MANAGER_TMP_DIR}/${main_hook_point_name}.orig.md" # Write the hook point documentation received via stdin to a tmp file for later processing.
|
||||
shift
|
||||
echo -n "$@" >"${EXTENSION_MANAGER_TMP_DIR}/${main_hook_point_name}.compat" # log the 2nd+ arguments too (those are the alternative/compatibility names), separate file.
|
||||
compgen -A export >"${EXTENSION_MANAGER_TMP_DIR}/${main_hook_point_name}.exports" # capture the exported env vars.
|
||||
compgen -A variable >"${EXTENSION_MANAGER_TMP_DIR}/${main_hook_point_name}.vars" # capture all env vars.
|
||||
|
||||
# add to the list of hook points called, in order.
|
||||
echo "${main_hook_point_name}" >>"${EXTENSION_MANAGER_TMP_DIR}/hook_point_calls.txt"
|
||||
}
|
||||
|
||||
# Helper function, to get clean "stack traces" that do not include the hook/extension infrastructure code.
|
||||
get_extension_hook_stracktrace() {
|
||||
local sources_str="$1" # Give this ${BASH_SOURCE[*]} - expanded
|
||||
local lines_str="$2" # And this # Give this ${BASH_LINENO[*]} - expanded
|
||||
local sources lines index final_stack=""
|
||||
IFS=' ' read -r -a sources <<<"${sources_str}"
|
||||
IFS=' ' read -r -a lines <<<"${lines_str}"
|
||||
for index in "${!sources[@]}"; do
|
||||
local source="${sources[index]}" line="${lines[((index - 1))]}"
|
||||
# skip extension infrastructure sources, these only pollute the trace and add no insight to users
|
||||
[[ ${source} == */.tmp/extension_function_definition.sh ]] && continue
|
||||
[[ ${source} == *lib/extensions.sh ]] && continue
|
||||
[[ ${source} == */compile.sh ]] && continue
|
||||
# relativize the source, otherwise too long to display
|
||||
source="${source#"${SRC}/"}"
|
||||
# remove 'lib/'. hope this is not too confusing.
|
||||
source="${source#"lib/"}"
|
||||
# add to the list
|
||||
arrow="$([[ "$final_stack" != "" ]] && echo "-> ")"
|
||||
final_stack="${source}:${line} ${arrow} ${final_stack} "
|
||||
done
|
||||
# output the result, no newline
|
||||
# shellcheck disable=SC2086 # I wanna suppress double spacing, thanks
|
||||
echo -n $final_stack
|
||||
}
|
||||
|
||||
show_caller_full() {
|
||||
local frame=0
|
||||
while caller $frame; do
|
||||
((frame++))
|
||||
done
|
||||
}
|
||||
# can be called by board, family, config or user to make sure an extension is included.
|
||||
# single argument is the extension name.
|
||||
# will look for it in /userpatches/extensions first.
|
||||
# if not found there will look in /extensions
|
||||
# if not found will exit 17
|
||||
declare -i enable_extension_recurse_counter=0
|
||||
declare -a enable_extension_recurse_stack
|
||||
enable_extension() {
|
||||
local extension_name="$1"
|
||||
local extension_dir extension_file extension_file_in_dir extension_floating_file
|
||||
local stacktrace
|
||||
|
||||
# capture the stack leading to this, possibly with a hint in front.
|
||||
stacktrace="${ENABLE_EXTENSION_TRACE_HINT}$(get_extension_hook_stracktrace "${BASH_SOURCE[*]}" "${BASH_LINENO[*]}")"
|
||||
|
||||
# if LOG_ENABLE_EXTENSION, output useful stack, so user can figure out which extensions are being added where
|
||||
[[ "${LOG_ENABLE_EXTENSION}" == "yes" ]] &&
|
||||
display_alert "Extension being added" "${extension_name} :: added by ${stacktrace}" ""
|
||||
|
||||
# first a check, has the extension manager already initialized? then it is too late to enable_extension(). bail.
|
||||
if [[ ${initialize_extension_manager_counter} -gt 0 ]]; then
|
||||
display_alert "Extension problem" "already initialized -- too late to add '${extension_name}' (trace: ${stacktrace})" "err"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# check the counter. if recurring, add to the stack and return success
|
||||
if [[ $enable_extension_recurse_counter -gt 1 ]]; then
|
||||
enable_extension_recurse_stack+=("${extension_name}")
|
||||
return 0
|
||||
fi
|
||||
|
||||
# increment the counter
|
||||
enable_extension_recurse_counter=$((enable_extension_recurse_counter + 1))
|
||||
|
||||
# there are many opportunities here. too many, actually. let userpatches override just some functions, etc.
|
||||
for extension_base_path in "${SRC}/userpatches/extensions" "${SRC}/extensions"; do
|
||||
extension_dir="${extension_base_path}/${extension_name}"
|
||||
extension_file_in_dir="${extension_dir}/${extension_name}.sh"
|
||||
extension_floating_file="${extension_base_path}/${extension_name}.sh"
|
||||
|
||||
if [[ -d "${extension_dir}" ]] && [[ -f "${extension_file_in_dir}" ]]; then
|
||||
extension_file="${extension_file_in_dir}"
|
||||
break
|
||||
elif [[ -f "${extension_floating_file}" ]]; then
|
||||
extension_dir="${extension_base_path}" # this is misleading. only directory-based extensions should have this.
|
||||
extension_file="${extension_floating_file}"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# After that, we should either have extension_file and extension_dir, or throw.
|
||||
if [[ ! -f "${extension_file}" ]]; then
|
||||
echo "ERR: Extension problem -- cant find extension '${extension_name}' anywhere - called by ${BASH_SOURCE[1]}"
|
||||
exit 17 # exit, forcibly. no way we can recover from this, and next extensions will get bogus errors as well.
|
||||
fi
|
||||
|
||||
local before_function_list after_function_list new_function_list
|
||||
|
||||
# store a list of existing functions at this point, before sourcing the extension.
|
||||
before_function_list="$(compgen -A function)"
|
||||
|
||||
# error handling during a 'source' call is quite insane in bash after 4.3.
|
||||
# to be able to catch errors in sourced scripts the only way is to trap
|
||||
declare -i extension_source_generated_error=0
|
||||
trap 'extension_source_generated_error=1;' ERR
|
||||
|
||||
# source the file. extensions are not supposed to do anything except export variables and define functions, so nothing should happen here.
|
||||
# there is no way to enforce it though, short of static analysis.
|
||||
# we could punish the extension authors who violate it by removing some essential variables temporarily from the environment during this source, and restore them later.
|
||||
# shellcheck disable=SC1090
|
||||
source "${extension_file}"
|
||||
|
||||
# remove the trap we set.
|
||||
trap - ERR
|
||||
|
||||
# decrement the recurse counter, so calls to this method are allowed again.
|
||||
enable_extension_recurse_counter=$((enable_extension_recurse_counter - 1))
|
||||
|
||||
# test if it fell into the trap, and abort immediately with an error.
|
||||
if [[ $extension_source_generated_error != 0 ]]; then
|
||||
display_alert "Extension failed to load" "${extension_file}" "err"
|
||||
exit 4
|
||||
fi
|
||||
|
||||
# get a new list of functions after sourcing the extension
|
||||
after_function_list="$(compgen -A function)"
|
||||
|
||||
# compare before and after, thus getting the functions defined by the extension.
|
||||
# comm is oldskool. we like it. go "man comm" to understand -13 below
|
||||
new_function_list="$(comm -13 <(echo "$before_function_list" | sort) <(echo "$after_function_list" | sort))"
|
||||
|
||||
# iterate over defined functions, store them in global associative array extension_function_info
|
||||
for newly_defined_function in ${new_function_list}; do
|
||||
extension_function_info["${newly_defined_function}"]="EXTENSION=\"${extension_name}\" EXTENSION_DIR=\"${extension_dir}\" EXTENSION_FILE=\"${extension_file}\" EXTENSION_ADDED_BY=\"${stacktrace}\""
|
||||
done
|
||||
|
||||
# snapshot, then clear, the stack
|
||||
local -a stack_snapshot=("${enable_extension_recurse_stack[@]}")
|
||||
enable_extension_recurse_stack=()
|
||||
|
||||
# process the stacked snapshot, finally enabling the extensions
|
||||
for stacked_extension in "${stack_snapshot[@]}"; do
|
||||
ENABLE_EXTENSION_TRACE_HINT="RECURSE ${stacktrace} ->" enable_extension "${stacked_extension}"
|
||||
done
|
||||
|
||||
}
|
||||
Reference in New Issue
Block a user