mirror of
https://github.com/armbian/build
synced 2025-09-24 19:47:06 +07:00
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
This commit is contained in:
Ricardo Pardini
158
lib/functions/cli/entrypoint.sh
Normal file
158
lib/functions/cli/entrypoint.sh
Normal file
@@ -0,0 +1,158 @@
|
||||
function cli_entrypoint() {
|
||||
# array, readonly, global, for future reference, "exported" to shutup shellcheck
|
||||
declare -rg -x -a ARMBIAN_ORIGINAL_ARGV=("${@}")
|
||||
|
||||
if [[ "${ARMBIAN_ENABLE_CALL_TRACING}" == "yes" ]]; then
|
||||
set -T # inherit return/debug traps
|
||||
mkdir -p "${SRC}"/output/call-traces
|
||||
echo -n "" > "${SRC}"/output/call-traces/calls.txt
|
||||
trap 'echo "${BASH_LINENO[@]}|${BASH_SOURCE[@]}|${FUNCNAME[@]}" >> ${SRC}/output/call-traces/calls.txt ;' RETURN
|
||||
fi
|
||||
|
||||
# @TODO: allow for a super-early userpatches/config-000.custom.conf.sh to be loaded, before anything else.
|
||||
# This would allow for custom commands and interceptors.
|
||||
|
||||
# Decide what we're gonna do. We've a few hardcoded, 1st-argument "commands".
|
||||
declare -g -A ARMBIAN_COMMANDS_TO_HANDLERS_DICT ARMBIAN_COMMANDS_TO_VARS_DICT
|
||||
armbian_register_commands # this defines the above two dictionaries
|
||||
|
||||
# Process the command line, separating params (XX=YY) from non-params arguments.
|
||||
# That way they can be set in any order.
|
||||
declare -A -g ARMBIAN_PARSED_CMDLINE_PARAMS=() # A dict of PARAM=VALUE
|
||||
declare -a -g ARMBIAN_NON_PARAM_ARGS=() # An array of all non-param arguments
|
||||
parse_cmdline_params "${@}" # which fills the above vars.
|
||||
|
||||
# Now load the key=value pairs from cmdline into environment, before loading config or executing commands.
|
||||
# This will be done _again_ later, to make sure cmdline params override config et al.
|
||||
apply_cmdline_params_to_env "early" # which uses ARMBIAN_PARSED_CMDLINE_PARAMS
|
||||
# From here on, no more ${1} or stuff. We've parsed it all into ARMBIAN_PARSED_CMDLINE_PARAMS or ARMBIAN_NON_PARAM_ARGS and ARMBIAN_COMMAND.
|
||||
|
||||
declare -a -g ARMBIAN_CONFIG_FILES=() # fully validated, complete paths to config files.
|
||||
declare -g ARMBIAN_COMMAND_HANDLER="" ARMBIAN_COMMAND="" ARMBIAN_COMMAND_VARS="" # only valid command and handler will ever be set here.
|
||||
declare -g ARMBIAN_HAS_UNKNOWN_ARG="no" # if any unknown params, bomb.
|
||||
for argument in "${ARMBIAN_NON_PARAM_ARGS[@]}"; do # loop over all non-param arguments, find commands and configs.
|
||||
parse_each_cmdline_arg_as_command_param_or_config "${argument}" # sets all the vars above
|
||||
done
|
||||
|
||||
# More sanity checks.
|
||||
# If unknowns, bail.
|
||||
if [[ "${ARMBIAN_HAS_UNKNOWN_ARG}" == "yes" ]]; then
|
||||
exit_with_error "Unknown arguments found. Please check the output above and fix them."
|
||||
fi
|
||||
|
||||
# @TODO: Have a config that is always included? "${SRC}/userpatches/config-default.conf" ?
|
||||
|
||||
# If we don't have a command decided yet, use the undecided command.
|
||||
if [[ "${ARMBIAN_COMMAND}" == "" ]]; then
|
||||
display_alert "No command found, using default" "undecided" "debug"
|
||||
ARMBIAN_COMMAND="undecided"
|
||||
fi
|
||||
|
||||
# If we don't have a command at this stage, we should default either to 'build' or 'docker', depending on OS.
|
||||
# Give the chosen command a chance to refuse running, or, even, change the final command to run.
|
||||
# This allows for example the 'build' command to auto-launch under docker, even without specifying it.
|
||||
# Also allows for launchers to keep themselves when re-launched, yet do something diferent. (eg: docker under docker does build).
|
||||
# Or: build under Darwin does docker...
|
||||
# each _pre_run can change the command and vars to run too, so do it in a loop until it stops changing.
|
||||
declare -g ARMBIAN_CHANGE_COMMAND_TO="${ARMBIAN_COMMAND}"
|
||||
while [[ "${ARMBIAN_CHANGE_COMMAND_TO}" != "" ]]; do
|
||||
display_alert "Still a command to pre-run, this time:" "${ARMBIAN_CHANGE_COMMAND_TO}" "debug"
|
||||
|
||||
ARMBIAN_COMMAND="${ARMBIAN_CHANGE_COMMAND_TO}"
|
||||
armbian_prepare_cli_command_to_run "${ARMBIAN_COMMAND}"
|
||||
|
||||
ARMBIAN_CHANGE_COMMAND_TO=""
|
||||
armbian_cli_pre_run_command
|
||||
done
|
||||
|
||||
# IMPORTANT!!!: it is INVALID to relaunch compile.sh from here. It will cause logging mistakes.
|
||||
# So the last possible moment to relaunch is in xxxxx_pre_run!
|
||||
# Also form here, UUID will be generated, output created, logging enabled, etc.
|
||||
|
||||
# Init basic dirs.
|
||||
declare -g DEST="${SRC}/output" USERPATCHES_PATH="${SRC}"/userpatches # DEST is the main output dir, and USERPATCHES_PATH is the userpatches dir.
|
||||
mkdir -p "${DEST}" "${USERPATCHES_PATH}" # Create output and userpatches directory if not already there
|
||||
display_alert "Output directory created! DEST:" "${DEST}" "debug"
|
||||
|
||||
# set unique mounting directory for this execution.
|
||||
# basic deps, which include "uuidgen", will be installed _after_ this, so we gotta tolerate it not being there yet.
|
||||
declare -g ARMBIAN_BUILD_UUID
|
||||
if [[ "${ARMBIAN_BUILD_UUID}" != "" ]]; then
|
||||
display_alert "Using passed-in ARMBIAN_BUILD_UUID" "${ARMBIAN_BUILD_UUID}" "debug"
|
||||
else
|
||||
if [[ -f /usr/bin/uuidgen ]]; then
|
||||
ARMBIAN_BUILD_UUID="$(uuidgen)"
|
||||
else
|
||||
display_alert "uuidgen not found" "uuidgen not installed yet" "info"
|
||||
ARMBIAN_BUILD_UUID="no-uuidgen-yet-${RANDOM}-$((1 + $RANDOM % 10))$((1 + $RANDOM % 10))$((1 + $RANDOM % 10))$((1 + $RANDOM % 10))"
|
||||
fi
|
||||
ARMBIAN_BUILD_UUID="$(uuidgen)"
|
||||
display_alert "Generated ARMBIAN_BUILD_UUID" "${ARMBIAN_BUILD_UUID}" "debug"
|
||||
fi
|
||||
display_alert "Build UUID:" "${ARMBIAN_BUILD_UUID}" "debug"
|
||||
|
||||
# Super-global variables, used everywhere. The directories are NOT _created_ here, since this very early stage.
|
||||
export WORKDIR="${SRC}/.tmp/work-${ARMBIAN_BUILD_UUID}" # WORKDIR at this stage. It will become TMPDIR later. It has special significance to `mktemp` and others!
|
||||
export LOGDIR="${SRC}/.tmp/logs-${ARMBIAN_BUILD_UUID}" # Will be initialized very soon, literally, below.
|
||||
# @TODO: These are used by actual build, move to its cli handler.
|
||||
export SDCARD="${SRC}/.tmp/rootfs-${ARMBIAN_BUILD_UUID}" # SDCARD (which is NOT an sdcard, but will be, maybe, one day) is where we work the rootfs before final imaging. "rootfs" stage.
|
||||
export MOUNT="${SRC}/.tmp/mount-${ARMBIAN_BUILD_UUID}" # MOUNT ("mounted on the loop") is the mounted root on final image (via loop). "image" stage
|
||||
export EXTENSION_MANAGER_TMP_DIR="${SRC}/.tmp/extensions-${ARMBIAN_BUILD_UUID}" # EXTENSION_MANAGER_TMP_DIR used to store extension-composed functions
|
||||
export DESTIMG="${SRC}/.tmp/image-${ARMBIAN_BUILD_UUID}" # DESTIMG is where the backing image (raw, huge, sparse file) is kept (not the final destination)
|
||||
|
||||
# Make sure ARMBIAN_LOG_CLI_ID is set, and unique.
|
||||
# Pre-runs might change it, but if not set, default to ARMBIAN_COMMAND.
|
||||
declare -g ARMBIAN_LOG_CLI_ID="${ARMBIAN_LOG_CLI_ID:-${ARMBIAN_COMMAND}}"
|
||||
|
||||
LOG_SECTION="entrypoint" start_logging_section # This creates LOGDIR. @TODO: also maybe causes a spurious group to be created in the log file
|
||||
add_cleanup_handler trap_handler_cleanup_logging # cleanup handler for logs; it rolls it up from LOGDIR into DEST/logs @TODO: use the COMMAND in the filenames.
|
||||
|
||||
# @TODO: So gigantic contention point here about logging the basic deps installation.
|
||||
if [[ "${ARMBIAN_COMMAND_REQUIRE_BASIC_DEPS}" == "yes" ]]; then
|
||||
if [[ "${OFFLINE_WORK}" == "yes" ]]; then
|
||||
display_alert "* " "You are working offline!"
|
||||
display_alert "* " "Sources, time and host will not be checked"
|
||||
else
|
||||
# check and install the basic utilities;
|
||||
LOG_SECTION="prepare_host_basic" do_with_logging prepare_host_basic # This includes the 'docker' case.
|
||||
fi
|
||||
fi
|
||||
|
||||
# Source the extensions manager library at this point, before sourcing the config.
|
||||
# This allows early calls to enable_extension(), but initialization proper is done later.
|
||||
# shellcheck source=lib/extensions.sh
|
||||
source "${SRC}"/lib/extensions.sh
|
||||
|
||||
# Loop over the ARMBIAN_CONFIG_FILES array and source each. The order is important.
|
||||
for config_file in "${ARMBIAN_CONFIG_FILES[@]}"; do
|
||||
local config_filename="${config_file##*/}" config_dir="${config_file%/*}"
|
||||
display_alert "Sourcing config file" "${config_filename}" "debug"
|
||||
|
||||
# use pushd/popd to change directory to the config file's directory, so that relative paths in the config file work.
|
||||
pushd "${config_dir}" > /dev/null || exit_with_error "Failed to pushd to ${config_dir}"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
LOG_SECTION="userpatches_config:${config_filename}" do_with_logging source "${config_file}"
|
||||
|
||||
# reset completely after sourcing config file
|
||||
set -e
|
||||
#set -o pipefail # trace ERR through pipes - will be enabled "soon"
|
||||
#set -o nounset ## set -u : exit the script if you try to use an uninitialised variable - one day will be enabled
|
||||
set -o errtrace # trace ERR through - enabled
|
||||
set -o errexit ## set -e : exit the script if any statement returns a non-true return value - enabled
|
||||
|
||||
popd > /dev/null || exit_with_error "Failed to popd from ${config_dir}"
|
||||
|
||||
# Apply the params received from the command line _again_ after running the config.
|
||||
# This ensures that params take precedence over stuff possibly defined in the config.
|
||||
apply_cmdline_params_to_env "after config '${config_filename}'" # which uses ARMBIAN_PARSED_CMDLINE_PARAMS
|
||||
done
|
||||
|
||||
display_alert "Executing final CLI command" "${ARMBIAN_COMMAND}" "debug"
|
||||
armbian_cli_run_command
|
||||
display_alert "Done Executing final CLI command" "${ARMBIAN_COMMAND}" "debug"
|
||||
|
||||
# Build done, run the cleanup handlers explicitly.
|
||||
# This zeroes out the list of cleanups, so it"s not done again when the main script exits normally and trap = 0 runs.
|
||||
run_cleanup_handlers
|
||||
}
|
||||
Reference in New Issue
Block a user