mirror of
https://github.com/armbian/build
synced 2025-09-24 19:47:06 +07:00
armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
- this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
- apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
- debootstrap, by passing it `--cache-dir`
- utility function to help understand what is happening to cache during usage
- apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
- introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
- don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
- clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
- it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
- source configs in a logging section.
- Docker: silent, fast retries to make sure `docker system df` works
- shut-up `chown` (no `-v`) output related to `SET_OWNER_TO_UID`
- ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
- use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
- introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
- introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
- introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
- use it for userpatches files created, logs, and output files, including images and debs.
- @TODOs ref. `$SUDO_USER` which I think the old version of this?
- add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
- initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
- using image in my private repo.
- this has significant speedup to "start building time" on the 1st run
- move some Linux specific stuff to its own if
- add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
- gen .dockerignore together with Dockerfile
- split in funcs
- hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
- mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
- this fails for the right reasons now, causing retries, which are then retried and work ;-)
- this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
- IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
- armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
- armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
- uses REQUIREMENTS_DEFS_ONLY
- works on Docker Desktop on Mac;
- linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
- can't assume things, for example:
- that `sudo` will be available; it might not, and might be already root, no reason to fail
- that `/etc/timezone` will exist
- that `systemd-detect-virt` will be available
- that `git` will be available
- that `locale-gen` will be available
This commit is contained in:
Ricardo Pardini
@@ -10,6 +10,13 @@ function logging_init() {
|
||||
if [[ "${CI}" == "true" ]]; then # ... but that is too dark for Github Actions
|
||||
export tool_color="${normal_color}"
|
||||
fi
|
||||
if [[ "${ARMBIAN_RUNNING_IN_CONTAINER}" == "yes" ]]; then # if in container, add a cyan "whale emoji" to the left marker wrapped in dark gray brackets
|
||||
local container_emoji="🐳" # 🐳 or 🐋
|
||||
export left_marker="${gray_color}[${container_emoji}|${normal_color}"
|
||||
elif [[ "$(uname -s)" == "Darwin" ]]; then # if on Mac, add a an apple emoji to the left marker wrapped in dark gray brackets
|
||||
local mac_emoji="🍏" # 🍏 or 🍎
|
||||
export left_marker="${gray_color}[${mac_emoji}|${normal_color}"
|
||||
fi
|
||||
}
|
||||
|
||||
function logging_error_show_log() {
|
||||
@@ -295,7 +302,7 @@ function export_ansi_logs() {
|
||||
----------------------------------------------------------------------------------------------------------------
|
||||
ANSI_HEADER
|
||||
|
||||
if [[ -n "$(command -v git)" ]]; then
|
||||
if [[ -n "$(command -v git)" && -d "${SRC}/.git" ]]; then
|
||||
display_alert "Gathering git info for logs" "Processing git information, please wait..." "debug"
|
||||
cat <<- GIT_ANSI_HEADER > "${target_file}"
|
||||
----------------------------------------------------------------------------------------------------------------
|
||||
@@ -388,51 +395,76 @@ function export_html_logs() {
|
||||
display_alert "Built HTML log file" "${target_file}"
|
||||
}
|
||||
|
||||
function discard_logs_tmp_dir() {
|
||||
# Linux allows us to be more careful, but really, those are log files we're talking about.
|
||||
if [[ "$(uname)" == "Linux" ]]; then
|
||||
rm -rf --one-file-system "${LOGDIR}"
|
||||
else
|
||||
rm -rf "${LOGDIR}"
|
||||
fi
|
||||
}
|
||||
|
||||
# Cleanup for logging.
|
||||
function trap_handler_cleanup_logging() {
|
||||
[[ "x${LOGDIR}x" == "xx" ]] && return 0
|
||||
[[ "x${LOGDIR}x" == "x/x" ]] && return 0
|
||||
[[ ! -d "${LOGDIR}" ]] && return 0
|
||||
|
||||
display_alert "Cleaning up log files" "LOGDIR: '${LOGDIR}'" "debug"
|
||||
|
||||
# `pwd` might not even be valid anymore. Move back to ${SRC}
|
||||
cd "${SRC}" || exit_with_error "cray-cray about SRC: ${SRC}"
|
||||
|
||||
# Just delete LOGDIR if in CONFIG_DEFS_ONLY mode.
|
||||
if [[ "${CONFIG_DEFS_ONLY}" == "yes" ]]; then
|
||||
display_alert "Discarding logs" "CONFIG_DEFS_ONLY=${CONFIG_DEFS_ONLY}" "debug"
|
||||
rm -rf --one-file-system "${LOGDIR}"
|
||||
discard_logs_tmp_dir
|
||||
return 0
|
||||
fi
|
||||
|
||||
local target_path="${DEST}/logs"
|
||||
mkdir -p "${target_path}"
|
||||
mkdir_recursive_and_set_uid_owner "${target_path}"
|
||||
|
||||
# Before writing new logfile, compress and move existing ones to archive folder. Unless running under CI.
|
||||
if [[ "${CI}" != "true" ]]; then
|
||||
# Before writing new logfile, compress and move existing ones to archive folder.
|
||||
# - Unless running under CI.
|
||||
# - Also not if signalled via SKIP_LOG_ARCHIVE=yes
|
||||
if [[ "${CI:-false}" != "true" && "${SKIP_LOG_ARCHIVE:-no}" != "yes" ]]; then
|
||||
declare -a existing_log_files_array
|
||||
mapfile -t existing_log_files_array < <(find "${target_path}" -maxdepth 1 -type f -name "armbian-logs-*.*")
|
||||
mapfile -t existing_log_files_array < <(find "${target_path}" -maxdepth 1 -type f -name "armbian-*.*")
|
||||
declare one_old_logfile old_logfile_fn target_archive_path="${target_path}"/archive
|
||||
for one_old_logfile in "${existing_log_files_array[@]}"; do
|
||||
old_logfile_fn="$(basename "${one_old_logfile}")"
|
||||
display_alert "Archiving old logfile" "${old_logfile_fn}" "debug"
|
||||
mkdir -p "${target_archive_path}"
|
||||
if [[ "${old_logfile_fn}" == *${ARMBIAN_BUILD_UUID}* ]]; then
|
||||
display_alert "Skipping archiving of current logfile" "${old_logfile_fn}" "warn"
|
||||
continue
|
||||
fi
|
||||
display_alert "Archiving old logfile" "${old_logfile_fn}" "warn"
|
||||
mkdir_recursive_and_set_uid_owner "${target_archive_path}"
|
||||
|
||||
# Check if we have `zstdmt` at this stage; if not, use standard gzip
|
||||
if [[ -n "$(command -v zstdmt)" ]]; then
|
||||
zstdmt --quiet "${one_old_logfile}" -o "${target_archive_path}/${old_logfile_fn}.zst"
|
||||
reset_uid_owner "${target_archive_path}/${old_logfile_fn}.zst"
|
||||
else
|
||||
# shellcheck disable=SC2002 # my cat is not useless. a bit whiny. not useless.
|
||||
cat "${one_old_logfile}" | gzip > "${target_archive_path}/${old_logfile_fn}.gz"
|
||||
reset_uid_owner "${target_archive_path}/${old_logfile_fn}.gz"
|
||||
fi
|
||||
rm -f "${one_old_logfile}"
|
||||
done
|
||||
else
|
||||
display_alert "Not archiving old logs." "CI=${CI:-false}, SKIP_LOG_ARCHIVE=${SKIP_LOG_ARCHIVE:-no}" "debug"
|
||||
fi
|
||||
|
||||
if [[ "${EXPORT_HTML_LOG}" == "yes" ]]; then
|
||||
local target_file="${target_path}/armbian-logs-${ARMBIAN_BUILD_UUID}.html"
|
||||
local target_file="${target_path}/armbian-${ARMBIAN_LOG_CLI_ID}-${ARMBIAN_BUILD_UUID}.html"
|
||||
export_html_logs
|
||||
reset_uid_owner "${target_file}"
|
||||
fi
|
||||
|
||||
local target_file="${target_path}/armbian-logs-ansi-${ARMBIAN_BUILD_UUID}.txt.log"
|
||||
local target_file="${target_path}/armbian-${ARMBIAN_LOG_CLI_ID}-${ARMBIAN_BUILD_UUID}.ansitxt.log"
|
||||
export_ansi_logs
|
||||
reset_uid_owner "${target_file}"
|
||||
|
||||
rm -rf --one-file-system "${LOGDIR}"
|
||||
discard_logs_tmp_dir
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user