JDM170/build
1
0
Fork 0
mirror of https://github.com/armbian/build synced 2025-09-24 19:47:06 +07:00
Code Issues Packages Projects Releases Wiki Activity

armbian-next: the great cli entrypoint (+docker) rewrite; introduce USE_LOCAL_APT_DEB_CACHE replacing apt-cacher-ng

Browse Source 
- armbian-next: introduce `USE_LOCAL_APT_DEB_CACHE` (default `=yes`) as alternative/in addition to `apt-cacher-ng` (eg, in Docker)
  - this uses `cache/aptcache/${RELEASE}-${ARCH}` (in the host) for
      - apt cache, by bind-mounting it to `${SDCARD}/var/cache/apt` in the `chroot_sdcard_apt_get()` runner and its usages
      - debootstrap, by passing it `--cache-dir`
  - utility function to help understand what is happening to cache during usage
  - apt itself mantains this cache, removing old packages when new ones are installed. apt does this _by default_
      - introduce `DONT_MAINTAIN_APT_CACHE=yes` to skip out of automatic apt maintenance of apt cache, eg, during `remove`s
      - don't do `apt clean` and such if using local cache, that would clean the cache, not the chroot
  - clean up `install_deb_chroot()` a little, find an unrelated bug there
- WiP: the great cli entrypoint (+docker) rewrite, Phase 6: relaunching structure; re-pass ARMBIAN_BUILD_UUID; use ARMBIAN_COMMAND for log filename; fix for output/logs dir perms
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 4/x; better logging, check & force `DEST_LANG`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 3/x; don't write to stderr in generated Dockerfile
  - it's `drastic red` on non-buildx dockers
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 2/x, logging
- WiP: the great cli entrypoint (+docker) rewrite, Phase 5: cleanups 1/x
  - source configs in a logging section.
  - Docker: silent, fast retries to make sure `docker system df` works
  - shut-up `chown` (no `-v`) output related to  `SET_OWNER_TO_UID`
  - ask user to wait while `DESTIMG` is rsync'ed to `FINALDEST` -- it's potentially very slow
  - use green apple for Mac logging, instead of red apple which might imply error...
- WiP: the great cli entrypoint (+docker) rewrite, Phase 4: run as non-root, maybe-with-Docker
  - introduce `is_docker_ready_to_go()`; if it is, and we're not root, use Docker instead of sudo. <- GOOD IDEA? BAD IDEA? lol
  - introduce `SET_OWNER_TO_UID` var to be passed to Docker/sudo so written files are owned by the launching user, not root.
    - introduce `mkdir_recursive_and_set_uid_owner()` and `reset_uid_owner()` to reset owner based on `SET_OWNER_TO_UID`
    - use it for userpatches files created, logs, and output files, including images and debs.
  - @TODOs ref. `$SUDO_USER` which I think the old version of this?
  - add a lot of @TODOs, ref being able to relaunch something that's not `build` inside Docker, also add/change params and configs and command.
    - initially add `ARMBIAN_DOCKER_RELAUNCH_EXTRA_ARGS`
- WiP: the great cli entrypoint (+docker) rewrite, Phase 3: rpardini is demented, v3
- WiP: the great cli entrypoint (+docker) rewrite, Phase 2: rpardini is demented
- WiP: the great cli entrypoint (+docker) rewrite, Phase 1
- armbian-next: WiP: Docker: actually use the GHA-image as base; pull it every 24hs.
  - using image in my private repo.
  - this has significant speedup to "start building time" on the 1st run
  - move some Linux specific stuff to its own if
  - add comments and todo
- armbian-next: WiP: Docker, high-WiP, beginnings of Armbian mount dict, with linux/darwin preferences
- armbian-next: WiP: Docker, configure `BUILDKIT_COLORS`
- armbian-next: WiP: Docker, make docker image from Dockerfile more compact by flattening layers
- armbian-next: `logging`: add whale indicator if build running under Docker
- armbian-next: WiP: `docker`: working with `bookworm`, `sid`, and `jammy` on Darwin & Linux; works with `bullseye` on Linux only
- armbian-next: WiP: `docker`: force ARMBIAN_RUNNING_IN_CONTAINER both in Dockerfile and passed as `--env`; apt update and install in same layer; back to jammy
- armbian-next: introduce `armbian_is_running_in_container()` and `armbian_is_host_running_systemd()`, replacing `systemd-detect-virt` in multiple spots
- WiP: try with debian:bullseye -- can't detect docker at all
- armbian-next: WiP: 2nd stab at new Docker support; Darwin still works; Linux `docker.io` working
  - gen .dockerignore together with Dockerfile
  - split in funcs
  - hacks for Linux and `/dev/loop` stuff, CONTAINER_COMPAT=yes
  - mac still works, Linux stuff would break it but I if'fed
- armbian-next: the secrets of `CONTAINER_COMPAT` revealed; add size checking to check_loop_device() and avoid retry when `mknod`ing
  - this fails for the right reasons now, causing retries, which are then retried and work ;-)
  - this is related to building under Docker on Linux, using docker.io package (not docker-ce)
- armbian-next: remove `.dockerignore` and add it to `.gitignore`; it's going to be auto-generated
- armbian-next: `.dockerignore`: Docker context should only have minimal files and folders, to speed up Dockerfile build
  - IMPORTANT: `.dockerignore` is going to be generated from now on: so this is the last commit with changes before removal
-  armbian-next: WiP: initial stab at new Docker support; really run the passed cmdline; add Dockerfile to gitignore
-  armbian-next: WiP: initial stab at new Docker support; generate Dockerfile; introduce REQUIREMENTS_DEFS_ONLY
  - uses REQUIREMENTS_DEFS_ONLY
  - works on Docker Desktop on Mac;
  - linux TBA
- armbian-next: don't error out if `.git` not present; other small fixes
- armbian-next: general "work or at least don't misbehave when run on a very bare ubuntu:latest instance"
  - can't assume things, for example:
  - that `sudo` will be available; it might not, and might be already root, no reason to fail
  - that `/etc/timezone` will exist
  - that `systemd-detect-virt` will be available
  - that `git` will be available
  - that `locale-gen` will be available
This commit is contained in:
Ricardo Pardini
2022-10-09 17:58:23 +02:00
parent 2c6751f584
commit d24d3327a8
37 changed files with 1476 additions and 504 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
lib/functions/logging/runners.sh
View File

@@ -9,16 +9,26 @@ function chroot_sdcard_apt_get_install_download_only() {
chroot_sdcard_apt_get --no-install-recommends --download-only install "$@"
}
function chroot_sdcard_apt_get_remove() {
DONT_MAINTAIN_APT_CACHE="yes" chroot_sdcard_apt_get remove "$@"
}
function chroot_sdcard_apt_get() {
acng_check_status_or_restart # make sure apt-cacher-ng is running OK.
local -a apt_params=("-${APT_OPTS:-y}")
local -a apt_params=("-y")
[[ $NO_APT_CACHER != yes ]] && apt_params+=(
-o "Acquire::http::Proxy=\"http://${APT_PROXY_ADDR:-localhost:3142}\""
-o "Acquire::http::Proxy::localhost=\"DIRECT\""
)
apt_params+=(-o "Dpkg::Use-Pty=0") # Please be quiet
if [[ "${DONT_MAINTAIN_APT_CACHE:-no}" == "yes" ]]; then
# Configure Clean-Installed to off
display_alert "Configuring APT to not clean up the cache" "APT will not clean up the cache" "debug"
apt_params+=(-o "APT::Clean-Installed=0")
fi
# Allow for clean-environment apt-get
local -a prelude_clean_env=()
if [[ "${use_clean_environment:-no}" == "yes" ]]; then
@@ -26,7 +36,26 @@ function chroot_sdcard_apt_get() {
prelude_clean_env=("env" "-i")
fi
chroot_sdcard "${prelude_clean_env[@]}" DEBIAN_FRONTEND=noninteractive apt-get "${apt_params[@]}" "$@"
local use_local_apt_cache apt_cache_host_dir
local_apt_deb_cache_prepare use_local_apt_cache apt_cache_host_dir "before 'apt-get $*'" # 2 namerefs + "when"
if [[ "${use_local_apt_cache}" == "yes" ]]; then
# prepare and mount apt cache dir at /var/cache/apt/archives in the SDCARD.
local apt_cache_sdcard_dir="${SDCARD}/var/cache/apt"
run_host_command_logged mkdir -pv "${apt_cache_sdcard_dir}"
display_alert "Mounting local apt cache dir" "${apt_cache_sdcard_dir}" "debug"
run_host_command_logged mount --bind "${apt_cache_host_dir}" "${apt_cache_sdcard_dir}"
fi
local chroot_apt_result=1
chroot_sdcard "${prelude_clean_env[@]}" DEBIAN_FRONTEND=noninteractive apt-get "${apt_params[@]}" "$@" && chroot_apt_result=0
local_apt_deb_cache_prepare use_local_apt_cache apt_cache_host_dir "after 'apt-get $*'" # 2 namerefs + "when"
if [[ "${use_local_apt_cache}" == "yes" ]]; then
display_alert "Unmounting apt cache dir" "${apt_cache_sdcard_dir}" "debug"
run_host_command_logged umount "${apt_cache_sdcard_dir}"
fi
return $chroot_apt_result
}
# please, please, unify around this function.
@@ -101,7 +130,7 @@ function host_apt_get_install() {
# For running apt-get stuff host-side. Not chroot!
function host_apt_get() {
local -a apt_params=("-${APT_OPTS:-y}")
local -a apt_params=("-y")
apt_params+=(-o "Dpkg::Use-Pty=0") # Please be quiet
run_host_command_logged DEBIAN_FRONTEND=noninteractive apt-get "${apt_params[@]}" "$@"
}
@@ -217,12 +246,20 @@ run_on_sdcard() {
function do_with_retries() {
local retries="${1}"
shift
local sleep_seconds="${sleep_seconds:-5}"
local silent_retry="${silent_retry:-no}"
local counter=0
while [[ $counter -lt $retries ]]; do
counter=$((counter + 1))
"$@" && return 0 # execute and return 0 if success; if not, let it loop;
display_alert "Command failed, retrying in 5s" "$*" "warn"
sleep 5
if [[ "${silent_retry}" == "yes" ]]; then
: # do nothing
else
display_alert "Command failed, retrying in ${sleep_seconds}s" "$*" "warn"
fi
sleep ${sleep_seconds}
done
display_alert "Command failed ${counter} times, giving up" "$*" "warn"
return 1