- both artifacts use functions in the bsp/utils-bsp.sh file, which wasn't being hashed
- bsp-cli hashed non-existent 'packages/bsp-cli' directory, remove
- bsp-cli: be verbose when rsync'ing `packages/bsp/common`
- force sync in a few strategic spots
- utils-bsp.sh: be verbose when copying files (incl `config/optional` stuff)
- in preparation for tightening the shellcheck severity level
- it needs to be able to follow all sources; dynamic ones are ignored, static ones need root-relative prefix
Sometimes we need to fix minor issues like changing the key or fixing other small problem on live OS. We can ship this as a part of BSP package, but its handling should be made easy.
This downloads script from CDN, verify its signature and executes after apt upgrade starts installing packages.
* Record patch output to syslog
* Enable armbian-live-patch as additional service and run patch mechanism at startup
* Additional security check
Improve naming confusion by changing build framework defaults. Set framework defaults VENDOR to Armbian-unofficial for self build images. Unsupported boards and unsupported distribution have framework defaults, except VENDOR and VENDORURL. We build images with predefined values.
- bsp-cli: `postinst` is now generated _after_ the hooks are run
- hooks are allowed to append to `postinst_functions` array
- hooks are _not_ allowed to modify `DEBIAN/postinst` directly anymore (it won't even be there)
- rewrite the postinst stuff in `helios4`'s `family_tweaks_bsp()` to use new way
- introduce `add_file_from_stdin_to_bsp_destination()`, a helper for easy heredoc-based adding of files to bsp-cli
- First and only argument is the destination path, relative to the root of the package -- do NOT include $destination -- it is already included.
- Containing directory, if any, is created automatically (no more "mkdir -p x; cp y x/z")
- The full path (including $destination) is set in $file_added_to_bsp_destination, declare in outer scope to get it if needed.
> tl-dr:
> - maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
> - publish simple `Version:`'s that don't include a crazy hash in repo and images
> - introduce `output/packages-hashed` directory
> - radically change the `output/debs` directory structure
- simplify artifact's `prepare_version()` method for `deb` and `deb-tar` artifacts:
- `artifact_base_dir` and `artifact_final_file` will now be auto-calculated; thus removed from each artifact (except `rootfs`)
- `artifact_deb_repo` ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
- `artifact_deb_arch` is now required, "all" is arch-independent, otherwise use `${ARCH}`
- `artifact_map_debs` is now auto-calculated based on the above, and shouldn't be specified manually
- `artifact_final_version_reversioned` is optional, and can force the final version of the artifact (specific for the `base-files` case)
- artifacts that need special handling for reversioning can add function names to `artifact_debs_reversion_functions` array (`base-files` and `bsp-cli` cases)
- artifacts `prepare_version()` should set `artifact_version`, but _never_ include it in other variables; `artifact_version` is now changed by framework after `prepare_version()` returns
- no longer use/refer/mention `${REVISION}` when building packages. All packages should be `${REVISION}`-agnostic.
- `${REVISION}` (actually, `artifact_final_version_reversioned`) will be automatically swapped in the `control` file during reversioning
- `fakeroot_dpkg_deb_build()` now takes exactly two arguments: the directory to pack, and the deb ID (key of `artifact_map_packages` dict); add this change in all the artifact's code for this
- `obtain_complete_artifact()`:
- automatically adds `-Rxxxx` "revisioning-hash" to `artifact_version`, by hashing the revisioning functions and any `artifact_debs_reversion_functions` set
- calculates more complex subdirectory paths for both the `output/packages-hashed` and `output/debs`/`output/debs-beta` directories
- with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial `download-debs` re-runs)
- in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
- exports a lot more information to JSON, including the new subdirectory paths
- comment-out code that implemented `skip_unpack_if_found_in_caches`, I'm very unsure why we had this in the first place
- `obtain_artifact_from_remote_cache()`
- for `deb` type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
- this is not needed for `deb-tar`, since that can preserve the dir structure itself
- introduce `artifacts-reversion.sh` and its main function `artifact_reversion_for_deployment()`
- this has the logic for reversioning .deb's, by `ar`-unpacking them, changing `control.tar` (and possibly `data.tar`), handling `.xz` compression, etc.
- also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
- by default, it changes `control` file only:
- replace `Version:` (which is the hash-version originally) with `artifact_final_version_reversioned` (which is mostly just `${REVISION}`)
- add a custom field `Armbian-Original-Hash:` with the original hash-version
- `artifact_reversion_for_deployment()` is called by
- new CLI wrapper `cli_obtain_complete_artifact()`, used for CLI building of specific artifact, but also for `download-artifact`
- `build_artifact_for_image()` used during image build
- `armbian-bsp-cli-deb.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `artifact-armbian-base-files.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `kernel`:
- add some custom fields to `DEBIAN/control`:
- `Armbian-Kernel-Version:` / `Armbian-Kernel-Version-Family:` (for future use: cleanup of usage of `Source: ` field which should be removed)
- declutter the `Description:` field, moving long description out of the first line
- obtain `IMAGE_INSTALLED_KERNEL_VERSION` from the reversioned deb (this is still a hack and has not been fixed)
- `uboot`:
- declutter the `Description:` field, moving long description out of the first line
- use the reversioned .deb when deploying u-boot to the image
- `main_default_build_packages()` now stores reversioned values and complete paths to reversioned .deb's
- `list_installed_packages()` now compares custom field `Armbian-Original-Hash: `, and not the `Version:` to make sure debs in the image are the ones we want
- `install_artifact_deb_chroot()` is a new wrapper around `install_deb_chroot()` for easy handling of reversioned debs
- use it everywhere `install_deb_chroot()` was used in `distro-agnostic.sh` and `distro-specific.sh`
- bsp-cli: now depends on `base-files (>= ${REVISION})`, this way upgrading the bsp-cli causes our base-files to be installed
- bsp-cli no longer does gymnastics with /etc/os-release et al, all done in armbian-base-files now
- general/apt-utils.sh: introduce `apt_find_upstream_package_version_and_download_url()`
- base-files: add release to version, in order to comply with repo restrictions (valid repos can't have two different debs with same name and version, md5 must match)
- bsp-cli: does not require aggregation
- bsp-desktop / desktop: requires aggregation
- 'desktop' packaged moved out of 'bsp' folder, it is NOT a bsp
- make consistent naming/folders
- extract .sh generation workflow into 'declare -f' magic functions
- `SHOW_DEBIAN=yes` to bump most of `DEBIAN/xxx` stuff to screen
- artifact armbian-bsp-desktop: use aggregation, require certain vars, CLI forces `BUILD_DESKTOP=yes`; requires BOARD
- artifact armbian-desktop: use aggregation, require certain vars, CLI forces `BUILD_DESKTOP=yes`, not require `BOARD` anymore
- cli: commands: fix artifact aliases, reorder
- desktop/bsp-desktop: add `config_dump` method; fix `artifact_map_debs` ref subdir and name
- artifact bsps: require aggregation to build, not to list what needs to be built
- distro-agnostic: fix install of `armbian-bsp-cli` (not `armbian-bsp`)
- armbian-bsp-cli: add `_config_dump` method, use configuration with aggregation, add required vars check
- distro-specific: fix installation of `fake-ubuntu-advantage-tools`
* Add / modify (c) in bash scripts
Signed-off-by: Igor <igor@armbian.com>
* Add (c) to the source config files
---------
Signed-off-by: Igor <igor@armbian.com>
- also compile.sh
- shellfmt lib
- split off shell and python tools under lib
- revert removal of stuff a-n no longer uses (ref. compilation): general packaging, mkdeb etc
- editoconfig split off
- extension changes split off
- sources and sources/families split off
- some undue stuff removed or split
armbian-next: manual merge (30) of lib changes between 882f995e21 and 31ac6383e1
armbian-next: manual merge (30) of family/board changes between 882f995e21 and 31ac6383e1
armbian-next: manual merge (29) of family/board changes between 3435c46367 and 882f995e21 (A LOT!)
armbian-next: manual merge (29) of lib changes between 3435c46367 and 882f995e21 (A LOT!)
armbian-next: manual merge (28) of lib changes between revisions af6ceee6c5 and 38df56fbf3
armbian-next: manual merge (28) of sources/families changes between revisions af6ceee6c5 and 38df56fbf3
armbian-next: manual merge (27) of `lib` changes between revisions 9c52562176 and af6ceee6c5
armbian-next: manual merge (27) of `sources/families` changes between revisions 9c52562176 and af6ceee6c5
armbian-next: move `ROOTFSCACHE_VERSION` resolution from GitHub from `main-config` down to `create-cache`
- this way config does not depend on remote...
armbian-next: move `ARMBIAN_MIRROR` selection (network) from `main-config` to `prepare-host`
- this way CONFIG_DEFS_ONLY can run without touching the network
armbian-next: manual merge (26) of MD5-checking via debsums (3955) re-imagined
- @TODO make sure
armbian-next: manual merge (26) of sources/families changes between revisions 20ee8c5450 and 9c52562176
armbian-next: manual merge (26) of lib changes between revisions 20ee8c5450 and 9c52562176
- @TODO NOT including the md5/debsums check, that needs further rewrite
armbian-next: manual merge (25) of lib changes between revisions fe972621c6 and 20ee8c5450
- @TODO hmm Igor is now going out to the network for rootfs cache version during configuration phase!!! BAD BAD BAD
armbian-next: manual merge (25) of family changes between revisions fe972621c6 and 20ee8c5450
armbian-next: manual merge (24) of families changes between revisions 9ca9120420 and 560531a635
armbian-next: manual merge (24) of lib changes between revisions 9ca9120420 and 560531a635
armbian-next: manual merge (23) of all changes between revisions 17b4fb913c and 9ca9120420
armbian-next: manual merge (22) of all changes between revisions 0eb8fe7497 and 1dddf78cd0
- @TODO EXCEPT the insanity about locales/eval/VERYSILENT in #3850, requires deep review
armbian-next: manual merge (21) of all changes between revisions e7d7dab1bb and 0eb8fe7497
armbian-next: fix: patching CREATE_PATCHES=yes
- needed to create output dir
armbian-next: add `python2-dev` dep for old uboots
- cleanup some comments
armbian-next: manual merge (20) of all changes between revisions 6b72ae3c86 and 247c4c45fd
armbian-next: fix: pass `TERM` to kernel's make, so `make menuconfig` can work
armbian-next: fix: git: read commit UNIX timestamp/local date correctly
- `checked_out_revision_ts` was correct; git outputs `%ct` as a UNIX timestamp, UTC-based
- `checked_out_revision_mtime` was incorrect: git output it without converting to local time
- manually convert using `date @xx` so it has correct local time, whatever it is.
- add debugging to `get_file_modification_time()` too
armbian-next: abstract `$QEMU_BINARY` to `qemu-static.sh`: `deploy_qemu_binary_to_chroot()`/`undeploy_qemu_binary_from_chroot()`
- add hackish logic to avoid removing binary that would be needed if image actually contains `qemu-user-static` package
armbian-next: fix `uuidgen` basic dep check; use fake bash `$RANDOM` if uuidgen not available
- not good: we need uuidgen to begin logging, but it may not be installed yet. workaround.
armbian-next: retry 3 times download-only also for `PACKAGE_LIST_BOARD`
- acng is really not helping
armbian-next: allow customizing UBUNTU_MIRROR (ports mirror) with `CUSTOM_UBUNTU_MIRROR_ARM64=host/path`
armbian-next: WiP: kernel make via `env -i` for clean env; show produced /boot tree
armbian-next: manual merge (19) of all changes between revisions b23498b949 and e621d25adc
- the ssh firstrun revert stuff mostly
armbian-next: *breaking change* remove `LIB_TAG` and `.ignore_changes` completely
- one day should be replaced with an "update checker" extension, or even "update-enforcer"
- for now this just causes chaos
armbian-next: `python2` is required for some u-boot builds
- would be "use `python-is-python2` so /usr/bin/python exists and points to Python 2.x" but Jammy does not have that anymore
- python2 is required for some u-boot builds.
- that said, python 2.x is deprecated for a while and needs work thus @TODO
armbian-next: bump Python info gatherer to RELEASE=jammy too
armbian-next: add `KERNEL_MAJOR_MINOR` info to `media` kernel (@balbes150)
- 5.18 is not yet released so might be a problem here
armbian-next: allow to skip submodules during `fetch_from_repo`; introduce hook `fetch_custom_uboot`
- via GIT_SKIP_SUBMODULES=yes, which disables all submodules everywhere
- via UBOOT_GIT_SKIP_SUBMODULES=yes, which disables fetching of submodules during uboot fetch (hidden rkbins anyone?)
- extension hook `fetch_custom_uboot` so we can fetch our own stuff if needed
armbian-next: `initrd` caching fixes (always enable hook; if cache hit, convert to uImage too)
armbian-next: introduce `initramfs`/`initrd` caching
- using hashes of (hopefully) all involved files
- cache hits are rewarded with sprinkly hearts.
- why? this proves we got a reproducible kernel modules build!
- also, you just saved yourself 2-10 minutes of pain
armbian-next: manual merge (18) of changes between revisions 08cf31de73 and c8855aa08d
- heh; most bash code changes are for things already done in -next, or no longer used
- some version bumps, etc
armbian-next: cleanup entrypoint and shuffle `prepare_host_basic()` into logging section
armbian-next: *breaking change* add global extlinux killswitch `ALLOW_EXTLINUX`
- unless you set `ALLOW_EXTLINUX=yes`, then `SRC_EXTLINUX` will be disabled globally.
- add a bunch of logging regarding extlinux, armbianEnv and bootscripts for clarity during build
- this is due to nand-sata-install problems with extlinux
- some boards _only work_ with extlinux; we'll have to handle it later
armbian-next: extensions: `image-output-{qcow2|ovf}`: virtual output formats
- which use `qemu-utils` for `qemu-img` conversion of the .img
armbian-next: extension: `kernel-localmodconfig`: faster/slimmer kernel builds with `make localmodconfig`
armbian-next: extension: `cleanup-space-final-image`: zerofree, slim down firmware, show used space
armbian-next: introduce `do_with_ccache_statistics` and use it for kernel compile
- some TODOs
- better logging for .config copying
armbian-next: *breaking change* really disable apt sources for non-desktop builds
armbian-next: fix: don't manage apt-cacher-ng if told NOT to, not the other way around
armbian-next: `JUST_UBOOT=yes` + hooks `build_custom_uboot()`/`post_write_uboot_platform()`
- post_write_uboot_platform()
- only runs during build, for now (not on device)
- build_custom_uboot()
- allow fully custom, extension driven, building of u-boot
- also partial preparation of uboot source combined with default Armbian build
- HACK: u-boot: downgrade some errors to warnings via KCFLAGS
- fix copy of atf bins to uboot, don't do it if atf's not there
armbian-next: fix: no use testing the host for resolvconf if we're manipulating the SDCARD
armbian-next: sunxi_common: avoid shortcircuit error on family_tweaks_bsp when family_tweaks_bsp_s is not defined
armbian-next: fix: add `zstd` and `parallel` to hostdeps
armbian-next: manual merge (17) of all changes between revisions 64410fb74b and 08cf31de73
- changes about `git safe dir` ignored, I've done the same in a different way
- hash calculation changes ignored, fasthash is completely different
armbian-next: add `crossbuild-essential-armel` so `arm-linux-gnueabi-gcc` is available with system toolchains
- need to for some ATF builds, at least.
armbian-next: rockchip64_common: lotsa logging and debugging
- supposedly no practical changes
armbian-next: grub: better logging
armbian-next: fix for chaos caused by git's fix of CVE-2022-24765 otherwise "fatal: unsafe repository"
- might not be the best solution, but it's the only one I found
partitioning: fix: don't try fixing a bootscript that's not there
- this fixes a bug when "rootpart=2" without rootpart 1 being /boot
armbian-next: cleanups: umount tmpfs-based $SDCARD during cleanup too
armbian-next: indented heredoc, no functional changes
armbian-next: fix shortcircuit as last statement in case of extlinux
- yes, I wasted 3 hours on this tiny bit, so *you* don't have to!
- better logging for rootfs `mkfs` et al
- introduce `PRESERVE_SDCARD_MOUNT=yes` to preserve SDCARD, MOUNT, and LOOP for debugging
armbian-next: kernel-headers: less verbose, trimmed down tools a bit (perf and testing)
khadas-vim3l: add asound.state for Khadas VIM3L
armbian-next: introduce hook `extension_finish_config()` - late hook for ext configuration
- `extension_finish_config()` is the last thing done in config phase
- use it for determining stuff based on kernel version details, package names, etc
- also tune down some logging which was too verbose
- CI logs with no ANSI escape codes
armbian-next: shuffle around code and logic of `add_desktop_package_sources()`
- @TODO: still needs proper asset logging for sources.list(.d)
- @TODO: tunes down adding of sources/packages to CLI builds, check with Igor
armbian-next: 4.x can't build objtool in kernel-headers; allow for handling that later
- 4.x has a lot more obtuse dependencies
- introduce KERNEL_HAS_WORKING_HEADERS calculated based on KERNEL_MAJOR_MINOR
armbian-next: downgrade `error=misleading-indentation` to warning
- some 4.x kernels patches are really messy
- newer gcc's make that an error now
armbian-next: *allow cross compilation*, even the so-called "reverse cross-compile" (amd64 on arm64)
armbian-next: add `zfs` extension, which installs headers and builds ZFS via DKMS in chroot
- similar to how `nvidia` extension does it
armbian-next: x86: enable `nvidia` extension for all releases (only desktop)
armbian-next: `headers-debian-byteshift.patch` is dead; long-live cross-compiled source-only kernel-headers
- kernel-headers package now only includes _sources_
- postinst does the compilation and preparation for DKMS compatibility
- `tools` dir is included now, which includes the byteshift utilities
- handle special scripts/module.lds case after 5.10
- tested on a 6 combinations of `x86` / `arm64` / `armhf` (3x targets, 2x hosts)
- @TODO: we might be able to reduce the size of tools a bit (perf/tests/etc)
- @TODO: still missing ARCH vs ARCHITECTURE vs SRC_ARCH clarity elsewhere
armbian-next: allow `use_clean_environment=yes` for `chroot_sdcard_apt_get()` and descendants
- this causes command to be run under `env -i`, for a clean environment
armbian-next: manual merge (16) of all changes between revisions be9b5156a4 and 2a8e1ecac1
- many `traps` ignored: we don't use them anymore
armbian-next: fix logging for apt sources/gpg keys
armbian-next: don't leak `if_error_xxx` vars across runner helper invocations; always clean then (even if no error)
- also: fix wireguard-tools install, had a double parameter there
bcm2711: rpi4b: add `pi-bluetooth` which provides working Bluetooth
armbian-next: fixes for (non-)logging when interactively configuring kernel (`KERNEL_CONFIGURE=yes`)
armbian-next: move `lz4` rootfs caches to `zstd`, multithreaded
armbian-next: customize.sh: error handling, do not mount overlay if it doesn't exist
armbian-next: extra info for runners; `if_error_detail_message` and `if_error_find_files_sdcard` globals
- those are unset after running any command
- if error occur, message and/or found files will be included in log, for clarity
armbian-next: manual merge (15) of all changes between revisions 0f7200c793 and 101eaec907
armbian-next: better logging for `rsync` calls everywhere
- make rsync verbose
armbian-next: downloads: skip download if no `ARMBIAN_MIRROR` nor `DOWNLOAD_MIRROR`; less logs
armbian-next: update rockchip.conf from master and use runners
armbian-next: update mvebu64.conf from master and use functions
armbian-next: git: fix `fetch_from_repo` with actual submodules usage
armbian-next: `armbian-next`ify the `nvidia` extension after rebase from master
- driver version is configurable via `NVIDIA_DRIVER_VERSION`
- use runner function to log/error-handle/use apt cache/etc
rpi4b: there's no legacy branch anymore, remove it from KERNEL_TARGET
armbian-next: `download_and_verify` non-error handled; logging is messy [WiP] [HACK]
armbian-next: logging: let ANSI colors pass to logfile; CALLER_PID instead of BASHPID in subshell
armbian-next: enable HTTPS CONNECT in Armbian-managed apt-cacher-ng configuration
- PPAs require it
armbian-next: don't loop forever if we can't obtain ARMBIAN_MIRROR from redirector
- also, don't even try to do it if `SKIP_ARMBIAN_REPO=yes`
armbian-next: manual merge (14) of all changes between revisions 13469fd8a9 and 09e416e31c
- also editorconfig and compile.sh (root) changes
armbian-next: *much* improved logging to HTML; log archiving; consistency
- keep only current logfile
- log to LOGFILE also if SHOW_LOG=yes
- log cmd runtime and success/error directly in runner
armbian-next: *breaking change* use `MemAvailable` (not including swap) and up requirements for tmpfs
- of course add debugging logs
- rename vars
- should really only use this if we've really a lot of completely free RAM to spare
- otherwise OOM killer comes knocking
- or swapping to disk, that is counter-productive
armbian-next: *breaking change* `DEB_COMPRESS=none` by default if not running in CI/GHA
armbian-next: *breaking change* `CLEAN_LEVEL=make` is no more; new `make-kernel`, `make-atf`, `make-uboot`
- allows individual control of what to clean
- this effectively disables `make clean` by default
- rebuilds work and timestamping works for patching, so no reason to clean everytime by default.
armbian-next: refactor `prepare_host`, give `apt-cacher-ng` some much needed attention
- library dir for host-related stuff, pull it out of "general" finally
armbian-next: hostdeps: all toolchains via `crossbuild-essential-arm64`/`armhf`/`amd64`
- trying to sort out hostdeps for Jammy [WiP]
armbian-next: remove `eatmydata` usage, leftover from failed tries to make git faster
armbian-next: fix git origin check, recreate working copy if origin does not match
- fix cold bundle https download progress reporting
armbian-next: finally consolidating logs into output/logs; colorized HTML logs
armbian-next: introduce `do_with_retries()` and use it for apt remote operations during image build
armbian-next: another round of logging tuning/fixes; log assets; git logging
- introduce `do_with_log_asset()` and `LOG_ASSET=xxx`
- separate "git" logging level
- add `trap_handler_cleanup_destimg()` to cleanup DESTIMG
armbian-next: kernel: use parallel compressors; reproducible kernel builds
- also remove leftover mkdebian/builddeb parameters in make invocation
- add pbzip2 to hostdeps
armbian-next: tuning logging for timestamp/fasthash related stuff which is very verbose
- idea is to not completely overwhelm `SHOW_DEBUG=yes` case
- make patching quieter and use file instead of stdin
- set checked_out_revision_ts during git checkout (timestamp version of _mtime)
- timestamp | fasthash logging level (via `SHOW_FASTHASH=yes`)
armbian-next: completely remove mkdebian/builddeb/general-packaging kernel packaging stuff
armbian-next: manual merge (12) of all changes between revisions 34d4be6b7b and 5fe0f36aa8
armbian-next: introduce `PRESERVE_WORKDIR=yes` for heavy debugging
armbian-next: packaging linux-headers again
- do NOT use any output from `make headers_install` - that's for libc headers
- grabs "headers" (and binary tools) directly from the kernel build tree, not install target
- does not produce headers if cross compiling, for now
- produces kernel-headers package for the architecture on which it was built
- doing a single make invocation with build and install for packaging
- using 'make all' in place of vmlinuz/bzImage/image/zImage
armbian-next: apt download-only retried 3 times before installing main packages
armbian-next: fix `VER=` read from kernel-image package, also add `linux` default
armbian-next: some logging for atf compilation
armbian-next: rewrite hostdeps as array, add armhf toolchains
armbian-next: distro-agnostic: cleanups
armbian-next: armbianEnv hooks/debugs (bsp / image)
armbian-next: rpi: completely remove dtb hacks, allowing flash-kernel to work again
armbian-next: refactor new kernel packaging; add linux-dtb package back in finally, upgrades untested
armbian-next: refactor new kernel packaging; extract hook helper, fix kernel symlink
armbian-next: refactor new kernel packaging; add linux-dtb package back in finally, all hooks untested
flash-kernel: fix short-circuits as last statement in functions
armbian-next: do not force `SHOW_LOG=yes` if `CI=true`; let's _trust_ logging and error handling works
armbian-next: back out of setting mtime to the revision time during git checkout.
- of course this causes huge recompiles when wanted revision moves forward
armbian-next: sync 'config' dir from master revision ed589b248a
- this is _also_ getting out of hand... gotta merge soon
armbian-next: sync 'packages' dir from master revision ed589b248a
armbian-next: manual merge (11) of all lib/*.sh changes between revisions 3305d45b81 and ed589b248a
armbian-next: more refactorings, general logging; fixes; split image stuff
- logging flowing correct to LOGDIR, still needs packaging
armbian-next: complete removal of usages of `LOG_SUBPATH`; 100% error handled
- loose ends, use new LOGDIR
- remove the last shortcircuit in extensions execution, now it's 100% error handled
- many logging fixes
- still missing: final log consolidation/cleanup
logging: blue groups if `SHOW_DEBUG=yes` or `SHOW_GROUPS=yes` (console equivalent of CI's avocado)
armbian-next: shut down some too-verbose logging: logo building and update-initramfs
armbian-next: git/patching, kernel: use date from git as mtime minimum for patched files
- use revision's date from git log as mtime for all fetch_from_repo
- fix patched files date at least checkout date, otherwise some patches never build
armbian-next: first attempt at kernel packaging rework; just linux-image pkg, no dtbs yet
- correctly predict KERNELRELEASE, put image-dtbs in the right spot for flash-kernel
- remove dpkg-gencontrol, do it all directly
armbian-next: rework kernel source packaging, now exporting from git, to .tar.zst
- compress with zstdmt (multi-thread zstd), remove pv indicator, it's much faster anyway
- export from git (soon will have all patches et al too)
- better logging, show pkg name
- much, much faster due to zstdmt and deb with none compression
armbian-next: a bit atrocious, nameref loaded, `get_list_of_all_buildable_boards()`
- in the process, add support for userpatches/config structure mirroring core, for easy upstreaming
armbian-next: make `SKIP_EXTERNAL_TOOLCHAINS=yes` default. lets see what breaks [WiP]
armbian-next: keeping stdout clean, use display_alert() for cleanup logging
armbian-next: library cleanups; remove `build-all.sh` for good; bring `patching.sh` back
armbian-next: `interactive_desktop_main_configuration()` and stderr'ed + error handl'ed dialog
- use redirection to fd 3 for dialog, now cleanly on stderr
- `show_menu()` -> `dialog_menu()` et al
- interactive configuration now works again!
armbian-next: logging: `SHOW_PIDS=yes`
armbian-next: refactor and error-handle most of desktop configuration, incl menus/dialog
- `dialog_if_terminal_set_vars()` in place of `dialog_if_terminal()`
[WiP] ddk stuff, allow if not in `$KERNEL_TARGET`
armbian-next: split `compile_kernel()` function into smaller functions (+logging)
- `do_with_logging()` galore, much better error reporting for kernel
- `do_with_hooks()` is for the future, just a placeholder for now
armbian-next: `do_with_hooks()` placeholder for future ideas
armbian-next: logging: small refactor and `do_with_logging` admit it does not do error control
armbian-next: fix: traps: `trap_manager_error_handled` is integer (`-i`) not array (`-a`)
armbian-next: sunxi-tools: fix logging for sunxi-tools compilation
armbian-next: runners now run bash with `-o pipefail` in addition to `-e`
- attention, only affects stuff run through the functions in runners.sh
armbian-next: kernel: reduce logging clutter (CC,LD,AR)
- hide fasthash_debug under `SHOW_FASTHASH`
armbian-next: `armhf` should make `zImage` -- or should it?
armbian-next: show logs through ccze; avoid ANSI escapes in file; `SHOW_xxx` control
- `SHOW_DEBUG` shows the debug level
- `SHOW_COMMAND` shows all invoked commands through the runners
- `SHOW_TRAPS` to show 'cleanup' and 'trap' level
- `SHOW_TIMING` to show $SECONDS but pretty
- replace hardcoded traps/cleanups logging
armbian-next: add `KERNEL_MAJOR_MINOR=x.z` to every family, manually from the `KERNELBRANCH`
armbian-next: cold/warm bundles import/download/export for fetch_from_repo
- warm remote, if present, can be exported shallow
- if warm remote bundle is present, can be imported shallow too
- fallback to cold bundle if warm not present
- export (from cold, if exists + warm) shallow bundle
- use temp clone and DATE (not rev or tag) for shallowing, WORKS!
- info JSON/CSV, include "config_ok" true/false, kernel/uboot info
- include logs for failed configs too
- core reports ARMBIAN_WILL_BUILD_KERNEL and ARMBIAN_WILL_BUILD_UBOOT now with package names
armbian-next: `KERNELDIR` is out, `KERNEL_MAJOR_MINOR` is in for all `meson64`, `rpi4b` and `uefi`
armbian-next: new kernel workdir layout: cache/sources/kernel/<ARCH>-<KERNEL_MAJOR_MINOR>-<LINUXFAMILY>
- `GIT_FIXED_WORKDIR` is used to ignore 2nd param and use a specific dir
- this now REQUIRES `KERNEL_MAJOR_MINOR` to be set.
- prepare some `WARM_REMOTE_NAME` and related, based on it
armbian-next: JUST_KERNEL=yes (opposed to KERNEL_ONLY=yes) is really just the kernel build
armbian-next: fetch_from_repos now done when actually compiling atf/uboot/kernel, not before
- lib regen after removing empty files (sources.sh and errors.sh are now gone)
armbian-next: linux: back to Torvalds bundle, no tags; reminder about export
armbian-next: full cached kernel build; refactor all make's in a single place, even for packaging
- 2nd+ runs build in less than a minute
- kernel: compile and package in a single step, more efficient?
- KERNEL_BUILD_DTBS yes/no to build or not dtbs, replaces KERNEL_EXTRA_TARGETS
- dtbs_install, modules_install and headers_install now called by Armbian, not packaging
- kernel with split, but identical, build and install makes for modules/headers/dtbs
- make mkdebian and builddeb idempotent as possible
- keep a lot more cache, specially 'debian' folder
- filtering logging of install stuff
- might be a few leftovers, revisit gains with packaging later
- keeping the arm64 makefile Image.gz vs Image hack
- fix order of packaging patch byteshift, but still there
- cleaning of scripts tools on cross compile removed (!)
armbian-next: minor stylistic changes that I end up doing while working on other stuff
- I am `OFFLINE_WORK`'ing, I don't wanna waste 3 seconds, thanks
- OCD-fix of double `local` declarations
[giga-wip] rework patching, introducing fasthash
[wip] git: experiment with stable kernel bundle, and all tags. nice, but for what?
- also: never delete working copy, exit with error instead.
[wip] disable make clean during packaging. I wanna rebuild fast, always [NO PR?]
armbian-next: export CHOSEN_KERNEL_WITH_ARCH for reporting info
- fix info gathering, parse all boards first, and stop if some failed
- fix KERNEL_TARGET regex by including optional "export "
- add export from info to CSV, very basic stuff, but works
[squash] remove ddk bullshit from KERNEL_TARGET
armbian-next: remove file logging of aggregation stuff. config phase can't touch disk anymore.
[WiP] git cold bundle; no: shallow clones/fetched; yes: locally packed repo
armbian-next: reorder functions in file, they have a ~logical call-tree order
armbian-next: move `fingerprint_image()` out of `git.sh` into its own file
logging: fix for double stderr redirect during `fakeroot_dpkg_deb_build`
logging: subdued "command" logging for debugging low level cmd invocations ("frog")
armbian-next: when showing log, emit all host-side invocations as `debug` too.
[WiP] trap handler abstraction, somewhat works!
armbian-next: manual merge (10) of all lib/*.sh changes between revisions a4ae3a2270 and 3305d45b81 - but NOT the git unshallow stuff, that will be rewritten
armbian-next: trapmanager pt1, identifying spots for trap manager intervention
armbian-next: `install_pkg_deb` -> `install_host_side_packages` which is completely rewritten version
- much simplified; compare installed packages vs wanted, and only update if some missing
armbian-next: force u-boot and kernel's gcc to output colors to make easy to spot warnings and errors
docker: pass the `CI` env var through Docker invocation, for GitHub Actions
armbian-next: avoid warning if "file" utility not installed
- should not happen, but better safe than sorry
armbian-next: disable long_running `pv` progress bar for custom case too
- will rework later, for now pipe causes subshell and caos
armbian-next: if `CI=true` then `SHOW_LOG=yes` always
docker: add arm64 toolchain to Dockerfile; warn, but don't break, on modprobe failure
armbian-next: docker: use ubuntu:rolling, fix deps, use `$SRC/cache` as container's cache dir
armbian-next: logging fixes (padding, don't show error more than once, don't remove trap)
armbian-next: fixes for early error handling and logging
- split stacktrace-related functions into their own lib file
- simplify the traps
- some stacktrace formatting for readability
armbian-next: fix: don't `trap` `ERR` twice, it causes bash to go bananas regarding `caller`
armbian-next: `UPSTEM_VER` -> `UBOOT_REPO_VERSION` and related fixes
armbian-next: oops, fix some non-lib changes I missed, up to revision ff4346c468
armbian-next: manual merge (9) of all lib/*.sh changes between revisions 3b7f5b1f34 and ff4346c468
armbian-next: more error handling fixes. avoid shortcircuits.
- store previous error message in `MSG_IF_ERROR` (still to be actually shown somewhere during error)
armbian-next: more error handling fixes. avoid subshells, shortcircuits, and pipes
- add `CFLAGS=-fdiagnostics-color=always` to kernel compile; would need also for u-boot soon
WiP: indexing JSON into OpenSearch, all-hardcoded version
rpi: add DTB symlink in Debian/Ubuntu standard location /lib/firmware/$version/device-tree; remove build-time-only hacks
- this allows us to remove the most horrible hack
- should allow for correctly working DTB upgrades
- should NOT impact other families, although a new symlink will be created, nothing uses it.
rpi: fix: flash-kernel fix to ignore kernel 'flavour' for all raspi's
armbian-next: don't try to remove packages that are not installed to begin with
- much faster
- new chroot_sdcard_with_stdout() runner, without bash or any escaping.
armbian-next: don't try to enable systemd services for units that don't exist
- those might be removed by a bsp extension, so check for existence before trying to enable
armbian-next: don't error/warn on failure to enable bootsplash when systemd units missing
armbian-next: use indented HEREDOCS for all call_extension_method's
armbian-next: manual merge (8) of all lib/*.sh changes between revisions 1d499d9ac2 and 3b7f5b1f34
armbian-next: manual merge (7) of all lib/*.sh changes between revisions d885bfc97d and 1d499d9ac2
armbian-next: manual merge (6) of all lib/*.sh changes between revisions c7f3c239fe and d885bfc97d
armbian-next: avoid writing to disk during configuration; `ANSI_COLOR=none` logging; make CONFIG_DEFS_ONLY=yes runnable without sudo
- when `CONFIG_DEFS_ONLY=yes`, avoid writing the config summary output.log file.
- refactor that into a function as to be easy to if-out-of
- don't write to disk during aggregate_content() if `CONFIG_DEFS_ONLY=yes`
- don't write to disk during show_checklist_variables() if `CONFIG_DEFS_ONLY=yes`
- don't write to disk during write_deboostrap_list_debug_log() if `CONFIG_DEFS_ONLY=yes`
- don't compress and rotate logs if `CONFIG_DEFS_ONLY=yes`
- don't pretend to be handling errors we can't handle during var capture
- I foresee a world we can build all .debs without sudo
- and a some kind of split of codebase entrypoint due to that future feature
- some python info.py enhancements, not ready yet
armbian-next: shellfmt and regen library (after rebase from master n.5)
tools/shellfmt.sh: exclude "cache" and ".tmp" from formatting, for obvious reasons
tools/gen-library.sh: sort function files, so it does not keep changing between runs on different machines.
- order should not be important, since files only contain functions, but avoid git churn
armbian-next: manual merge (5) of all lib/*.sh changes between revisions 1b18df3c83 and e7962bb2b5
- most PKG_PREFIX work was already done
armbian-next: `TMPDIR` for all, many logging fixes, error handling: leave-no-garbage-behind without needing traps.
- set `MOUNT_UUID` and `WORKDIR`/`MOUNT`/`SDCARD`/`EXTENSION_MANAGER_TMP_DIR`/`DESTIMG` early in do_main_configuration()
- but, they're just _set_ there, dirs are not created early, but on demand later
- still @TODO: actually clean those during error trap. (unhappy path leaves garbage still)
- but does not leave garbage behind during "successful" runs at least (happy path works)
- actually export `TMPDIR` (== `WORKDIR`) during start of build (not config!), so all `mktemp` are subject to it
- `runners.sh` has helpers to avoid passing `TMPDIR` to chroot. Use the helpers! don't call `chroot` directly.
- don't trap/cleanup individual `mktemp` temp dirs during .deb packaging's, all is handled at once now.
- kernel packaging, for example, automatically picks up `TMPDIR` too. So now hosts `/tmp` is mostly left alone.
- fix some "infodumps" that are done into `.log` files directly.
- don't use sudo if `CONFIG_DEFS_ONLY=yes`; we'll only be collecting info, not doing anything.
- simpler logging for `rsync` operations (just dump to stdout, logging will handle it!)
- use padded counter for section logfiles, so we know which order things ran. exported as `CURRENT_LOGGING_COUNTER`
- no reason to use `apt-get` with `-yqq` anymore, since all logging is handled, so now `-y` by default
- desktop: using runners helpers for rootfs-desktop.sh, which should help a lot with acng caching and finding of problems
- extensions: correctly cleanup temp stuff, extensions has its own tmp/workdir now, and is always cleaned up at end of build.
armbian-next: bye `PKG_PREFIX`, hello `run_host_x86_binary_logged()` wrapper function; better error handling
- we've x86-only utilities that might need to be run on non-x86 build machines
- previously duplicated logic in PKG_PREFIX variable refactored into logged function
- added centralized debug logging
- replace all PKG_PREFIX usage with the new wrapper function, which already handles logging and errors.
- mostly FIP tooling invocations
- but also the boot_logo image builder
- wrapper function delegates to common `run_host_command_logged`
- wrap other FIP invocations with `run_host_command_logged` too, for tidy logging
- avoid using conditionals when invoking functions; that completely disables error handling inside the called function
- use explicit bash opts instead of shortcuts like `set -e`
- a _lot_ of debug logging added
armbian-next: always use UPPERCASE labels for FAT32 UEFI filesystems (rpi4b, uefi-*)
armbian-next: shellfmt after rebase onto master
armbian-next: manual merge (4) of all lib/*.sh changes between revisions 23afccf56e and e610f00bc7
- plus ooops
atf: fix for `set -e` mode; fix CROSS_COMPILE quoting
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: predict the future location of .img file
- otherwise it's really unhelpful
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
uefi: alias `BRANCH=ddk` to `current`'s `DISTRO_GENERIC_KERNEL=yes`
- no real change, just to match rpi4b's BRANCH=style
- opens space for Armbian-built `current` soon
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
rpi: `legacy`->`ddk` (distro default kernel), remove overclock
- common vars in bcm2711.conf moved to top
- removed overclock/overvolt that was leftover my old setup
- confirmed: works with rpi3b too, should work with CM4/CM3 and others
- use valid UPPERCASE FAT label for RPICFG (in place of `rpicfg`)
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
armbian-next: shellfmt again after rebase
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
armbian-next: manual merge (3) of all lib/*.sh changes between revisions 1035905760 and e4e4ab0791
- missed non lib change on Several improvements for RPi builds (#3391)
- I just realized I will have to drop all non lib changes
rockship: fixes for `set -e` mode in rockship armhf family and bsp tweaks
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
armhf: enable building armhf targets on amd64 using system toolchains
- SKIP_EXTERNAL_TOOLCHAINS=yes on amd64 should use the same system toolchains as an arm64 build
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: better logging about family_tweaks and family_tweaks_bsp
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
kernel: unblock cross compilation, warn about headers package
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: fixes for sunxi/megous stuff with `set -e`
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: fix shellcheck references generation
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: manual merge (2) of all lib/*.sh changes between revisions 117633687e and 3083038855
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: renaming function files a bit more consistently
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: removing leftover empty file after all the moving around
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: really insist on set -e during library loading
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: shellfmt again after rebasing master
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: manual merge of all lib/*.sh changes between revisions f6143eff67 and f3388b9aee
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: generic do_capturing_defs wrapper; Python parser
- enabled by passing CONFIG_DEFS_ONLY=yes; in this case does not build anything
- [WiP] Python3 info reader / matrix expander
- multithreaded version
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: move some interactive parts of config into its own functions
- mostly from config-prepare;
- there is still a lot of others in main-config
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: use chroot_custom for grub and flash-kernel extension logging
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: use line buffering, fix runner output color for GHA
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: wrap dpkg-deb; set TMPDIR (not in chroot); refactor kernel make
- And a huge amount of @TODO's added
- Add "debug" and "deprecation" `display_alert()` levels
- insist that `install_common` is now `install_distribution_agnostic`
- unrelated: realtek 8822CS is EXTRAWIFI=yes only now, sorry.
- many debug statements for desktop
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: don't bail out on patching error
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: bunch of fixes; no-stdin; traps; better stacks
- mostly no-stdin dialog handling (desktop et al)
- let ERR trap run together with unmount trap (EXIT etc)
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
logging: trap ERR very early, pass-in caller info
Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
armbian-next: huge refactor, shellfmt, codesplit, logging/error handling
- *this commit changes most/all the lines of bash code in armbian/build*
- *author is set to IgorPec for historical reasons, rpardini is to blame for the actual changes*
- logging: refactorings, pt.4: autogen lib, shellfmt tool, extract cli
- shellfmt: auto-downloader and runner of shellfmt
- darwin/linux
- amd64/arm64
- find ~correct files to format
- run formatting
- check formatting soon
- refactor compile's CLI stuff out into function/cli
- gen, and use genned library with tool
- logging: refactoring pt3: HUGE split of everything into everything else
- plus rebasing fixes
- logging: refactorings, pt. 2: error handling
- fix: no errors during umount_chroot()
- no progress for CI=true builds
- WiP disable kernel hashing. too crazy.
- a few builds now working with "set -e"
- wtf. or something
- kernel logging and long_running stuff - a mess - needs moving around in files
- rewrite uboot compile loop without using subshells. remove ALL traps. refactor host command
- better logging for u-boot
- more fixes, u-boot
- more fixes for logging et al
- git stuff
- many (many) fixes
- new color scheme
- a monster. make sure unmounted at the end. remove set -e's, to-be-readded.
- remove set -e "for safety reasons"
- more alerts. we gotta know whats failing
- some more logging stuff and fixes for error checking
- more logging and error handling stuff
- fixes; some set -e's
- more logging stuff
- logging: refactoring codebase, pt.1: functions-only
- Refactor the hell out of everything into functions
- rename build-all-ng to build-multi; other fixes, extensions init
- slight nudge
- some were already good, like this one.
- syntax fixes
- some need a little nudge
- another clean one
- some just need a better name (and splitting later)
- syntax fixes
- some were already good, like this desktop one
- some were already good, like this other one
- some were already good, like this one.
- debootstrap is gone.
- extract functions from compile.sh
- add logging to main_default_build
- more stuff
- cleanups and refactors of main.sh
- logging: first steps
- logging: pt. 0: shellfmt everything
- add riscv64 do SRC_ARCH/ARCH/ARCHITECTURE mess; add warn
