mirror of
https://github.com/armbian/build
synced 2025-09-24 19:47:06 +07:00
hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages
> tl-dr:
> - maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
> - publish simple `Version:`'s that don't include a crazy hash in repo and images
> - introduce `output/packages-hashed` directory
> - radically change the `output/debs` directory structure
- simplify artifact's `prepare_version()` method for `deb` and `deb-tar` artifacts:
- `artifact_base_dir` and `artifact_final_file` will now be auto-calculated; thus removed from each artifact (except `rootfs`)
- `artifact_deb_repo` ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
- `artifact_deb_arch` is now required, "all" is arch-independent, otherwise use `${ARCH}`
- `artifact_map_debs` is now auto-calculated based on the above, and shouldn't be specified manually
- `artifact_final_version_reversioned` is optional, and can force the final version of the artifact (specific for the `base-files` case)
- artifacts that need special handling for reversioning can add function names to `artifact_debs_reversion_functions` array (`base-files` and `bsp-cli` cases)
- artifacts `prepare_version()` should set `artifact_version`, but _never_ include it in other variables; `artifact_version` is now changed by framework after `prepare_version()` returns
- no longer use/refer/mention `${REVISION}` when building packages. All packages should be `${REVISION}`-agnostic.
- `${REVISION}` (actually, `artifact_final_version_reversioned`) will be automatically swapped in the `control` file during reversioning
- `fakeroot_dpkg_deb_build()` now takes exactly two arguments: the directory to pack, and the deb ID (key of `artifact_map_packages` dict); add this change in all the artifact's code for this
- `obtain_complete_artifact()`:
- automatically adds `-Rxxxx` "revisioning-hash" to `artifact_version`, by hashing the revisioning functions and any `artifact_debs_reversion_functions` set
- calculates more complex subdirectory paths for both the `output/packages-hashed` and `output/debs`/`output/debs-beta` directories
- with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial `download-debs` re-runs)
- in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
- exports a lot more information to JSON, including the new subdirectory paths
- comment-out code that implemented `skip_unpack_if_found_in_caches`, I'm very unsure why we had this in the first place
- `obtain_artifact_from_remote_cache()`
- for `deb` type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
- this is not needed for `deb-tar`, since that can preserve the dir structure itself
- introduce `artifacts-reversion.sh` and its main function `artifact_reversion_for_deployment()`
- this has the logic for reversioning .deb's, by `ar`-unpacking them, changing `control.tar` (and possibly `data.tar`), handling `.xz` compression, etc.
- also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
- by default, it changes `control` file only:
- replace `Version:` (which is the hash-version originally) with `artifact_final_version_reversioned` (which is mostly just `${REVISION}`)
- add a custom field `Armbian-Original-Hash:` with the original hash-version
- `artifact_reversion_for_deployment()` is called by
- new CLI wrapper `cli_obtain_complete_artifact()`, used for CLI building of specific artifact, but also for `download-artifact`
- `build_artifact_for_image()` used during image build
- `armbian-bsp-cli-deb.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `artifact-armbian-base-files.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `kernel`:
- add some custom fields to `DEBIAN/control`:
- `Armbian-Kernel-Version:` / `Armbian-Kernel-Version-Family:` (for future use: cleanup of usage of `Source: ` field which should be removed)
- declutter the `Description:` field, moving long description out of the first line
- obtain `IMAGE_INSTALLED_KERNEL_VERSION` from the reversioned deb (this is still a hack and has not been fixed)
- `uboot`:
- declutter the `Description:` field, moving long description out of the first line
- use the reversioned .deb when deploying u-boot to the image
- `main_default_build_packages()` now stores reversioned values and complete paths to reversioned .deb's
- `list_installed_packages()` now compares custom field `Armbian-Original-Hash: `, and not the `Version:` to make sure debs in the image are the ones we want
- `install_artifact_deb_chroot()` is a new wrapper around `install_deb_chroot()` for easy handling of reversioned debs
- use it everywhere `install_deb_chroot()` was used in `distro-agnostic.sh` and `distro-specific.sh`
This commit is contained in:
Ricardo Pardini
@@ -30,18 +30,30 @@ function compile_armbian-bsp-cli-transitional() {
|
||||
Maintainer: $MAINTAINER <$MAINTAINERMAIL>
|
||||
Section: oldlibs
|
||||
Priority: optional
|
||||
Depends: ${artifact_name} (= ${artifact_version})
|
||||
Description: Armbian CLI BSP for board '${BOARD}' - transitional package
|
||||
EOF
|
||||
|
||||
# Build / close the package. This will run shellcheck / show the generated files if debugging
|
||||
fakeroot_dpkg_deb_build "${destination}"
|
||||
fakeroot_dpkg_deb_build "${destination}" "armbian-bsp-cli-transitional"
|
||||
|
||||
done_with_temp_dir "${cleanup_id}" # changes cwd to "${SRC}" and fires the cleanup function early
|
||||
|
||||
display_alert "Done building BSP CLI transitional package" "${destination}" "debug"
|
||||
}
|
||||
|
||||
function reversion_armbian-bsp-cli-transitional_deb_contents() {
|
||||
if [[ "${1}" != "armbian-bsp-cli-transitional" ]]; then
|
||||
return 0 # Not our deb, nothing to do.
|
||||
fi
|
||||
display_alert "Reversion" "reversion_armbian-bsp-cli-transitional_deb_contents: '$*'" "debug"
|
||||
|
||||
# Depends on the new package
|
||||
cat <<- EOF >> "${control_file_new}"
|
||||
Depends: ${artifact_name} (= ${REVISION})
|
||||
EOF
|
||||
|
||||
}
|
||||
|
||||
function compile_armbian-bsp-cli() {
|
||||
: "${artifact_version:?artifact_version is not set}"
|
||||
: "${artifact_name:?artifact_name is not set}"
|
||||
@@ -63,14 +75,6 @@ function compile_armbian-bsp-cli() {
|
||||
declare -a extra_description=()
|
||||
[[ "${EXTRA_BSP_NAME}" != "" ]] && extra_description+=("(variant '${EXTRA_BSP_NAME}')")
|
||||
|
||||
# Replaces: base-files is needed to replace /etc/update-motd.d/ files on Xenial
|
||||
# Depends: linux-base is needed for "linux-version" command in initrd cleanup script
|
||||
# Depends: fping is needed for armbianmonitor to upload armbian-hardware-monitor.log
|
||||
# Depends: base-files (>= ${REVISION}) is to force usage of our base-files package (not the original Distro's).
|
||||
declare depends_base_files=", base-files (>= ${REVISION})"
|
||||
if [[ "${KEEP_ORIGINAL_OS_RELEASE:-"no"}" == "yes" ]]; then
|
||||
depends_base_files=""
|
||||
fi
|
||||
cat <<- EOF > "${destination}"/DEBIAN/control
|
||||
Package: ${artifact_name}
|
||||
Version: ${artifact_version}
|
||||
@@ -78,10 +82,7 @@ function compile_armbian-bsp-cli() {
|
||||
Maintainer: $MAINTAINER <$MAINTAINERMAIL>
|
||||
Section: kernel
|
||||
Priority: optional
|
||||
Depends: bash, linux-base, u-boot-tools, initramfs-tools, lsb-release, fping${depends_base_files}
|
||||
Suggests: armbian-config
|
||||
Replaces: zram-config, armbian-bsp-cli-${BOARD}${EXTRA_BSP_NAME} (<< ${artifact_version})
|
||||
Breaks: armbian-bsp-cli-${BOARD}${EXTRA_BSP_NAME} (<< ${artifact_version})
|
||||
Recommends: bsdutils, parted, util-linux, toilet
|
||||
Description: Armbian CLI BSP for board '${BOARD}' branch '${BRANCH}' ${extra_description[@]}
|
||||
EOF
|
||||
@@ -95,7 +96,6 @@ function compile_armbian-bsp-cli() {
|
||||
BOARDFAMILY=${BOARDFAMILY}
|
||||
BUILD_REPOSITORY_URL=${BUILD_REPOSITORY_URL}
|
||||
BUILD_REPOSITORY_COMMIT=${BUILD_REPOSITORY_COMMIT}
|
||||
VERSION=${REVISION}
|
||||
LINUXFAMILY=$LINUXFAMILY
|
||||
ARCH=$ARCHITECTURE
|
||||
IMAGE_TYPE=$IMAGE_TYPE
|
||||
@@ -104,7 +104,6 @@ function compile_armbian-bsp-cli() {
|
||||
KERNEL_IMAGE_TYPE=$KERNEL_IMAGE_TYPE
|
||||
FORCE_BOOTSCRIPT_UPDATE=$FORCE_BOOTSCRIPT_UPDATE
|
||||
VENDOR=$VENDOR
|
||||
REVISION=$REVISION
|
||||
EOF
|
||||
|
||||
# copy general overlay from packages/bsp-cli
|
||||
@@ -210,13 +209,53 @@ function compile_armbian-bsp-cli() {
|
||||
fi
|
||||
|
||||
# Build / close the package. This will run shellcheck / show the generated files if debugging
|
||||
fakeroot_dpkg_deb_build "${destination}"
|
||||
fakeroot_dpkg_deb_build "${destination}" "armbian-bsp-cli"
|
||||
|
||||
done_with_temp_dir "${cleanup_id}" # changes cwd to "${SRC}" and fires the cleanup function early
|
||||
|
||||
display_alert "Done building BSP CLI package" "${destination}" "debug"
|
||||
}
|
||||
|
||||
# Reversion function is called with the following parameters:
|
||||
# ${1} == deb_id
|
||||
function reversion_armbian-bsp-cli_deb_contents() {
|
||||
if [[ "${1}" != "armbian-bsp-cli" ]]; then
|
||||
return 0 # Not our deb, nothing to do.
|
||||
fi
|
||||
display_alert "Reversion" "reversion_armbian-bsp-cli_deb_contents: '$*'" "debug"
|
||||
|
||||
# Replaces: base-files is needed to replace /etc/update-motd.d/ files on Xenial
|
||||
# Depends: linux-base is needed for "linux-version" command in initrd cleanup script
|
||||
# Depends: fping is needed for armbianmonitor to upload armbian-hardware-monitor.log
|
||||
# Depends: base-files (>= ${REVISION}) is to force usage of our base-files package (not the original Distro's).
|
||||
declare depends_base_files=", base-files (>= ${REVISION})"
|
||||
if [[ "${KEEP_ORIGINAL_OS_RELEASE:-"no"}" == "yes" ]]; then
|
||||
depends_base_files=""
|
||||
fi
|
||||
cat <<- EOF >> "${control_file_new}"
|
||||
Depends: bash, linux-base, u-boot-tools, initramfs-tools, lsb-release, fping${depends_base_files}
|
||||
Replaces: zram-config, armbian-bsp-cli-${BOARD}${EXTRA_BSP_NAME} (<< ${REVISION})
|
||||
Breaks: armbian-bsp-cli-${BOARD}${EXTRA_BSP_NAME} (<< ${REVISION})
|
||||
EOF
|
||||
|
||||
artifact_deb_reversion_unpack_data_deb
|
||||
: "${data_dir:?data_dir is not set}"
|
||||
|
||||
cat <<- EOF >> "${data_dir}"/etc/armbian-release
|
||||
VERSION=${REVISION}
|
||||
REVISION=$REVISION
|
||||
EOF
|
||||
|
||||
# Show results if debugging
|
||||
if [[ "${SHOW_DEBUG}" == "yes" ]]; then
|
||||
run_tool_batcat --file-name "armbian-release.sh" "${data_dir}"/etc/armbian-release
|
||||
fi
|
||||
|
||||
artifact_deb_reversion_repack_data_deb
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
function get_bootscript_info() {
|
||||
bootscript_info[has_bootscript]="no"
|
||||
bootscript_info[has_extlinux]="no"
|
||||
|
||||
@@ -59,7 +59,7 @@ function compile_armbian-bsp-desktop() {
|
||||
eval "${AGGREGATED_DESKTOP_BSP_PREPARE}"
|
||||
display_alert "Done with bsp-desktop -specific aggregated prepare script" "AGGREGATED_DESKTOP_BSP_PREPARE" "debug"
|
||||
|
||||
fakeroot_dpkg_deb_build "${destination}"
|
||||
fakeroot_dpkg_deb_build "${destination}" "armbian-bsp-desktop"
|
||||
|
||||
done_with_temp_dir "${cleanup_id}" # changes cwd to "${SRC}" and fires the cleanup function early
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user