JDM170/build
1
0
Fork 0
mirror of https://github.com/armbian/build synced 2025-09-24 19:47:06 +07:00
Code Issues Packages Projects Releases Wiki Activity

hashed-OCI-revisioned-debs: introduce "reversioning" of .deb packages

Browse Source 
> tl-dr:
> - maximize OCI cache hit ratio across nightlies/releases/PRs/etc;
> - publish simple `Version:`'s that don't include a crazy hash in repo and images
> - introduce `output/packages-hashed` directory
> - radically change the `output/debs` directory structure

- simplify artifact's `prepare_version()` method for `deb` and `deb-tar` artifacts:
  - `artifact_base_dir` and `artifact_final_file` will now be auto-calculated; thus removed from each artifact (except `rootfs`)
  - `artifact_deb_repo` ("global", "jammy", "bookworm") is now required; "global" means common across all RELEASES
  - `artifact_deb_arch` is now required, "all" is arch-independent, otherwise use `${ARCH}`
  - `artifact_map_debs` is now auto-calculated based on the above, and shouldn't be specified manually
  - `artifact_final_version_reversioned` is optional, and can force the final version of the artifact (specific for the `base-files` case)
  - artifacts that need special handling for reversioning can add function names to `artifact_debs_reversion_functions` array (`base-files` and `bsp-cli` cases)
  - artifacts `prepare_version()` should set `artifact_version`, but _never_ include it in other variables; `artifact_version` is now changed by framework after `prepare_version()` returns
- no longer use/refer/mention `${REVISION}` when building packages. All packages should be `${REVISION}`-agnostic.
- `${REVISION}` (actually, `artifact_final_version_reversioned`) will be automatically swapped in the `control` file during reversioning
- `fakeroot_dpkg_deb_build()` now takes exactly two arguments: the directory to pack, and the deb ID (key of `artifact_map_packages` dict); add this change in all the artifact's code for this
- `obtain_complete_artifact()`:
  - automatically adds `-Rxxxx` "revisioning-hash" to `artifact_version`, by hashing the revisioning functions and any `artifact_debs_reversion_functions` set
  - calculates more complex subdirectory paths for both the `output/packages-hashed` and `output/debs`/`output/debs-beta` directories
    - with the new subdirectories we can be sure a re-version is already done correctly and can skip it (eg, for partial `download-debs` re-runs)
    - in the future we can automatically clean/remove old versions that are no longer relevant based on the dir structure
    - exports a lot more information to JSON, including the new subdirectory paths
  - comment-out code that implemented `skip_unpack_if_found_in_caches`, I'm very unsure why we had this in the first place
- `obtain_artifact_from_remote_cache()`
  - for `deb` type artifacts, OCI won't preserve the subdirectory structure, so move downloaded files to the correct subdirectory manually
  - this is not needed for `deb-tar`, since that can preserve the dir structure itself
- introduce `artifacts-reversion.sh` and its main function `artifact_reversion_for_deployment()`
  - this has the logic for reversioning .deb's, by `ar`-unpacking them, changing `control.tar` (and possibly `data.tar`), handling `.xz` compression, etc.
  - also handles hashing those functions, for consistency. Any changes in reversioning code actually change the artifact itself so we're not caught by surprise
  - by default, it changes `control` file only:
    - replace `Version:` (which is the hash-version originally) with `artifact_final_version_reversioned` (which is mostly just `${REVISION}`)
    - add a custom field `Armbian-Original-Hash:` with the original hash-version
  - `artifact_reversion_for_deployment()` is called by
    - new CLI wrapper `cli_obtain_complete_artifact()`, used for CLI building of specific artifact, but also for `download-artifact`
    - `build_artifact_for_image()` used during image build
- `armbian-bsp-cli-deb.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `artifact-armbian-base-files.sh`: move `${REVISION}` related stuff from the main package build to new reversioning functions.
- `kernel`:
  - add some custom fields to `DEBIAN/control`:
    - `Armbian-Kernel-Version:` / `Armbian-Kernel-Version-Family:` (for future use: cleanup of usage of `Source: ` field which should be removed)
  - declutter the `Description:` field, moving long description out of the first line
  - obtain `IMAGE_INSTALLED_KERNEL_VERSION` from the reversioned deb (this is still a hack and has not been fixed)
- `uboot`:
  - declutter the `Description:` field, moving long description out of the first line
  - use the reversioned .deb when deploying u-boot to the image
- `main_default_build_packages()` now stores reversioned values and complete paths to reversioned .deb's
- `list_installed_packages()` now compares custom field `Armbian-Original-Hash: `, and not the `Version:` to make sure debs in the image are the ones we want
- `install_artifact_deb_chroot()` is a new wrapper around `install_deb_chroot()` for easy handling of reversioned debs
  - use it everywhere `install_deb_chroot()` was used in `distro-agnostic.sh` and `distro-specific.sh`
This commit is contained in:
Ricardo Pardini
2023-08-02 16:45:16 +02:00
committed by Igor
parent a876d0f882
commit 2142f73b97
33 changed files with 528 additions and 240 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
lib/functions/compilation/kernel-debs.sh
View File

@@ -68,17 +68,17 @@ function prepare_kernel_packaging_debs() {
# package the linux-image (image, modules, dtbs (if present))
display_alert "Packaging linux-image" "${LINUXFAMILY} ${LINUXCONFIG}" "info"
create_kernel_deb "linux-image-${BRANCH}-${LINUXFAMILY}" "${debs_target_dir}" kernel_package_callback_linux_image
create_kernel_deb "linux-image-${BRANCH}-${LINUXFAMILY}" "${debs_target_dir}" kernel_package_callback_linux_image "linux-image"
# if dtbs present, package those too separately, for u-boot usage.
if [[ -d "${tmp_kernel_install_dirs[INSTALL_DTBS_PATH]}" ]]; then
display_alert "Packaging linux-dtb" "${LINUXFAMILY} ${LINUXCONFIG}" "info"
create_kernel_deb "linux-dtb-${BRANCH}-${LINUXFAMILY}" "${debs_target_dir}" kernel_package_callback_linux_dtb
create_kernel_deb "linux-dtb-${BRANCH}-${LINUXFAMILY}" "${debs_target_dir}" kernel_package_callback_linux_dtb "linux-dtb"
fi
if [[ "${KERNEL_HAS_WORKING_HEADERS}" == "yes" ]]; then
display_alert "Packaging linux-headers" "${LINUXFAMILY} ${LINUXCONFIG}" "info"
create_kernel_deb "linux-headers-${BRANCH}-${LINUXFAMILY}" "${debs_target_dir}" kernel_package_callback_linux_headers
create_kernel_deb "linux-headers-${BRANCH}-${LINUXFAMILY}" "${debs_target_dir}" kernel_package_callback_linux_headers "linux-headers"
else
display_alert "Skipping linux-headers package" "for ${KERNEL_MAJOR_MINOR} kernel version" "info"
fi
@@ -88,6 +88,7 @@ function create_kernel_deb() {
declare package_name="${1}"
declare deb_output_dir="${2}"
declare callback_function="${3}"
declare artifact_deb_id="${4}"
declare cleanup_id="" package_directory=""
prepare_temp_dir_in_workdir_and_schedule_cleanup "deb-k-${package_name}" cleanup_id package_directory # namerefs
@@ -138,7 +139,7 @@ function create_kernel_deb() {
#display_alert "Package dir" "for package ${package_name}" "debug"
#run_host_command_logged tree -C -h -d --du "${package_directory}"
fakeroot_dpkg_deb_build "${package_directory}"
fakeroot_dpkg_deb_build "${package_directory}" "${artifact_deb_id}"
done_with_temp_dir "${cleanup_id}" # changes cwd to "${SRC}" and fires the cleanup function early
}
@@ -228,8 +229,8 @@ function kernel_package_callback_linux_image() {
# Clean up symlinks in lib/modules/${kernel_version_family}/build and lib/modules/${kernel_version_family}/source; will be in the headers package
run_host_command_logged rm -v -f "${package_directory}/lib/modules/${kernel_version_family}/build" "${package_directory}/lib/modules/${kernel_version_family}/source"
display_alert "Showing contents of Kbuild produced modules" "linux-image" "debug"
if [[ -d "${package_directory}/lib/modules/${kernel_version_family}/kernel" ]]; then
display_alert "Showing contents of Kbuild produced modules" "linux-image" "debug"
run_host_command_logged tree -C --du -h -d -L 1 "${package_directory}/lib/modules/${kernel_version_family}/kernel" "|| true" # do not fail
fi
@@ -246,14 +247,16 @@ function kernel_package_callback_linux_image() {
Package: ${package_name}
Version: ${artifact_version}
Source: linux-${kernel_version}
Armbian-Kernel-Version: ${kernel_version}
Armbian-Kernel-Version-Family: ${kernel_version_family}
Architecture: ${ARCH}
Maintainer: ${MAINTAINER} <${MAINTAINERMAIL}>
Section: kernel
Priority: optional
Provides: linux-image, linux-image-armbian, armbian-$BRANCH
Description: Armbian Linux $BRANCH kernel image ${artifact_version_reason:-"${kernel_version_family}"}
This package contains the Linux kernel, modules and corresponding other
files, kernel_version_family: $kernel_version_family.
Description: Armbian Linux $BRANCH kernel image $kernel_version_family
This package contains the Linux kernel, modules and corresponding other files.
${artifact_version_reason:-"${kernel_version_family}"}
CONTROL_FILE
# Install the maintainer scripts
@@ -328,8 +331,9 @@ function kernel_package_callback_linux_dtb() {
Architecture: ${ARCH}
Priority: optional
Provides: linux-dtb, linux-dtb-armbian, armbian-$BRANCH
Description: Armbian Linux $BRANCH DTBs ${artifact_version_reason:-"${kernel_version_family}"} in /boot/dtb-${kernel_version_family}
This package contains device blobs from the Linux kernel, version ${kernel_version_family}
Description: Armbian Linux $BRANCH DTBs in /boot/dtb-${kernel_version_family}
This package contains device tree blobs from the Linux kernel, version ${kernel_version_family}
${artifact_version_reason:-"${kernel_version_family}"}
CONTROL_FILE
kernel_package_hook_helper "preinst" <(
@@ -464,10 +468,11 @@ function kernel_package_callback_linux_headers() {
Priority: optional
Provides: linux-headers, linux-headers-armbian, armbian-$BRANCH
Depends: make, gcc, libc6-dev, bison, flex, libssl-dev, libelf-dev
Description: Armbian Linux $BRANCH headers ${artifact_version_reason:-"${kernel_version_family}"}
Description: Armbian Linux $BRANCH headers ${kernel_version_family}
This package provides kernel header files for ${kernel_version_family}
.
This is useful for DKMS and building of external modules.
${artifact_version_reason:-"${kernel_version_family}"}
CONTROL_FILE
# Make sure the target dir is clean/not-existing before installing.