JDM170/build
1
0
Fork 0
mirror of https://github.com/armbian/build synced 2025-09-24 19:47:06 +07:00
Code Issues Packages Projects Releases Wiki Activity
Files
76e276c6a9373453eb476157ff6dddd3324e9248
build/lib/tools/aggregation.py
Ricardo Pardini 76e276c6a9 armbian-next: Python patching delusion, pt1 & pt2 & pt3 
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00

202 lines
11 KiB
Python
Raw Blame History

#! /bin/env python3
# Disclaimer: this script was written solely using GitHub Copilot.
# I wrote "prompt" comments and the whole thing was generated by Copilot.
# Unfortunately I removed most original comments/prompts after code was generated, I should have kept them all in...
# I'm not sure if I should be proud or ashamed of this. <-- this was suggested by Copilot too.
# -- rpardini, 23/11/2022
import hashlib
import logging
import os
import common.aggregation_utils as util
import common.armbian_utils as armbian_utils
from common.md_asset_log import SummarizedMarkdownWriter
# Prepare logging
armbian_utils.setup_logging()
log: logging.Logger = logging.getLogger("aggregation")
# Read SRC from the environment, treat it.
armbian_build_directory = armbian_utils.get_from_env_or_bomb("SRC")
if not os.path.isdir(armbian_build_directory):
raise Exception("SRC is not a directory")
# OUTPUT from the environment, treat it.
output_file = armbian_utils.get_from_env_or_bomb("OUTPUT")
with open(output_file, "w") as bash:
bash.write("")
BUILD_DESKTOP = armbian_utils.yes_or_no_or_bomb(armbian_utils.get_from_env_or_bomb("BUILD_DESKTOP"))
INCLUDE_EXTERNAL_PACKAGES = True
ARCH = armbian_utils.get_from_env_or_bomb("ARCH")
DESKTOP_ENVIRONMENT = armbian_utils.get_from_env("DESKTOP_ENVIRONMENT")
DESKTOP_ENVIRONMENT_CONFIG_NAME = armbian_utils.get_from_env("DESKTOP_ENVIRONMENT_CONFIG_NAME")
RELEASE = armbian_utils.get_from_env_or_bomb("RELEASE") # "kinetic"
LINUXFAMILY = armbian_utils.get_from_env_or_bomb("LINUXFAMILY")
BOARD = armbian_utils.get_from_env_or_bomb("BOARD")
USERPATCHES_PATH = armbian_utils.get_from_env_or_bomb("USERPATCHES_PATH")
# Show the environment
armbian_utils.show_incoming_environment()
util.SELECTED_CONFIGURATION = armbian_utils.get_from_env_or_bomb("SELECTED_CONFIGURATION") # "cli_standard"
util.DESKTOP_APPGROUPS_SELECTED = armbian_utils.parse_env_for_tokens(
"DESKTOP_APPGROUPS_SELECTED") # ["browsers", "chat"]
util.SRC = armbian_build_directory
util.AGGREGATION_SEARCH_ROOT_ABSOLUTE_DIRS = [
f"{armbian_build_directory}/config",
f"{armbian_build_directory}/config/optional/_any_board/_config",
f"{armbian_build_directory}/config/optional/architectures/{ARCH}/_config",
f"{armbian_build_directory}/config/optional/families/{LINUXFAMILY}/_config",
f"{armbian_build_directory}/config/optional/boards/{BOARD}/_config",
f"{USERPATCHES_PATH}"
]
util.DEBOOTSTRAP_SEARCH_RELATIVE_DIRS = ["cli/_all_distributions/debootstrap", f"cli/{RELEASE}/debootstrap"]
util.CLI_SEARCH_RELATIVE_DIRS = ["cli/_all_distributions/main", f"cli/{RELEASE}/main"]
util.DESKTOP_ENVIRONMENTS_SEARCH_RELATIVE_DIRS = [
f"desktop/_all_distributions/environments/_all_environments",
f"desktop/_all_distributions/environments/{DESKTOP_ENVIRONMENT}",
f"desktop/_all_distributions/environments/{DESKTOP_ENVIRONMENT}/{DESKTOP_ENVIRONMENT_CONFIG_NAME}",
f"desktop/{RELEASE}/environments/_all_environments",
f"desktop/{RELEASE}/environments/{DESKTOP_ENVIRONMENT}",
f"desktop/{RELEASE}/environments/{DESKTOP_ENVIRONMENT}/{DESKTOP_ENVIRONMENT_CONFIG_NAME}"]
util.DESKTOP_APPGROUPS_SEARCH_RELATIVE_DIRS = [
f"desktop/_all_distributions/appgroups",
f"desktop/_all_distributions/environments/{DESKTOP_ENVIRONMENT}/appgroups",
f"desktop/{RELEASE}/appgroups",
f"desktop/{RELEASE}/environments/{DESKTOP_ENVIRONMENT}/appgroups"]
# Debootstrap.
debootstrap_packages = util.aggregate_all_debootstrap("packages")
debootstrap_packages_remove = util.aggregate_all_debootstrap("packages.remove")
# both main and additional result in the same thing, just different filenames.
rootfs_packages_main = util.aggregate_all_cli("packages")
rootfs_packages_additional = util.aggregate_all_cli("packages.additional")
rootfs_packages_external = util.aggregate_all_cli("packages.external") # @TODO: enable/disable this
rootfs_packages_all = util.merge_lists(rootfs_packages_main, rootfs_packages_additional, "add")
rootfs_packages_all = util.merge_lists(rootfs_packages_all, rootfs_packages_external, "add")
rootfs_packages_remove = util.aggregate_all_cli("packages.remove")
# Desktop environment packages; packages + packages.external
desktop_packages_main = util.aggregate_all_desktop("packages")
desktop_packages_external = util.aggregate_all_desktop("packages.external")
desktop_packages_additional = util.aggregate_all_desktop("packages.additional")
desktop_packages_all = util.merge_lists(desktop_packages_main, desktop_packages_external, "add")
desktop_packages_all = util.merge_lists(desktop_packages_all, desktop_packages_additional, "add")
desktop_packages_remove = util.aggregate_all_desktop("packages.remove")
env_list_remove = util.parse_env_for_list("REMOVE_PACKAGES")
env_list_extra_rootfs = util.parse_env_for_list("EXTRA_PACKAGES_ROOTFS")
env_list_extra_image = util.parse_env_for_list("EXTRA_PACKAGES_IMAGE")
env_package_list_board = util.parse_env_for_list(
"PACKAGE_LIST_BOARD", {"function": "board", "path": "board.conf", "line": 0})
env_package_list_family = util.parse_env_for_list(
"PACKAGE_LIST_FAMILY", {"function": "family", "path": "family.conf", "line": 0})
env_package_list_board_remove = util.parse_env_for_list(
"PACKAGE_LIST_BOARD_REMOVE", {"function": "board_remove", "path": "board.conf", "line": 0})
env_package_list_family_remove = util.parse_env_for_list(
"PACKAGE_LIST_BOARD_REMOVE", {"function": "family_remove", "path": "family.conf", "line": 0})
# Now calculate the final lists.
# debootstrap is the aggregated list, minus the packages we want to remove.
AGGREGATED_PACKAGES_DEBOOTSTRAP = util.merge_lists(debootstrap_packages, debootstrap_packages_remove, "remove")
AGGREGATED_PACKAGES_DEBOOTSTRAP = util.merge_lists(AGGREGATED_PACKAGES_DEBOOTSTRAP, env_list_remove, "remove")
# components for debootstrap is just the aggregated list; or is it?
AGGREGATED_DEBOOTSTRAP_COMPONENTS = util.aggregate_all_debootstrap("components")
AGGREGATED_DEBOOTSTRAP_COMPONENTS_COMMA = ','.join(AGGREGATED_DEBOOTSTRAP_COMPONENTS).replace(' ', ',')
# The rootfs list; add the extras, and remove the removals.
AGGREGATED_PACKAGES_ROOTFS = util.merge_lists(rootfs_packages_all, env_list_extra_rootfs, "add")
AGGREGATED_PACKAGES_ROOTFS = util.merge_lists(AGGREGATED_PACKAGES_ROOTFS, rootfs_packages_remove, "remove")
AGGREGATED_PACKAGES_ROOTFS = util.merge_lists(AGGREGATED_PACKAGES_ROOTFS, env_list_remove, "remove")
# The desktop list.
AGGREGATED_PACKAGES_DESKTOP = util.merge_lists(desktop_packages_all, desktop_packages_remove, "remove")
AGGREGATED_PACKAGES_DESKTOP = util.merge_lists(AGGREGATED_PACKAGES_DESKTOP, env_list_remove, "remove")
# the image list; this comes from env only; apply the removals.
AGGREGATED_PACKAGES_IMAGE = util.merge_lists(env_list_extra_image, env_package_list_board, "add")
AGGREGATED_PACKAGES_IMAGE = util.merge_lists(AGGREGATED_PACKAGES_IMAGE, env_package_list_family, "add")
AGGREGATED_PACKAGES_IMAGE = util.merge_lists(AGGREGATED_PACKAGES_IMAGE, env_package_list_board_remove, "remove")
AGGREGATED_PACKAGES_IMAGE = util.merge_lists(AGGREGATED_PACKAGES_IMAGE, env_package_list_family_remove, "remove")
AGGREGATED_PACKAGES_IMAGE = util.merge_lists(AGGREGATED_PACKAGES_IMAGE, env_list_remove, "remove")
# Calculate a md5 hash of the list of packages, so we can use it as a cache key.
# Attention: ROOTFS does not include DESKTOP. @TODO have 2 hashes, one for cli, one for cli+desktop.
AGGREGATED_ROOTFS_HASH = hashlib.md5(
(" ".join(AGGREGATED_PACKAGES_DEBOOTSTRAP) + " ".join(AGGREGATED_PACKAGES_ROOTFS)).encode("utf-8")).hexdigest()
# We need to aggregate some desktop stuff, which are not package lists, postinst contents and such.
# For this case just find the potentials, and for each found, take the whole contents and join via newlines.
AGGREGATED_DESKTOP_POSTINST = util.aggregate_all_desktop(
"debian/postinst", util.aggregate_simple_contents_potential)
AGGREGATED_DESKTOP_CREATE_DESKTOP_PACKAGE = util.aggregate_all_desktop(
"armbian/create_desktop_package.sh", util.aggregate_simple_contents_potential)
AGGREGATED_DESKTOP_BSP_POSTINST = util.aggregate_all_desktop(
"debian/armbian-bsp-desktop/postinst", util.aggregate_simple_contents_potential)
AGGREGATED_DESKTOP_BSP_PREPARE = util.aggregate_all_desktop(
"debian/armbian-bsp-desktop/prepare.sh", util.aggregate_simple_contents_potential)
# Aggregate the apt-sources; only done if BUILD_DESKTOP is True, otherwise empty.
AGGREGATED_APT_SOURCES = {}
if BUILD_DESKTOP:
apt_sources_debootstrap = util.aggregate_all_debootstrap("sources/apt", util.aggregate_apt_sources)
apt_sources_cli = util.aggregate_all_cli("sources/apt", util.aggregate_apt_sources)
apt_sources_desktop = util.aggregate_all_desktop("sources/apt", util.aggregate_apt_sources)
AGGREGATED_APT_SOURCES = util.merge_lists(apt_sources_debootstrap, apt_sources_cli, "add")
AGGREGATED_APT_SOURCES = util.merge_lists(AGGREGATED_APT_SOURCES, apt_sources_desktop, "add")
# ----------------------------------------------------------------------------------------------------------------------
output_lists: list[tuple[str, str, object, object]] = [
("debootstrap", "AGGREGATED_PACKAGES_DEBOOTSTRAP", AGGREGATED_PACKAGES_DEBOOTSTRAP, None),
("rootfs", "AGGREGATED_PACKAGES_ROOTFS", AGGREGATED_PACKAGES_ROOTFS, None),
("image", "AGGREGATED_PACKAGES_IMAGE", AGGREGATED_PACKAGES_IMAGE, None),
("desktop", "AGGREGATED_PACKAGES_DESKTOP", AGGREGATED_PACKAGES_DESKTOP, None),
("apt-sources", "AGGREGATED_APT_SOURCES", AGGREGATED_APT_SOURCES, util.encode_source_base_path_extra)
]
with open(output_file, "w") as bash, SummarizedMarkdownWriter("aggregation.md", "Aggregation") as md:
bash.write("#!/bin/env bash\n")
# loop over the aggregated lists
for id, name, value, extra_func in output_lists:
stats = util.prepare_bash_output_array_for_list(bash, md, name, value, extra_func)
md.add_summary(f"{id}: {stats['number_items']}")
# The rootfs hash (md5) is used as a cache key.
bash.write(f"declare -g -r AGGREGATED_ROOTFS_HASH='{AGGREGATED_ROOTFS_HASH}'\n")
# Special case for components: debootstrap also wants a list of components, comma separated.
bash.write(
f"declare -g -r AGGREGATED_DEBOOTSTRAP_COMPONENTS_COMMA='{AGGREGATED_DEBOOTSTRAP_COMPONENTS_COMMA}'\n")
# Single string stuff for desktop packages postinst's and preparation. @TODO use functions instead of eval.
bash.write(util.prepare_bash_output_single_string(
"AGGREGATED_DESKTOP_POSTINST", AGGREGATED_DESKTOP_POSTINST))
bash.write(util.prepare_bash_output_single_string(
"AGGREGATED_DESKTOP_CREATE_DESKTOP_PACKAGE", AGGREGATED_DESKTOP_CREATE_DESKTOP_PACKAGE))
bash.write(util.prepare_bash_output_single_string(
"AGGREGATED_DESKTOP_BSP_POSTINST", AGGREGATED_DESKTOP_BSP_POSTINST))
bash.write(util.prepare_bash_output_single_string(
"AGGREGATED_DESKTOP_BSP_PREPARE", AGGREGATED_DESKTOP_BSP_PREPARE))
# 2) @TODO: Some removals...
# aggregate_all_cli "packages.uninstall" " "
# aggregate_all_desktop "packages.uninstall" " "
# PACKAGE_LIST_UNINSTALL="$(cleanup_list aggregated_content)"
# unset aggregated_content
log.debug(f"Done. Output written to {output_file}")
Reference in New Issue View Git Blame Copy Permalink