Commit Graph

1867 Commits

Author SHA1 Message Date
Ricardo Pardini
ca46f86dec artifacts: download-artifact CLI. makes sure to only used local .deb, or download from OCI, never build 2023-05-16 20:09:47 +02:00
Ricardo Pardini
1fa54e030d artifacts: include the artifact maps info (debs/packages) both as keys and values in the artifact JSON info
- this is valuable for the debs-to-repo process; we can now know which exact .debs are produced, and all the ways we can refer to them
2023-05-16 20:09:47 +02:00
Ricardo Pardini
234232085d docker: remove dead code 2023-05-16 20:09:47 +02:00
Igor Velkov
9b68c4e42a Add support for nilfs2 fs based image
Signed-off-by: Igor Velkov <iav@iav.lv>
2023-05-16 08:51:05 +02:00
Ricardo Pardini
78700cf139 logging/retries: curb CI/GHA warning emitted _for each retry_ 2023-05-15 17:43:25 +02:00
Ricardo Pardini
c351936cb0 binfmts: hide output when loading qemu-arm under aarch64 (some are native, some are emulated, all are good, errors can be ignored) 2023-05-15 17:43:25 +02:00
Ricardo Pardini
ceea84ff28 logging: curb warning that always came up when SRC_EXTLINUX=yes
- SRC_EXTLINUX is quite prevalent, no use warning, it was more a debugging marker
2023-05-15 17:43:25 +02:00
Ricardo Pardini
f0f9159faa logging: curb CI/GHA special errors for cascading errors during build failure
- otherwise we get 4 errors for each "error" in CI
  - one for the real error
  - one for "wait for cleanups"
  - one for "Docker build failed"
  - one from GHA itself since command failed (we can't get rid of this)
2023-05-15 17:43:25 +02:00
Ricardo Pardini
25929ea6ac apt_purge_unneeded_packages_and_clean_apt_caches(): count files, don't use du: avoid WARNs when not needed; tolerate 1 file
- 'lock' might or not be there, tolerate 1 file (not 0)
2023-05-15 17:43:25 +02:00
Ricardo Pardini
99e065381a squash more warnings:
- `odroidxu4`'s firmware hook for vendor kernel. such is life, no use warning.
- `sun50iw9`: vendor kernel with extension, proven working, stop warning
- old kernels don't have working headers, it's a fact of life, remove warning
2023-05-15 17:43:25 +02:00
Ricardo Pardini
d164f6850b rootfs/aggregation/helios64: fix: include PACKAGE_LIST_BOARD_REMOVE (and FAMILY version) into rootfs' artifact_input_variables
- `PACKAGE_LIST_BOARD_REMOVE` is used only in `helios64` board
- the fact is `PACKAGE_LIST_BOARD_REMOVE` removes the packages _from the rootfs_ and not _from the image_
  - this only accounts for that fact, but in the future we should actually change how this works
2023-05-15 17:43:25 +02:00
Ricardo Pardini
e49d91ec40 popular demand: bring back KERNEL_CONFIGURE=yes *during image build*
- partially revert 6ef394d95d (thus bring back the TUI/dialog for selecting KERNEL_CONFIGURE=yes/no)
- partially revert d890b418c7 (thus bring back the capacity to config & build image in one go)
- stop after configuring kernel, but only if command is `kernel-config`, not regular image-build KERNEL_CONFIGURE=yes
2023-05-09 10:05:09 +02:00
Ricardo Pardini
83e35b1473 binfmts: don't raise warning if qemu-arm failed to load under aarch64 (some are native, some are emulated, all are good)
- reported by @igorpecovnik on Discord, during massive CI build, when armhf builds landed on arm64 builders
2023-05-09 10:04:57 +02:00
Ricardo Pardini
8f14f33f78 logging: squash undue warnings rootfs/aptcache
- rootfs is built with BOARD= so just live with it (idea about this warn was that families might hook rootfs build process)
- count, instead of du, apt caches. seems some empty dirs are reported as "1Mb" by du?
2023-05-09 10:04:57 +02:00
Ricardo Pardini
5316c6685b host-utils: more logging for when junk is found in image apt caches 2023-05-09 10:04:57 +02:00
Ricardo Pardini
b18cd8aaf4 logging: curb some warnings into info's 2023-05-09 10:04:57 +02:00
Ricardo Pardini
ab696d0776 rootfs: bump non-desktop tmpfs_estimated_size from 2000mb to 2300mb (determined empirically, after building all images) 2023-05-09 10:04:57 +02:00
Ricardo Pardini
b7c659c0a9 pipeline: gha: fix IMAGES_ONLY_OUTDATED_ARTIFACTS and SKIP_IMAGES by exporting them so Python can see; tune logging 2023-05-09 10:04:57 +02:00
Ricardo Pardini
637c4fec01 include (clean/real) apt lists in built images; update xapian index during image build, not rootfs; don't leave junk in the rootfs cache
- revert #5139 Revert "Apt lists is needed for synpatic index generation"
  This reverts commit 63e9bd3baa
  - this actually only left junk in rootfs cache for the future to find
2023-05-09 10:04:50 +02:00
Ricardo Pardini
a872fe8773 config-prepare: config_pre_main(): try to be more helpful when BRANCH is unset or wrong instead of just "bad KERNEL_MAJOR_MINOR" 2023-05-08 11:21:47 +02:00
amazingfate
0b5dc0c921 fix uwe5622 for kernel v6.3 2023-05-06 12:55:10 +02:00
Igor
63e9bd3baa Apt lists is needed for synpatic index generation 2023-05-05 19:37:19 +02:00
Ricardo Pardini
de8bbcec11 pipeline: correctly quote params passed over pure strings (eg in the GHA JSON matrixes) so values with spaces are not mangled 2023-05-04 16:08:11 +02:00
Ricardo Pardini
1b36bb63d7 bash_declare_parser: parse space-separated single-quoted array values correctly ('THIS=has space' is a single token, not two) 2023-05-04 16:08:11 +02:00
Ricardo Pardini
875960ddb7 aggregation: armbian_utils.parse_env_for_tokens() now actually does what it said on the box "split by whitespace, line breaks, commas, and semicolons"; add appgroups summary 2023-05-04 16:08:11 +02:00
Ricardo Pardini
1ad9f7cbd2 artifact-rootfs: add desktop info (environment, config_name, appgrous) to artifact_version_reason (so we can debug when I mess up later) 2023-05-04 16:08:11 +02:00
Ricardo Pardini
7c0fb70a2b configdump: alias config-dump, config-dump-json and new config-dump-no-json (bash declare format) 2023-05-04 16:08:11 +02:00
Ricardo Pardini
0ff694cfbd produce_repeat_args_array: @mhoffrog was right, we need to quote repeat params; (here I'm being stubborn and only quoting the ones that have spaces in them) 2023-05-04 16:08:11 +02:00
Ricardo Pardini
73e9c89807 docker: fix: always mount .docker/config.json if it exists (in GHA) 2023-05-03 15:09:56 +02:00
Ricardo Pardini
6ef394d95d kernel: get better rid of KERNEL_CONFIGURE, remove UI asking for it, it's now a separate command kernel-config and wouldn't work anyway 2023-05-03 15:09:56 +02:00
Ricardo Pardini
724a67d3e5 pipeline: cli: add proper gha- prefix to CLI commands that have something to do with GitHub Actions (thus gha-matrix now, and the new gha-template)
- stop at the correct places for `inventory` and `targets` non-GHA-specific commands
- handle `gha-template` first (and return), since it does not really depend on any info/targets/etc
  - batcat the output for show-off
2023-05-03 07:49:53 +02:00
Ricardo Pardini
23a2b34847 pipeline: add output-gha-workflow-template.py utility, for rendering GHA workflow YAML templates with chunks
- python-tools: add Jinja2. Incredible how we made it this far without it.
- output-gha-workflow-template: a double-templater, first runs jinja with a custom syntax, then "moar magic" to be useful for GHA
2023-05-03 07:49:53 +02:00
Ricardo Pardini
0aac5a4945 pipeline: targets-compositor: whole target can be enabled: no to quickly disable it (as if it wasn't there) 2023-05-03 07:49:53 +02:00
Ricardo Pardini
46cd0a2cfc pipeline: output-gha-matrix: directly GHA-output images-chunk-json-2, images-chunk-not-empty-2, images-chunk-size-2 and artifacts-xxx with no fake entry
- as alternative to the full output, this allows us to skip chunkjobs when they're empty, and hopefully makes GHA-JSON-parsing 10x faster
2023-05-03 07:49:53 +02:00
Ricardo Pardini
d6bdf4221e pipeline: output-gha-matrix: new input.pipeline.gha.runners in the targets.yaml for mapping GHA's runs_on based on name and arch 2023-05-03 07:49:53 +02:00
Ricardo Pardini
4bf98ebfab artifact kernel/uboot: add ARCH to artifact_input_variables; used for (maybe) mapping to runner tags in JSON pipeline 2023-05-03 07:49:53 +02:00
igorpecovnik
ba1a84a5e8 Adjusting ORAS cache location, runners (#5129)
* Move packages to new location
* Adjust build runners
* Switch to OS

Signed-off-by: Igor <igor@armbian.com>
---------

Signed-off-by: Igor <igor@armbian.com>
Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
2023-05-02 11:08:34 +02:00
Vyacheslav Bocharov
64f434ab33 Revert "fix rtl88x2cs compilation for kernel 6.3" as fixed in rtl88x2cs repository.
This reverts commit dc9f751012.
2023-05-02 12:04:51 +03:00
Ricardo Pardini
3767729dc5 rootfs: add artifact name prefix and EXTRA_ROOTFS_NAME suffix (for extensions), yyyymm one-per-month auto-version in place of REVISION for artifact version
- rootfs is not a deb package, and thus not subject to the same strict ordering rules of `artifact_prefix_version`
  - replace with `yyyymm` (`202305`) in version so we get one-per-month automatically
2023-05-01 22:47:12 +02:00
Ricardo Pardini
f992dcd466 git-ref2info: retries for getting Makefile body (https://raw.githubusercontent.com has been failing a lot) 2023-05-01 22:47:03 +02:00
Ricardo Pardini
d890b418c7 kernel/uboot/atf: introduce kernel-patch, uboot-patch, atf-patch, uboot-config, kernel-config CLI commands; enhanced manual patching; block deprecated ways
- all interactive commands now **don't build the artifact** anymore; just patches/.configs are produced and then build ends
  - user is required to put the produced patches in the right place and build again, for full consistency
- split ATF and U-BOOT manual patching process; use CLI command `atf-patch` to patch ATF, and `uboot-patch` to patch u-boot
- non-interactive artifact builds are now 100% sans-stdin
- introduce `uboot-config` CLI command; still experimental, only produces a defconfig and not a patch
- reworked `userpatch_create()` to be (hopefully) more useful:
  - detects a previous patch and offers to apply it before continuing
  - enters a loop showing the diff, and only proceeds when user indicates he's happy with the patch
  - produces `mbox`-formatted patches via `format-patch` and standard Armbian parameters
  - uses MAINTAINER and MAINTAINERMAIL instead of git configuration (so it works in containers)
- don't allow image builds with any patching or configuring _at all_ (it has been deprecated with a warning for months already, and results are inconsistent)
2023-05-01 22:46:55 +02:00
Ricardo Pardini
f77edd0591 rtl8822bs: fix: remove binaries (firmware?) being added to kernel driver source and breaking patching; related to #5117 2023-05-01 22:46:43 +02:00
Ricardo Pardini
e74ab59f48 kernel-make: introduce KERNEL_EXTRA_CFLAGS; fill it to add to what's passed to KCFLAGS; related to #5117 2023-05-01 22:46:43 +02:00
Ricardo Pardini
369eb80f9c driver_uwe5622_allwinner: only apply "pre-6.1" park-link patch if kernel >= 5 (eg: not for rockchip64-legacy); possibly fixes #5120 2023-05-01 22:46:43 +02:00
Ricardo Pardini
91bb00bfa2 artifact-uboot: fix: include UBOOT_TARGET_MAP in variables hash of version 2023-05-01 22:46:43 +02:00
Ricardo Pardini
c692676da7 kernel-debs: postinst/etc: fix: unify is_boot_dev_vfat() function; make work during image build too via env var
- determinining if `/boot` is vfat is more complex than it looks;
  - use a common function to unify across all scripts
- during image build, the kernel install is always done with a non-fat, non-mounted /boot
  - use a special variable passed through the env so unified function can know ahead of time
  - introduce `extra_apt_envs` param to `chroot_sdcard_apt_get()`; array with extra environment vars
- if vfat /boot:
  - `linux-update-symlinks` is not called (Debian would just choke on this)
  - don't even try to symlink Armbian stuff, move instead
- more/better logging board-side (unfortunately mixed in all the set -x debugs)
2023-05-01 22:46:43 +02:00
Ricardo Pardini
f8ddf7f9e2 🔥 JSON info pipeline: v18
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
  - pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
2023-05-01 22:46:25 +02:00
Ricardo Pardini
3143db5df4 kernel-debs: call linux-update-symlinks in postinst; expected of Debian-compatible kernels
- this handles /vmlinuz etc symlinks (doesn't know about uInitrd, etc)
2023-04-28 21:32:52 +02:00
Ricardo Pardini
59d6690a11 git-ref2info: many fixes for nonstandard repos; try a few different dereference and refs/ combinations in a certain order; remove codeaurora repo handling
- git-ref2info: when resolving SHA1's for 'xxx' tag or branch, try first `refs/heads/xxx`, then just 'xxx'; validate SHA1 with 40-char regexp
  - this works-around Gerrit-style repositories that have a 'refs/for/xxx' ref that gets in the way
  - case in point: https://github.com/hardkernel/u-boot/tree/odroidc-v2011.03 (has both refs/heads/odroidc-v2011.03 and refs/for/odroidc-v2011.03)
- git-ref2info: when processing tags for a SHA1, try both dereference and non-dereferenced, in that order, to get a SHA1
  - this solves old problem reported in #4916: GitHub incorrectly processes dereference requests for tags that are not actually annotated
  - case in point: https://github.com/ayufan-rock64/linux-kernel/releases/tag/4.4.202-1237-rockchip-ayufan
- `codeaurora` is no more, https://bye.codeaurora.org/
2023-04-28 21:32:52 +02:00
Ricardo Pardini
dc9f751012 fix rtl88x2cs compilation for kernel 6.3 2023-04-28 21:32:29 +02:00