Commit Graph

115 Commits

Author SHA1 Message Date
Ricardo Pardini
c825966857 pipeline: targets-compositor: force variable BETA the same for all targets, reading from param/env passed to prepare/gha-matrix step
- this way it's consistent across all artifacts and images
- debs-to-repo-download: fix: use DEB_STORAGE instead of hardcoding output/debs (so BETA=yes works)

also:
- fix `Duplicate oci_target` message in mapper-oci-uptodate
2023-05-25 19:30:40 +02:00
Ricardo Pardini
eff56a7909 pipeline: debs-to-repo (v7, working with reprepro)
- pipeline: artifacts: use better description for artifacts (artifact_name+artifact_version instead of artifact_final_file_basename)
2023-05-16 20:09:47 +02:00
Ricardo Pardini
b7c659c0a9 pipeline: gha: fix IMAGES_ONLY_OUTDATED_ARTIFACTS and SKIP_IMAGES by exporting them so Python can see; tune logging 2023-05-09 10:04:57 +02:00
Ricardo Pardini
de8bbcec11 pipeline: correctly quote params passed over pure strings (eg in the GHA JSON matrixes) so values with spaces are not mangled 2023-05-04 16:08:11 +02:00
Ricardo Pardini
1b36bb63d7 bash_declare_parser: parse space-separated single-quoted array values correctly ('THIS=has space' is a single token, not two) 2023-05-04 16:08:11 +02:00
Ricardo Pardini
875960ddb7 aggregation: armbian_utils.parse_env_for_tokens() now actually does what it said on the box "split by whitespace, line breaks, commas, and semicolons"; add appgroups summary 2023-05-04 16:08:11 +02:00
Ricardo Pardini
23a2b34847 pipeline: add output-gha-workflow-template.py utility, for rendering GHA workflow YAML templates with chunks
- python-tools: add Jinja2. Incredible how we made it this far without it.
- output-gha-workflow-template: a double-templater, first runs jinja with a custom syntax, then "moar magic" to be useful for GHA
2023-05-03 07:49:53 +02:00
Ricardo Pardini
0aac5a4945 pipeline: targets-compositor: whole target can be enabled: no to quickly disable it (as if it wasn't there) 2023-05-03 07:49:53 +02:00
Ricardo Pardini
46cd0a2cfc pipeline: output-gha-matrix: directly GHA-output images-chunk-json-2, images-chunk-not-empty-2, images-chunk-size-2 and artifacts-xxx with no fake entry
- as alternative to the full output, this allows us to skip chunkjobs when they're empty, and hopefully makes GHA-JSON-parsing 10x faster
2023-05-03 07:49:53 +02:00
Ricardo Pardini
d6bdf4221e pipeline: output-gha-matrix: new input.pipeline.gha.runners in the targets.yaml for mapping GHA's runs_on based on name and arch 2023-05-03 07:49:53 +02:00
igorpecovnik
ba1a84a5e8 Adjusting ORAS cache location, runners (#5129)
* Move packages to new location
* Adjust build runners
* Switch to OS

Signed-off-by: Igor <igor@armbian.com>
---------

Signed-off-by: Igor <igor@armbian.com>
Co-authored-by: Ricardo Pardini <ricardo@pardini.net>
2023-05-02 11:08:34 +02:00
Ricardo Pardini
f8ddf7f9e2 🔥 JSON info pipeline: v18
- pipeline: add `pipeline` context object to targets; use it to filter artifacts and images to build; warn about oci-name with multiple oci-tags
- pipeline: better error messages when info's fail; show some (out-of-order) progress messages during parallel info gathering
- pipeline: targets-compositor: add `not-eos` inventory
- TARGETS_FILENAME, log all OCI lookups
- SKIP_IMAGES
- IMAGES_ONLY_OUTDATED_ARTIFACTS
- no dash in chunk id in JSON
- pipeline: very initial chunking, using the same outputs
- pipeline: template targets, `items-from-inventory:` inventory expansion, CHECK_OCI=yes, CLEAN_MATRIX=yes, CLEAN_INFO=yes, many fixes
- cli: `inventory` / `targets` / `matrix` / `workflow`
- pipeline: workflow beginnings
- pipeline: general log cleanup + OCI stats / better miss handling
- pipeline: fixes/reorg
- pipeline: catch & log JSON parsing errors
- pipeline: gha matrix: use IMAGE_FILE_ID as job description
- pipeline (delusion): gha workflow output, based on old matrix code
- pipeline: better parsing and reporting of stderr log lines (under `ANSI_COLOR=none`)
- pipeline: mapper-oci-uptodate: use separate positive/negative cache dirs (GHA will only cache positives); cache negs for 5 minutes locally
- pipeline: output-gha-matrix artifacts + images
  - pipeline: output-gha-matrix artifacts + images: "really" and fake 1-item matrix if empty
- pipeline: move files into subdir; update copyright & cleanup
- pipeline: refactor bash jsoninfo driver a bit
- pipeline: outdated-artifact-image-reducer
- pipeline: introduce `target_id` at the compositor, aggregate it at the reducer, carry it over in the artifact info mapper
- pipeline: mapper-oci-uptodate
- pipeline: info-gatherer-artifact, with PRE_PREPARED_HOST
- pipeline: refactor/rename info-gatherer-image.py
- pipeline: beginnings
2023-05-01 22:46:25 +02:00
Ricardo Pardini
22059c2669 lib: shellfmt & fix gen-library to match 2023-04-18 09:27:38 +02:00
Ricardo Pardini
e631e0fc97 patching: python: useful debug log with value of BASE_GIT_TAG 2023-04-15 22:43:06 +02:00
Ricardo Pardini
f75b9c110f aggregation: fix: don't eat value if empty during bash_string_multiline(); write end-of-file comment; show aggregation bash results if debugging 2023-03-31 09:12:44 +02:00
Ricardo Pardini
7d90b10f79 configdump: better logging; insert (still unsupported) array/dict raw value and 2do markers into produced dump JSON 2023-03-31 09:12:44 +02:00
Ricardo Pardini
42fc56697b configdump/json-info-boards: revamp, all-JSON now; use_board=yes skip_kernel=no for config; refactor & use new Python bash-declare-to-JSON utility
- use new capture'd vars scheme
- so `./compile.sh BOARD=xxx BRANCH=yyyy config-dump-json | jq .` now works and is consistent/newline tolerant
- introduce internal `skip_host_config=yes` for `prep_conf_main_minimal_ni()` to skip calling `check_basic_host()`
2023-03-31 09:12:44 +02:00
Ricardo Pardini
a13383143c lib/tools/shellcheck.sh: add (very low-bar "error") checks to config/sources and config/boards (keeping lib/ code in custom "critical" level) 2023-03-20 17:54:25 -03:00
Ricardo Pardini
6b965c605c completely remove any traces of KERNEL_ONLY and exit with error if set 2023-03-17 08:32:11 +01:00
Igor Pečovnik
d59fa58875 Add (c) also to python scripts (#4926) 2023-03-09 20:52:06 +01:00
Igor Pečovnik
b7b8eb7b72 Add / modify (c) in bash scripts (#4922)
* Add / modify (c) in bash scripts

Signed-off-by: Igor <igor@armbian.com>

* Add (c) to the source config files

---------

Signed-off-by: Igor <igor@armbian.com>
2023-03-09 18:30:40 +01:00
Ricardo Pardini
2f7423d0e4 shellfmt/.editorconfig: enable trim whitespaces, run shellfmt, no actual code changes (#4849)
- shellfmt: bump to 3.6.0 (run with `bash lib/tools/shellfmt.sh`)
- .editorconfig: rationalize, remove duplicated logic
2023-02-23 09:24:41 +01:00
Ricardo Pardini
49947ec373 armbian-next: aggregation: don't include packages.additional if BUILD_MINIMAL=yes 2023-02-18 07:45:58 -03:00
Ricardo Pardini
2767cf8fc2 armbian-next: shellcheck: raise the bar, stop excluding "SC2154 is referenced but not assigned" 2023-02-18 07:45:32 -03:00
Ricardo Pardini
010760fa92 armbian-next: shellcheck: raise the bar, stop excluding "SC2155 Declare and assign separately to avoid masking return values" 2023-02-18 07:45:30 -03:00
Ricardo Pardini
7d540d8a2e armbian-next: shellcheck: update, cleanup script, invent some new severities 2023-02-18 07:45:28 -03:00
Ricardo Pardini
1fc13a57a7 armbian-next: json-info: cleanup info/json/csv/opensearch, add logging, add some very basic instructions; add Kibana dashboards & docker-compose to bring it OS+Kibana up 2023-02-18 07:45:27 -03:00
Ricardo Pardini
e3a0f949e1 armbian-next: call-stack-analyze.py: detect missing function_to_file and stop properly (half-finished traces?) 2023-02-18 07:45:26 -03:00
Ricardo Pardini
5d3ba1bade armbian-next: patching: rewrite: don't gag on bad "/dev/null.c" rename patches; archeology: --find-copies-harder; try to get GHA to run GH pages
- try to convince GHA to actually run my kernel patching summary workflow on push
2023-02-18 07:44:51 -03:00
Ricardo Pardini
4219533054 armbian-next: aggregation: quick & dirty "potential paths" in Markdown 2023-02-18 07:44:43 -03:00
Ricardo Pardini
872463f2f0 armbian-next: aggregation: dont' require or consider BOARD or LINUXFAMILY for aggregation anymore
- it was only used for "bsp-aggregation" of `rk3318-box` which I will come up with something extensions/hooks based later
2023-02-18 07:44:25 -03:00
Ricardo Pardini
f71e6f4865 armbian-next: patching: accept b4's magic marker git@z Thu Jan 1 00:00:00 1970 for mbox validations 2023-02-18 07:44:15 -03:00
Ricardo Pardini
eb30aac310 armbian-next: extensions: move 'extensions.sh' into library; replace manual source with init function
- wraps globals in `extension_manager_declare_globals()` -- and make them actually global
- whoever wrote this original code is insane, add copyright
2023-02-18 07:44:05 -03:00
Ricardo Pardini
2c9582f9ba armbian-next: add Debian Ports keyring hostdep to strap Debian riscv64 from ports
- rpardini: actually use ports keyring for Debian Risc-V
- `risc-v`: correctly include `debian-ports-archive-keyring` in Risc-V CLI packages
  - include in debootstrap list, via `config/optional/architectures/riscv64/_config/cli/_all_distributions/main/packages`
  - can't have two `--include` in debootstrap call
- riscv64: move `debian-ports-archive-keyring` from main to debootstrap; also only for `sid`
2023-02-18 07:43:49 -03:00
Ricardo Pardini
3e012306c4 armbian-next: patching: less horrible patching log, pt 38: downgrade utf-8 mbox to log warning, mark as problem 2023-02-18 07:43:47 -03:00
Ricardo Pardini
861b07305e armbian-next: patching: less horrible patching log, pt 37
- don't show stats for patches that are bind applied blind (failed to parse/autogen)
- don't show Author, it's too long
- don't dump Python exceptions in log
- extra: expand Python imports
2023-02-18 07:43:44 -03:00
Ricardo Pardini
b0bef8b1c4 armbian-next: patching.py vs decent logging: better, but not there yet
- mostly downgrading stuff to .debug()
- some stats
2023-02-18 07:43:37 -03:00
Ricardo Pardini
a3d8b5891b armbian-next: aggregation.py vs create-cache.sh: better hashing, much desktop
- aggregation.py:
  - only include non-removed pkgs in hash (do'h)
  - use a sorted unique of desktop+rootfs+debootstrap pkgs for pkg hash
  - make space for having apt sources/gpg keys in hash @TODO
  - write a hash-text variable for bash with info, for easy comparison later
  - refactor out bash_string_multiline()
- create-cache.sh:
  - don't armhf unless `ARMHF_ARCH=yes`
  - remove the uninstall stuff until we decide @TODO
  - don't show "free space of MOUNT" when there's only SDCARD
  - better logging for resolv.conf/machine-id shenanigans does in rootfs cache
  - drop dead code, squash some quoting issues
  - show the size of the zstd ball after compressing
  - don't even go to github armbian/cache nor cache.armbian.com for list of caches if SKIP_ARMBIAN_REPO=yes
  - remove mentions of BOARD or BRANCH, not relevant in rootfs cache
2023-02-18 07:43:36 -03:00
Ricardo Pardini
e540e52207 armbian-next: patching: parse renamed source files, don't swallow them during rewrite 2023-02-18 07:42:57 -03:00
Ricardo Pardini
a254643268 armbian-next: patching: avoid mangling valid utf-8 from mbox'es 2023-02-18 07:42:56 -03:00
Ricardo Pardini
7096cf70d3 armbian-next: patching: remove quotes from author name during parsing 2023-02-18 07:42:55 -03:00
Ricardo Pardini
0207d5f517 armbian-next: patching: don't rewrite empty desc as "None" 2023-02-18 07:42:54 -03:00
Ricardo Pardini
3c7587ef84 armbian-next: patching: don't complain about missing files when the patch parsed OK and file was actually deleted 2023-02-18 07:42:53 -03:00
Ricardo Pardini
4307bac073 armbian-next: patching: use --zero-commit instead of find/replace for rewrite 2023-02-18 07:42:02 -03:00
Ricardo Pardini
14ecc86b60 armbian-next: patching: grouping tryout, very basic by-dir: SPLIT_PATCHES=yes 2023-02-18 07:42:01 -03:00
Ricardo Pardini
f0a0fa3000 armbian-next: patching: patches-to-git: don't eat newline between subject and message 2023-02-18 07:42:00 -03:00
Ricardo Pardini
a57e128cfa armbian-next: patching: don't gag if empty hexsha / no archeology possible 2023-02-18 07:41:59 -03:00
Ricardo Pardini
360f1ba4f9 armbian-next: patching: go through parsing & splitting all patches before stopping if there are critical errors 2023-02-18 07:41:56 -03:00
Ricardo Pardini
0b0fa6630b armbian-next: patching: add sanity checks for magic markers matching mbox len & patch_contents not containing magic markers
- case in point, mvebu-edge's disappearing patch in `12-net-dsa-mv88e6xxx.patch`
2023-02-18 07:41:55 -03:00
Ricardo Pardini
b7efd70f1a armbian-next: patching: to-git: don't fail if is_autogen_dir; only change Subject if not rewriting in place 2023-02-18 07:41:54 -03:00