Commit Graph

38 Commits

Author SHA1 Message Date
Ricardo Pardini
e3a0f949e1 armbian-next: call-stack-analyze.py: detect missing function_to_file and stop properly (half-finished traces?) 2023-02-18 07:45:26 -03:00
Ricardo Pardini
5d3ba1bade armbian-next: patching: rewrite: don't gag on bad "/dev/null.c" rename patches; archeology: --find-copies-harder; try to get GHA to run GH pages
- try to convince GHA to actually run my kernel patching summary workflow on push
2023-02-18 07:44:51 -03:00
Ricardo Pardini
4219533054 armbian-next: aggregation: quick & dirty "potential paths" in Markdown 2023-02-18 07:44:43 -03:00
Ricardo Pardini
872463f2f0 armbian-next: aggregation: dont' require or consider BOARD or LINUXFAMILY for aggregation anymore
- it was only used for "bsp-aggregation" of `rk3318-box` which I will come up with something extensions/hooks based later
2023-02-18 07:44:25 -03:00
Ricardo Pardini
f71e6f4865 armbian-next: patching: accept b4's magic marker git@z Thu Jan 1 00:00:00 1970 for mbox validations 2023-02-18 07:44:15 -03:00
Ricardo Pardini
eb30aac310 armbian-next: extensions: move 'extensions.sh' into library; replace manual source with init function
- wraps globals in `extension_manager_declare_globals()` -- and make them actually global
- whoever wrote this original code is insane, add copyright
2023-02-18 07:44:05 -03:00
Ricardo Pardini
2c9582f9ba armbian-next: add Debian Ports keyring hostdep to strap Debian riscv64 from ports
- rpardini: actually use ports keyring for Debian Risc-V
- `risc-v`: correctly include `debian-ports-archive-keyring` in Risc-V CLI packages
  - include in debootstrap list, via `config/optional/architectures/riscv64/_config/cli/_all_distributions/main/packages`
  - can't have two `--include` in debootstrap call
- riscv64: move `debian-ports-archive-keyring` from main to debootstrap; also only for `sid`
2023-02-18 07:43:49 -03:00
Ricardo Pardini
3e012306c4 armbian-next: patching: less horrible patching log, pt 38: downgrade utf-8 mbox to log warning, mark as problem 2023-02-18 07:43:47 -03:00
Ricardo Pardini
861b07305e armbian-next: patching: less horrible patching log, pt 37
- don't show stats for patches that are bind applied blind (failed to parse/autogen)
- don't show Author, it's too long
- don't dump Python exceptions in log
- extra: expand Python imports
2023-02-18 07:43:44 -03:00
Ricardo Pardini
b0bef8b1c4 armbian-next: patching.py vs decent logging: better, but not there yet
- mostly downgrading stuff to .debug()
- some stats
2023-02-18 07:43:37 -03:00
Ricardo Pardini
a3d8b5891b armbian-next: aggregation.py vs create-cache.sh: better hashing, much desktop
- aggregation.py:
  - only include non-removed pkgs in hash (do'h)
  - use a sorted unique of desktop+rootfs+debootstrap pkgs for pkg hash
  - make space for having apt sources/gpg keys in hash @TODO
  - write a hash-text variable for bash with info, for easy comparison later
  - refactor out bash_string_multiline()
- create-cache.sh:
  - don't armhf unless `ARMHF_ARCH=yes`
  - remove the uninstall stuff until we decide @TODO
  - don't show "free space of MOUNT" when there's only SDCARD
  - better logging for resolv.conf/machine-id shenanigans does in rootfs cache
  - drop dead code, squash some quoting issues
  - show the size of the zstd ball after compressing
  - don't even go to github armbian/cache nor cache.armbian.com for list of caches if SKIP_ARMBIAN_REPO=yes
  - remove mentions of BOARD or BRANCH, not relevant in rootfs cache
2023-02-18 07:43:36 -03:00
Ricardo Pardini
e540e52207 armbian-next: patching: parse renamed source files, don't swallow them during rewrite 2023-02-18 07:42:57 -03:00
Ricardo Pardini
a254643268 armbian-next: patching: avoid mangling valid utf-8 from mbox'es 2023-02-18 07:42:56 -03:00
Ricardo Pardini
7096cf70d3 armbian-next: patching: remove quotes from author name during parsing 2023-02-18 07:42:55 -03:00
Ricardo Pardini
0207d5f517 armbian-next: patching: don't rewrite empty desc as "None" 2023-02-18 07:42:54 -03:00
Ricardo Pardini
3c7587ef84 armbian-next: patching: don't complain about missing files when the patch parsed OK and file was actually deleted 2023-02-18 07:42:53 -03:00
Ricardo Pardini
4307bac073 armbian-next: patching: use --zero-commit instead of find/replace for rewrite 2023-02-18 07:42:02 -03:00
Ricardo Pardini
14ecc86b60 armbian-next: patching: grouping tryout, very basic by-dir: SPLIT_PATCHES=yes 2023-02-18 07:42:01 -03:00
Ricardo Pardini
f0a0fa3000 armbian-next: patching: patches-to-git: don't eat newline between subject and message 2023-02-18 07:42:00 -03:00
Ricardo Pardini
a57e128cfa armbian-next: patching: don't gag if empty hexsha / no archeology possible 2023-02-18 07:41:59 -03:00
Ricardo Pardini
360f1ba4f9 armbian-next: patching: go through parsing & splitting all patches before stopping if there are critical errors 2023-02-18 07:41:56 -03:00
Ricardo Pardini
0b0fa6630b armbian-next: patching: add sanity checks for magic markers matching mbox len & patch_contents not containing magic markers
- case in point, mvebu-edge's disappearing patch in `12-net-dsa-mv88e6xxx.patch`
2023-02-18 07:41:55 -03:00
Ricardo Pardini
b7efd70f1a armbian-next: patching: to-git: don't fail if is_autogen_dir; only change Subject if not rewriting in place 2023-02-18 07:41:54 -03:00
Ricardo Pardini
6c11914e38 armbian-next: cli: json-info: make errors clearer; don't include logs on success; small fixes 2023-02-18 07:41:43 -03:00
Ricardo Pardini
c4a85c4771 armbian-next: json-info: do not replace ${SRC} 2023-02-18 07:41:18 -03:00
Ricardo Pardini
9973010972 armbian-next: Python patching: pass $HOME/$PATH to Python, so git can find .gitconfig and Python can find git
- add sanity check before handling the Git repo in Python (call `status()` to make sure access is OK)
2023-02-18 07:41:16 -03:00
Ricardo Pardini
a3c64bb6e7 armbian-next: aggregation: fix PACKAGE_LIST_BOARD_REMOVE/PACKAGE_LIST_FAMILY_REMOVE; also remove pkgs from DESKTOP aggregation
- case in point: `hostapd`, again.
2023-02-18 07:41:08 -03:00
Ricardo Pardini
770d508ca0 armbian-next: Python patching delusion pt 5 + EXTRAWIFI patch generator/harness + consistent patch mtime targets
- Python patching: tune some logging all-around
- Python patching: *FINALLY* set the dates on the patched files to `max(patch_date, root_makefile_date)` thus making lighting-fast rebuilds again
- new EXTRAWIFI patch generator harness; Python patching EXTRA_PATCH_FILES_FIRST
- Python patching: use temp file for patching rejects; clean it up
- new EXTRAWIFI patch generator harness; Python: mark drivers as autogen, don't split or parse them, read as bytes, apply bytes directly
- new EXTRAWIFI patch generator harness; somewhat-works, but patches are per-family
- Python patching: add the `cache/patch` as a mountpoint
  - Darwin Docker performance is 20x with namedvolume; the cached patches are very large
2023-02-18 07:40:55 -03:00
Ricardo Pardini
97f6836705 armbian-next: Python patching delusion, pt4: series & better markdown
- Python patching: hopefully better Markdown: split series dir; detect more status/problems; arche only for missing Subject
  - Python patching: archeology only for missing Subject:, not description; clarify CLI pushing
  - Python patching: use `{file_base_name}.patch` for archeology as `file_name` might include a dir
- Python patching: fix: don't skip board/target patches for no reason
- Python patching: fix for series.conf patches (sunxi/sunxi64): don't sort; mark as series and show on summary
- Python patching: don't fail if archeology found no commits (but add warning)
- Python patching: CLI command `kernel-patches-to-git` with archeology summary, and git pushing of results & summary
  - Python patching: patches-to-git small fixes, auto-push if it's rpardini
  - Python patching: add `patch-kernel` CLI command
- Python patching: commit README.md & gh-pages workflow when apply_patches_to_git
  - Python patching: hopefully better markdown
- Python patching: `git add` everything all at once, for speed
2023-02-18 07:40:54 -03:00
Ricardo Pardini
2c0e9182ed armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles
- armbian-next: fixes to ORAS-related logging
- armbian-next: actually enable ORAS-based kernel git bare tree seeding/bundles; enable cleaning of bundle articfacts after confirmed working
- armbian-next: introduce ORAS-based kernel git bare tree seeding/bundles (although it is a .tar, not a bundle); this is 20x faster than cloning
- armbian-next: introduce `ORAS` tooling; pull and push functions & downloader/launcher
2023-02-18 07:40:53 -03:00
Ricardo Pardini
76e276c6a9 armbian-next: Python patching delusion, pt1 & pt2 & pt3
- WiP: Python patching delusion, pt 1: finding & parsing patches; apply & git commit with pygit2; Markdown summaries (also for aggregation); git-to-patches tool
  - Python: Markdown aggregation and patching summaries; collapsible; SummarizedMarkdownWriter
  - Python: Markdown aggregation and patching summaries
  - Python: reorg a bit into common/armbian_utils; define the `ASSET_LOG_BASE` in preparation for Markdown delusion
  - Python patching: initial apply patches & initial commit patches to git (using pygit2)
  - Python patching: add basic `series.conf` support
  - Python patching: force use of utf-8; better error handling; use realpath of dirs
  - Python patching: `git-to-patches` initial hack. not proud. half-reused some of the patches-to-git
  - Python patching: "tag" the git commits with info for extracting later; introduce REWRITE_PATCHES/rewrite_patches_in_place
  - Python patching: commented-out, recover-bad-patches hacks
  - Python patching: shorten the signature
  - Python patching: allow BASE_GIT_TAG as well as BASE_GIT_REVISION
  - Python patching: git-archeology for patches missing descriptions; avoid UTF-8 in header/desc (not diff)
  - Python patching: use modern-er email.utils.parsedate_to_datetime to parse commit date
  - Python patching: unify PatchInPatchFile; better git-commiting; re-exporting patches from Git (directly)
  - Python patching: switch to GitPython
    - GitPython is like 100x slower than pygit2, but actually allows for date & committer
    - also allows to remove untracked files before starting
  - Python aggregation: fix missing `AGGREGATED_APT_SOURCES_DICT`
  - Python patching: add `unidecode` dependency to pip3 install
  - Python patching: don't try archeology if SRC is not a Git Repo (eg, in Docker)
  - Python patching: don't try archeology if not applying patches to git
- WiP: Python patching delusion, pt2: actually use for u-boot & kernel patching
  - Python patching: much better problem handling/logging; lenient with recreations (kernel)
  - Python patching: don't force SHOW_LOG for u-boot patching
  - Python patching: don't bomb for no reason when there are no patches to apply
  - Python patching: fully (?) switch kernel patching to Python
  - Python patching: more logging fixups
  - Python patching: capture `kernel_git_revision` from `fetch_from_repo()`'s `checked_out_revision`
  - Python patching: fully switch u-boot patching to Python
  - Python aggregation/patching: colored logging; patching: always reset to git revision
  - Python aggregation/patching: better logging; introduce u-boot Python patching
- Python patching pt3: recovers and better Markdown
  - Python patching: detect, and rescue, `wrong_strip_level` problem; don't try to export patches that didn't apply, bitch instead
  - Python patching: Markdown patching summary table, complete with emoji
  - Python patching: include the problem breakdown in Markdown summary
  - Python patching: sanity check against half-bare, half-mbox patches
  - Python patching: try to recover from 1) bad utf-8 encoded patches; 2) bad unidiff patches; add a few sanity checks
  - Python patching: try, and fail, to apply badly utf-8 encoded patches directly as bytes [reverted]
  - Python patching: try to recover from patch *parse* failures; show summary; better logging
      - set `GIT_ARCHEOLOGY=yes` to do archeology, default not

- armbian-next: Python `pip` dependencies handling, similar to `hostdeps`
  - same scheme for Dockerfile caching
  - @TODO: still using global/shared environment; should move to a dir under `cache` or some kinda venv
- WiP: add `python3-pip` to hostdeps; remove `python-setuptools`
  - remove `python-setuptools` (Python2, no longer exists in Sid) from hostdeps
  - add `python3-pip` to hostdeps; part of virtualenv saga
- WiP: split `kernel.sh` a bit, into `kernel-patching.sh`, `kernel-config.sh` and `kernel-make.sh`
  - `advanced_patch()`: rename vars for clarity; no real changes
- Python patching: introduce FAST_ARCHEOLOGY; still trying for Markdown links
2023-02-18 07:40:52 -03:00
Ricardo Pardini
a9a9428bf7 armbian-next: shellcheck downloader/runner; only for lib/ 2023-02-18 07:40:45 -03:00
Ricardo Pardini
adce39d3b7 armbian-next: aggregation.py rewrite; introduce package-lists.sh; usage in core lib 2023-02-18 07:40:22 -03:00
Ricardo Pardini
578b21922e armbian-next: call trace analyzer and call-graph generator 2023-02-18 07:40:20 -03:00
Ricardo Pardini
a9e88f58e5 armbian-next: json-info command, wrapping up the Python tooling; running in Docker; fix config dump 2023-02-18 07:40:09 -03:00
Ricardo Pardini
2ec5619745 armbian-next: python: info: replace instances of ${SRC} with literal ${SRC} so infos are more comparable 2023-02-18 07:39:03 -03:00
Ricardo Pardini
7ad5f1144c armbian-next: lib/tools - python - config definitions capture and massaging 2023-02-18 07:38:48 -03:00
Ricardo Pardini
3ee0c015cd armbian-next: lib/tools - shell - gen library and shellfmt
- armbian-next: `lib/tools/shellfmt.sh` - format only `lib` and `config/sources/`, `extensions` and `compile.sh`
2022-10-08 14:37:21 +02:00