diff --git a/justfile b/justfile index c7ce2b705cd36d810b148e470d0983574baf30b7..168ca98af268929931f2a6ebb636f239a0462f5e 100644 --- a/justfile +++ b/justfile @@ -2,7 +2,7 @@ _default: @just --list # Skills backed up in this repo, installed into ~/.claude/skills (repo -> ~/.claude) -skills := "sourcehut-ci sourcehut-ci-cacher sourcehut-custom-service" +skills := "sourcehut-ci sourcehut-ci-cache sourcehut-custom-service" # Absolute path to this repo (the SourceHut documentation mirror), substituted # into skills for the __SRHT_MIRROR__ placeholder at install time. diff --git a/skills/sourcehut-ci-cache/SKILL.md b/skills/sourcehut-ci-cache/SKILL.md new file mode 100644 index 0000000000000000000000000000000000000000..170166f59afa46f376426649f5d172f75cc3099a --- /dev/null +++ b/skills/sourcehut-ci-cache/SKILL.md @@ -0,0 +1,262 @@ +--- +name: sourcehut-ci-cache +description: Add or fix build caching in a builds.sr.ht manifest with `art cache`, the cache subcommands of the artifacts.sr.ht client — directory trees and docker images in the Garage `docker-cache` bucket, reached through artifacts.srht.bigb.es with an sr.ht token and no S3 credential in the build. Use whenever a .build.yml needs to cache Go module/build trees, an assembled SCSS tree, a docker image or any other directory between builds; whenever a build is slow because it recompiles or re-clones the same dependencies every time; whenever the user mentions art cache, cache_restore, cache_save, art_init, install_art, CACHE_NS, docker-cache, `~/.srht-token`, or asks how to cache anything on sourcehut. Companion to the `sourcehut-ci` skill, which covers builds.sr.ht itself — upstream sourcehut genuinely has no cache feature, and this is the local answer to that. +license: BSD-3-Clause +metadata: + audience: developers + workflow: ci + tags: [ci, sourcehut, builds.sr.ht, cache, art, artifacts.sr.ht, garage, go] + version: 2.0.0 + author: bigbes +--- + +# Build caching on sourcehut + +builds.sr.ht has no cache feature: every build is a fresh VM. The local answer +is **`art cache`** — the cache subtree of `art`, the artifacts.sr.ht client +(`sr-ht-monorepo/services/artifacts/cmd/art`). It tars a directory (or a docker +image) under a content-derived key and stores it through +`artifacts.srht.bigb.es/api/v1/cache` in the Garage `docker-cache` bucket, so +the build carries a working sr.ht token rather than a raw bucket credential — +and that token is usually mounted already for publishing, coverage or bench. + +**Reference implementations:** + +- `~/data/home/sr-ht-monorepo/ci/lib/art.sh` (install/init), `ci/lib/cache.sh` + (Go restore/save), `ci/lib/scss.sh` (`--exec`), driven from `ci/.yml` + — `ci/tokens.yml` is a representative manifest. The reasoning per task is in + `services//docs/ci.md` (`#cache_restore`, `#cache_save`, `#scss`). +- `~/data/home/mcp-as/.build.yml` — the same pattern inlined into a single-repo + manifest, no helper scripts. Copy this shape for a standalone repo. +- `sr-ht-monorepo/ci/artifacts.yml` `cache_setup` — the one project that builds + `art` from its own checkout instead of fetching it (see below). + +## Manifest shape + +```yaml +secrets: + # Carries artifacts:read (cache reads) AND artifacts:upload (cache writes). + - c7968415-1a6d-4ca0-a188-150fb7f57b65 # ~/.srht-token +environment: + ARTIFACTS_ENDPOINT: https://artifacts.srht.bigb.es + # Objects land at /. One namespace per project. + CACHE_NS: +tasks: + - install_art: | + mkdir -p ~/.local/bin + curl -sSL "$ARTIFACTS_ENDPOINT/~bigbes/main/blob/art/art-linux-amd64" \ + -o ~/.local/bin/art + chmod +x ~/.local/bin/art + echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.buildenv + export PATH="$HOME/.local/bin:$PATH" # ~/.buildenv reaches only LATER tasks + art version + - art_init: | + if [ ! -r ~/.srht-token ]; then + echo "the art cache backend needs a working token at ~/.srht-token" >&2 + exit 1 + fi + art init --endpoint "$ARTIFACTS_ENDPOINT" --cache-ns "$CACHE_NS" \ + --token-file ~/.srht-token + - cache_restore: | + KEY_MOD=$(art cache key "deps/gomod/{hash}.tar.zst" --hash-from "$REPO/go.sum") + KEY_GOC=$(art cache key "deps/gocache/{hash}.tar.zst" --hash-from "$REPO/go.sum") + echo "export KEY_MOD=$KEY_MOD KEY_GOC=$KEY_GOC" >> ~/.buildenv + art cache dir download "$KEY_MOD" ~/go/pkg/mod --optional + art cache dir download "$KEY_GOC" ~/.cache/go-build --optional + # see "The half-restored module cache" below — do not drop, do not soften + cd "$REPO" + chmod -R u+w ~/go/pkg/mod 2>/dev/null || true + if ! go mod verify >/dev/null 2>&1; then + echo "restored module cache did not verify — discarding it" + rm -rf ~/go/pkg/mod + fi + go mod download # NEVER `all` — see below + go mod verify + git status --porcelain # the tree must still be clean here + # … build / test / publish … + - cache_save: | + art cache dir upload "$KEY_MOD" ~/go/pkg/mod + art cache dir upload "$KEY_GOC" ~/.cache/go-build +``` + +Verbs: `art cache key TEMPLATE --hash-from FILE`, `art cache dir +download|upload KEY DIR`, `art cache docker download|upload KEY IMAGE:TAG`, +`art cache download|upload KEY FILE`, `art cache exists|ls|rm`. `--endpoint`, +`--token-file` and `--cache-ns` are persistent root flags; `ART_TOKEN` +overrides the token file at use time. + +## Installing `art` + +- **The binary is a blob, not an apk and not a build step.** `sr-ht-artifacts` + republishes `linux/amd64` `art` to `~bigbes/main/blob/art/art-linux-amd64` on + every `master` build (its `publish_client` task). The channel is public, so a + consuming build fetches it anonymously and needs the token only for the cache + itself. Static `CGO_ENABLED=0`, so Alpine and Ubuntu workers both run it. +- **The download is not verified.** The digest is checked at *publish* time — + `art blob push` refuses to report success unless the daemon's sha256 matches + what it sent — which is a guarantee about what is stored, not about what a + consumer received. +- **The artifacts service itself must not fetch it that way.** Its manifest + builds `art` from its own checkout (`go build ./cmd/art` in `cache_setup`), so + the cache does not depend on the service it is about to build. +- **`~/.local/bin` is not on PATH** in the default Ubuntu VM, hence both the + `~/.buildenv` line and the in-task export. + +## Rules that are load-bearing + +**`install_art` and `art_init` stay separate from the tasks that use `art`.** +builds.sr.ht sources `~/.buildenv` *before each task*, so the `PATH` line is in +effect from the NEXT task onward. The in-task `export` covers the rest of the +install task and nothing after it. + +**A green `art_init` proves nothing about the token.** `art init` probes +`GET /api/v1/channels`, which succeeds *anonymously* against a public channel, +and writes its config either way. Gate on the token file yourself (as above) so +a submission without the secret fails at `art_init` rather than several tasks +later. What proves `artifacts:read` is a later `hit / -> …` line; what +proves `artifacts:upload` is a `cache_save` that does not fail. + +**Restore before the first compiler runs, not before `build`.** If the manifest +has a `test` task, that is usually where the tree first meets a compiler; a +restore placed after it warms nothing but the final relink. + +**`--optional`, never `|| true`.** `||` fires on every non-zero exit, so it also +swallows a bad token, a missing grant and a truncated tarball — each becomes one +line in the log and a silently cold build forever after. `--optional` exits 0 +for a missing key and nothing else. + +**`cache_save` goes after `publish`, and stays fatal.** It is an optimisation for +the *next* build; placed earlier, a storage hiccup fails a build whose artifact +was already built but not yet published. Keep it fatal so a broken token is +noticed rather than quietly making every build cold. + +**No `art cache exists "$KEY" ||` guard before an upload.** Without `--force`, +`dir upload` checks the key itself and leaves an existing one alone, exit 0. +The guard duplicates that one level up and wrongly, since two separate commands +can disagree about a key uploaded between them. + +**The namespace is the tenant boundary.** Every project shares the bucket and +the token. Rename a project and `CACHE_NS` must follow — a stale namespace is +not an error, it is a cache that never hits. Keys stored as `/` with no +rewriting, so `CACHE_NS` plus the key is the whole object name; the family keeps +Go and SCSS keys under `deps/`. + +## `dir download --exec` for build-it-if-missing + +`--exec` replaces the restore-or-build block: on a miss it creates the directory, +runs the script through `sh -c`, and seeds the cache from what the script leaves +behind. The script's own exit status becomes `art`'s. + +```yaml + art cache dir download "deps/scss/${CORE_VER}-${BOOTSTRAP_REV}.tar.zst" ~/scss --exec ' + git clone --depth 1 --branch "$CORE_VER" … /tmp/core + cp /tmp/core/scss/*.scss ~/scss/ + ' +``` + +- **Write the key inline; do not hoist it to a variable.** The script runs in a + separate shell and sees only exported values — `environment:` entries and + whatever an earlier task appended to `~/.buildenv`. A shell-local `KEY=…` in + the task body is invisible inside the script. Spelling the template out in the + argument lets the *outer* shell expand it before `art` sees it. +- **Single-quote the whole `--exec` argument.** The script's own `"$VAR"` + references must reach `sh -c` unexpanded. Double quotes would expand them + outside and happen to produce the same text today — then substitute the wrong + thing the day a variable is set inside the script. +- **`--exec` has no `set -e`.** The LAST command's status decides whether the + cache is seeded, not the first failure. Add `set -e` at the top of the script + the moment it grows a step the following ones do not depend on, or you will + seed a half-built tree under a key that claims to be good. +- Seeding after `--exec` is best-effort: the tree already exists locally, so a + failed upload there does not fail the build. + +## Docker images + +`art cache docker download KEY IMAGE:TAG --pull` loads a cached image, and on a +miss pulls it and seeds the cache. **Check the buildx driver first:** with the +`docker-container` driver the builder has its own image store and cannot see +what was loaded into the host daemon, so the task warms nothing. Mirror the base +image to a registry on the LAN and pin `FROM` to it instead, or use the plain +`docker` driver. That is a property of buildx, not of the cache. + +## Never `go mod download all` + +`all` resolves the whole module graph — test-only dependencies of dependencies +included — and **appends their hashes to the TRACKED `go.sum`**. It exits 0 and +prints nothing, so the only trace is a modified file sitting in the checkout. +Measured in the sr.ht services: 157 lines to 426. + +**`-mod=readonly` does not prevent it.** Readonly governs updates to the module +*requirements*, not writes to `go.sum`. + +It matters wherever the pipeline reads a VCS stamp: a dirtied `go.sum` makes +every later `go build` record `vcs.modified=true`, so the packaged binary calls +itself `-dirty` and a `check-version` gate fails on `M go.sum`. + +Plain `go mod download` writes nothing and still fetches everything the module +builds and tests (verified against an empty module cache with `GOPROXY=off`). +Keep a `git status --porcelain` at the end of `cache_restore` regardless — it +makes this class of failure visible at the task that causes it. + +## The half-restored module cache + +The most important paragraph here, and the one that looks like removable +boilerplate. + +Go writes its module cache read-only — module directories are `0555`. The +tarball preserves those modes, so an extraction can create a module directory +and then be **denied its own next file inside it**. The result is a cache HIT, +one swallowed `permission denied`, and a handful of packages failing to compile +against a dependency missing a single file — while unrelated packages pass, so +the log reads like a code bug. cover.sr.ht build **#284** is that failure. + +Two halves to the fix, and both are needed: + +- **`-modcacherw`** on the `go build` that populates the cache (in the APKBUILD's + `build()` for the apk-packaging projects), so the stored tree is writable. +- **the repair block** in `cache_restore`: `chmod -R u+w`, then `go mod verify` + as the gate, then `rm -rf ~/go/pkg/mod` on failure. `go mod verify` recomputes + each extracted module's hash and is the only instrument that sees a file + missing from a directory Go otherwise trusts on sight. + +Never soften the verify to `|| true`. A bad restore costs one cold download, +which is the optimisation failing, not the build failing. The first build after +a `go.sum` change can never detect this: a miss extracts nothing, `go mod +download` fills a clean tree, and `cache_save` then stores the tarball that +breaks every build after it. + +## abuild interaction (apk-packaging projects) + +abuild redirects the Go caches into its throwaway `$tmpdir`, so exports from +`cache_restore` do not survive into the package build. Worse, `abuild.in` reads: + +```sh +export GOCACHE="${GOCACHE:-$tmpdir/go}" +export GOMODCACHE="${GOCACHE:-$tmpdir/gomod}" # upstream typo: GOCACHE, not GOMODCACHE +``` + +Because `$GOCACHE` is never empty by the second line, `GOMODCACHE` is +**overwritten with GOCACHE's value** regardless of what the APKBUILD set at top +level. So the pin has to sit **inside the function that compiles**: + +```sh +build() { + export GOCACHE="$HOME/.cache/go-build" + export GOMODCACHE="$HOME/go/pkg/mod" + CGO_ENABLED=0 make build GOFLAGS="-trimpath -modcacherw" +} +``` + +`package()` runs in a *fresh* abuild process under fakeroot and never calls +`build()`, so nothing `build()` exported reaches it. If `package()` also +compiles, it compiles from a cold cache — make `package()` install only. + +## Manifest size + +builds.sr.ht stores a submitted manifest in a `varchar(16384)`. Cache stages plus +their comments add roughly 3 KB. Over the cap, `hut builds submit` fails with +`pq: value too long for type character varying(16384)` — and git.sr.ht's push +hook fails the same way, which is the dangerous one: not a red build but no +build, with nobody mailed. Check `wc -c` on the manifest after adding stages, and +put the reasoning in `docs/ci.md` with a one-line pointer in the manifest — or +move the stages into helper scripts, as `sr-ht-monorepo/ci/lib/` does. diff --git a/skills/sourcehut-ci-cacher/SKILL.md b/skills/sourcehut-ci-cacher/SKILL.md deleted file mode 100644 index 74b5e9f3a1175110e8c58cbbf2f0fa4b6130b391..0000000000000000000000000000000000000000 --- a/skills/sourcehut-ci-cacher/SKILL.md +++ /dev/null @@ -1,362 +0,0 @@ ---- -name: sourcehut-ci-cacher -description: Add or fix build caching in a builds.sr.ht manifest with either of the user's two cache backends — `cacher` (go.bigb.es/cacher, straight to Garage S3) or `art cache` (the artifacts.sr.ht client, same bucket through the daemon and no S3 credential in the build). Use whenever a .build.yml needs to cache Go module/build trees, an assembled SCSS tree, a docker image or any other directory between builds; whenever a build is slow because it recompiles or re-clones the same dependencies every time; whenever the user mentions cacher, art cache, cache_restore, cache_save, cache_setup, docker-cache, ci-cacher, `~/.s3-cache-key-id`, `~/.srht-token`, or asks how to cache anything on sourcehut; and whenever a manifest is being moved from cacher to art. Also use when copying cache stages between the `~/data/home/sourcehut-*` sibling repos, since those carry two different generations of the pattern and the older one is easy to copy by mistake. Companion to the `sourcehut-ci` skill, which covers builds.sr.ht itself — upstream sourcehut genuinely has no cache feature, and this is the local answer to that. -license: BSD-3-Clause -metadata: - audience: developers - workflow: ci - tags: [ci, sourcehut, builds.sr.ht, cache, cacher, art, artifacts.sr.ht, s3, garage, go] - version: 1.1.0 - author: bigbes ---- - -# Build caching on sourcehut - -builds.sr.ht has no cache feature: every build is a fresh VM. Two local helpers -answer that, and they are interchangeable because they address the same objects: - -- **`cacher`** (`go.bigb.es/cacher`) tars a directory to S3 under a - content-derived key, speaking S3 itself. Needs the Garage key pair. -- **`art cache`** (the artifacts.sr.ht client) does the same through - `artifacts.srht.bigb.es/api/v1/cache`, so the build carries a working token - instead of a raw bucket credential. See "The `art` backend" below. - -Both land in the Garage `docker-cache` bucket on `s3.bigb.es`. Most of this -document is about the shape of the cache stages, which is the same either way; -where a rule differs, it says so. - -**Reference implementations:** `__SRHT_FAMILY__/sourcehut-coverage` and -`__SRHT_FAMILY__/sourcehut-tokens` for the overall shape, but take `cache_restore` -from `__SRHT_FAMILY__/sourcehut-bench` — the other two still carry a `go mod download -all` that dirties the tracked `go.sum` (see "Never `go mod download all`"). All -three keep the reasoning in `docs/ci.md` alongside the manifest; read -`docs/ci.md#cacher`, `#cache_restore`, `#never-go-mod-download-all` and -`#cache_save` before writing anything new. - -**For `art`:** `__SRHT_FAMILY__/sourcehut-artifacts` carries the switchable -`cache_setup` and `publish_client`, the task that publishes the binary every -other build installs — read `docs/ci.md#cache_setup` and `#publish_client` there. -`__SRHT_FAMILY__/bencher/.build.yml` on `license-free/v0.6.6` is the plain hardwired -form, and the only non-Alpine consumer. - -## Do not copy from the older siblings - -`sourcehut-spec`, `sourcehut-dolt` and `sourcehut-compare` still carry the -pre-v0.2.0 shape. Grepping `cache` across the family returns the stale form -three times and the current form twice, so a majority vote picks the wrong -donor. - -| stale | current | -|---|---| -| hand-written `curl` + `chmod` + `PATH` bootstrap | `curl -sSL …/ci-cacher/install.sh \| sh` | -| `if ! cacher dir download …; then …; cacher dir upload …; fi` | `cacher dir download --exec '