diff --git a/.build.yml b/.build.yml index 195d6e0984fbc779eb61a2ed9c97843c3f05f904..b6bb7aff8cfd1c1d4bd73570829eefeb0de14e03 100644 --- a/.build.yml +++ b/.build.yml @@ -24,6 +24,9 @@ # S3 credentials for the cacher CI cache (Garage `docker-cache` bucket), # same pair the bencher and ci-cacher builds use. - 7dde4219-0783-4581-a67d-c94749de3600 # ~/.s3-cache-key-id - 0e5b3530-6f19-4f30-9b73-9339dd382e46 # ~/.s3-cache-key-secret + # A tokens.sr.ht working token carrying artifacts:upload, the same secret the + # sibling services mount. It is what publish_artifacts sends. + - c7968415-1a6d-4ca0-a188-150fb7f57b65 # ~/.srht-token sources: - https://git.srht.bigb.es/~bigbes/sr-ht-compare environment: @@ -31,6 +34,12 @@ REPO: sr-ht-compare APK_REPO: alpine/v3.22/bigbes/x86_64 S3_BUCKET: repo S3_ENDPOINT: https://s3.bigb.es + # The second destination of the same apk: one channel of artifacts.sr.ht + # holds the packages of every sibling service, so a consumer adds one + # repository line. See docs/ci.md#publish_artifacts. + ARTIFACTS_ENDPOINT: https://artifacts.srht.bigb.es + ARTIFACTS_CHANNEL: "~bigbes/main" + ARTIFACTS_DIST: v3.22 # CORE_VER must track the deployment's SRHT_CORE_VER; BOOTSTRAP_REV is the # submodule commit core.sr.ht pins at that tag. See docs/ci.md#environment. CORE_VER: "0.84.5" @@ -184,6 +193,48 @@ rclone copyto "$f" "garage:$S3_BUCKET/$APK_REPO/$(basename "$f")" echo "uploaded $(basename "$f")" done echo "published; apk-mirror on phoebe re-indexes within 15 minutes" + - publish_artifacts: | + # The same apk into the artifacts.sr.ht channel, beside the S3 copy above. + # The two destinations are independent on purpose: S3 is the road phoebe + # still walks, this one is the road being opened. Its failure is its own, + # and it never rolls back the upload that already succeeded. + # See docs/ci.md#publish_artifacts. + if [ ! -r ~/.srht-token ]; then + echo "no ~/.srht-token: nothing was published to artifacts.sr.ht" + exit 0 + fi + count=$(find "$HOME/packages" -name '*.apk' | wc -l) + test "$count" -gt 0 || { echo "no .apk under $HOME/packages" >&2; exit 1; } + url="$ARTIFACTS_ENDPOINT/api/v1/pkg/$ARTIFACTS_CHANNEL/apk/$ARTIFACTS_DIST" + # A `find | while read` would run the body in a subshell under ash and + # lose $failed with it, so the loop reads a word list instead. + failed= + for file in $(find "$HOME/packages" -name '*.apk'); do + set +x # the token must not reach the log + code=$(curl -sS -o /tmp/artifacts.out -w '%{http_code}' -X PUT \ + -H "Authorization: Bearer $(cat ~/.srht-token)" \ + --data-binary "@$file" "$url") + set -x + case "$code" in + 200|201) + echo "published $(basename "$file") -> $code" + ;; + 409) + # Same version, other bytes. abuild stamps mtimes into the archive, + # so resubmitting one commit builds a byte-different apk under the + # same pkgver; the published copy stands and the build stays green. + echo "WARNING: $(basename "$file") already published with other bytes; kept the published copy" + ;; + *) + echo "FAILED $(basename "$file") -> $code" >&2 + cat /tmp/artifacts.out >&2 + echo >&2 + failed=1 + ;; + esac + done + test -z "$failed" || exit 1 + echo "index: $ARTIFACTS_ENDPOINT/$ARTIFACTS_CHANNEL/apk/$ARTIFACTS_DIST/x86_64/APKINDEX.tar.gz" - cache_save: | # AFTER publish so an S3 hiccup cannot strand a good apk, and fatal on # purpose. Without --force an upload skips a key already there, so no diff --git a/docs/ci.md b/docs/ci.md index fd98898c6cc90e3bafe2d9bdac44e43d97252c08..5657c69859a9771db005c947fce14e263d0e320a 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -33,13 +33,14 @@ `ecoretest`. Nothing in the tree talks to a live git.sr.ht API. ## secrets -Three, all account-level and shared with the sibling services: +Four, all account-level and shared with the sibling services: | secret | lands at | used by | |---|---|---| | `apk-ci-s3` | `~/.apk-ci.env` | `publish` | | `7dde4219-…` | `~/.s3-cache-key-id` | `cacher_init` | | `0e5b3530-…` | `~/.s3-cache-key-secret` | `cacher_init` | +| `c7968415-…` | `~/.srht-token` | `publish_artifacts` | They are file secrets. Listing them is what turns `publish` and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part @@ -287,6 +288,62 @@ `$REPODEST///`, so each file is copied to a fixed prefix rather than the tree being mirrored. `apk-mirror` on phoebe re-indexes within 15 minutes; nothing here waits for it. + +## publish_artifacts + +The same apk, uploaded a second time — into the `~bigbes/main` channel of +`artifacts.sr.ht`, which indexes and signs it in the same request rather than on +a 15-minute timer. One `PUT` per file to +`/api/v1/pkg/~bigbes/main/apk/v3.22`, with the working token from +`~/.srht-token` as the bearer. No client binary is involved: the upload route +takes the file as the request body, and `curl` is already installed for the +cacher. + +It is a task of its own and not two more lines inside `publish` for one reason: +the S3 copy is what phoebe re-indexes today and this one is the road being +opened. A refusal here has to be legible as *this* destination refusing, and it +must not be able to undo an upload that already succeeded. + +`set +x` around every `curl`, for the same reason `publish` has it: the token is +on the command line and the log is world-readable. + +Three answers are expected and only two of them are good: + +- `201` — published, indexed, signed. +- `200` — the same version with the same bytes was already there. A rerun of a + job that got as far as this task is idempotent, and stays green. +- `409` — the same version with *different* bytes. `abuild` stamps mtimes into + the archive, so resubmitting one commit produces a byte-different apk under an + identical `pkgver`; a published version is immutable, so the daemon keeps what + it has. That is a `WARNING` and not a failure — it means the build was run + twice, not that anything is wrong. Republishing means removing the version + through the API first. + +Anything else prints the daemon's error body and fails the task, after trying +every remaining file: one bad package must not hide the fate of the others. + +The loop is `for file in $(find …)` rather than `find … | while read`, because +under ash the second form runs its body in a subshell and the failure flag set +in it is lost when the pipeline ends — the task would report success it never +had. + +Gated on `~/.srht-token` being readable, exactly as `publish` is gated on +`~/.apk-ci.env`: a manual submission without secrets still builds the apk, and +says that nothing was published. + +Installing what it publishes, on any Alpine box: + +```sh +wget -qO /etc/apk/keys/bigbes@artifacts.srht.bigb.es.rsa.pub \ + https://artifacts.srht.bigb.es/~bigbes/keys/apk.rsa.pub +echo https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22 >> /etc/apk/repositories +apk update && apk add diff.sr.ht +``` + +The name of the key file is not free: apk looks a signature up by the name +carried in the index's `.SIGN.RSA256.` entry, so it has to be +`bigbes@artifacts.srht.bigb.es.rsa.pub` and nothing else. With the key in place +no `--allow-untrusted` is needed — the daemon signs the index it builds. ## cache_save