diff --git a/justfile b/justfile index 09e8f533d18a39e4c4066a5fe198df2cbb7642a3..bf5e254e1ab81d1c70dda827475f6aeb836d0121 100644 --- a/justfile +++ b/justfile @@ -8,6 +8,13 @@ # Absolute path to this repo (the SourceHut documentation mirror), substituted # into skills for the __SRHT_MIRROR__ placeholder at install time. mirror := justfile_directory() +# Absolute path to the directory holding the sibling repos this instance is +# actually built from — the custom services (sourcehut-*), the shared library +# (sr-ht-ecore) and the forks (sr-ht-core, sr-ht-api). Substituted into skills +# for the __SRHT_FAMILY__ placeholder. Citations in the skills are rooted at one +# of the two: `mirror::` for upstream's published source, `family::` for ours. +family := parent_directory(justfile_directory()) + # Install the backed-up skills into ~/.claude/skills install: #!/usr/bin/env bash @@ -17,9 +24,10 @@ src="skills/$skill" dst="$HOME/.claude/skills/$skill" mkdir -p "$dst" rsync -a --delete --exclude='.*' "$src/" "$dst/" - # Substitute the mirror-root placeholder with this repo's absolute path + # Substitute the two root placeholders with their absolute paths find "$dst" -type f -name '*.md' -print0 | while IFS= read -r -d '' f; do - sed "s|__SRHT_MIRROR__|{{mirror}}|g" "$f" > "$f.tmp" && mv "$f.tmp" "$f" + sed -e "s|__SRHT_MIRROR__|{{mirror}}|g" \ + -e "s|__SRHT_FAMILY__|{{family}}|g" "$f" > "$f.tmp" && mv "$f.tmp" "$f" done echo "Installed $skill -> $dst ($(find "$dst" -type f | wc -l | tr -d ' ') files)" done diff --git a/skills/sourcehut-custom-service/SKILL.md b/skills/sourcehut-custom-service/SKILL.md index be9ffafa2597c0a8dab20365d34f86ad50f60257..0e7bfbdfa07b6dfb0190865de881215e868c2f6d 100644 --- a/skills/sourcehut-custom-service/SKILL.md +++ b/skills/sourcehut-custom-service/SKILL.md @@ -1,267 +1,129 @@ --- name: sourcehut-custom-service -description: How to write your OWN service that integrates into a self-hosted SourceHut — appearing in the shared nav/service-switcher, reusing the unified login session, the shared theme, GraphQL federation, and webhooks, WITHOUT modifying upstream SourceHut source. Covers the integration mechanisms (all config-driven) and concrete recipes in Python, Go, and any other language. Use when the user asks about building a custom/third-party SourceHut service, a plugin, extending SourceHut, adding a service to their instance, or making something "show up like paste/todo/git" in the web UI. ---- - -# Writing a custom service that integrates into SourceHut - -## TL;DR - -SourceHut has **no plugin system** — nothing loads third-party code into a running service. But the things that make a service feel "integrated" (the nav service-switcher, the shared login session, the theme, the unified GraphQL endpoint, webhooks) are all **config-driven** and reusable from any language. Your service runs as its own separate process/daemon; you wire it in via `config.ini` + nginx, and optionally reuse `core.sr.ht` (Python) or `core-go` (Go) libraries. - -**This only works on a self-hosted instance you control** (config + nginx + DNS), e.g. the user's `*.srht.bigb.es`. You cannot add a service to hosted sr.ht. SourceHut services are tightly-coupled siblings sharing `core.sr.ht`/`core-go`, **not** a versioned plugin API — when you bump those on a refresh, your service can break and it's on you to track it. - -### Reference notation - -Citations are written **`repo::::`**: open `` **relative to the mirror root** and search for `` (a function, class, template anchor, or config section). Symbol names survive refreshes; line numbers don't, so they are deliberately omitted. The mirror root is the **single absolute path** in this skill, substituted at install time: - -> **`__SRHT_MIRROR__`** - -(In this workspace that is the repo holding the SourceHut documentation mirror. The `just install` target rewrites the token to the real path.) - ---- - -## The integration surfaces (all config-driven, language-agnostic) - -### 1. The nav / service-switcher -The list of services in the top nav is computed from config — **every section name ending in `.sr.ht`**: - -```python -# repo::core.sr.ht/srht/app/flask.py::_network -_network = [ - s for s in config - if s.endswith(".sr.ht") and s not in ["paste.sr.ht", "pages.sr.ht"] -] -``` - -The nav template loops over `network` and links each via `get_origin()`, which just reads `[service] origin=` from config (`repo::core.sr.ht/srht/templates/nav.html::for _site in network`, `repo::core.sr.ht/srht/config.py::get_origin`). - -→ **Add a `[myservice.sr.ht]` section with `origin=` to each service's `config.ini` and your service appears in the nav of every SourceHut web UI.** (paste/pages are hardcoded out of the switcher; your service won't be.) - -### 2. The unified login session (shared cookie — this is the important one) -Services do **not** each run a separate web-login OAuth dance. There is a single shared cookie, `sr.ht.unified-login.v1`, set on the **global domain** (e.g. `.srht.bigb.es`), httponly, containing the user's profile as Fernet-encrypted JSON: - -```python -# repo::core.sr.ht/srht/app/flask.py::get_session_cookie (read on every request) -cookie = request.cookies.get("sr.ht.unified-login.v1") -user_info = json.loads(fernet.decrypt(cookie.encode()).decode()) -user = self.oauth_service.lookup_user(user_info["name"]) -# ...g.current_user = user - -# repo::core.sr.ht/srht/app/flask.py::make_response (set after login) -response.set_cookie("sr.ht.unified-login.v1", - fernet.encrypt(user_info.encode()).decode(), - domain=global_domain, httponly=True, max_age=...) -``` - -The Fernet key is the shared **`[sr.ht] network-key`** from config (`repo::core.sr.ht/srht/crypto.py::fernet`). meta.sr.ht sets this cookie when the user logs in; **every service — in any language — can read it, decrypt it with the same `network-key`, and know who the user is.** No per-service OAuth callback is needed just to render pages as the logged-in user. - -Login/logout are plain redirects to meta (`repo::core.sr.ht/srht/app/flask.py::login_url`, `::logout_url`): -``` -{meta-origin}/login?return_to={your_url} -{meta-origin}/logout?return_to={your_url} -``` - -> Fernet = AES-128-CBC + HMAC-SHA256, base64url, with a version byte + timestamp + IV (the `cryptography` library's spec). Reimplementable in any language; Go has `github.com/fernet/fernet-go` (the same library `core-go` uses). - -### 3. The shared theme -Just a compiled CSS asset. `repo::core.sr.ht/scss/` is the Bootstrap-derived theme; each service's `scss/main.scss` imports it and the Makefile compiles to a hashed `main.min..css`. To match the look, **link/serve that compiled CSS** — you do not reimplement SCSS. - -### 4. The GraphQL federation gateway (`api.sr.ht`) -The aggregator merges per-service GraphQL schemas into one endpoint, also config-driven: - -```go -// repo::api.sr.ht/main.go::main -for name := range conf { - if strings.HasSuffix(name, ".sr.ht") { - services = append(services, thistle.NewService(name, getOrigin(conf, name))) - } -} -``` - -It fetches each service's `/query` schema and runs `thistle.BuildSchema(...)` (`repo::api.sr.ht/main.go::updateSchema`). Add a config section pointing at your service's GraphQL `/query` endpoint and your types join the unified API — **no edit to api.sr.ht**, just config + a SIGHUP to reload (handled in `repo::api.sr.ht/main.go::main`). Internal service-to-service calls use HMAC auth (`repo::api.sr.ht/auth.go::InternalAuthTransport` on the caller side, `repo::core-go/auth/middleware.go::internalAuth` on the receiver side). - -### 5. Webhooks -React to events from other services: GraphQL-native (`repo::core-go/webhooks/queue.go::NewQueue`, plus per-service `api/webhooks/`) or legacy HTTP (`repo::core.sr.ht/srht/webhook/`). Smaller services (`paste`, `man`, `pages`) may not emit them. - +description: How to write your OWN service for a self-hosted SourceHut instance — one that appears in the shared nav, reuses the unified login session and theme, accepts the instance's tokens, joins the federated GraphQL endpoint and exposes MCP, WITHOUT modifying upstream SourceHut source. Covers the family of nine Go services already running on this instance (sourcehut-* on sr-ht-ecore + the sr-ht-core fork), their shared libraries, the credential model, packaging and deployment; plus the language-agnostic integration contract for a foreign service. Use when the user asks about building a custom/third-party SourceHut service, extending SourceHut, adding a service to their instance, or making something "show up like paste/todo/git" in the web UI. +license: BSD-3-Clause +metadata: + audience: developers + workflow: service-development + tags: [sourcehut, sr.ht, srht.bigb.es, self-hosted, go, graphql, mcp, federation] + version: 2.0.0 + author: bigbes --- -## The two halves of a service +# Writing a service for a self-hosted SourceHut -| Half | What it is | Go support | Python support | -| --- | --- | --- | --- | -| **API / backend** | GraphQL API, DB, jobs, federation | **First-class** (`core-go` + gqlgen + thistle). The blessed path. | Yes (`core.sr.ht` GraphQL helpers) | -| **Web chrome** | nav switcher, login session, theme, rendered pages | **No shared code** — reimplement (small; see below) | **Free** via `srht.app.Flask` | +## Start here: which question is this? -The web-chrome code (shared Jinja `layout.html`/`nav.html` + `repo::core.sr.ht/srht/app/flask.py::Flask`) lives **only in `core.sr.ht` (Python)**. `core-go` is purely API-side (`auth`, `config`, `database`, `redis`, `server`, `webhooks`, `crypto`, `objects`) — no HTML templating. (It does have API-side cookie auth in `repo::core-go/auth/middleware.go::cookieAuth`, but no rendered web UI / Jinja chrome.) The Go-only services (`api.sr.ht`, `pages.sr.ht`, `sourcehut-ssh`) render **no** integrated web UI; `pages.sr.ht` has zero `.html` files. - ---- - -## Recipe A — Python web tier (the standard pattern, chrome for free) - -This is how every user-facing service is built. Subclass `srht.app.Flask` and you inherit nav + unified login + theme + GraphQL blueprint + error pages. paste's entire bootstrap is ~35 lines (`repo::paste.sr.ht/pastesrht/app.py::PasteApp`): - -```python -from srht.app import Flask -from srht.config import cfg -from srht.database import DbSession - -db = DbSession(cfg("myservice.sr.ht", "connection-string")); db.init() - -class MyServiceApp(Flask): - def __init__(self): - super().__init__("myservice.sr.ht", __name__, user_class=User) - from myservicesrht.blueprints.public import public - from srht.graphql import gql_blueprint - self.register_blueprint(public) - self.register_blueprint(gql_blueprint) +| The situation | Where to go | +| --- | --- | +| **Writing a new service for this instance** (the common case) | The family path, below. You are writing **service #11** of an existing, tightly-conventional Go family — copy it, don't design it. | +| **Integrating a service that already exists**, in a language that is not Go | The foreign-service contract, near the end. It is just HTTP, a config file and a cookie format. | +| **Explaining how upstream SourceHut itself works** | `references/upstream.md` | -app = MyServiceApp() -``` +The rest of this page assumes the first. Getting this fork wrong is the most expensive mistake available here: the family path is nine worked examples and a shared library, and re-deriving any of it from upstream source produces a service that looks right and is subtly alone. -Templates `{% extends "layout.html" %}`. Build against an installed `core.sr.ht` (`pip install -e ../core.sr.ht`). Best when you want the integrated shell with minimal work. +## The mental model -## Recipe B — Pure Go (the user's preference) +**SourceHut has no plugin system.** Nothing loads third-party code into a running service. There is no registry, no entry point, no versioned extension API. Your service is a **separate process** that agrees with the others about a cookie, a config file and a set of URLs. Every mechanism in this skill is config-driven and reusable from any language. -**API side:** straightforward and blessed — `core-go` gives you `auth` (validate meta OAuth bearer tokens: `repo::core-go/auth/bearer.go::DecodeBearerToken`, `repo::core-go/auth/middleware.go::Middleware`), `config`, `database`, `redis`, gqlgen scaffolding (`repo::core-go/server/server.go::WithDefaultMiddleware`), `webhooks`, `objects` (S3). Federate into `api.sr.ht` via thistle (config only). +**This only works on an instance you control** — config, routing, DNS. You cannot add a service to hosted sr.ht. -**Web side — reimplement the chrome, which is small:** -1. **Nav**: ~30 lines of `html/template` mirroring `nav.html`, reading the same config (`network` = config sections ending `.sr.ht`). -2. **Theme**: link/serve the already-compiled `main.min.css` from `core.sr.ht`'s build output. No reimplementation. -3. **Login/identity**: read the `sr.ht.unified-login.v1` cookie, Fernet-decrypt with `[sr.ht] network-key` (`github.com/fernet/fernet-go`), get the user. For login, redirect to `{meta-origin}/login?return_to=...`; meta sets the shared cookie on the parent domain. For **write/API** actions on behalf of the user, also obtain an OAuth token (register an OAuthClient on meta, validate via `repo::core-go/auth/middleware.go::Middleware`). +**You couple to internals, not to a contract.** `core.sr.ht` and `core-go` are libraries upstream's services share *with each other*. Upstream changes them freely. That is why this instance runs forks. -That's the whole gap between Go and "looks like paste": a small nav template + a cookie-reading login handler. Everything else is `core-go`. +**There are two generations, and you are in the second one.** Upstream's family is Python web tiers plus Go APIs. This instance's family is nine pure-Go services built on a forked core and a shared library that upstream does not have. They are not the same stack and their advice does not transfer in either direction. -## Recipe C — Go API only, no web page +### The stack, in three lines -If you just need your types in the unified GraphQL endpoint (no browser page in the shell): build a Go gqlgen API, register it in `api.sr.ht`'s config, done. Fully Go. +- **`sr-ht-core`** — a fork of upstream `core-go`, module path `sourcecraft.dev/bigbes/sr-ht-core`. Upstream plus a module rename, the S3 checksum patch, and a real `errors.OnPath` data-race fix. Pinned by all nine services; there is no `replace` directive anywhere. +- **`sr-ht-ecore`** — module `sourcecraft.dev/bigbes/sr-ht-ecore`. Sixteen packages that upstream has no equivalent of: the shared **Go web chrome**, the credential validators, CSRF, assets, MCP plumbing, instance config, logging. It exists because the same code was copied into six-to-nine services and drifted into real bugs. +- **`sourcehut-federation`** — the federation gateway that actually runs, because upstream's `api.sr.ht` does not. -## Recipe D — Any other language +### Reference notation -The integration contract is just HTTP + config + a known cookie format, so any stack works if it can: -- serve under a subdomain wired in nginx; -- read `[sr.ht] network-key` and Fernet-decrypt the `sr.ht.unified-login.v1` cookie for identity (or treat all traffic as anonymous + redirect to meta for login); -- (optional) speak the gqlgen/thistle federation conventions to join `api.sr.ht`; -- link the shared `main.min.css` and replicate the ~30-line nav from config. +Citations are `::::`: open `` under the named root and search for ``. Symbols survive refactors; line numbers don't, so they are omitted. The two roots are substituted at install time by `just install`: -There is no language lock-in — `core.sr.ht`/`core-go` are conveniences, not requirements. +- **`mirror::`** → `__SRHT_MIRROR__` — the read-only documentation mirror of **upstream** SourceHut. +- **`family::`** → `__SRHT_FAMILY__` — the directory holding **this instance's** repos: the nine `sourcehut-*` services, `sr-ht-ecore`, `sr-ht-core`, `sr-ht-api`. ---- +So `family::sr-ht-ecore/chrome/chrome.go::BuildNav` is ours; `mirror::core.sr.ht/srht/app/flask.py::_network` is upstream's. -## Wiring checklist (self-hosted) +## Which reference to read -1. **Build** your service (Recipe A/B/C/D). -2. **Config** — add to the `config.ini` of **every** service (each independently builds its own nav): - ```ini - [myservice.sr.ht] - origin=https://myservice.srht.bigb.es - #api-origin=https://myservice.srht.bigb.es # if federating - connection-string=postgresql://.../myservice - ``` - Your service also needs `[sr.ht] network-key` (the shared Fernet key) and `[meta.sr.ht] origin=` to read the login cookie and link login/register. -3. **nginx** — add a `server`/subdomain route to reach your app (`repo::sr.ht-nginx/` style). -4. **api.sr.ht** (optional federation) — the matching config section + SIGHUP; types merge in. -5. **DNS** — `myservice.srht.bigb.es` under the same global domain as the rest, so the shared cookie (`domain=.srht.bigb.es`) is visible to your service. +This page routes; the references carry the material. Read the ones your task touches — they are written for an agent about to write code, not for browsing. ---- +- `references/anatomy.md` — the repo skeleton, the `main.go` wiring order, the Makefile gate set, `go.mod`, the DB and migration layer, the test conventions. **Read this first when scaffolding.** +- `references/ecore.md` — what `sr-ht-ecore` and the `sr-ht-core` fork provide, package by package, with real signatures; and the core-vs-ecore-vs-neither table for every concern. +- `references/auth.md` — the five credential planes, tokens.sr.ht, the grant vocabulary, the validation order and the status-code contract, CSRF, and a checklist per surface. **Read this before writing any handler that reads a credential.** +- `references/chrome.md` — rendering integrated pages: `pages`, `chrome.Page`, the nav, the theme and hashed assets, forms, the startup invariants. +- `references/federation.md` — how federation really works here, what a service must satisfy to join, the gqlgen setup, and the MCP surface. +- `references/caching.md` — the cache catalogue and the shape to imitate, the `cachex`/`blobx` pattern, S3 against Garage, and the HTTP cache-correctness rules. +- `references/config.md` — the corrected config field reference, which keys must match instance-wide, key generation, port allocation. +- `references/deploy.md` — apk packaging, the compose/Traefik deployment, migrations, CI, and the day-one runbook. +- `references/pitfalls.md` — the accumulated judgement: ~110 rules with their reasons, mined from the family's commit history. Cheap to read, expensive to rediscover. +- `references/upstream.md` — upstream's own mechanisms, and which two of them the family reuses. -## Configuration field reference +## The integration surfaces -SourceHut config is INI (`config.ini`, parsed by `repo::core-go/config/config.go::LoadConfig` and `repo::core.sr.ht/srht/config.py::load_config`). A custom service draws from four kinds of section. **Generate keys with `sr.ht-keygen {service,network,webhook}`** — never hand-roll them. +Six surfaces make a service feel like part of the instance. All are config-driven; none require touching upstream source. -### `[sr.ht]` — global, must match the rest of the instance -| Field | Required | Who reads it | Notes | +| # | Surface | How you get it | Detail | | --- | --- | --- | --- | -| `network-key` | **Yes** | `repo::core-go/crypto/crypto.go::InitCrypto`, `repo::core.sr.ht/srht/crypto.py::fernet` | **The single most important field.** Shared Fernet key used to decrypt the `sr.ht.unified-login.v1` cookie and sign internal messages. **Must be byte-identical across every service** or you can't read who's logged in. `sr.ht-keygen network`. | -| `service-key` | Recommended | `core.sr.ht` session layer | Encrypts this service's own session cookies. May differ per service; identical is fine if you share one config. `sr.ht-keygen service`. | -| `redis-host` | If you use Redis | `repo::core-go/server/server.go::WithDefaultMiddleware` | Cache / pubsub / webhook queue. Shared between nodes of a service. | -| `internal-ipnet` | Recommended | `repo::core-go/config/config.go::LoadConfig` | CIDRs trusted as internal (service-to-service) callers. Defaults to loopback+private ranges; set explicitly to match your cluster. | -| `owner-name`, `owner-email` | **Yes (Go)** | `repo::core-go/config/config.go::GetOwner` — **panics if missing** | Required by any `core-go` service at startup. | -| `site-name` | Recommended | nav/templates (`site_name`) | Brand text shown in the nav across all services. | -| `environment` | Recommended | `repo::core.sr.ht/srht/templates/layout.html::ENVIRONMENT` | Anything other than `production` shows a colored banner; admins always see it. | -| `site-info`, `site-blurb`, `source-url` | Optional | templates | Cosmetic / footer links. | +| 1 | **Nav / service-switcher** | A config section whose name ends in `.sr.ht`, with `origin=`. Every service builds its nav from its own config copy, so the section must be in *every* service's config — hence one shared `config.ini` on this instance. | `references/chrome.md` | +| 2 | **Unified login session** | Read the `sr.ht.unified-login.v1` cookie, Fernet-sealed with `[sr.ht] network-key`, set by meta on the parent domain. Use `family::sr-ht-ecore/login`; never hand-roll the decoder. | `references/auth.md` | +| 3 | **Theme** | One `scss/main.scss` importing the shared base; CI assembles `core.sr.ht` + Bootstrap at pinned revisions and `make css` compiles a hashed stylesheet you embed. Not vendored, not served from core.sr.ht. | `references/chrome.md` | +| 4 | **GraphQL `/query`** | Mount it on the **anonymous** router, serve `api-meta.json`, accept a meta PAT beside the working token. Then it can be namespaced and merged by `fedgw`. | `references/federation.md` | +| 5 | **MCP `/mcp`** | The family's majority surface — six of seven services expose one. Stateless transport, always. | `references/federation.md` | +| 6 | **Webhooks** | Upstream's two generations exist; **no custom service here uses them.** Note that `[webhooks] private-key` is still mandatory for an unrelated reason. | `references/upstream.md` | -### `[meta.sr.ht]` — the identity provider, always needed -| Field | Required | Notes | -| --- | --- | --- | -| `origin` | **Yes** | Used to build `login`/`logout`/register URLs (`repo::core.sr.ht/srht/app/flask.py::login_url`) and to validate tokens. Without it your service can't send users to log in. | -| `oauth-client-id`, `oauth-client-secret` | If you make API calls *as a client* | Register an OAuth client on meta and put its credentials here (pattern: `repo::git.sr.ht/config.example.ini::[builds.sr.ht]` declares the *integrated* service's id under its own section). Only needed for write/API actions on the user's behalf — **not** needed merely to read the login cookie. | +## Recipe: service #11, in Go -### `[myservice.sr.ht]` — your own section (the name MUST end in `.sr.ht`) -The `.sr.ht` suffix is what puts you in the nav `network` list (`repo::core.sr.ht/srht/app/flask.py::_network`) and the federation loop (`repo::api.sr.ht/main.go::main`). Pick a service "prefix" (the part before `.sr.ht`) that's unique on the instance. -| Field | Required | Who reads it | Notes | -| --- | --- | --- | --- | -| `origin` | **Yes** | nav, `repo::core.sr.ht/srht/config.py::get_origin`, `repo::core-go/config/config.go::GetOrigin` | External `scheme://host` your web UI is served at. Drives every cross-service link to you. | -| `connection-string` | If you have a DB | `repo::core-go/server/server.go::WithDefaultMiddleware`, `core.sr.ht` `DbSession` | Postgres DSN. | -| `internal-origin` | Optional | `GetOrigin` (internal pref.) | LAN/cluster URL preferred for service-to-service traffic; falls back to `origin`. | -| `api-origin` | If you expose an API | `repo::core-go/config/config.go::GetAPI` | External API URL. `api.sr.ht` federates `api-origin + "/query"`. | -| `api-internal-origin` | Optional | `GetAPI` (internal pref.) | Internal API URL for the federation gateway / other services. | -| `migrate-on-upgrade` | Optional | packaging | `yes` to auto-run migrations on package upgrade. | -| `webhooks` | If you emit webhooks | webhook worker | Redis URL/db for the webhook queue (e.g. `redis://localhost:6379/1`). | -| `debug-host`, `debug-port` | Dev only | debug server | Bind address for `run.py` / local dev. Pick a port not used by another service. | -| `s3-bucket`, `s3-prefix` | If you store objects | `core-go/objects` | Leave bucket empty to disable object storage. | +The short version. Every step has a reference behind it; the numbers are the order to do them in. -### `[webhooks]` — only if your service signs outgoing webhook payloads -| Field | Required | Who reads it | Notes | -| --- | --- | --- | --- | -| `private-key` | If emitting webhooks | `repo::core-go/crypto/crypto.go::InitCrypto` | base64 Ed25519 signing key, **shared across services**. `sr.ht-keygen webhook`. Distribute the public half to consumers. | -| `queue-size` | Optional | `repo::core-go/webhooks/queue.go::NewQueue` | Defaults to `config.DefaultQueueSize`. | +1. **Choose the service name first.** It is `.sr.ht`, and everything derives from it: the binary (`srht`), the migrate binary, the env prefix, the apk name, the config section, the subdomain. **The repository directory name derives nothing** — `sourcehut-curator` builds `gosrht` because the service is `go.sr.ht`. Claim the next free bind port while you are here; `5097` is next. Set it as `bind-address=0.0.0.0:` — the compiled-in default binds loopback, which inside a container is a service that starts cleanly and is reachable by nobody. +2. **Scaffold from a sibling, not from scratch.** `references/anatomy.md` gives the layout, the annotated `main.go` wiring order and the full Makefile gate set. `sourcehut-tokens` and `sourcehut-bench` are the cleanest donors. +3. **Depend on the forks.** `sourcecraft.dev/bigbes/sr-ht-core` and `sourcecraft.dev/bigbes/sr-ht-ecore`, by pseudo-version, no `replace`. +4. **Write the config section** from `references/config.md`, and add it to the instance's shared config so you appear in everyone's nav. +5. **Wire identity before anything else.** Resolve the viewer exactly once, in front of every surface, per `references/auth.md`. Decide per *surface* which credential planes it accepts — never fold a plane into a shared resolver, because a plane on the resolver is a plane on every surface. +6. **Build the web surface** with `family::sr-ht-ecore/{chrome,pages,assets,csrf,middleware,chimw}` — `references/chrome.md` has a copy-pasteable minimal example. Two middlewares are not optional: `csrf.Require` mounted **router-wide** (there is no CSRF token to fall back on, and there must never be an "is this an API path?" exemption), and `middleware.PrivateCache` on **every** router the daemon builds, not just the site one. +7. **Add `/query` and `/mcp`** if the service has an API worth federating, per `references/federation.md`. `/query` goes on the anonymous router and ships `api-meta.json` alongside it. +8. **Package and deploy** per `references/deploy.md`. Production is **Docker Compose + Traefik**, not nginx and systemd — the `contrib/` units in each repo are the plain-host fallback, not what runs. The steps are the APKBUILD, the CI manifest, the compose service and its Traefik router, the DB role, and the deployment-repo file list whose omission has broken the whole stack's deploys before. -### `[mail]` — only if your service sends email -`smtp-host`, `smtp-port`, `smtp-user`, `smtp-password`, `smtp-from`, `smtp-encryption` (`starttls`/`tls`/`insecure`), `smtp-auth` (`plain`/`none`), `error-to`, `error-from`, and `pgp-privkey`/`pgp-pubkey`/`pgp-key-id` for signing. Skip the whole section if you don't email. +### The five rules that cost the most when broken -### Reuse the names and accessors — don't invent your own +Independently surfaced by several passes over the family's history. Each has its full reasoning in `references/pitfalls.md`. -Two reasons to mirror upstream's config vocabulary instead of designing fresh keys: (1) you can read config with the **shared accessor helpers**, which already encode SourceHut's resolution rules; (2) anyone who knows SourceHut configs (or runs one shared `config.ini`) can configure your service without surprises. +- **Import `sr-ht-ecore`; do not reimplement the chrome, the cookie decoder, the CSRF check or the asset handler.** The cost of ignoring this is not style: the hand-rolled generation produced six cookie decoders, two of which validated the username not at all. +- **Do not call `core-go`'s `WithDefaultMiddleware`.** It opens a second, uncapped Postgres pool against your own database in an unreachable field, plus an email queue and a Redis client you do not want — and `log.Fatal`s on a Redis URL you never meant to configure. Build the router with `AnonRouter().Group(...)` instead. Related: `server.New` **freezes** the router, so every later mount is a `Group`, and a bare `r.Use` panics. +- **`[webhooks] private-key` is mandatory even with zero webhooks.** `InitCrypto` panics without it, *and* it derives the instance's bearer HMAC key — rotating it silently invalidates every working token on the instance. +- **The auth error contract has four arms**: cookie failure → anonymous (never an error), bad bearer → 401, valid credential without the grant → 403, **backend unreachable → 503, never 401**. Answering 401 for an outage locks every credential on the instance out for its duration. Check order is itself the security property: visibility (404) → identity (401) → ownership (403). +- **`make css` before `go build`, and never `sed` the APKBUILD.** The stylesheet is `go:embed`ed, so a later `make css` restyles nothing; editing a tracked file dirties `git status`, which is what Go reads for `vcs.modified`, stamping every binary in that abuild run `+dirty`. -**Read config through the shared helpers, not raw INI parsing** — they give you internal/external origin fallback, `api-origin` defaulting, S3 resolution, and the internal-IP trust check for free: +Two more, both cheap to state and silent when violated: a `.build.yml` over 16 KiB makes a **push create no build at all**; and an `api-meta.json` whose scope list marshals to `null` instead of `[]` returns a 500 on meta's `/oauth2/personal-token` page **for the whole instance**, not just for your service. -| Concept | Go — `repo::core-go/config/config.go::` | Python — `repo::core.sr.ht/srht/config.py::` | -| --- | --- | --- | -| Load file | `LoadConfig` | `load_config` | -| String / int / bool | `GetString` / `GetInt` / `GetBool` | `cfg` / `cfgi` / `cfgb` | -| A service's web URL | `GetOrigin` | `get_origin` | -| A service's API URL | `GetAPI` | `get_api` | -| Owner name/email | `GetOwner` | — | -| Internal-caller check | `IsInternalIP` | — | -| Global domain | — | `get_global_domain` | -| S3 upstream | — | `get_s3_upstream` | +## Recipe: a foreign service, any language -**Use the canonical key names for your `[myservice.sr.ht]` section** — every other service uses these exact spellings, so reuse them rather than coining synonyms: +The integration contract is HTTP, a config file and a known cookie format. Nothing is Go-specific. A service in any stack integrates if it can: -| Use this (canonical) | Not these (invented) | Why | -| --- | --- | --- | -| `origin` | `url`, `base-url`, `web-url` | `GetOrigin`/`get_origin` look up exactly `origin` (+ `internal-origin`). | -| `internal-origin` | `lan-url`, `private-origin` | The internal-preferred fallback partner of `origin`. | -| `api-origin` / `api-internal-origin` | `graphql-url`, `api-url` | `GetAPI`/`get_api` probe these names in a fixed order. | -| `connection-string` | `db`, `dsn`, `database-url`, `pg` | `repo::core-go/server/server.go::WithDefaultMiddleware` and `DbSession` read `connection-string`. | -| `redis-host` | `redis`, `redis-url` | Read from `[sr.ht]`, not your section (`repo::core-go/server/server.go::WithDefaultMiddleware`). | -| `migrate-on-upgrade` | `auto-migrate` | Packaging convention. | -| `webhooks` | `webhook-redis`, `hooks-db` | Redis URL for the webhook queue. | -| `debug-host` / `debug-port` | `host` / `port`, `bind` | Dev-server bind convention. | -| `s3-bucket` / `s3-prefix` | `bucket` / `object-store` | `core-go/objects` + `get_s3_upstream` expect these. | +- serve under a subdomain of the instance's global domain, so the parent-domain cookie is visible to it; +- read `[sr.ht] network-key` and Fernet-decrypt `sr.ht.unified-login.v1` for identity — or treat all traffic as anonymous and redirect to meta for login; +- link the instance's compiled stylesheet and reproduce the nav, which is a loop over the config sections ending in `.sr.ht` (`mirror::core.sr.ht/srht/templates/nav.html::for _site in network`); +- optionally serve a GraphQL `/query` plus `api-meta.json` to be federated. -**Reuse the global keys in place — don't re-declare them.** `network-key`, `service-key`, `redis-host`, `internal-ipnet`, `owner-*` live in `[sr.ht]` and are read there by the helpers. Read them via `cfg("sr.ht", "network-key")` / `config.Get("sr.ht", ...)`; do **not** add a private copy under your own section. +Fernet is implementable anywhere (AES-128-CBC + HMAC-SHA256, base64url, version byte + timestamp + IV). Python is the one non-Go stack with a shortcut: subclassing `mirror::paste.sr.ht/pastesrht/app.py::PasteApp`'s base gives nav, login and theme for free — at the cost of being the only Python service on an instance whose packaging and deployment are built for Go binaries. -**Reference other services by their canonical section name.** To talk to meta/git/etc., call `get_origin("meta.sr.ht")` / `GetAPI(conf, "git.sr.ht", false)` — you integrate by *reading* their well-known sections, never by hardcoding URLs. This is also why your section name must be the real `myservice.sr.ht` string everywhere: other tools (and `api.sr.ht`) find you by that exact key. +## Caveats to state every time -### Two rules that bite people -- **Shared-vs-per-service.** `network-key` and `[webhooks] private-key` **must be identical everywhere**; `service-key` and `redis-host` may differ per service. Getting `network-key` wrong = "logged out" on your service even though the user is logged in. -- **Every service reads its OWN copy of config to build the nav.** Adding `[myservice.sr.ht]` to *your* config alone is not enough — each existing service must also have an `[myservice.sr.ht] origin=` entry, or you won't appear in *their* nav. The simplest operationally-correct setup is one shared `config.ini` distributed to all services (which is why upstream notes you *may* use one `service-key` for all). +- **Self-hosted only.** Requires control of config, routing and DNS. +- **No stable plugin API.** You couple to `core.sr.ht`/`core-go` internals through forks. An upstream refresh can break you, and tracking it is your job. +- **The theme is pinned by CI, not vendored.** Your `CORE_VER`/`BOOTSTRAP_REV` must track the deployment's `SRHT_CORE_VER`, or your service silently drifts away from the instance's look. Bumping the shared chrome is a CI-variable change in every service at once. +- **Nothing auto-propagates.** After a config template edit, every consumer must be re-rendered and restarted deliberately. +- **`api.sr.ht` federation does not work** — not misconfigured, dead. See `references/federation.md` before promising anyone a unified endpoint. +- **`sourcehut-epaste` is work in progress**, not a tenth precedent; it has no `go.mod` yet. Don't copy from it. -## Caveats (state these every time) +## The family, for orientation -- **Self-hosted only.** Requires control of config + nginx + DNS. Not possible on hosted sr.ht. -- **No stable plugin API.** You couple to `core.sr.ht`/`core-go` internals; refreshes can break you. -- **Cookie ⇒ identity, not authorization.** The unified-login cookie tells you *who* is browsing; for API writes you still need a real OAuth token scoped via meta.sr.ht. -- **`dispatch.sr.ht` is gone** — the old "third-party integrations" service was removed upstream. (The `dispatch` package in `sourcehut-ssh` is unrelated SSH shell dispatch.) +Nine services, all Go, all on the two shared libraries: -## Key files to re-read when working on this +`sourcehut-artifacts` (package/blob hosting), `sourcehut-bench` (benchmark results), `sourcehut-compare` (diff/compare), `sourcehut-coverage` (coverage reports), `sourcehut-curator` (`go.sr.ht`, Go module proxy), `sourcehut-dolt` (Dolt hosting), `sourcehut-federation` (`fedgw`), `sourcehut-specs` (specs/RFCs), `sourcehut-tokens` (the instance's token authority). -- `repo::core.sr.ht/srht/app/flask.py` — `::_network` (nav list), `::get_session_cookie` (unified-login read) + `::make_response` (write), `::login_url` / `::logout_url`, `::Flask` (the base class) -- `repo::core.sr.ht/srht/templates/nav.html`, `repo::core.sr.ht/srht/templates/layout.html` — the shared chrome -- `repo::core.sr.ht/srht/config.py::get_origin` / `::get_api`; `repo::core.sr.ht/srht/crypto.py::fernet` — Fernet key -- `repo::core-go/config/config.go` — `::LoadConfig` (incl. `internal-ipnet`), `::GetOwner` (panics), `::GetOrigin`, `::GetAPI`; `repo::core-go/crypto/crypto.go::InitCrypto` (webhook key + `network-key`); `repo::core-go/server/server.go::WithDefaultMiddleware` (`connection-string` + `redis-host`); `repo::core-go/auth/middleware.go::Middleware` / `::cookieAuth` / `::internalAuth` -- `*/config.example.ini` — real field names per service (`paste.sr.ht` minimal, `git.sr.ht` rich: S3, dispatch, optional `repo::git.sr.ht/config.example.ini::[builds.sr.ht]` integration block) -- `repo::paste.sr.ht/pastesrht/app.py::PasteApp` — minimal Python service bootstrap -- `repo::api.sr.ht/main.go::main` + `::updateSchema` — federation gateway (config-driven, thistle) +When you need a donor: **tokens** for the auth and service shape, **bench** for the Makefile gates and CI, **coverage** or **curator** for a read-heavy web service, **artifacts** for object storage. Check `references/pitfalls.md` before copying anything wholesale — donor prose is unverified, and a comment next to code is a claim, not documentation. diff --git a/skills/sourcehut-custom-service/references/anatomy.md b/skills/sourcehut-custom-service/references/anatomy.md new file mode 100644 index 0000000000000000000000000000000000000000..e4a6da8cf54800bb57f127d7c7b19e47ef53b06e --- /dev/null +++ b/skills/sourcehut-custom-service/references/anatomy.md @@ -0,0 +1,513 @@ +# Anatomy of a service in this family + +> Citations: `family::/::` — this instance's own repos; `mirror::::` +> — the upstream documentation mirror. Both roots are substituted at install time. Symbols, never +> line numbers. + +The canonical skeleton, derived from the nine Go services already on the instance (`artifacts`, +`bench`, `compare`, `coverage`, `curator`, `dolt`, `federation`, `specs`, `tokens`). Where they +disagree, this page states the correct variant and names the service to copy from. + +**Copy `tokens`** — the newest full build and the only one carrying the complete Makefile gate set, +a written `config.example.ini` and a `README.md` that explains the layout. Consult `bench` for the +GraphQL/MCP surfaces and the background goroutines it adds. + +--- + +## 1. Names, and the trap + +Five names derive from **one** string — the SourceHut service name `.sr.ht` — and **none** of +them derives from the repository directory name. + +| Thing | Rule | Example (tokens) | Example (curator) | +|---|---|---|---| +| Service name | `.sr.ht` | `tokens.sr.ht` | `go.sr.ht` | +| Config section | `[.sr.ht]` — byte-identical, the `.sr.ht` suffix is what puts you in the shared nav | `[tokens.sr.ht]` | `[go.sr.ht]` | +| Daemon binary | strip the dots: `srht` | `tokensrht` | `gosrht` | +| Migrate binary | `srht-migrate` | `tokensrht-migrate` | `gosrht-migrate` | +| Test env var | `SRHT_TEST_PG`, uppercased | `TOKENSSRHT_TEST_PG` | `GOSRHT_TEST_PG` | +| apk package | the service name verbatim | `pkgname=tokens.sr.ht` | `pkgname=go.sr.ht` | +| Go module | `sourcecraft.dev/bigbes/sr-ht-` | `sr-ht-tokens` | `sr-ht-curator` | +| Repo directory | `sourcehut-` — **derives nothing** | `sourcehut-tokens` | `sourcehut-curator` | + +The trap is real and already bit: `sourcehut-coverage` is module `sr-ht-cover`, service `cov.sr.ht`, +binary `coversrht`, env `COVERSRHT_TEST_PG`. `sourcehut-specs` is `sr-ht-spec` / `spec.sr.ht` / +`specsrht`. `sourcehut-compare` is `diff.sr.ht` but its binary is `comparesrht`, the one place the +strip-the-dots rule is broken — do not copy that. **Pick the service name first, write it once as a +constant, derive everything else from it.** + +Spell the name exactly once in Go, in `core/`: + +```go +// family::sourcehut-tokens/core/servicename.go::ServiceName +const ServiceName = "tokens.sr.ht" +``` + +`service.ConfigSection` aliases it; `cmd/*/main.go`, the migrate binary and every log line read it +from there. + +**Bind port.** Sequentially allocated across the instance and cross-referenced in each +`config.example.ini`: 5090 diff, 5091 spec, 5092 bench, 5093 cov, 5094 tokens, 5095 artifacts, +5096 go. **Service #11 takes 5097.** The key is spelled `bind-address` and nothing else — `bind`, +`listen`, `addr` appear nowhere in the family, and the SPEC that once said `bind` was corrected +rather than the daemon made to accept both (`family::sourcehut-tokens/config.example.ini`). A +service needing several listeners names each `-listen` instead, as dolt does +(`remotesapi-listen`, `credsapi-listen`). + +--- + +## 2. Directory layout + +Minimal set for a service with a database and a web UI: + +``` +cmd/srht/ cmd/srht-migrate/ +core/ db/ authn/ service/ api/ web/ +migrations/ schema.sql +scss/ Makefile config.example.ini APKBUILD .build.yml +go.mod go.sum +README.md SPEC.md AGENTS.md CLAUDE.md docs/ +``` + +| Dir | What lives there | Notes | +|---|---|---| +| `cmd/` | one directory per binary, `main.go` each | see §3 | +| `core/` | the service name, config struct, process-wide vocabulary | **imports nothing else from this tree** | +| `db/` | `database/sql` + `lib/pq`, hand-written SQL, `Store` | no ORM anywhere in the family | +| `authn/` | credential resolution — cookie plane + bearer plane, `Resolver.Middleware()` | a read-only service with no identity of its own may have `authz/` instead (compare) | +| `service/` | the composition root and all business rules | the only package allowed to touch `db/` and `authn/` | +| `api/` | REST surface, mounted under `/api` | | +| `web/` | browser UI in the shared chrome, `/healthz`, `//go:embed`-ed static assets | | +| `graph/` | gqlgen GraphQL at `/query`, read-only | add if the service has a public read model — 6/9 do | +| `mcpsrv/` | MCP surface at `/mcp` | same 6/9; bearer-only | +| `migrations/` | brant-format `NNNN_name.sql` | §6 | +| `scss/` | `main.scss` importing the shared `base` partial | §4 | +| `docs/` | `DESIGN.md`, `ANALYSIS.md`, `ci.md` — rationale prose | §8 | +| `contrib/` | packaging/deploy misc | optional | + +The layering rule, stated in `family::sourcehut-tokens/CLAUDE.md`: + +> `core/` (vocabulary, no dependencies) ← `db/` and `authn/` ← `service/` ← `api/` and `web/`. +> `service/` is the only package allowed to talk to `db/` and `authn/`, and the only one `api/` and +> `web/` may talk to. The two surfaces never import each other: shared helpers are copied. + +Two naming hazards. `docs/` (prose) and a Go package named `doc/` (document parsing, in specs) are +different things — do not create the singular by habit. `internal/` has no fixed meaning here; name +packages for what they hold rather than filling a dir called `internal`. And there is **no** +`srht/` top-level package directory: that is the upstream Python+Go shape, not this family's. +A `gosrht` or `doltsrht` file sitting next to `cmd/` in a checkout is the compiled binary +(`.gitignore`d, written there by the Makefile), not a second source tree. + +--- + +## 3. `cmd/` and the `main.go` wiring order + +**Exactly two binaries** unless there is a reason for a third: the daemon and `-migrate`. There is +no worker process in this family — retention sweeps, reconcilers and mirror ticks are goroutines +inside the daemon (`svc.StartRetention(ctx, ...)`), started **before** `srv.Run()` so a daemon that +was down for a week catches up on boot instead of waiting for a schedule. + +The order below is identical in `tokensrht`, `benchsrht`, `coversrht`, `gosrht`, `doltsrht` and +`specsrht`. It is load-bearing, not stylistic; the annotations say why. + +```go +// family::sourcehut-tokens/cmd/tokensrht/main.go::main +func main() { + // Config BEFORE the logger: logging.Defaults reads [.sr.ht] log-level, and a + // verbosity that only $LOG_LEVEL can set is one nobody can write down. + conf := config.LoadConfig() // never fails; returns a nil ini.File if no config.ini + log := installLogger(conf) + + if err := run(log, conf); err != nil { + // Plain stderr, not a log record: a daemon that did not start has no log format yet. + fmt.Fprintf(os.Stderr, "%s did not start: %v\n", service.ConfigSection, err) + os.Exit(1) + } +} + +// family::sourcehut-tokens/cmd/tokensrht/main.go::installLogger +func installLogger(conf ini.File) *slog.Logger { + opts := logging.Defaults(conf, service.ConfigSection) // sr-ht-ecore: -d > $LOG_LEVEL > log-level > info + return logging.Install(scribe.NewTintHandler( // auxilia/scribe — every sibling uses this + scribe.WithWriter(os.Stderr), + scribe.WithLevel(opts.Level), scribe.WithSource(opts.AddSource), + scribe.WithTimeFormat(opts.TimeFormat), scribe.WithNoColor(!opts.Color), + // Partial mask FIRST — first matching rule wins, and ecore's blanket pattern + // would otherwise replace a value this wants a readable prefix of. + scribe.WithMaskPartial(logging.PartialMaskPattern, logging.PartialMaskKeep), + scribe.WithMaskKeys(opts.MaskKeys...), + scribe.WithMask(opts.MaskPattern, opts.MaskReplacement), + )) +} + +// family::sourcehut-tokens/cmd/tokensrht/main.go::run +func run(log *slog.Logger, conf ini.File) error { + // ONE refusal for every configuration problem: core-go's required keys and the + // service's own, merged. An operator fixing config.ini should see them all at once, + // not one per restart. + cfg, err := validateConfig(conf) + if err != nil { return err } + for _, w := range cfg.Warnings { log.Warn(w) } + + // core-go binds metrics to ":0" — every interface. Insert this daemon's loopback + // default right after argv[0] so a user-supplied -m still wins (getopt takes the last). + args := withMetricsDefault(os.Args) + binds, err := bindAddresses(args, cfg.BindAddress) // for the startup line only; core-go re-parses + if err != nil { return culpa.Wrap(err, "read the command line") } + + // Open and PROVE reachable before anything is mounted on it. Pool sizing happens + // exactly here, once; a Store never reconfigures a pool it does not own. + pool, err := openDatabase(cfg) // sql.Open + SetMaxOpenConns + PingContext(pingTimeout) + if err != nil { return err } + defer pool.Close() + + store := db.NewStore(pool) + svc, err := service.New(service.Options{Config: cfg, Store: store}) + resolver, err := newResolver(cfg, store) // cookie plane always; token plane may be nil + + rest, err := api.New(svc) + site, err := web.New(web.Options{Conf: conf, Config: cfg, Service: svc, Resolver: resolver}) + // optional surfaces: + // agents, err := mcpsrv.New(mcpsrv.Options{Service: svc, Resolver: resolver, Version: build.Version, Origin: cfg.Origin}) + // gql, err := newGraphServer(svc, resolver) + + // coreserver.New calls crypto.InitCrypto, which log.Fatalf's on a missing + // [sr.ht] network-key / [webhooks] private-key. Nothing above this line may seal or + // verify anything — which is why validateConfig runs first, to refuse with a sentence. + srv := coreserver.New(service.ConfigSection, cfg.BindAddress, conf, args) // full os.Args, not a slice + mountRoutes(srv.AnonRouter(), conf, pool, resolver, rest, site) + + ctx, stop := context.WithCancel(context.Background()) + defer stop() + retention := svc.StartRetention(ctx, service.DefaultRetentionInterval, log) // BEFORE Run + + bridgeSIGTERM(log) + log.Info(service.ConfigSection+" starting", "version", build.Version, + "bind", binds, "origin", cfg.Origin) // report the LISTENERS, not the configured value + + srv.Run() // blocks until SIGINT, then drains the listeners + + stop() + select { + case <-retention: + case <-time.After(shutdownGrace): // 30s + log.Warn("background work did not stop in time; exiting anyway", "grace", shutdownGrace.String()) + } + log.Info(service.ConfigSection + " stopped") + return nil +} +``` + +### Traps in that sequence + +**`server.New` freezes the anonymous router.** A bare `router.Use(...)` after it panics. Every mount +goes inside `router.Group(func(r chi.Router){...})`, and the groups exist precisely so different +credential planes get different middleware chains on one routing tree +(`family::sourcehut-bench/cmd/benchsrht/main.go::mountRoutes`): + +```go +router.Group(func(r chi.Router) { // unauthenticated: liveness + assets + r.Use(chimw.RealIP) + r.Handle("/healthz", pages) // must answer even when meta.sr.ht is down + r.Handle("/static/*", pages) +}) +// Each remaining group opens with the same chain — RealIP, RequestID, requestLog(), +// Recoverer, then config.Middleware(conf, service.ConfigSection) + database.Middleware(pool) — +// and differs only in what it mounts and whether it adds the resolver: +// /mcp r.Handle(mcpPrefix, agents.Handler()) // mcpsrv gates itself, bearer only +// /query r.Handle(queryRoute, gql) // graph.Server gates itself +// r.Get(apimeta.Path, apimeta.Handler(graph.GrantScopes...)) +// web+REST r.Use(resolver.Middleware(&surfaceDenier{api: rest, web: site})) +// rest.Mount(r); r.Mount("/", site.Handler()) +``` + +Ordering inside the resolver's group is forced: `authn`'s cookie plane calls core-go's +`auth.LookupUser`, which reads config and the pool out of the request context, so the resolver must +sit downstream of `config.Middleware` and `database.Middleware`. + +**Pass the full `os.Args` to `coreserver.New`** — it runs its own getopt over `-b/-d/-m/-p` and +calls `crypto.InitCrypto`. Reading `-b` yourself for the startup log is a second, independent parse +(`family::sourcehut-tokens/cmd/tokensrht/main.go::bindAddresses` via `git.sr.ht/~sircmpwn/getopt`). + +**Inject a metrics default before that parse.** Left alone, core-go's `-m` default publishes the +metrics/pprof listener on **every** interface. The family's fix is two lines and relies on getopt +taking the last occurrence, so an explicit `-m` from the operator still wins +(`family::sourcehut-tokens/cmd/tokensrht/main.go::withMetricsDefault`): + +```go +// injects "-m localhost:0" right after argv[0] +func withMetricsDefault(args []string) []string { + out := make([]string, 0, len(args)+2) + out = append(out, args[0], "-m", metricsAddrDefault) // metricsAddrDefault = "localhost:0" + return append(out, args[1:]...) +} +``` + +Note the asymmetry this leaves: the service's own traffic binds where `bind-address` says, while +pprof is deliberately pinned to loopback. That is intentional, and `family::sourcehut-bench/cmd/benchsrht/main.go` +carries the comment explaining it plus the test that pins the last-occurrence-wins behaviour. + +**SIGTERM must be bridged.** `coreserver.Server.Run` installs a SIGINT handler only; systemd sends +SIGTERM. Lift this verbatim (`family::sourcehut-tokens/cmd/tokensrht/main.go::bridgeSIGTERM`): + +```go +func bridgeSIGTERM(log *slog.Logger) { + sig := make(chan os.Signal, 2) + signal.Notify(sig, syscall.SIGTERM, os.Interrupt) + go func() { + for s := range sig { + if s != syscall.SIGTERM { continue } + log.Info("SIGTERM received; starting the warm shutdown core-go waits for SIGINT to begin") + if err := syscall.Kill(os.Getpid(), syscall.SIGINT); err != nil { + log.Error("could not raise SIGINT for the warm shutdown", scribe.Err(err)) + } + } + }() +} +``` +The alternative — `KillSignal=SIGINT` in the unit file — is what compare does and what the rest +deliberately moved away from: a daemon that only shuts down warmly under one particular unit file is +a daemon that shuts down coldly everywhere else. + +**Doc-comment density is house style.** These `main.go` files run 400–850 lines for a structurally +150-line program; the comments *are* the design record. Match the density of the file you are +editing rather than trimming it. + +--- + +## 4. Makefile + +Copy `family::sourcehut-tokens/Makefile` whole. The full gate set below is the standard even though +only bench, coverage, curator, tokens and federation currently carry all of it — the older repos are +behind, not right. + +```make +SERVICE = .sr.ht +BIN = srht +MIGRATE_BIN = srht-migrate + +PREFIX ?= /usr/local +BINDIR ?= $(PREFIX)/bin +ASSETS ?= /usr/share/sourcehut +MIGRATIONDIR ?= $(ASSETS)/migrations/$(SERVICE) +SCHEMAFILE ?= $(ASSETS)/$(SERVICE).sql + +VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev) +VERSION_PKG = sourcecraft.dev/bigbes/sr-ht-/internal/build +GOLDFLAGS = -ldflags "-X $(VERSION_PKG).Version=$(VERSION)" + +CHECK_BIN ?= ./$(BIN) # packaging points this into $(DESTDIR) so the gates inspect the shipped file +CSS = web/static/main.min.*.css +``` + +| Target | Does | Catches | +|---|---|---| +| `all` → `build` | `css-warn` then both binaries | | +| `build` | `$(GOBUILD) $(GOLDFLAGS) -o $@ ./cmd/$(BIN)`; targets are `.PHONY` because make cannot see three packages deep | | +| `test` | `go test ./...`; DB tests skip themselves | | +| `test-pg` | same suite with `SRHT_TEST_PG` set; **errors if the var is empty** rather than passing vacuously | a green run that tested no database | +| `vet` | `go vet ./...` | | +| `fmt` / `fmt-check` | `gofmt -w .` / `out=$$(gofmt -l .); [ -z "$$out" ] \|\| exit 1` — `gofmt -l` exits 0 on findings, so the check must be written this way | unformatted code passing an `&&` chain | +| `tidy` | `go mod tidy` | | +| `check` | `fmt-check vet test` — what CI runs and what to run before pushing | | +| `css` | sassc → minify → rename to `main.min..css`, remove the unminified intermediate | stale stylesheet served from a browser cache | +| `css-warn` | soft warning if no CSS; prerequisite of `build` | a developer build silently unstyled — warns, does not refuse | +| `check-css` | hard failure if no CSS; prerequisite of `install` | a *package* shipped with no stylesheet | +| `check-version` | `go version -m $(CHECK_BIN)`; fails on a missing `vcs.revision` or `vcs.modified=true` | a binary with no record of its commit — Go's `auto` mode omits the stamp **silently** when git is not on PATH — and one built from a dirty tree (`git status --porcelain` counts untracked files) | +| `check-embedded-css` | greps `main\.min\.[0-9a-f]{8}\.css` out of the binary and requires it to equal the name on disk | the one thing `check-css` cannot see: a binary compiled *before* `make css` ran. It embeds nothing (or an old hash), links, starts, and serves every page unstyled while `check-css` is perfectly happy | +| `install` | `check-css build` then `@$(MAKE) install-files` | | +| `install-files` | the copies only — the packaging contract | | +| `init-db` / `migrate` | `./$(MIGRATE_BIN) init` / `up` | | +| `run-dev` | `./$(BIN) -b localhost:` | | +| `clean` | binaries + `web/static/main*.css` | | + +**Build order is CSS, then Go.** `//go:embed static` fixes the stylesheet at compile time, so +`make css` after a build restyles nothing. `check-embedded-css` is the gate that enforces it; write +it on day one rather than backporting it. + +Two mechanical details worth copying verbatim rather than re-deriving: + +- `install-files` is invoked through a **sub-`$(MAKE)` in a recipe line**, not listed as a third + prerequisite of `install`. abuild exports `MAKEFLAGS=-j$(nproc)`, prerequisites of one target run + in parallel under `-j`, and the copying must not start beside the build it is supposed to follow. + A recipe line always runs after prerequisites; a prerequisite does not. +- `grep -a -o` in `check-embedded-css`: `-a` is not optional (GNU grep suppresses `-o` on a file it + decides is binary), and the `tr -c '[:print:]'` spelling is ruled out — busybox `tr` on the Alpine + builders makes it a silent no-op. + +```make +install-files: + install -Dm755 $(BIN) $(DESTDIR)$(BINDIR)/$(BIN) + install -Dm755 $(MIGRATE_BIN) $(DESTDIR)$(BINDIR)/$(MIGRATE_BIN) + mkdir -p $(DESTDIR)$(MIGRATIONDIR) + install -Dm644 -t $(DESTDIR)$(MIGRATIONDIR) migrations/*.sql + install -Dm644 schema.sql $(DESTDIR)$(SCHEMAFILE) +``` + +**Codegen is not a make target.** gqlgen runs from a `//go:generate` directive in +`graph/generate.go`, invoked by hand, with the version **pinned**: +`go run github.com/99designs/gqlgen@v0.17.94 generate`. No dataloaden, no DataLoader packages — +that upstream convention is not used here. + +`scss/main.scss` is `@import "base";` plus what is unique to the service — the base chrome is not +yours to redesign. The shared partials ship in no package: `-I$(ASSETS)/scss`, materialized by +core.sr.ht's `make install` locally and by the `scss:` task of `.build.yml` in CI. + +--- + +## 5. `go.mod` + +Go **1.26.4**. Module path `sourcecraft.dev/bigbes/sr-ht-`. + +``` +git.sr.ht/~bitfehler/brant v0.5.1 migrations +git.sr.ht/~sircmpwn/getopt v1.0.0 core-go's own flag spec, re-parsed for the startup log +github.com/alexflint/go-arg v1.6.0 the -migrate binary's CLI +github.com/go-chi/chi/v5 v5.3.1 router +github.com/lib/pq v1.10.9 the only Postgres driver in the family +github.com/stretchr/testify v1.11.1 require/assert +github.com/vaughan0/go-ini ... the ini.File core-go's LoadConfig returns +go.bigb.es/auxilia v0.7.0 culpa (error wrapping) + scribe (slog handler) +sourcecraft.dev/bigbes/sr-ht-core ... the fork of upstream core-go: config/database/server/webhooks/crypto +sourcecraft.dev/bigbes/sr-ht-ecore ... the extension layer: logging/apimeta/bearer/chimw/instconf +``` + +Add per surface: `github.com/99designs/gqlgen v0.17.94` + `github.com/vektah/gqlparser/v2 v2.5.36` +for `graph/`; `github.com/modelcontextprotocol/go-sdk` for `mcpsrv/`; +`github.com/fernet/fernet-go` if decoding the unified-login cookie directly. + +`sr-ht-core` and `sr-ht-ecore` are **forks** on sourcecraft, not the upstream `git.sr.ht/~sircmpwn/core-go` +paths — they carry the instance's patches. Every daemon depends on both. + +**No `replace` directives.** One exists in the whole family (dolt, pinning a vendored pure-Go zstd +shim under `third_party/`) and it points at a vendored subtree, never at a sibling checkout. Do not +add a `replace` to a local `core-go`/`sr-ht-core` clone. + +`go.bigb.es/auxilia` is a standing preference: `culpa.Wrap(err, "open the database")` rather than +`fmt.Errorf`, `scribe.NewTintHandler` for the root handler, `scribe.Err(err)` for error attributes. + +--- + +## 6. Database + +Plain `database/sql` + `lib/pq`, hand-written SQL under `db/`. **No ORM, no query builder, no +sqlc.** The `Store` wraps one handle and nothing else +(`family::sourcehut-tokens/db/store.go::NewStore`): + +```go +type Querier interface{ /* Query/QueryRow/Exec, satisfied by *sql.DB and *sql.Tx */ } +type Store struct{ q Querier } + +func NewStore(q Querier) *Store // production passes the shared pool; tests a scratch one +func FromContext(ctx context.Context) *Store // over the *sql.DB core-go's database.Middleware installed +func (s *Store) WithTx(tx *sql.Tx) *Store // same queries, inside a caller's transaction +func (s *Store) InTx(ctx, fn) error // rolls back on error (unwrapped, so errors.Is still works) + // and re-panics after rolling back on a panic +``` + +A `Store` **never** reconfigures the pool: `SetMaxOpenConns` happens once, in the daemon's +`openDatabase`, before the pool is handed to anything. + +**Migrations** are brant-format `migrations/NNNN_name.sql`, `-- +brant Up` / `-- +brant Down` +markers, bookkeeping in `brant_db_version`. `schema.sql` at the repo root is a full current snapshot +for fresh installs. + +``` +srht-migrate init apply schema.sql wholesale, stamp bookkeeping to head (fresh install) +srht-migrate up replay pending migrations one at a time (existing database) +srht-migrate down roll the last one back +srht-migrate status what has been applied +srht-migrate -a the package-upgrade entry point: honours [.sr.ht] migrate-on-upgrade + and exits quietly when the instance has turned it off +``` + +**`init` and `up` are alternatives, not a sequence.** `init` declares every migration already +applied; running `up` afterwards is a deliberate no-op. + +The binary is brant's CLI (`git.sr.ht/~bitfehler/brant` + `brant/cli`) with three additions: the +connection string comes from the shared `config.ini` instead of a flag, the migration directory is +discovered (`./migrations`, else `/usr/share/sourcehut/migrations/.sr.ht`), and `init` exists. +Copy `family::sourcehut-tokens/cmd/tokensrht-migrate/main.go`. + +`-a` is only useful if something calls it. Wire it into packaging the way curator does — +`install="$pkgname.post-upgrade"` in the APKBUILD, backed by a shell script running +`srht-migrate -a` — or the config key is inert. + +--- + +## 7. Tests + +- `_test.go` next to the code it tests. Package-**internal** (`package db`, not `package db_test`) + in the DB layer, where fixtures need unexported helpers. +- testify throughout: `require` for fatal checks, `assert` for the rest. +- **No docker, no testcontainers, no dockertest** — zero uses in the family. +- No top-level `integration/`. A genuinely heavy suite goes behind a build tag in its own dir, as + artifacts' `//go:build e2e` suite does. + +Postgres comes from one env var naming a DSN whose role may `CREATE SCHEMA`. Every test that needs +it creates a throwaway schema, applies `schema.sql` into it, and drops it in `t.Cleanup`; when the +var is unset the test **skips cleanly** so `go test ./...` is green and meaningful on a machine with +no Postgres (`family::sourcehut-tokens/db/db_test.go::newSchemaNamed`): + +```go +const testEnv = "TOKENSSRHT_TEST_PG" +const testSchemaPrefix = "tokenssrht_test_" // a leftover schema names the package that made it + +base := os.Getenv(testEnv) +if base == "" { + t.Skipf("%s not set; skipping Postgres-backed test (set it to a DSN to run)", testEnv) +} +``` + +Isolation is a **schema**, not a database: no `CREATE DATABASE` privilege is needed and cleanup is +one `DROP SCHEMA ... CASCADE`. The scratch pool pins connections to it with lib/pq's +`options=-c search_path=` startup parameter, which is what lets unqualified table names in +`schema.sql` and in the package's queries resolve. + +Copy these two for #11: + +- `newTestStore(t)` / `newSchema(t, applySchema bool)` from tokens' `db_test.go`. +- **`TestSchemaAndMigrationsAgree`** — builds one scratch schema from `schema.sql` and another by + replaying `migrations/*.sql`, then compares the catalogue. It is the load-bearing test that keeps + the fresh-install and the upgrade path from silently diverging, and it needs a real Postgres, so + `make test` alone does not exercise it. Present in every DB-backed service under a slightly + different name; tokens' spelling is the clearest. + +--- + +## 8. Documentation set + +| File | Lang | Role | +|---|---|---| +| `README.md` | en | Short, human-facing: what it does, how to boot it locally (`config.example.ini` → `make init-db` → `make run-dev`), a metadata table (service name, binaries, apk package, config section, port). Points at `SPEC.md` for normative behaviour rather than duplicating it. | +| `SPEC.md` | **ru** | The normative document, RFC-2119 keywords declared at the top (должен / не должен / следует). Author-owned — never translate it, never rewrite it as a side effect of code changes. Some services carry a large `docs/DESIGN.md` in this role instead; either name is accepted, one must exist. | +| `docs/DESIGN.md`, `docs/ANALYSIS.md` | ru | Decision log and precedent research behind the SPEC, referenced from its header. Split when large (`DESIGN.mcp.md`, `DESIGN.views.md` in dolt). | +| `docs/ci.md` | en | `.build.yml` notes — see the `sourcehut-ci` skill. | +| `AGENTS.md` | — | beads-integration boilerplate, byte-identical across repos. | +| `CLAUDE.md` | en | The same boilerplate **plus** the project-specific tail: an Architecture Overview stating the layering rule, and a Conventions & Patterns section. The one a fresh session actually reads. | +| `config.example.ini` | en | Write it even when SPEC.md covers the same ground — it is the only file an operator copies. | + +`config.example.ini`'s own structure is a convention: in production the `[.sr.ht]` section is +merged into the **one** shared instance `config.ini` that every `*.sr.ht` service reads, so +everything outside that section is a read-only reference copy of keys another service owns, present +only so a standalone dev config boots. Each tuning key carries a *measured* derivation in its +comment (concurrency × heap-per-upload arithmetic, and so on) rather than a restatement of its name. + +--- + +## 9. Scaffolding order for service #11 + +1. Pick `.sr.ht` and port 5097. Write `core/servicename.go`; derive binary, module, apk and env + names from it mechanically (§1). +2. `go.mod` with the full dep set from §5 in **one** commit, plus a smoke-import build to populate + `go.sum`, so later parallel work never touches `go.mod`. +3. `core/` (config struct, `LoadConfig`, `MissingKeys`), then `db/` (`Store`, `schema.sql`, + `migrations/0001_initial.sql`, the two test helpers and `TestSchemaAndMigrationsAgree`). +4. `service/`, `authn/`, then the surfaces `api/` and `web/` — disjoint, parallelizable. +5. `cmd/srht-migrate/` (copy tokens'), then `cmd/srht/` in the §3 order. +6. `Makefile` with the **full** gate set from §4 — `check-version` and `check-embedded-css` + included, from day one. +7. `scss/main.scss`, `config.example.ini`, `APKBUILD` (with the `post-upgrade` hook), `.build.yml`. +8. `README.md`, `SPEC.md`, `CLAUDE.md` with its Architecture Overview tail, `docs/`. +9. Add `[.sr.ht] origin=` to **every** service's config.ini, not just this one — each service + builds its own nav from its own copy. diff --git a/skills/sourcehut-custom-service/references/auth.md b/skills/sourcehut-custom-service/references/auth.md new file mode 100644 index 0000000000000000000000000000000000000000..5a8a05079e5164a58bd5fa3246305b4dcfb9cb8d --- /dev/null +++ b/skills/sourcehut-custom-service/references/auth.md @@ -0,0 +1,567 @@ +# Authentication and authorization on the instance + +The old model — "the cookie gives identity, an OAuth token from meta is what you need for +writes" — is one third of the truth and the least often applicable third. There are **five +credential planes**, they are validated by **shared packages in `sr-ht-ecore`** rather than +by per-service code, and the credential a machine caller should hold is a **tokens.sr.ht +working token**: an offline-verifiable signed bearer token in a per-action grant vocabulary. +A meta PAT survives only where GraphQL federation forces it. + +Two citation prefixes, substituted at install time: `family::` is the sibling repos +(`sr-ht-ecore`, `sr-ht-core`, `sourcehut-`), `mirror::` is the upstream +documentation mirror. Symbols, never line numbers. + +--- + +## 1. The credential taxonomy + +Two of these five are the same wire format sealed with the same key and are told apart by +exactly one field. + +| Plane | Issuer | Carrier | Validator | Proves | Does **not** prove | +|---|---|---|---|---|---| +| **Unified-login cookie** `sr.ht.unified-login.v1` | meta.sr.ht, on the parent domain | `Cookie:` — fernet over JSON `{"name":""}` (`family::sr-ht-core/auth/middleware.go::AuthCookie`) | `family::sr-ht-ecore/login/login.go::Username` | a browser session for that meta username | any permission, any scope, that the account has a local row, and **that the request was intentional** (§8) | +| **meta.sr.ht PAT** | meta `/oauth2/personal-token` (`mirror::meta.sr.ht/metasrht/blueprints/oauth2.py::personal_token_POST`) | `Authorization: Bearer `; HTTP Basic password on dolt's remote protocol | `family::sr-ht-ecore/metapat/metapat.go::(*Validator).Resolve` | the named meta account, that meta has not revoked it, its OAuth scopes | which daemon minted it (shared signing key — only `ClientID` says); and **empty grants means universal**, so "no grants" ≠ "no permission" | +| **tokens.sr.ht working token** | `tokensrht` UI mint, or `POST /api/v1/exchange` | `Authorization: Bearer `; HTTP Basic password on artifacts' OCI registry (`family::sourcehut-artifacts/api/router.go::(Authorizer).BasicPrincipal`) | `family::sr-ht-ecore/bearer/bearer.go::(*Validator).Validate` / `::Inspect` | the named meta username, the granted actions, the expiry, and — if **registered** — that it is not revoked | which daemon minted it by signature alone; the payload is **signed, not encrypted**, so the holder reads username/grants/expiry straight out of it | +| **tokens.sr.ht parent token** `tsrht_…` | `tokensrht` UI, or `POST /api/v1/tokens` kind `parent` | `Authorization: Bearer tsrht_<43>` | `family::sourcehut-tokens/authn/parent.go::(*ParentBackend).Verify` — **inside tokensrht only** | the right to call `POST /api/v1/exchange` | anything to any other service. **No service on the instance accepts one.** It cannot mint, revoke, or read a page | +| **Internal auth** | any holder of `[sr.ht] network-key` — `family::sr-ht-ecore/internalauth/internalauth.go::Authorization` / `::AuthorizationAs` | `Authorization: Internal ` | `family::sr-ht-ecore/internalauth/internalauth.go::Identify` | the caller holds the network key **and** its `RemoteAddr` is inside `[sr.ht]internal-ipnet` | *which* service, unless `Guard(clientID, nodeID, …)` pins it | + +**Service-local planes** sit *beside* these, never instead of them. dolt adds the dolt CLI's +own EdDSA JWT (`family::sourcehut-dolt/authn/jwt.go::ResolveDoltJWT`) because the `dolt` +client speaks `dolt login` and nothing else. Add one only when a foreign client protocol +forces it. + +Wire format of the two Bearer planes: `family::sr-ht-core/auth/bearer.go::BearerToken` = +BARE(`Version, Expires, Grants, ClientID, Username`) ‖ HMAC-SHA256, base64 raw-std. The key +is `bearerKey = HMAC-SHA256([webhooks]private-key, "sr.ht HMAC key")`, derived in +`family::sr-ht-core/crypto/crypto.go::InitCrypto`. The cookie's fernet key is +`[sr.ht] network-key`, from the same function. + +--- + +## 2. Which credential for which job + +This is the decision you actually have to make. + +| Job | Credential | Why not the others | +|---|---|---| +| A CI job, an agent, a script uploading to your service | **working token** (`bearer.Validate`) | Per-action grants, short-lived, revocable, verified with one local HMAC. A PAT is instance-wide, long-lived, and its vocabulary does not contain your service's actions. | +| A long-lived agent that mints its own short credentials | **parent token** held by the agent, exchanged for a working token per run | The parent never leaves the agent; only sub-24h stateless tokens travel. | +| Your `/query` GraphQL endpoint | **working token *and* meta PAT** | `api.sr.ht`'s `AuthMiddleware` copies **one** client `Authorization` header to every service a federated query touches (`mirror::api.sr.ht/auth.go::AuthMiddleware`). A `/query` that refuses PATs can never be federated. | +| Your REST / MCP surface | **working token only** | Not federated, so nothing forces the PAT, and a narrow revocable grant is worth its cost. bench exemplifies: its `/mcp` sits in a route group with no resolver, so a cookie cannot structurally reach it (`family::sourcehut-bench/cmd/benchsrht/main.go`). | +| Rendering a page as the logged-in viewer | **cookie** (`login.Optional` / `login.Required`) | Nothing else is present in a browser request. | +| A **mutation** from a browser form | **cookie + same-origin check** — this pair *is* the auth (§9) | There is no CSRF token on this instance and nowhere to keep one. The cookie alone does not prove the request was intentional. | +| Calling another service on the user's behalf | **internal auth minted as that user** (`internalauth.AuthorizationAs`) | No service in this family holds a user's OAuth token, and there is no user-OAuth flow. | +| Accepting a call from a sibling service | **`internalauth.Guard(clientID, nodeID, deny)`** | Pin both ids; the source-IP half is much weaker than it looks. | + +--- + +## 3. tokens.sr.ht + +### What it solves that meta does not + +Three services had each grown a byte-identical Mint/Verify and three `/tokens` pages — and +**none of them had permissions**: the token equalled the owner's full write access. A meta PAT +meanwhile is instance-wide, long-lived, and scoped in a vocabulary (`svc.sr.ht/SCOPE:RW`) in +which custom services do not appear at all. tokens.sr.ht gives one issuing point, per-action +grants, exchange-based narrowing, short-lived credentials that need no revocation, and a +revocation check that is **off the hot path**. + +### Endpoints + +`family::sourcehut-tokens/api/api.go::(*Handler).routes`: + +```go +h.mux.HandleFunc("POST /v1/tokens", h.cookieOnly(h.handleMint)) +h.mux.HandleFunc("DELETE /v1/tokens/{id}", h.cookieOnly(h.handleRevoke)) +h.mux.HandleFunc("POST /v1/exchange", h.parentOnly(h.handleExchange)) +h.mux.Handle("GET /v1/revocations/{id}", authn.InternalGuard(http.HandlerFunc(h.handleRevocation))) +``` + +The two gates are mutually exclusive on purpose. `cookieOnly` refuses a parent token with an +explicit **403**, not as "not the owner"; `parentOnly` refuses a cookie with 403, because a +browser that could exchange would let any page the viewer visits mint working tokens with +their cookie. + +### Exchange (the agent flow) + +``` +curl -X POST https://tokens.srht.bigb.es/api/v1/exchange \ + -H "Authorization: Bearer tsrht_..." \ + -d '{"grants": "bench:upload", "ttl": "1h"}' +``` + +`family::sourcehut-tokens/service/tokens.go::(*Service).Exchange`: + +- the parent is authenticated by hash lookup (stored as SHA-256, compared with + `subtle.ConstantTimeCompare` — `family::sourcehut-tokens/db/token.go::TokenMatches`); +- omitting `grants` **inherits** the parent's; supplying them requires + `asked.IsSubsetOf(held)` or **403**. This is the **narrowing rule**: an exchange only shrinks; +- TTL is capped at `exchange-max-ttl` (default 24h), and `(*Service).checkTTL` refuses + `ttl <= 0`, so **`"ttl": "never"` is impossible on the exchange path** — only the UI mint + can issue a never-expiring token; +- an `exchange_log` row is written for **every** exchange, whether or not a token row is. + +UI mint (`(*Service).mintWorking`): `ttl` and `grants` are both **mandatory** — a missing +field is 400, not an unbounded or universal token. The empty-string-means-universal rule +applies only when *reading a stored column*, never to a request. + +### Registered vs stateless ("short") tokens + +`family::sourcehut-tokens/service/tokens.go::(*Service).mintWorking`: + +| TTL | Result | +|---|---| +| `<= stateless-ttl` (default 24h) | **stateless** — sealed and returned, no database row, no `id:` member, **no revocation possible**, no revocation check at validation | +| `> stateless-ttl` | **registered** — `registerWorking` INSERTs first, *then* seals around `row.ID`. The order is forced: sealing first would hand out a credential whose revocation check names a row that may never exist, and a 404 there means every validator refuses it | +| `0` (`"ttl": "never"`, UI only) | **always registered**, whatever `stateless-ttl` says; `working-max-ttl` is skipped. An unbounded stateless token would be the one credential on the instance that nothing can stop | + +The row id travels **inside the signed payload**, as a grant member `id:`, because +`auth.BearerToken` has no field for it. It is an address, not a permission: +`grants.Grants.IsSubsetOf` ignores it on both sides and `Grants.Members()` omits it. + +### Revocation and the liveness cache + +`GET {tokens-origin}/api/v1/revocations/{id}` has exactly **two** answers — **204 = live**, +**404 = revoked / expired / unknown** — behind an internal guard. Anything that is neither is +not a third answer; it is the absence of one. + +Client side, `family::sr-ht-ecore/bearer/bearer.go::(*Validator).checkRevocation`: + +- a 500, a timeout, a proxy's HTML page → `ErrUnavailable` → **503, never 401**; +- `ErrUnavailable` is **never cached**; only real verdicts are; +- TTL `DefaultCacheTTL = 60s`, bounded at `maxCacheEntries = 4096` with sweep-then-drop (no + LRU). `(*Validator).Forget(id)` invalidates out of band; +- **the stated trade**: a revocation takes up to 60s to take effect instance-wide. + +Revoking a **parent** does not revoke the working tokens it already issued — short ones burn +down, long ones must be revoked individually. The only mass revocation is rotating +`[webhooks]private-key`, which invalidates every working token *and* every meta PAT at once. +That is an emergency stop, not a procedure. + +**What is not verifiable offline is only the revocation state.** Signature, version, expiry, +grants and issuer are all local. A stateless token therefore touches the network **zero** +times during validation — the common case under default configuration. tokens.sr.ht being +down stops *minting*, not *using*. + +--- + +## 4. The grant vocabulary + +`family::sr-ht-ecore/grants/grants.go`. This is **not** core-go's `auth.Grants`; that one is +meta's OAuth vocabulary (`git.sr.ht/OBJECTS:RW`). The two grammars share a token format and +nothing else. + +### Grammar + +- Members separated by **ASCII whitespace only** — `::asciiFields` uses `strings.FieldsFunc` + over `' ' \t \n \v \f \r`, deliberately **not** `strings.Fields`. Under `unicode.IsSpace` a + field holding only U+00A0 splits into zero members, and zero members means *universal*. +- A member is `:`, split at the **first** colon; the action may carry further + colon segments (`bench:upload:~bigbes/foo` is reserved for a later version). +- Every byte printable ASCII (`0x21..0x7e`). **Upper case is refused, not folded** — grants + are compared literally. +- No empty segment, no `::`, no leading or trailing colon. `MaxGrantsLen = 4096`. +- `*` (`grants.Universal`) is every action of every service; `"* cov:upload"` normalises to + `"*"`. +- **An empty string parses to universal**, but the **zero value `Grants{}` grants nothing** — + the meaning of `""` lives in `Parse`, so a caller that forgot to parse ends up admitting + nobody. +- `id:` is reserved. `::Parse` accepts it (reading a stored column or a presented token); + `::ParseRequested` **refuses** it. That refusal is a privilege boundary: a caller who could + choose the id would point their own revocation check at somebody else's live row, and their + revoke would stop revoking anything. +- `(Grants).IsSubsetOf`: universal is a subset only of universal. There is **no wildcard below + `*`** — `cov:*` is an ordinary member matching only a validator asking for literally `cov:*`. + +### Real grant strings in use + +Each service owns its vocabulary in its own package; **the daemon does not validate it**, so +an unknown grant is not an error — it is a grant that admits nobody anywhere, which is the +safe direction. + +| Service | Symbol | String | +|---|---|---| +| bench | `family::sourcehut-bench/authn/instance.go::GrantUpload` / `::GrantRead` | `bench:upload`, `bench:read` | +| coverage | `family::sourcehut-coverage/authn/principal.go::GrantUpload` / `::GrantRead` | `cov:upload`, `cov:read` | +| specs | `family::sourcehut-specs/authn/bearer.go::ActionPropose` / `::ActionRead` | `spec:propose`, `spec:read` | +| artifacts | `family::sourcehut-artifacts/core/grants.go::GrantUpload` / `::GrantDelete` / `::GrantRead` | `artifacts:upload`, `artifacts:delete`, `artifacts:read` | +| dolt | `family::sourcehut-dolt/core/grants.go::GrantRead` | `dolt:read` | +| curator | `family::sourcehut-curator/authn/principal.go::GrantRead` / `::GrantWrite` | `go:read`, `go:write` | + +The coverage prefix is **`cov:`**, not `cover:` — `family::sourcehut-tokens/SPEC.md` ch. 2 and +ch. 3 spell it `cover:` and the SPEC is wrong there; the service's own constant is the +authority, per that same chapter. + +**dolt deliberately has no write grant.** A grant nobody checks is a promise to an operator +that no code keeps: `dolt:write` would be indistinguishable in effect from `dolt:read`, and +the operator who chose the narrow one would believe in a restriction that does not exist. +Declare a grant only where a check consumes it. + +The **meta PAT scope** is a second, non-overlapping vocabulary published beside the first: + +| Service | Symbol | Full scope | Published in api-meta.json | +|---|---|---|---| +| artifacts | `family::sourcehut-artifacts/authn/meta.go::ScopeRead` | `artifacts.sr.ht/ARTIFACTS` | `ARTIFACTS` | +| bench | `family::sourcehut-bench/authn/meta.go::ScopeRead` | `bench.sr.ht/RESULTS` | `RESULTS` | +| coverage | `family::sourcehut-coverage/authn/meta.go::ScopeRead` | `cov.sr.ht/REPORTS` | `REPORTS` | +| curator | `family::sourcehut-curator/authn/meta.go::Scope` | `go.sr.ht/MODULES` | `MODULES` | +| dolt | `family::sourcehut-dolt/authn/token.go::RepoScope` | `dolt.sr.ht/repos` (`:RO`/`:RW`) | `repos` | +| specs | `family::sourcehut-specs/authn/meta.go::ScopeRead` | `spec.sr.ht/SPECS` | `SPECS` | + +No PAT can ever carry `spec:read`, and no working token can ever carry `SPECS`. A credential +has one grammar or the other, never both. + +### The literal-comparison rule and its consequence + +Grants are compared literally and **nothing on the instance cross-checks a spelling**. The +mint form's checkbox list (`family::sourcehut-tokens/web/tokens.go::grantVocabulary`) says so +in its own comment — *THE DAEMON DOES NOT VALIDATE THE VOCABULARY* — and carries a free-text +field beside it so a service that grows an action is mintable the day it ships. Its tests +assert the list is well-formed and say nothing about whether any service honours it. + +**Therefore: renaming your service's grant prefix silently invalidates every token already +minted against it.** The old string still parses, still stores, still renders on the token +page, and admits nobody — surfacing hours later as a 403 naming a grant the caller does not +hold. Same on the PAT side, where the scope is built from the config section (`ScopeRead = +ConfigSection + "/SPECS"`): renaming the section changes the scope and every already-granted +PAT stops carrying it. If you must rename, treat it as a credential rotation, not a refactor. + +Two mitigations worth copying: derive the published scope from the checked one +(`metapat.ScopeName(authn.ScopeRead)` — specs and bench both do), and **assert in a test that +the scope you publish and the scope you check are the same string**. + +--- + +## 5. Validation order, and why + +`family::sr-ht-ecore/bearer/bearer.go` — four steps: + +1. **Decode and verify** — `auth.DecodeBearerToken`: base64, length, HMAC, version, **and + expiry against the real clock**. Local, no network. +2. **Is it ours?** — `bt.ClientID != TokensClientID` → `ErrNotOurs`, returned *with* the + decoded token, and the package stops. Per-service policy decides. +3. **Does it grant the action?** — `Token.Authorize(action)` → `ErrForbidden`. +4. **Is it still live?** — only if `TokenID != 0`; the revocation endpoint through the 60s + cache. + +Every step that can refuse locally runs before the one that cannot, so the network is touched +only for a token already proved well-formed, ours, unexpired and sufficient — and an instance +under a flood of junk tokens does not turn that flood into traffic against tokens.sr.ht. Step +3 before step 4 is deliberate: a token missing the grant costs no round trip. + +**The split for middleware.** `Validate` needs the action; middleware does not know the route +yet. `Inspect` is steps 1, 2, 4 — call it in the resolver, carry `Token.Grants` on the +principal, call `Token.Authorize(action)` in the handler. Do **not** lift the bearer plane out +of the shared middleware to get an action; that is how a surface ends up with two different +ideas of who is calling. + +### The status contract + +`family::sr-ht-ecore/bearer/status.go::StatusFor`: + +| sentinel | status | +|---|---| +| `ErrForbidden` | 403 | +| `ErrUnavailable` | **503** | +| `ErrInvalid`, `ErrRevoked`, `ErrNotOurs`, anything unrecognised | 401 | + +`ErrRevoked` is 401 and not 403 on purpose: the token is no longer a credential at all, and a +client shown 403 keeps presenting it. The three caller-fault refusals share one status and one +sentence — telling a prober "that token exists but is revoked" is information they have not +earned; which it was belongs in the log. **503 is the arm to defend**: reading an unreachable +token daemon as "revoked" refuses every registered token on the instance until it restarts. +`metapat` mirrors the same table for meta outages. + +**Guard the table so a DB outage cannot answer 401.** `StatusFor`'s default arm answers 401 to +*anything* — including your own Postgres error or a dead context. Wrap the delegation: + +```go +if bearer.IsRefusal(err) { + http.Error(w, msg, bearer.StatusFor(err)) + return +} +// anything else is ours, not the caller's — 500 +``` + +`family::sr-ht-ecore/bearer/status.go::IsRefusal` answers for *that package's* vocabulary and +nothing else, which is the trap in the guard: a service whose own refusals do not wrap those +sentinels sends its "bad token" into the else branch and answers 503 to a caller whose +credential really was the problem. Either wrap (your `ErrInvalidToken` unwraps to +`bearer.ErrInvalid`) or ask your own predicate first. + +Send `bearer.Challenge(section)` — `WWW-Authenticate: Bearer realm="bench.sr.ht"`, RFC 9110 +§11.6.1-quoted — on every 401. + +--- + +## 6. Plane routing + +Route with `family::sr-ht-ecore/metapat/metapat.go::PlaneOf`, **not** by catching +`bearer.ErrNotOurs`. One local HMAC, no network, and it works on an instance whose config has +no `[tokens.sr.ht]` section at all — where there is no `bearer.Validator` to call and the meta +PAT plane must keep working anyway. + +`family::sourcehut-bench/graph/server.go::authenticate` is the exemplar: + +```go +if presented := authn.BearerFromRequest(r); presented != "" { + if metapat.PlaneOf(presented) == metapat.PlaneMeta { + resolved, err = meta.VerifyToken(r.Context(), presented) + } else { + resolved, err = auth.Resolve(r.Context(), r) + if err == nil { + err = resolved.Authorize(authn.GrantRead) + } + } + // ... bearer.StatusFor(err) ... +} +``` + +`PlaneUnknown` is not a verdict about the issuer — an expired token of either plane lands +there, because `auth.DecodeBearerToken` checks expiry before it reports anything. Answer it +with the same 401 as `ErrInvalid`, never as "no credential presented". + +`metapat.Allows(ac, metapat.Scope("cov.sr.ht", "REPORTS"), auth.RO)` is the PAT-side scope +check, and it **passes unconditionally** for `ac == nil` or `ac.BearerToken == nil` — a cookie +session, an anonymous request, or a working token resolved by the other plane was never scoped +in meta's vocabulary. A PAT minted with no grants is universal by core-go's own definition. +Neither is a hole; both are surprises if you did not read them here. + +`metapat.New` requires `Service` (e.g. `"cov.sr.ht"`) because `auth.DecodeGrants` reads the +calling service's name off the context and `config.ServiceName` **panics** when nothing put it +there — so a validator relying on ambient context works behind an HTTP router and takes the +process down in a background job, a CLI, or a test. + +**Mount the PAT plane structurally, not by convention.** On bench, coverage, curator and specs +`/query` sits in a route group that does **not** install the shared resolver, and the GraphQL +server does its own `PlaneOf` routing; the other surfaces' resolver holds no `MetaAuth` and +therefore *cannot* produce a meta principal. Copy that shape — it is not enforceable by review. + +--- + +## 7. The login cookie + +`family::sr-ht-ecore/login/login.go`. Do not hand-roll the decode. Five decisions, and two of +the six donor services had already dropped one of them: + +1. **`crypto.DecryptWithoutExpiration`, never a decrypt with a TTL.** The cookie's lifetime is + the browser's `Expires` plus key rotation, both instance-wide facts. A service-side TTL logs + a viewer out of that one service on a schedule no sibling shares, and the viewer reads that + as "this service is broken". (The mirror image is internal auth, which *must* have a TTL.) +2. **Unmarshal into `auth.AuthCookie` and take `.Name`.** +3. **Strip a leading `~` — one, and only at the front.** `"~~x"` is not a name and must not be + repaired into one. +4. **Every failure is anonymity, never an error, and never logged.** No cookie, forged, + truncated, wrong key, wrong JSON, no name, invalid name → `""`. The reasons are + indistinguishable to the viewer and all mean "log in again"; reporting them turns a tab left + open across a key rotation into a broken site rather than a logged-out one. Not logged + because the value is attacker-supplied on every request — a log-flood anybody can turn on. +5. **Validate the name before it reaches a path, a log line or a SQL parameter.** + `::ValidName`: non-empty, ≤ `MaxUsernameLen = 64`, `[A-Za-z0-9._-]` only, not starting `-`, + not `.` or `..`. It catches `/` and `\` (filepath escape), control bytes and NUL (log + forging), non-ASCII (two spellings comparing unequal in Go and equal in Postgres), and a + leading `-` (a flag to something exec'd). It is a **default, not a parameter**; + `WithValidator(nil)` **restores** it, and there is no spelling of "accept anything". + +Upstream does all of this in its **unexported** `mirror::core-go/auth/middleware.go::cookieAuth`, +which is exactly why every custom service had to re-derive it. **Serve from under the shared +parent domain or the cookie is never sent** and every viewer is anonymous with no second door. + +**Two middlewares, one decode. Install one or the other, never both.** `login.Optional()` never +refuses and stores `""` for anonymous, so a handler always reads a resolved answer. +`login.Required(deny)` refuses through a handler you supply, because the right refusal is +usually a 302 to `{meta}/login?return_to=…` and `login` holds no configuration literal. Build +that URL with `family::sr-ht-ecore/chrome/chrome.go::(*Service).LoginURLFor`: + +```go +return s.metaOrigin + "/login?return_to=" + url.QueryEscape(s.selfOrigin+r.URL.RequestURI()) +``` + +`login.FromContext(ctx)` returns `""` for both "anonymous" and "no middleware ran" — +deliberately the same answer, so nobody writes a fail-open branch for the second. + +**What stays per-service**: turning the username into a local row (`auth.LookupUser` plus your +own `user` table mirroring meta's profile). A name is instance-wide; a row is not. Refuse a +`UserID == 0` result — every ownership row keys on the id, and a zero matches whichever row has +an unset owner. + +--- + +## 8. `Authorization: Internal` + +`family::sr-ht-ecore/internalauth/internalauth.go` holds **both ends** of the protocol over one +struct, which upstream does not: core-go implements the check unexported inside +`mirror::core-go/auth/middleware.go::internalAuth`, so a service wanting the guard without the +whole middleware rewrites thirty lines and the minting half lands in another program spelling +the payload by hand. Change the payload shape at one end and nothing fails to compile. + +**Minting**: `::Authorization(clientID, nodeID)`, or `::AuthorizationAs(username, clientID, +nodeID)` for a call on a user's behalf — what core-go's own `client.Do` does for every GraphQL +call, the receiving service resolving its whole auth context from the name. Both refuse an +empty `clientID`/`nodeID` at the call site. Mint **per call**; never cache. + +**Verifying**: `::Guard(clientID, nodeID, deny)` / `::Verify` / `::Identify`. Two required +checks in fixed order — `config.IsInternalIP(RemoteAddr)`, then fernet open with +`Expiry = 30 * time.Second`. + +- **The address check must never stand alone.** Behind a reverse proxy the address is the + proxy's, internal for every forwarded request. `X-Forwarded-For` is deliberately **not** + consulted: honouring it would turn the weaker check into one an outsider can pass by asking. +- **The real acceptance window is `[now-30s, now+60s]`.** Fernet's verifier also accepts a + token dated up to 60s in the future, and shortening `Expiry` does not shorten the forward + half. There is no nonce and no seen-token set; replay is bounded by the window alone. +- **Pin the peer.** `Guard("", "", …)` accepts any non-empty client/node — core-go's own + behaviour. A custom service is typically reached by exactly one sibling for exactly one + purpose, so pin both, and write `Guard("", "", …)` explicitly if you genuinely mean "any + service on the instance". +- Status mapping (`::Status`): **401** for `ErrSourceIP` / `ErrMissing`, **500** for + `ErrNetworkKey`, **403 for everything else** including unrecognised errors. + +### What it bypasses and what it does not + +Internal auth is a **full-authority credential**: it bypasses every `@access` scope check in +core-go — `AuthContext.Access` returns nil for `AUTH_INTERNAL` / `AUTH_ANON_INTERNAL`. + +It does **not** bypass the resolvers' own SQL filtering: `User.repositories` selects on the +authenticated user's id. So **mint it as the user whose data you want**, not as a service +identity, or you get a silently smaller result set — the failure mode nobody notices. +`family::sourcehut-curator/srhtapi/srhtapi.go::InternalAuth` does exactly this, per call. + +--- + +## 9. CSRF / same-origin + +`family::sr-ht-ecore/csrf/csrf.go`. **There is no CSRF token and nowhere to keep one**: +identity is meta's cookie on the parent domain, no individual service issues it, and none can +set its `SameSite`. A synchronizer-token scheme would mean every daemon inventing a session +store for two forms. What is left is what the browser says about where the request came from — +`Origin` and `Referer` are set by the user agent and cannot be forged from script cross-origin. + +The rule (`::claimMatches`, `::originMatches`): + +- **Safe methods exempt** — GET, HEAD, OPTIONS, TRACE (`::SafeMethod`). Everything else, + including a method invented tomorrow, is guarded. +- Read `Origin`; **when present it is used alone.** Falling back to `Referer` after `Origin` + already said "somewhere else" turns the stronger statement into the weaker. +- Failing that read `Referer` (whole URL; the path is ignored). +- Compare **scheme and host, case-insensitively, port included** (RFC 6454 §4). A value parsing + to no scheme or no host — the `"null"` a sandboxed iframe posts — matches nothing. +- **Neither header ⇒ refuse.** This is the clause holding the guard up. A request that will not + say where it came from cannot be shown to have come from us; waving it through reduces the + whole guard to a header an attacker's page simply omits. +- A `selfOrigin` that is not a scheme+host URL refuses **every** mutating request — fails closed. + +**Install `csrf.Require` as router-wide middleware, not per handler.** Of five donor services, +two checked per handler — and in those two the default for a new POST route was *unprotected*. +Installed on the router it also runs **before routing**, so a POST to an unserved path is +refused rather than 404'd; otherwise the difference enumerates which routes exist. Install it +after whatever sets the cache headers. + +**Exempting a bearer API is a fact about where you `Mount`, not a path test.** There is +deliberately no "is this `/api`?" option and there must never be one: every escape, every case +fold and every dot segment a client can spell would then be a way to ask for the exemption. +When one mux carries both planes, key the exemption on the **credential** — +`family::sourcehut-tokens/api/api.go::(*Handler).originAllows`: + +```go +if csrf.SafeMethod(r.Method) { return true } +if authn.PrincipalFromContext(r.Context()).Method != authn.MethodCookie { return true } +return csrf.SameOrigin(r, h.selfOrigin) +``` + +Refusal is **403, never a redirect** — a redirect after a POST drops the body and turns a +refused mutation into a page that looks like it worked. `csrf.Message` is the shared sentence; +pass your own `renderError` closure as `deny` so the refusal looks like the rest of the surface. + +Adjacent and load-bearing: `family::sr-ht-ecore/middleware/middleware.go::PrivateCache` sets +`Cache-Control: private, no-store` and `Vary: Cookie, Authorization` on **every** answer, +because these services render per-viewer documents at URLs that say nothing about the viewer. +`no-cache` is insufficient — it still permits a stored copy. + +--- + +## 10. Checklists + +### Process-global prerequisites (all planes depend on these) + +- [ ] `config.LoadConfig()` — populates `[sr.ht]internal-ipnet`; without it **every** address is + external and internal auth refuses the siblings it exists for. +- [ ] `crypto.InitCrypto(conf)` — installs the fernet key and derives the bearer HMAC key. It + `log.Fatal`s if `[sr.ht]network-key` or `[webhooks]private-key` is missing, so validate + both in your own config check first (`instconf.Require`, as tokens and curator do). +- [ ] Read origins through `family::sr-ht-ecore/instconf/instconf.go` — `ExternalOrigin` for + what a browser sees, `InternalOrigin` for the tokens daemon, `InternalAPIOrigin` (the + four-key ladder `api-internal-origin` → `internal-origin` → `api-origin` → `origin`) for + calling a sibling. Never hand-roll the trailing-slash trim. +- [ ] Your config section must be literally `.sr.ht`. +- [ ] **Resolve identity exactly once**, in middleware in front of *all* surfaces, onto a + principal on the context. Never re-resolve per surface. + +### (a) Pages rendered as the logged-in viewer + +- [ ] Served from under the shared parent domain, or no cookie ever arrives. +- [ ] `r.Use(login.Optional())` where anonymous is legitimate; `login.Required(s.denyLogin)` + where every page belongs to somebody. **One, not both.** +- [ ] Deny handler is a 302 to `chrome.Service.LoginURLFor(r)`. +- [ ] `login.FromContext(ctx)`; `""` is anonymous *and* "no middleware ran" — never branch on + the difference. +- [ ] Resolve the name to a local row yourself; refuse `UserID == 0`. +- [ ] `middleware.RecoverPanics(render)` outermost, then `middleware.PrivateCache`. +- [ ] `csrf.Require(selfOrigin, s.denyCSRF)` on the **router**, for every form. +- [ ] Do not hand-roll the cookie decode. If you think you must, re-read §7. + +### (b) An authenticated API + +- [ ] Plane policy **per surface**, not per service: `/query` takes PATs *and* working tokens + (federation forces it); REST and MCP take working tokens only. +- [ ] Route with `metapat.PlaneOf(presented)`, never by catching `bearer.ErrNotOurs`. +- [ ] Build one `bearer.New(bearer.Options{Origin: instconf.InternalOrigin(conf, + "tokens.sr.ht"), ClientID: , NodeID: })` at startup. Do not + override `CacheTTL` — the instance runs one 60s window everywhere. +- [ ] `Inspect` in middleware → `Token.Grants` on the principal → `Token.Authorize(action)` in + the handler. `bearer.IsRefusal(err)` → `bearer.StatusFor(err)`; anything else is a 500. + **`ErrUnavailable` is 503.** `bearer.Challenge(section)` on every 401. +- [ ] Taking PATs too: `metapat.New(metapat.Options{Service: ".sr.ht"})` (`Service` is + required), then `metapat.Allows(ac, metapat.Scope(".sr.ht", "SCOPE"), auth.RO)`. +- [ ] Serve `apimeta.Handler("SCOPE")` at `apimeta.Path` (`/query/api-meta.json`) if you mount + `/query` yourself. **The scope list must never marshal to `null`** — meta iterates it + when rendering `/oauth2/personal-token`, and a `null` 500s that page for the *whole + instance*. `apimeta.Handler` marshals `[]` for none; do not build the JSON by hand. +- [ ] Grant strings as named lower-case constants in one package; published scope derived from + the checked one; a test asserting the two agree; the grants added to + `family::sourcehut-tokens/web/tokens.go::grantVocabulary`. +- [ ] Mount the bearer surface **outside** the `csrf.Require` router, or key the exemption on + the credential (§9). + +### (c) Calling another service on the user's behalf + +- [ ] There is **no** user-OAuth-token flow here and no service holds a user's OAuth token. Two + mechanisms, both minting internal auth **as the user**: core-go's + `client.Do(ctx, viewer, "git.sr.ht", query, &out)`, or a hand-written client with an + injectable authorizer over `internalauth.AuthorizationAs` + (`family::sourcehut-curator/srhtapi/srhtapi.go::InternalAuth`) — prefer the second when + you want the calls testable without process-global crypto state. +- [ ] Mint per call (30s window, never reused, never cached). Post to + `instconf.InternalAPIOrigin(conf, ".sr.ht") + "/query"`, from inside + `[sr.ht]internal-ipnet` as the peer sees it. +- [ ] **This can be your whole authorization model.** compare owns no permission code: + `family::sourcehut-compare/authz/authz.go` asks git.sr.ht *as the viewer*, and a + repository the viewer may not see comes back as a null `user`/`repository` mapped to + `core.ErrNotFound` — so private-repo *existence* never leaks. Cache positives and + not-founds; never cache a transport error. +- [ ] Never forward the caller's own bearer token to a peer. Never `url.URL.Redacted()` a + token-bearing URL — it masks the password and leaves a bare username intact. + +### (d) Accepting calls from another service + +- [ ] `r.With(internalauth.Guard(peerClientID, peerNodeID, deny)).Post("/internal/…", h)`. + **Pin both ids** unless you genuinely mean any service, and then write `Guard("", "", …)` + explicitly. +- [ ] Read the caller with `internalauth.FromContext(ctx)`; render the refusal from + `internalauth.Reason(ctx)` + `internalauth.Status(err)`. +- [ ] Keep the endpoint out of the CSRF router and out of the public nginx location. +- [ ] Do not rely on the source-IP check alone; do not read `X-Forwarded-For`. +- [ ] **Return the smallest possible answer.** tokens' revocation endpoint has exactly two + (204/404) so that a third — a 500 from an out-of-range id — cannot be read by validators + as "the daemon is down" and take instance-wide uploads with it. diff --git a/skills/sourcehut-custom-service/references/caching.md b/skills/sourcehut-custom-service/references/caching.md new file mode 100644 index 0000000000000000000000000000000000000000..44202103565129a0fb343e65d191b712a1fd66d4 --- /dev/null +++ b/skills/sourcehut-custom-service/references/caching.md @@ -0,0 +1,560 @@ +# Caching and object storage + +> Citations: `family::/::` — this instance's own repos; `mirror::::` +> — the upstream documentation mirror. Both roots are substituted at install time. Symbols, never +> line numbers. + +Two corrections before anything else, because both names mislead: + +- **`cachex` is not a generic cache library.** It is the CI-cache *facet* of one service — + an HTTP API (`/api/v1/cache`) over an S3 bucket with quota and TTL accounting in its own + PostgreSQL tables. You cannot import it as a cache. +- **`blobx` is not content-addressed storage.** Its key is + `"blob/" + owner + "/" + channel + "/" + path` (`family::sourcehut-artifacts/blobx/path.go::objectKey`) + — path-addressed and human-readable. The sha256 exists, but as S3 user metadata, for a + different job (see §4). + +Both are HTTP facets of one daemon, mounted next to each other on one router. What service +#11 copies from them is **the pattern**, file seam for file seam — never the package. + +Two absences are deliberate and you should not repair them: + +- **No Redis.** Not for sessions, not for caching, not for rate limiting. Every custom + service carries `go-redis` as `// indirect` (dragged in by the core fork's `server` + package) and imports it in zero `.go` files; `sr-ht-ecore` does not have it in `go.mod` at + all. Do not add `redis-host` to your `config.ini`. The family's answer to "where does + cached state live" is **PostgreSQL for metadata, S3 for bytes, HTTP headers for the client, + a small bounded in-process map for auth verdicts.** +- **No `golang.org/x/sync/singleflight` anywhere.** Two cold-cache validations of the same + token both issue a request "and nothing worse: the answers agree, the second write to the + cache is idempotent, and collapsing them would buy one saved request in exchange for a + dependency and a shared failure mode where a single slow call holds up every goroutine + waiting behind it" (`family::sr-ht-ecore/bearer/bearer.go::Validator`). The one collapsing + construct in the family (`family::sourcehut-curator/service/inflight.go`) is hand-rolled and + documents itself as *not* a cache — the result is not remembered past the call. + +--- + +## 1. The catalogue + +| Cache | Where it lives | TTL / bound | Key | Invalidated by | +|---|---|---|---|---| +| CI build cache (`cachex`) | S3 bucket `docker-cache` + PG `cache_ns`/`cache_entry` | `cache-default-ttl` 2160h, per-ns override; **disuse-based** on `last_access`, not age | `/` verbatim, no hashing (`family::sourcehut-artifacts/cachex/keys.go::objectKey`) | GC pass, operator purge, PUT overwrite, quota refusal | +| Quota accounting | PG, under a per-namespace advisory lock | — | ns name → FNV-32a → `pg_advisory_xact_lock` | `Restore` compensation, reconcile | +| `last_access` touch | PG column | at most one write per **5 min** per key | `(ns, key)` | every read, throttled | +| Reconcile inventory | S3 listing ↔ PG | 64 namespaces/pass, 24h staging grace | bucket listing under `/` | background loop | +| goproxy pull-through | S3 + `mirror_object` | artifacts **never refetched**; pointers (`/@v/list`, `/@latest`) `pointer_ttl` 15 min | mirror + normalised upstream path | pointer TTL; retention on disuse | +| docker pull-through | S3 + `oci_tag` | digests **immutable, never revalidated**; tags `tag_ttl` 15 min then HEAD revalidation | digest / `(repo, tag)` | tag TTL; retention | +| apt/apk snapshots | S3, one `mirror_snapshot` per sync | `schedule`-driven; one previous kept as `old` | snapshot id + path | a new sync publishing `current` | +| Release blobs (`blobx`) | S3 `artifacts` bucket + PG row | none — the row is the truth | `blob///` | publication, delete | +| Bearer revocation verdict | in-process `map[int]verdict` + mutex | **60s**, bound **4096**, sweep-then-drop-all | integer token row id (**not** the secret) | `Forget(id)`, TTL | +| meta.sr.ht PAT resolution | in-process `map[[64]byte]entry` | **60s**, bound 4096, drop-all | `sha512.Sum512(presented_secret)` | `Forget(presented)`, TTL | +| Hashed static assets | the client's cache | `public, max-age=31536000, immutable`; unhashed fallback `public, max-age=3600` | filename matching `\.[0-9a-f]{8,}\.(css\|m?js)$` | a new hash from `make css` | +| git ACL verdicts | in-process map | 60s, lazy prune + one full sweep per TTL | `viewer \x00 owner \x00 name` | time only — transport errors are never cached | +| Rendered godoc pages | in-process LRU | **no TTL**, bound 128 | a ref that **carries the commit** | nothing; a rebuild is a different key | +| Negative discovery | in-process map | 5m, lazy eviction | module path | time; "it can never change an answer" | +| Beads projections | in-process generic cache | 60s **and** gated on the repo head hash, ceiling 256 | repo id + head hash | a head change; TTL | +| HTTP responses | the client's / proxy's cache | §5 | URL + `Vary` | `no-store` / `must-revalidate` | + +**The recurring shape — imitate it rather than inventing a policy:** + +1. **60 seconds for anything auth-derived.** Four independent caches landed on it. It is the + revocation-lag window, and it is a stated trade, not a default. +2. **No TTL at all when the key carries the content identity** — a commit sha, a module + version, a digest. "A moved tag or a rebuilt version is a different key and there is no + invalidation to get wrong." +3. **A bound plus a cheap drop, not an eviction policy.** Sweep the expired entries; if still + over the bound, throw the whole map away. 4096 in ecore, 256 in dolt's beads, 128 in + curator's godoc. +4. **Two correctness sources beat one.** The best cache in the family gates on a head hash + *and* a TTL: "the head hash is what makes it correct, the TTL is what makes it bounded." + +### The one rule that ties the disuse-based caches together + +`family::sourcehut-artifacts/core/retention.go::CheckTTLOutlastsThrottle`: + +```go +func CheckTTLOutlastsThrottle(ttl, throttle time.Duration, subject string) error { + if ttl > throttle { + return nil + } + return fmt.Errorf( + "%s does not outlast the %s last-access throttle, so %s being read would expire before a read could refresh them", + ttl, throttle, subject) +} +``` + +A read refreshes `last_access` at most once per throttle window, so a retention TTL that +does not outlast that window expires rows that are **still being read**: the reads inside +the window write nothing, and the collecting pass that follows finds the row untouched. +Measured with `ttl=2s`: a GET answered 200 and the very next GC pass collected the same key. +Equality is refused too — both the expiry query and the throttled touch compare +`last_access <= now() - interval`, so `ttl == throttle` makes a row collectable at the exact +instant the first refreshing read becomes possible. + +One copy, shared by both retention facets with the window passed as an argument, and +enforced in **three** places: at config load (so the operator sees it with the rest of the +file's problems), at the facet's `New` (a caller may pass its own throttle), and at the row +validator every REST verb writes through. If you age anything by a `last_access` column, +copy this function and its three call sites. + +Two more rules from the same family: + +- **Compare a timestamp against the clock that wrote it.** A column written by the daemon and + a cutoff taken from SQL `now()` differ by the clock skew between them — measured at 44ms + between one repo's test container and its host. +- **Revalidate a pull-through pointer on read, not on a schedule.** A cache whose keyspace + cannot be enumerated (`Get`/`Put`/`Delete` by path) cannot have a sweeping revalidator; a + scheduled one was deleted after it spent a dedicated backend connection per tick to find + nothing to do. + +--- + +## 2. The two in-process auth caches + +These are in `sr-ht-ecore`, which your service imports. **Use them as-is; do not build your +own token cache, and do not "harmonise" the differences between them** — the differences are +forced by who controls the key space. + +| | `family::sr-ht-ecore/bearer/bearer.go::Validator` | `family::sr-ht-ecore/metapat/metapat.go::Validator` | +|---|---|---| +| Caches | the *verdict* only (`alive bool`, `until time.Time`) | the whole successful resolution (`*auth.AuthContext`) | +| Key | the integer token row id from the grant string, **unhashed** | `sha512.Sum512(presented_secret)` | +| TTL / bound | 60s / 4096, sweep-expired-then-drop-all | 60s / 4096, drop-all | +| Negative caching | **yes for 404** (revoked, expired and unknown are all permanent); **never for a failure** | **none at all** | +| Copy semantics | value type | shallow copy on every read *and* write | + +**Why one key is hashed and the other is not.** `bearer`'s key is not a secret and its key +space cannot be inflated: "an entry can only be created by a token that already passed an +HMAC check, so the id space here is the daemon's real rows and not something a caller can +inflate." `metapat`'s key **is** the presented secret and is attacker-chosen, so it is +hashed before it becomes a map key, and nothing negative is stored: "a genuine refusal is +not cached either: it costs one local HMAC to reproduce, and the alternative is a data +structure that an attacker can grow by presenting garbage." + +**The rule to carry into service #11:** whether you may cache a negative answer, and whether +you may key on the raw credential, are both decided by *who controls the key space*. Write +the answer down next to the cache. + +Three more decisions worth copying verbatim: + +- **A failure is never cached.** "Caching it would let one blip, one timeout, one restart pin + every token that happened to be checked during it to failure for the whole TTL. That turns + a moment of unavailability into a minute of it, and it does so silently, because the daemon + is healthy again while the services are still refusing." +- **An unreachable token daemon is 503, never 401** (`family::sr-ht-ecore/bearer/status.go::StatusFor`). + Reading an outage as a revocation would refuse every registered token on the instance. + Note the mirror image upstream: meta.sr.ht's own `TokenRevocationStatus` fails **closed**, + because meta owns the truth and is answering about its own keyspace. A downstream service + must not. +- **Return a copy.** `metapat.copyOf` exists "so that a caller which annotates the context it + was handed — core-go's own middleware sets IPAddress on one — does not write through into + the cache and hand the next caller somebody else's address." Shallow is argued for; deep + would be wrong. + +**What ecore deliberately does not cache**, and neither should you: the unified-login cookie +(decoded per request, no TTL — "a service that added an expiry to the decrypt would log a +viewer out of that one service on a schedule no sibling shares"); templates at runtime +(parsed once at startup); upstream HTTP responses; the internal fernet authorization ("a +fernet blob the daemon accepts only for thirty seconds"). + +--- + +## 3. The `cachex` / `blobx` pattern + +### File seams + +``` +yourfacet/ + storage.go ObjectStore interface + NewS3Store (the block in §4) + repository.go Repository interface + NewSQLRepository — ALL SQL lives here + maintenance.go a narrower interface, for background loops only + server.go New(Options) *Server; APIMount(authorizer) http.Handler + keys.go normalize* + objectKey — the ONLY place the layout is written +``` + +Two seams, not one: `ObjectStore` (bytes) and `Repository` (metadata) are separate +interfaces, so **every handler test runs without S3 and without PostgreSQL**. A third, +narrower seam for the background passes +(`family::sourcehut-artifacts/cachex/maintenance.go::Maintenance`) so "a request handler must +not be able to adopt or resize accounting rows, and a loop has no use for the quota +decision" — backed by the same implementation type, so loop and handler take the same lock. + +**The bucket is the source of truth for reads; PostgreSQL is the source of truth for quota.** +`family::sourcehut-artifacts/cachex/server.go::handleGet` never consults the accounting row. +That is what lets a key written straight to the bucket be readable before any reconcile pass +has adopted it, and what makes a row without an object harmless. Honour it on **every** +verb, listing included — a namespace being filled right now answering 404 to a listing +"reads as one that does not exist, for a gc-interval and longer." + +### Quota decision and accounting write: one transaction, one lock + +Non-negotiable. The decision and the write are one transaction under one namespace-wide +**transaction-scoped** advisory lock (`pg_advisory_xact_lock`, never the session-scoped +form): "a handler cannot leak the lock by returning early, and a dropped connection cannot +leave the namespace unwritable." Without it, two concurrent uploads to different keys both +read pre-upload usage and jointly exceed the quota. + +The lock is taken **after** the body has been streamed, so it serialises bookkeeping, not +byte movement. + +A streamed body has no length, so the quota is approached from two sides: a declared +`Content-Length` over the headroom is refused before a byte is read; an undeclared one is +wrapped in a reader that stops one byte past the headroom. Neither is the decision — +`Account`, on the size actually read, is. The reader carries its own `exceeded` flag because +the SDK may wrap a read error several layers deep. + +The throttled `last_access` write belongs **in the `WHERE` clause**, not in a read-then-write +pair (`family::sourcehut-artifacts/cachex/repository.go::Touch`): two concurrent readers of a +hot key both run the statement, the second blocks on the row lock, PostgreSQL re-evaluates +the qualification against the committed row, so exactly one write happens per window however +many readers arrive together. A failed `Touch` is **logged, not returned** — "failing the +download instead would turn a metadata hiccup into a broken CI build." + +### Write ordering: every crash state repairable in one direction + +``` +PUT: bytes → staging prefix → accounting/row commits → copy onto the final key +DELETE: object first → row second +``` + +The forward order leaves, on a crash, a **row without an object** — which reconcile drops. +The reverse order leaves an **unaccounted object**, which that same reconcile pass *adopts* — +resurrecting a key the operator just purged. That asymmetry is the whole argument, and it is +why purge is deliberately not a delete-by-prefix. + +`blobx` runs the same shape at finer grain +(`family::sourcehut-artifacts/blobx/server.go::handlePut`): the body streams into a **unique +staging key outside the serving namespace** *before* the path lock is taken, so a slow upload +cannot block the currently published blob; then, inside the lock, the staged object is copied +to a **pending key named by `(final key, sha256)`**, the row is committed, the pending object +is promoted onto the final key, and the pending object is removed. A promotion that cannot +find its pending object leaves the old final object untouched "rather than exposing +mismatched bytes and checksum". A metadata write whose outcome is unknown *preserves* the +pending key and logs it for recovery. + +**GC must re-check the TTL before it deletes bytes, not after.** The candidate list is +already stale when it is used, so a PUT arriving after the list was taken commits a fresh +`last_access`, publishes its bytes, and has them removed by the pass a moment later — *after +the client was told 201 Created*. The TTL re-check then found the row fresh, kept it, and +counted the candidate as refreshed, so **nothing even reports the loss**. Make the +conditional row delete the *first* step so the TTL re-check is the decision, take it and the +object deletion under the namespace lock, and bound the deletion with its own timeout (30s) +— a bucket that stops answering would otherwise hold the lock, and with it every upload to +the namespace, for as long as the request hangs. The commit that fixed this notes: "the +comment defending the old order was wrong twice." + +### The torn-publication guard + +A publication is a row and an object, written at different moments. The read has to tell a +current object from one a crashed publication left behind — **without a second round trip**, +because "a read that asks twice holds the path lock for two round trips instead of one, which +is what starves publications" (measured: the wait reached the 10s lock ceiling and answered +503, which turned CI red). + +The mechanism is a sha256 stamped as S3 user metadata (`x-amz-meta-sha256`) on the pending +copy — the one place the digest first exists, since a streamed upload cannot know its own +digest until the body has been read. Every later copy carries it forward. The read +(`family::sourcehut-artifacts/blobx/server.go::servedObject`) opens the object and compares +`object.Digest` against the row's; agreement means one question asked and nothing to repair, +disagreement triggers the repair path. + +**Length cannot stand in for the digest** — two publications of the same path routinely have +the same size, so a stale object is invisible to a length check. The regression test +overwrites with equal-length payloads on purpose and asserts zero torn reads. + +Reads take the path lock in **shared** mode; mutations exclusively. + +### Other measured details + +- **A truncated listing page with no continuation token is an error, not the end** — silently + ending a reconcile walk drops the accounting rows of every namespace behind the cut. Name + the source that minted a cursor (`bucketCursorSource = "bucket"`): an S3 continuation token + and a row key cannot be told apart by shape, and Garage answers **400 InvalidRequest** to a + key offered in a token's place. +- **Canonicalise an identifier before it becomes a cache key.** `strconv.Atoi` accepts `+42` + and `0042`, so one resource had unboundedly many addresses — "many cache keys, many + rate-limiter buckets, many log lines." +- **Bound every pass, and do not let a bounded pass claim a sweep.** GC 200/batch, 20 000 + entries/pass; reconcile 64 namespaces/pass, 1000 per list page. The "swept" marker is + written only by a pass that reached the end of the backlog. +- **A brief PostgreSQL or bucket outage is 503 + `Retry-After: 1`, not 500.** +- **Do not hand-build the fake store's transcript.** The in-memory test store paged a + delimited listing by the next key — which reports a group a second time — and agreed with + itself until it was held to the transcript Garage produced for the same eight keys. + +--- + +## 4. S3 against Garage + +Every store in the family builds its own client. Copy this block exactly +(`family::sourcehut-artifacts/cachex/storage.go::NewS3Store`): + +```go +api := awss3.New(awss3.Options{ + Region: region, + Credentials: credentials.NewStaticCredentialsProvider(access, secret, ""), + UsePathStyle: true, // Garage has no per-bucket DNS + BaseEndpoint: aws.String(endpoint), + RequestChecksumCalculation: aws.RequestChecksumCalculationWhenRequired, + ResponseChecksumValidation: aws.ResponseChecksumValidationWhenRequired, +}) +uploader := manager.NewUploader(api, func(u *manager.Uploader) { + // One body is one client stream: parts cannot be produced ahead of the + // reader, so extra upload concurrency would only multiply the buffered + // part size per in-flight request. + u.Concurrency = 1 +}) +``` + +Why each line: + +- **`*ChecksumCalculation`/`*ChecksumValidation = WhenRequired`.** The SDK's newer default is + to add a CRC32 trailer to every request and demand one on every response. `WhenSupported` + also **seeks the request body** before `PutObject`, so a non-seekable streamed body fails or + is buffered whole into memory. Asking for checksums only where the protocol requires them + keeps the client to what an S3-compatible store must implement. +- **`UsePathStyle: true`** — Garage has no per-bucket DNS. +- **`manager.Uploader` with `Concurrency = 1`** — stream through it rather than calling + `PutObject` with a non-seekable body. Also validate bucket and credentials as non-empty at + construction; several copies in the family forgot to. + +### Relationship to the upstream `core-go-checksum` patch + +`mirror::core-go/objects/middleware.go::NewClient` builds its client with SDK defaults, which +breaks streaming `PutObject` against Garage (pages publish, builds artifact upload). The +production instance carries a patch, mirrored at `mirror::patches/core-go-checksum.patch` +(and live in the fork at `family::sr-ht-core/objects/middleware.go::NewClient`). + +**Read the patch, not its README summary.** The README leads with the checksum flags; the +diff's actual fix is a middleware swap plus a signing-region pin: + +```go +}, s3.WithSigV4SigningRegion(region), func(opts *s3.Options) { + opts.APIOptions = append(opts.APIOptions, func(stack *smithymw.Stack) error { + _, err := stack.Finalize.Swap("ComputePayloadHash", &v4.UnsignedPayload{}) + return err + }) +``` + +The flags alone are not enough: the SDK's default dynamic-payload middleware only uses +`UnsignedPayload` over **HTTPS**; over HTTP (`s3-insecure=true` to an internal Garage) it +falls through to `ComputePayloadSHA256`, which seeks the body and fails. +`WithSigV4SigningRegion` pins the region so the V1 endpoint resolver does not override it +back to `"default"`. + +**Service #11 does not need the patch** — it does not use `core-go/objects.NewClient` at all. +The block above plus `manager.Uploader` avoids the bug: the uploader buffers each part into a +seekable byte slice before signing, so the seeking middleware has something to seek. (That +last sentence is inference from the uploader's part-buffering behaviour, not a claim any +comment in the family makes — but no store in the family carries the swap and all of them +work.) + +### Never echo the endpoint with `%q` + +An S3 endpoint can carry credentials in its userinfo, and the ordinary "invalid S3 endpoint +%q" error puts them in a startup log. The correct variant is +`family::sourcehut-curator/blobx/blobx.go::normalizeEndpoint`, which **refuses** an endpoint +carrying userinfo and never echoes the value, with the trap named in its doc-comment: + +> `url.URL.Redacted` is not a way out: it masks the password and leaves a bare username, +> which is the shape a token takes. + +Config keys (`family::sourcehut-artifacts/config.example.ini`) — a bare `host:port` is +accepted and defaulted to `http://`: + +```ini +s3-upstream=garage:3900 +s3-access-key=... +s3-secret-key=... +s3-bucket=artifacts +s3-bucket-cache=docker-cache +cache-default-ttl=2160h +cache-default-quota=50G +``` + +--- + +## 5. HTTP cache correctness + +The family uses exactly **four** policies. There is no `stale-while-revalidate`, no +`s-maxage`, and no `Age` handling. + +| Policy | Header | Where | +|---|---|---| +| Private (the default) | `Cache-Control: private, no-store` + `Vary: Cookie, Authorization` | every credential-dependent surface | +| Public, revalidate always | `Cache-Control: public, max-age=0, must-revalidate` | resolved-public objects, public mirror paths | +| Immutable | `Cache-Control: public, max-age=31536000, immutable` | hashed assets; goproxy version artifacts | +| Anonymous 404 | `Cache-Control: no-cache, max-age=0` | a 404 that carried neither cookie nor `Authorization` | + +### Rule 1 — mount the private policy once, on the router + +```go +r.Use(middleware.RecoverPanics(render)) +r.Use(middleware.PrivateCache) // second, INSIDE RecoverPanics +``` + +`family::sr-ht-ecore/middleware/middleware.go::PrivateCache` / +`::SetPrivateCache` is the whole login-cookie story and it is one line. Inside +`RecoverPanics` "so that the error page it renders carries the same headers as any other +answer." Not in the render path, because "render is not the only writer: /healthz is +text/plain, static assets are bytes, a badge is an image, and a header this important must +not depend on which write path a future page picks." `SetPrivateCache` is exported +separately for the places the middleware chain does not reach — the deny path called by the +authentication middleware in front of the whole mount, which otherwise answers a rotated +token with a bare `http.Error`: plain text, no headers, and free for a shared cache to keep. + +`no-store` rather than `no-cache`, deliberately: "no-cache still permits a *stored* copy… +`private` bars the shared caches from keeping it at all, `no-store` bars the private ones +from writing it down." `Vary` is the pair `Cookie, Authorization`, because both a browser +session and a bearer token can change the answer; naming one header is correct only on a +surface that genuinely reads only one (a bearer-only `/mcp` may `Vary: Authorization` alone). + +**Mount it on every router, including the ones that are not the web UI.** A REST API group, a +`/query` mount and a GOPROXY mount each build their own chi router; a `PrivateCache` that +lives only inside the site handler covers none of them. + +### Rule 2 — start private, let a *resolved* public object opt out + +`family::sourcehut-artifacts/pkgrepo/router.go::distributionCacheBoundary` sets +`private, no-store` **on the way in**, and only `setDistributionCacheControl` — after +visibility has actually been resolved — relaxes it. "Authentication middleware can reject a +machine credential before the object handler has resolved visibility. Start +conservatively." + +The failure mode this prevents is a handler that computes the right policy but never reaches +the line that sets it. + +The dual is a URL that must be public for anonymous callers and private for credentialed +ones — a status badge. Two correct answers exist: +`family::sourcehut-coverage/web/badge.go::badgeCacheControl` is the more elegant, a bare +`max-age=300` with **no `public`, no `s-maxage`, no `must-revalidate`**: RFC 9111 §3.5 +forbids a shared cache from storing the answer to a request carrying `Authorization` unless +one of those three is present, so omitting all three makes a bearer-fetched badge private *by +protocol default*, with no branch to get wrong. + +### Rule 3 — the header must be set before the response is committed + +Setting headers on the way in is safe **only if nothing downstream overwrites them** — say so +in the doc-comment, which makes the assumption self-checking ("gqlgen writes no +`Cache-Control` of its own, so unlike the MCP surface's these can be set on the way in and +stay set"). When something downstream *does* write `Cache-Control` — the MCP SDK's streamable +transport sets it with `Set` from inside the handler — the headers must be written at commit +time through a `ResponseWriter` wrapper, and that wrapper must hook **all three** commit +routes: + +| Hook | Why | +|---|---| +| `WriteHeader` | the obvious one | +| `Write` | net/http commits on the first `Write` and **silently drops** every header set after that. A handler that never calls `WriteHeader` is the ordinary case | +| `Flush` | `http.ResponseController` prefers a `Flush` **on the writer it was handed** over one reached through `Unwrap`, so without it the flush commits at the writer below and neither header is ever written — on exactly the answers that stay open longest | + +`Unwrap` is still needed, but **not for flushing**: once the wrapper has its own `Flush`, it +is what reaches the writer below for read/write deadlines, `Hijack` and `EnableFullDuplex`. +The folk claim that "without `Unwrap` the wrapper hides the flusher and SSE streaming breaks" +is false and has now been retracted in writing twice; it was "the sentence standing where the +bug was." + +`family::sr-ht-ecore/mcphttp/cache.go::PrivateCache` is the correct implementation — import +it rather than writing a seventh copy. Six hand-copied clones exist and two are still broken; +the comment in two of them claiming ecore's version "is not resolvable" is verifiably false. +If you must write your own: + +```go +func (w *cacheWriter) Flush() { + w.commit() + _ = http.NewResponseController(w.ResponseWriter).Flush() +} +``` + +Second-order effect worth knowing: the wrapper embeds the `http.ResponseWriter` *interface*, +so without a `Flush` method a `w.(http.Flusher)` assertion in an SDK fails outright — not +merely a missed header. + +**Test it over a real socket.** `httptest.ResponseRecorder` structurally cannot detect any of +this, because `Header()` hands back the live map: a header set *after* the commit reads back +exactly like one that reached the client, so a recorder test passes for exactly this bug. The +regression tests that pin it spin an `httptest.NewServer`, **flush before writing anything**, +and assert client-side. Both discoveries in this family came from *mutating the method away* +and watching every test stay green — that is the transferable technique, not reading the +code. A test named after the flush path that writes before flushing is vacuous. + +### Rule 4 — `immutable` only where the client can catch a wrong answer + +Legitimate in exactly two places: + +- **Hashed static assets** (`family::sr-ht-ecore/assets/assets.go::CacheControl`). The package + never computes a hash — `make css` cuts sha256 to eight hex digits and puts it in the + filename; ecore only recognises the shape, with **one** regexp for all assets, matching + `.mjs` alongside `.js` because an unknown extension "would quietly demote a hashed file to + the hour an unhashed one gets — the failure is silent and shows up only as traffic." +- **goproxy version artifacts.** "The protocol makes these three paths content-addressed by + the client's `go.sum`, so a shared cache holding one for a year cannot serve a wrong answer: + a different answer would fail the client's own check rather than pass unnoticed." The same + test, applied without an HTTP header, is why a docker manifest fetched **by digest** is + never revalidated while the same manifest fetched **by tag** is. + +`assets.Handler` is the one route allowed to opt out of the private policy, and it opts out +in full — sets `Cache-Control` *and* `Del("Vary")`, because "an asset served identically to +everybody but declared to vary on Cookie is an asset no shared cache will ever reuse, which +is the whole point of hashing its name." + +**And it stamps at commit time, not on entry — for a leak reason.** This is the sharpest +argument in the family: + +> Writing it onto the header map before delegating is the obvious spelling and it is wrong, +> because the header map outlives the handler that filled it: a panic anywhere after that +> line is recovered by whatever middleware renders the 500 — **a page carrying a viewer's +> login block** — into a response that already says `public, max-age=3600` with the `Vary` +> deleted. + +A public cache policy set early can be inherited by a private error page. Two neighbours of +the same package: refuse directories before the file server sees them (`/static/` would +otherwise publish the binary's whole inventory "as a public, hour-cacheable page"), and never +pass `os.DirFS("")`, which "does not mean 'no assets', it means 'serve the filesystem root', +and it is one unset config key away." + +### Rule 5 — an anonymous 404 may be shared, a credentialed one may not + +Relax `private, no-store` to `no-cache, max-age=0` **only** when the request carried neither +`Authorization` nor `Cookie`: "the 404 carrying a credential… may say something about who is +asking. The anonymous one says only that the name is not public." + +### Rule 6 — test the policy through the router + +Every handler-level test is structurally blind to a header the router contributes. A review +once asked for `Cache-Control` in a handler; measured through the router the headers were +already there, and what was actually missing was the *check* — "which is exactly how the page +came to look unprotected to a careful reader." Also register read routes for HEAD (chi's +`GetHead` middleware): a route registered for GET alone answers HEAD with 405 plus a kilobyte +of rendered error page, to "every cache revalidating what it holds." + +### Conditional requests: there are none + +`ETag` and `Last-Modified` are emitted on some mirror paths, and **no `If-None-Match` or +`If-Modified-Since` handling exists anywhere in the family** — a grep over all repos including +tests returns zero. `ETag` elsewhere is purely an S3-protocol detail used to assemble +multipart copies, never an HTTP validator; `http.ServeContent` is the only thing answering a +304, and only for an `os.DirFS` tree (an `embed.FS` file has a zero `ModTime`, so no +`Last-Modified` is emitted and a conditional request is unanswerable). + +So a `must-revalidate` policy in this family means every revalidation is answered with a full +200 body. **If service #11 honours `If-None-Match`, it is adding a new thing** — not +following the pattern. Do it deliberately, in the origin handler, comparing against the +object's own ETag before opening the body, and say in the commit that it is new. + +Content integrity travels in custom headers instead: `X-Checksum-Sha256`, +`Docker-Content-Digest`. A stale answer served because upstream was unreachable is marked for +the operator with `X-Mirror-Stale: true` / `X-Cache: stale` — "for the operator reading a +response, not for the go command, which has no opinion about it." + +--- + +## 6. CI caching is a different subject + +Caching a Go module tree or an assembled SCSS tree **between builds** is not application +caching. It is `cacher` / `art cache` in `.build.yml`, against the same `docker-cache` +bucket — see the **`sourcehut-ci-cacher`** skill, which also carries the three measured +poisoning bugs (a half-restored read-only module cache swallowed by `|| true`, `-trimpath` +clobbering `-modcacherw`, and `go mod download all` dirtying a tracked `go.sum`). Do not +reinvent any of it here. diff --git a/skills/sourcehut-custom-service/references/chrome.md b/skills/sourcehut-custom-service/references/chrome.md new file mode 100644 index 0000000000000000000000000000000000000000..3cb3b5efb1784d41fab49967d87ec991f0a805f8 --- /dev/null +++ b/skills/sourcehut-custom-service/references/chrome.md @@ -0,0 +1,504 @@ +# Rendering integrated pages from a Go service + +> Citations: `family::/::` — this instance's own repos; `mirror::::` +> — the upstream documentation mirror. Both roots are substituted at install time. Symbols, never +> line numbers. + +A Go service does **not** reimplement the nav. `sr-ht-ecore` is a shared Go module that ships the nav, the service switcher, the brand, the login box, the environment banner, the section-tab row, the head links, two listing partials, an error page and four template helpers as embedded `html/template` partials. All eight web services on this instance (`sourcehut-{artifacts,bench,compare,coverage,curator,dolt,specs,tokens}`) import it; none of them writes nav markup. + +The whole nav integration is one constructor call and one template line: + +```go +chromeSvc := chrome.NewService(conf, "widget.sr.ht") +``` +```html +{{template "srht-nav" .}} +``` + +Module path: `sourcecraft.dev/bigbes/sr-ht-ecore`. Packages you will use: + +| package | role | +|---|---| +| `chrome` | the shared partials, `Service` (startup) and `Page` (per request), `Funcs()` | +| `pages` | template discovery, layout+content composition, buffered `Render`, the error page | +| `assets` | hashed-asset discovery (`Resolve`) and serving (`Handler`) with the cache policy | +| `csrf` | the same-origin guard for mutating requests | +| `middleware` | `RecoverPanics`, `PrivateCache`, `StatusClientClosedRequest` (net/http only) | +| `chimw` | `GetHead`, `RenderRefusals`, `RequestLogger` (the chi-aware half) | + +--- + +## 1. Minimal working example + +Four files. This is `sourcehut-tokens/web` condensed; it compiles as written. + +### `web/templates/layout.html` + +```html + + + + + + {{.Title}} + {{template "srht-head-links" .}} + {{block "head" .}}{{end}} + + + {{template "srht-env-banner" .}} + + {{template "srht-sections" .}} +
+ {{template "content" .}} +
+ {{block "scripts" .}}{{end}} + + +``` + +`content` is `{{template}}` and must **never** become `{{block}}` — see §5. `head` and `scripts` are `{{block}}` because they are optional seams. `srht-sections` renders nothing when the service declares no `Sections`, so leaving it in costs nothing. + +### `web/templates/index.html` + +```html +{{define "content"}} +
+
+

Widgets

+ {{template "srht-repo-list" .Data.Widgets}} +
+
+{{end}} +``` + +### `web/server.go` + +```go +//go:embed all:templates +var tmplFS embed.FS + +//go:embed static +var staticFS embed.FS + +const cssGlob = "static/main.min.*.css" + +// viewData is the root value every template is executed against. Embedding +// chrome.Page promotes its fields onto the dot, which is what the shared +// partials read; the page's own payload lives under Data. +type viewData struct { + chrome.Page + Data any +} + +func New(conf ini.File, svc *service.Service) (*Server, error) { + set, err := pages.Load(tmplFS, pages.Options{Funcs: funcMap()}) + if err != nil { + return nil, err // a page with no "content" fails the boot + } + + cssHref, err := assets.Resolve(staticFS, cssGlob, assets.DefaultPrefix) + if err != nil { + return nil, err // only ever a malformed glob + } + if cssHref == "" { + slog.Warn("no stylesheet embedded in this binary; pages will render unstyled", + "glob", cssGlob, "remedy", "run `make css` before `go build`") + } + + chromeSvc := chrome.NewService(conf, "widget.sr.ht") // the literal config section + chromeSvc.StyleHref = cssHref + if logo, err := assets.Resolve(staticFS, "static/logo.svg", assets.DefaultPrefix); err != nil { + return nil, err + } else if logo != "" { + chromeSvc.FaviconHref = template.URL(logo) + } + + staticSub, err := fs.Sub(staticFS, "static") + if err != nil { + return nil, err + } + s := &Server{svc: svc, chromeSvc: chromeSvc, pages: set} + // Built after the Server exists: the 404 for a name that is not embedded is + // this surface's own chrome-wrapped page. + s.static = assets.Handler(staticSub, assets.DefaultPrefix, + http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + s.renderError(w, r, http.StatusNotFound, "") + })) + return s, nil +} + +// view builds the frame for one request. username must be the *resolved* +// principal — "" for anonymous — so the nav and the page cannot disagree +// about who is looking. +func (s *Server) view(r *http.Request, title string) viewData { + username := authn.PrincipalFromContext(r.Context()).Username() + return viewData{Page: s.chromeSvc.Page(r, title, username)} +} +``` + +### the handler and the error renderer + +```go +func (s *Server) handleIndex(w http.ResponseWriter, r *http.Request) { + vd := s.view(r, "Widgets") + vd.Data = indexData{Widgets: chrome.RepoList{Items: items, Empty: "No widgets yet."}} + if err := s.pages.Render(w, http.StatusOK, "index", vd); err != nil { + slog.ErrorContext(r.Context(), "render index", + "method", r.Method, "path", r.URL.Path, scribe.Err(err)) + } // the response is already answered — log only, never call fail +} + +func (s *Server) renderError(w http.ResponseWriter, r *http.Request, status int, message string) { + vd := s.view(r, http.StatusText(status)) + vd.Data = pages.Error(status, message).BackTo("/widgets", "Back to your widgets") + if err := s.pages.Render(w, status, pages.ErrorPage, vd); err != nil { + slog.ErrorContext(r.Context(), "render the error page", + "method", r.Method, "path", r.URL.Path, "status", status, scribe.Err(err)) + } +} +``` + +### `web/router.go` — middleware order is load-bearing + +```go +func (s *Server) Handler() http.Handler { + r := chi.NewRouter() + + r.Use(middleware.RecoverPanics(func(w http.ResponseWriter, r *http.Request, _ any) { + s.renderError(w, r, http.StatusInternalServerError, "") + })) + r.Use(middleware.PrivateCache) + r.Use(csrf.Require(s.chromeSvc.SelfOrigin(), func(w http.ResponseWriter, r *http.Request) { + s.renderError(w, r, http.StatusForbidden, csrf.Message) + })) + + chimw.RenderRefusals(r, s.renderError) // chi's 404 and 405 → the service's page + + chimw.GetHead(r, "/healthz", s.handleHealthz) + chimw.GetHead(r, assets.DefaultPrefix+"*", s.handleStatic) + chimw.GetHead(r, "/", s.handleIndex) + r.Post("/widgets", s.handleCreate) + return r +} +``` + +The order is argued in `family::sr-ht-ecore/middleware/middleware.go` (package doc): + +- `RecoverPanics` outermost, so it covers the later middleware as well as the handlers. A `nil` render callback panics at construction — a wiring mistake must not surface at 3am inside a deferred function. +- `PrivateCache` inside it, so the error page it renders carries the same `private, no-store` + `Vary: Cookie, Authorization` as any other answer. +- `csrf.Require` after the cache headers, so a refusal is `private, no-store` too. It runs **before routing**, so an unrouted POST is refused rather than 404'd — a 404 there would enumerate which routes exist without ever passing the check. +- `chimw.RequestLogger` is the one that goes **outside** `RecoverPanics` (`family::sr-ht-ecore/chimw/logger.go::RequestLogger`), so it observes the 500 that was actually rendered — and after chi's `RequestID`/`RealIP`. +- `chimw.RenderRefusals` is a registration on the routing tree, not a link in the chain; install it once on the root and every sub-router inherits it. + +Mutating routes deliberately do **not** go through `GetHead` — a HEAD that writes is not a HEAD (`family::sr-ht-ecore/chimw/chimw.go::GetHead`). + +--- + +## 2. `chrome.Page` — the fields a template sees + +From `family::sr-ht-ecore/chrome/chrome.go::Page`: + +``` +Title string // +SiteName string // instance brand, from [sr.ht] site-name +SiteLabel string // red service suffix: Section minus ".sr.ht" +Nav []NavItem // the service switcher {Name, Origin, Active} +ExtraNav []NavItem // DEPRECATED — see §4 +Tabs []SectionTab // this service's own tab row, Active resolved +Username string // "" = anonymous +LoginURL string // meta login, return_to = the current URL +LogoutURL string // meta logout, return_to = this service's origin +RegisterURL string // meta origin +ProfileURL string // hub's ~username when hub is configured, else meta /profile +MetaOrigin string +SelfOrigin string +HubOrigin string // "" when the instance has no hub +StyleHref string // "" when built without `make css` +FaviconHref template.URL // "" renders no <link> +Assets map[string]string // extra hashed hrefs: {{index .Assets "uplot.js"}} +Environment string // uppercased +ShowBanner bool // environment != "" && != "production" +ContainerClass string // "container" | "container-fluid", per request +``` + +Two facts that bite: + +- **Embedding names the field `Page`.** A view struct that wants its own `Page` (a pagination counter) gets a compile error, not a silent shadow. Rename yours `PageNum`. +- **`FaviconHref` is `template.URL`.** `html/template` rewrites an `href` whose scheme is not http/https/mailto to `#ZgotmplZ`, and `chrome.DefaultFaviconHref` is a `data:` URI. The type is also the guard: a request value cannot land in it by accident. + +Page payload goes under `.Data`, never as a sibling field of the chrome — a page that wanted a `Username` of its own would otherwise silently replace the one the login block reads. + +--- + +## 3. The shared partials + +Parsed into every set by `pages.Load` (`family::sr-ht-ecore/chrome/templates.go::Attach` plus the error partial from `pages`). + +| partial | dot | renders | +|---|---|---| +| `srht-head-links` | `Page` | guarded `<link>` for stylesheet + favicon | +| `srht-env-banner` | `Page` | the green non-production strip; place first in `<body>` | +| `srht-nav` | `Page` | brand + switcher + login box — the navbar's **inner** content | +| `srht-sections` | `Page` | `.header-tabbed > .container > ul.nav.nav-tabs`; **owns its wrapper** | +| `srht-repo-list` | `chrome.RepoList` | the family's `.event-list` cards | +| `srht-repo-table` | `chrome.RepoList` | the same data as aligned columns | +| `srht-error` | `pages.ErrorData` | status heading, message, way back | + +`srht-nav` renders inner content because the services disagreed about the `<nav>`'s own classes; `srht-sections` owns its wrapper because nobody disagrees about that row, and a service that writes the wrapper is a service that can write it differently. + +The listing dot: + +```go +chrome.ListItem{Href, Title, Visibility, Description string; Updated time.Time; Meta []string} +chrome.RepoList{Items []ListItem; Empty string} +``` + +`Updated` is a `time.Time`, not a preformatted string, so the partial renders `reltime` with `abstime` in the `title` attribute. A zero `Updated` and a nil `Meta` render nothing — which is what lets a service with no timestamp in its schema use the same partial. `Visibility` is compared against the literals `"PUBLIC"` and `"UNLISTED"`. Adopters: **curator** (`sourcehut-curator/web/index.go` builds `chrome.RepoList`, `web/templates/index.html` invokes `srht-repo-table`), compare, dolt, specs. + +The error page ships with `pages` and every service takes it: `pages.Error(status, message)` → `ErrorData{Status, StatusText, Message, Back}`, chained with `.BackTo(href, text)`. Shared sentences: `NotFoundMessage`, `UnauthorizedMessage`, `ForbiddenMessage`, `MethodMessage`, `InternalMessage`, `UnavailableMessage`, plus an `API*Message` set and `pages.Message(status)` / `pages.APIMessage(status)`. **There is no constant for 400** and that is deliberate: a 400 describes something the viewer just typed, and a house phrase would send them back to the form with nothing to change. The 404 wording is shared on purpose — these surfaces must make "somebody else's private thing" and "no such thing" indistinguishable. + +--- + +## 4. The nav, the switcher, and your own tab row + +### Membership + +`family::sr-ht-ecore/chrome/chrome.go::BuildNav`: + +1. every config section whose name ends in `".sr.ht"` +2. minus `navExcluded` = `paste`, `pages`, `hub` +3. minus any section with no resolvable origin (`config.GetOrigin(conf, section, true)`) +4. ordered by `navCanonical` = `hub, git, hg, lists, todo, builds, man, meta`, then alphabetically +5. `Active` on the section equal to the service's own + +Against upstream (`mirror::core.sr.ht/srht/app/flask.py::_network`, rendered by `mirror::core.sr.ht/srht/templates/nav.html::for _site in network`): membership is identical (`.sr.ht` suffix), exclusions match except that hub is dropped in `navExcluded` rather than inside the loop, and the switcher is shown to authenticated viewers only — same as upstream's `current_user` gate. Ecore diverges in two harmless ways: it orders alphabetically where upstream falls back to config insertion order, and it skips a section with no origin where upstream still lists it. + +**The `.sr.ht` suffix is the entire membership rule.** Your config section must be named literally `<name>.sr.ht` regardless of the host you serve from: + +```ini +[widget.sr.ht] +origin = https://widget.example.org +``` + +Pass that same literal to `chrome.NewService`. Use one constant (`service.ConfigSection` in tokens) rather than two string literals — a service that spelled its section differently in two places appears in the switcher and fails to recognise itself in it. + +### Name, label, colour + +There is no knob, and that is the policy: + +- `SiteName` comes from `[sr.ht] site-name`; `SiteLabel` is `strings.TrimSuffix(Section, ".sr.ht")` — the config section chooses the label. +- No colour hook. The label is always `text-danger`, wrapped in a `<span>` rather than being a red `<a>`, because the theme colours `.navbar-light .navbar-brand a` and would repaint it white in dark mode. +- The brand carries **two** links where upstream has one: the site name → hub (falling back to `/` when the instance has no hub), the red label → `/`. Hub is excluded from the switcher, so the brand is the only route to it. +- The brand has a fixed `min-width: 15rem` so the switcher starts at the same x-coordinate on every service. +- Your freedom is everything below `@import "base"` in `scss/main.scss`, plus `FaviconHref`, `ContainerClass`, `Sections` and `Assets`. That is the complete list. + +### Your own tab row: `chrome.Section` + +```go +chromeSvc.Sections = []chrome.Section{ + {Name: "channels", Href: "/", Paths: []string{"/"}, Prefixes: []string{"/~"}}, + {Name: "images", Href: "/images", Paths: []string{"/images"}}, + {Name: "cache", Href: "/cache", Paths: []string{"/cache"}, Prefixes: []string{"/cache/"}}, + {Name: "mirrors", Href: "/mirrors", Paths: []string{"/mirrors"}}, +} +``` + +That is `sourcehut-artifacts/web/server.go`, the exemplar; `sourcehut-curator` is the second adopter. Matching, in `family::sr-ht-ecore/chrome/chrome.go::Section.matches`: + +- `Paths` match as **path segments**: `"/mirrors"` covers `/mirrors` and `/mirrors/alpine/rules`, not `/mirrorsomething`. The service root `"/"` matches only itself — a root treated as a prefix would light its tab on every page. +- `Prefixes` match as **literal string prefixes**, for the shapes a segment boundary cannot express: `"/~"` covers `/~owner/name`. +- A path in no section lights **nothing** rather than falling back to the first tab. +- The row is empty for an anonymous viewer, following the switcher's rule. + +Markup and classes are upstream's (`.header-tabbed` + `.nav.nav-tabs`, what meta uses for profile/security/keys and git for a repo's tree/log), so the row costs your service no CSS. + +**`Service.ExtraNav` is `Deprecated:` with no correct use left.** Both historical uses were the same mistake — putting a page of one service into the row that lists the instance's services. Your own pages go in `Sections`. Do not add entries to `ExtraNav`. + +--- + +## 5. Startup invariants `pages.Load` enforces + +`family::sr-ht-ecore/pages/pages.go::Load` takes an `fs.FS` and `pages.Options`. The zero `Options` is the family convention: `Dir="templates"`, `Layout="layout.html"`, `PartialPrefix="_"`, `ContentBlock="content"`, `Funcs` merged **over** `chrome.Funcs()`. + +Discovery: every `*.html` directly under `Dir` is a page, except `layout.html` and anything starting with `_`. **Adding `templates/foo.html` is the whole registration of a page** — there is no list to update, which is what a hand-maintained slice cost the donors. + +Each page gets **its own set** — every page defines `"content"`, and a shared set would let the last parse win. A set contains, in order: `chrome.Attach` (partials + `chrome.Funcs`), the service's `Funcs` layered on top, the shared `srht-error` partial, `layout.html`, **every** `_*.html` partial (not just the ones a page uses today), then that one page's `.html` **last**, so its `content` wins over any default the layout carries. If the FS ships no `error.html`, `Load` registers the shipped one under `pages.ErrorPage` (`"error"`). + +| refusal | error | why | +|---|---|---| +| a page defines no `content` | `pages: template %s defines no "content" block: %w` (`pages.ErrNoContent`) | executed, it renders the chrome around an empty hole and answers **200** — the one failure a viewer cannot report usefully | +| the layout file is missing | `pages: no layout %s in %s` | `ParseFS`'s own message for a pattern matching nothing never mentions that the missing file is the layout | +| no page templates in the directory | `pages: no page templates in %s` | only reachable if an embed pattern stops matching; otherwise the symptom is a daemon that boots happily and 500s on every route | +| a template does not parse | `pages: parse template %s: %w` | a broken template must fail the deploy, not the first viewer | +| the chrome partials do not parse | `pages: attach the shared chrome partials: %w` | reported with a sentence naming what it was doing | +| `RecoverPanics(nil)` / `RenderRefusals(nil)` | panic at construction | a wiring mistake must not surface inside a deferred function at 3am | +| `csrf.Require` with an unparseable origin | no panic — **fails closed**, every mutating request refused | a guard comparing against nothing admits everyone | + +**Why the content check parses each page standalone** (`Options.definesContent`): the obvious check — looking `"content"` up in the assembled set — works only while `layout.html` spells its hole `{{template "content" .}}` while its neighbours are `{{block "head"}}` / `{{block "scripts"}}`. `block` *defines* the name it invokes. The day somebody makes the three consistent — a tidying edit no reviewer would question — `Lookup` starts finding the layout's own empty default on every page and the guard silently stops guarding, producing exactly the failure it exists to prevent. Parsed on its own, a page has only what it defines itself, and no edit to the layout can reach it. Cost: one extra parse per page, once, at startup. + +**So: `{{template "content" .}}` in your layout. Never `{{block "content" .}}`.** + +Two things deliberately **not** startup failures: a missing stylesheet (`assets.Resolve` answers `""`; a service that will not boot without a build artefact cannot be run from a checkout) and a missing favicon (`NewService` has already put a `data:` URI there). + +--- + +## 6. Rendering discipline + +`family::sr-ht-ecore/pages/pages.go::Set.Render` executes into a `bytes.Buffer`, then writes `Content-Type`, `WriteHeader(status)`, and the body. + +1. **Render into a buffer, always.** `html/template` writes as it evaluates, so executing straight into the `ResponseWriter` commits the status line and however many kilobytes of chrome were produced before the expression that failed. On these surfaces the missing half is sometimes the one carrying a secret shown exactly once. +2. **A returned error means the response has already been answered.** Log it and do nothing else. Handing it to your own `fail()` either writes a second response over a committed one, or — when the error page itself is what broke — recurses until the stack runs out. On failure `Render` writes a bare 500 carrying the fixed string `internal server error`; never publish `err.Error()` to the browser, it names templates, field paths and whatever the payload's `String` produces. +3. The log line the family writes is `slog.ErrorContext(r.Context(), …, "method", r.Method, "path", r.URL.Path, scribe.Err(err))` — `ErrorContext` because `chimw.RequestLogger`'s request id lives in the context, and a render failure that cannot be correlated with its request line is half a record. +4. `pages.ErrUnknownPage` is returned for a name not in the set. It is always a bug in the calling package — the set is built from the files that exist — and is worth recognising in a log filter. + +Your own `fail(w, r, err)` maps your domain sentinels onto statuses and calls `renderError`. That mapping deliberately stays in the service: two surfaces of one service must agree about which object exists, and that agreement is a property of the domain. Use `middleware.StatusClientClosedRequest` (499) for `context.Canceled` / `DeadlineExceeded` — the viewer went away, nothing is broken, and a 500 there pages somebody for a healthy service. + +--- + +## 7. Theme and assets + +### The SCSS is compiled per service, from upstream's sources, not vendored + +`scss/main.scss` is one file that opens with `@import "base"` and adds only your own rules. `base` is upstream `mirror::core.sr.ht/scss/base.scss` — Bootstrap 4 plus the SourceHut chrome (contrast, variables, nav, events, dark, icons, highlight). The partials ship with no package manager; they are materialized at build time under `$(ASSETS)/scss`, i.e. `/usr/share/sourcehut/scss`, by core.sr.ht's own `make install` locally and by the `scss:` task of `.build.yml` in CI: + +```yaml +- scss: | + cacher dir download "scss/${CORE_VER}-${BOOTSTRAP_REV}.tar.zst" ~/scss --exec ' + git clone --depth 1 --branch "$CORE_VER" \ + https://git.sr.ht/~sircmpwn/core.sr.ht /tmp/core + mkdir -p ~/scss/bootstrap + cp -f /tmp/core/scss/*.scss /tmp/core/scss/*.css ~/scss/ + git init -q /tmp/bootstrap + git -C /tmp/bootstrap remote add origin https://github.com/twbs/bootstrap + git -C /tmp/bootstrap fetch -q --depth 1 origin "$BOOTSTRAP_REV" + git -C /tmp/bootstrap checkout -q FETCH_HEAD + cp -rf /tmp/bootstrap/scss ~/scss/bootstrap/scss + ' + sudo mkdir -p /usr/share/sourcehut + sudo cp -rf ~/scss /usr/share/sourcehut/scss +``` + +`CORE_VER` must track the deployment's `SRHT_CORE_VER` (`phoebe-lab/srht/versions.env`); `BOOTSTRAP_REV` is the submodule commit core.sr.ht pins at that tag. Both are cache-key components, so an upstream outage cannot fail a build. The `--exec` argument is in **single** quotes on purpose — `"$CORE_VER"` must reach `sh -c` unexpanded — and the cache key is spelled inline because `--exec` sees exported variables only. + +### `make css` + +`sourcehut-curator/Makefile` is the exemplar (preflight checks + a `SHA256SUM` variable, so it works on macOS as `make SHA256SUM="shasum -a 256" css`): + +```make +SHA256SUM ?= sha256sum +ASSETS ?= /usr/share/sourcehut +SASSC ?= sassc +MINIFY ?= minify +SASSC_INCLUDE = -I$(ASSETS)/scss + +css: + @command -v $(SASSC) >/dev/null 2>&1 || { echo "error: $(SASSC) not found — apk add sassc"; exit 1; } + @command -v $(MINIFY) >/dev/null 2>&1 || { echo "error: $(MINIFY) not found — apk add minify"; exit 1; } + @[ -f $(ASSETS)/scss/base.scss ] || { echo "error: no $(ASSETS)/scss/base.scss — run core.sr.ht's 'make install'"; exit 1; } + mkdir -p web/static + rm -f web/static/main.css web/static/main.min.*.css + $(SASSC) $(SASSC_INCLUDE) scss/main.scss web/static/main.css + $(MINIFY) -o web/static/main.min.css web/static/main.css + mv web/static/main.min.css \ + web/static/main.min.$$($(SHA256SUM) web/static/main.min.css | cut -c1-8).css + rm -f web/static/main.css +``` + +The `rm -f web/static/main.min.*.css` prologue is not optional: `assets.Resolve` takes the **first** glob match, so two hashed stylesheets in the tree mean the one served depends on the lexical order of two hex digests. Hardcoding `sha256sum` is a real bug on macOS/BSD — the substitution yields empty, `mv` produces `main.min..css`, which does not match the glob, and every page renders unstyled with nothing in the log. Gitignore the compiled CSS: it is a build product, and a tracked one gets deleted by `make css`, dirtying the tree and stamping every later `go build` `vcs.modified=true`. + +Build order is `make css`, **then** `go build`, then restart — the CSS is embedded into the binary. + +### Runtime discovery and serving + +`family::sr-ht-ecore/assets/assets.go::Resolve(fsys, glob, urlPrefix)` globs and returns `NormalizePrefix(urlPrefix) + path.Base(matches[0])`, or `""` when this build produced none. Absence is `""` and not an error — the `error` return is only ever a malformed glob. Warn on `""` naming the make target; do not refuse to boot. + +Extra hashed assets (a chart bundle, a front-end bundle) go into `Service.Assets` and are read as `{{index .Assets "uplot.js"}}` — the hash is a property of the binary, not of any page. + +`family::sr-ht-ecore/assets/assets.go::Handler(fsys, urlPrefix, notFound)`: + +- `hashedRe` = `\.[0-9a-f]{8,}\.(css|m?js)$` — **one** pattern, not one per asset: what makes a file cacheable forever is the hash in its name, not which build step produced it. `.mjs` is matched alongside `.js` because an unknown extension would silently demote a hashed file to the short lifetime. +- `CacheControl`: `public, max-age=31536000, immutable` for a hashed name, `public, max-age=3600` for anything else (favicon, logo). +- `Lookup` stats the same FS the file server reads, so a **directory** is never served (a `/static/` listing would publish the build's stylesheet hash) and a missing file goes to `notFound` — point it at your chrome-wrapped 404. +- `.js`/`.mjs` get an explicit `Content-Type: text/javascript; charset=utf-8`, because `mime.TypeByExtension` is seeded from the **host's** tables and a browser refuses a module script whose type is not a JavaScript MIME type — same binary, different image, chart missing. +- The public policy is written by a `ResponseWriter` wrapper at the moment the answer commits, never onto the header map beforehand: a panic after that line would otherwise be recovered into a 500 page carrying a viewer's login block under `public, max-age=3600`. `Vary` is `Del`'d, not overwritten. + +### embed vs `os.DirFS` + +The FS is a parameter everywhere. Embed (`//go:embed all:templates`, `//go:embed static`) is the default — seven of eight services. Use `assets.DirFS(cfg.StaticDir)` if you serve a static tree from disk; **never** `os.DirFS(dir)` directly, because `os.DirFS("")` resolves every name against the filesystem root, so one unset config key turns the static handler into a reader of the host. `assets.DirFS("")` returns an empty FS instead. + +Use `//go:embed all:templates`, not `templates/*.html`: without the `all:` prefix the directory walk drops every file whose name begins with `_`, which is exactly how this family names its partials. A glob form also silently misses a subdirectory or a non-`.html` fragment, and the failure is "no such template" at request time. + +### The empty-href guard + +`<link rel="stylesheet" href="">` **re-requests the page it sits on** — one extra page load per page load, for nothing. So `""` must be guarded, never emitted. `srht-head-links` does it: + +``` +{{if .StyleHref}}<link rel="stylesheet" href="{{.StyleHref}}">{{end}} +{{if .FaviconHref}}<link rel="icon" href="{{.FaviconHref}}">{{end}} +``` + +A service adding a second asset owes its own template the same `{{if}}`. And resolve a favicon through `assets.Resolve`, not as a literal path: a path a template asserts is one that 404s on every page load if the file is renamed or hashed. Leaving `FaviconHref` unresolved keeps `chrome.DefaultFaviconHref`, a `data:` URI that costs no request and cannot 404. + +--- + +## 8. Forms and mutations + +### There is no CSRF token, and nowhere to keep one + +Identity is meta.sr.ht's unified-login cookie, set on the parent domain; no individual service issues it and none can set its `SameSite`. A synchronizer-token scheme would mean every daemon inventing a session store for two forms. The guard is same-origin instead (`family::sr-ht-ecore/csrf/csrf.go::Require`): + +read `Origin`; failing that `Referer`; compare **scheme and host including port**, case-insensitively; **refuse a request that carries neither**. Only `GET`, `HEAD`, `OPTIONS`, `TRACE` are exempt — an unknown method is guarded, not waved through. `Origin`, when present, is consulted alone: treating `Referer` as a second chance turns the stronger statement into the weaker one. + +- Install it **on the whole router**, not per handler. The drift among the donors was not the wording — it was that in two of five services the default for a new POST route was unprotected. +- `deny` must answer **403, not 400** (the request is well-formed; what is missing is evidence the viewer asked for it) and **must not redirect** — a redirect after a POST drops the body and turns a refused mutation into a page that looks like it worked. Pass a closure rendering your own error page with `csrf.Message`; a `nil` deny falls back to a plain-text 403, which looks nothing like the rest of your surface. +- A bearer-token API in its own mux keeps its exemption by **not mounting this middleware there**. There is deliberately no "is this /api?" option and there must never be one — every escape, case fold and dot segment would become a way to ask for the exemption. +- `csrf.SameOrigin(r, origin)` is the bare predicate, exported only for a route outside the guarded router. Prefer `Require`. + +### Reading the body + +```go +form, err := pages.FormValues(w, r, pages.DefaultMaxFormBytes) // 64 KiB +``` + +`family::sr-ht-ecore/pages/form.go::FormValues` returns **`r.PostForm` and never `r.Form`**, and that is the whole reason a three-line function is shared. `r.Form` merges the query string into the body's values, so a mutation could be driven entirely from a URL somebody was linked to — precisely the request the same-origin guard sees nothing wrong with, because it really did come from our own page. Map `pages.ErrInvalidForm` onto 400. The 64 KiB bound replaces net/http's 10 MiB ceiling. + +### After the mutation + +- **Redirect** (`http.StatusSeeOther`) when the redirect target already says everything the mutation changed: a reload re-reads instead of re-posting. +- **Render in place** when the response carries something that exists only in it — a minted secret. A redirect would have to carry it in a URL, a flash cookie or a session, all three of which outlive the response and are readable by somebody other than the person who asked. `sourcehut-tokens/web/tokens.go` has both: `handleRevoke` redirects, `handleMint` renders and documents the accepted cost. +- **Never answer a POST from an anonymous viewer with a login redirect.** It throws the body away and the viewer comes back logged in to a page that did nothing. Answer a 401 page. A `requireLogin` redirect is for GETs only. + +### There are no flash messages + +None of the eight services has one, and there is no session store to keep one in. State after a mutation is carried by rendering in place or by the redirect target re-reading the data. + +--- + +## 9. Template helpers + +`family::sr-ht-ecore/chrome/funcs.go::Funcs` is merged into every set by `pages.Load`: + +| helper | signature | notes | +|---|---|---| +| `dict` | `dict(kv ...any) (map[string]any, error)` | `{{template "x" (dict "A" .A "B" .B)}}`; odd count or non-string key is a render error | +| `shortsha` | `shortsha(string) string` | first 8 characters | +| `reltime` | `reltime(time.Time) string` | `"3 hours ago"` / `"in 3 weeks"` / `"just now"` | +| `abstime` | `abstime(time.Time) string` | `2006-01-02 15:04:05 UTC` | + +`reltime` faces **forward** deliberately: the copies it replaced disagreed about the future, printing "in 3 hours" on one service and "just now" on the next for the same instant. Its `<1 minute` window covers both a just-past and a just-future instant, which is also what two unsynchronised clocks produce. + +Your `Options.Funcs` are merged **over** these, so a name can be shadowed. **Do not shadow one.** No service currently registers `dict`, `shortsha`, `reltime` or `abstime` over chrome's, and the layering exists for a deliberate override, not an accidental one — two spellings of `reltime` on one instance is the exact drift `chrome.Funcs` was hoisted to end. Keep your own map short: each entry should be a formatting rule this surface fixes once, or something a template genuinely cannot express (`tokens`' `deref`, because `html/template` cannot dereference a `*time.Time`). Presentation that belongs to one page belongs in that page's payload, computed in Go where it can be tested. + +--- + +## 10. Where the family has no shared convention yet + +Two gaps. Do not invent a third spelling. + +**Pagination.** There is no shared partial and no shared vocabulary. Four services spell one control four ways: `Older`; `« newest` / `older »` in a service-local CSS class; `Previous page — Next page`; and `Older commits →` as a `btn btn-secondary`. Upstream ships `mirror::core.sr.ht/srht/templates/pagination.html` (prev / `page / total_pages` / next, `btn btn-default` plus caret icons); ecore ships nothing. If your service paginates, the right move is to land a shared `srht-pager` partial in `family::sr-ht-ecore/chrome/templates/chrome.tmpl` over a `chrome.Pager{PrevHref, NextHref, Page, TotalPages}` dot and use it — not to copy any of the four. The `inc` / `dec` helpers copied into three services exist only because this partial does not; land them in `chrome.Funcs` alongside it. + +**Relative time in a shape `reltime` does not cover.** Two services carry incompatible `ago` helpers under the same name — one short-unit and forward-facing (`"3h"`, `"in 3h"`), one long-unit and past-facing with a swappable clock. `chrome.Funcs` already provides `reltime`, which is the past-facing spelling with the future left forward-facing. **Use `reltime`.** If you genuinely need short units, add `shorttime` to `chrome/funcs.go` with a swappable clock so it is testable at the boundary — do not register a third `ago`. + +Also unguarded family-wide, and worth honouring anyway: **no Bootstrap 5 utility classes** (`text-end`, `ms-3`, …). The theme is Bootstrap 4; only `sourcehut-artifacts` enforces this, with `web/templates_alignment_test.go`. Nothing has drifted yet — keep it that way. diff --git a/skills/sourcehut-custom-service/references/config.md b/skills/sourcehut-custom-service/references/config.md new file mode 100644 index 0000000000000000000000000000000000000000..56f172cd8d31227c40aee841ae65374007317ebc --- /dev/null +++ b/skills/sourcehut-custom-service/references/config.md @@ -0,0 +1,479 @@ +# Configuration for a custom service on this instance + +Citations: `family::<repo>/<path>::<symbol>` for the sibling repos under the user's home +(`sr-ht-core`, `sr-ht-ecore`, `sourcehut-*`); `mirror::<path>::<symbol>` for the upstream +documentation mirror. Both prefixes are substituted at install time. Symbols, never line +numbers. `phoebe-lab/srht/…` is the deployment repo and is named without a prefix. + +Everything below was read out of the running code and the deployed template, not out of +upstream's example files. Where upstream defines a key that no service in this family +reads, it gets one line saying so rather than a table row. + +--- + +## 1. There is one `config.ini`, and you are appending to it + +Production is a **single instance-wide file**: `phoebe-lab/srht/config/config.ini.tmpl`, +`envsubst`-rendered by a one-shot `config-init` container into `/etc/sr.ht/config.ini` on a +shared volume every container mounts read-only. Your service reads the *same* file meta, git +and builds read. + +`mirror::core-go/config/config.go::LoadConfig` searches `./config.ini`, `../config.ini`, +`/etc/sr.ht/config.ini`, `/etc/sr.ht/*.ini` and stops at the **first location that matches +anything**. It never returns an error — a missing file yields a nil `ini.File` and every +lookup answers "absent". Nothing tells you the file was not found. + +- Your `config.example.ini` is **not a deployment artefact** — it is a dev-boot file plus + documentation. Only its `[myservice.sr.ht]` block is merged into the shared file; every + other section in it is a read-only reference copy of something another service owns. Say + that in the header comment, as `family::sourcehut-tokens/config.example.ini` and six + siblings do verbatim. +- **A config-template edit does not propagate.** `docker compose up -d` never re-renders an + already-exited `config-init`, and a push only recreates containers whose *image* moved. + Force-recreate `config-init`, restart every consumer, and verify by container start time + against the rendered file's mtime — not by asking a running process what it believes. +- `sourcehut-specs` ships **no** `config.example.ini`, and it is the donor for half the + daemon skeleton. Template from tokens or bench instead. + +--- + +## 2. A complete `config.example.ini` for a new service + +Copy this, replace `myservice`, delete the tuning keys you do not have. The prose comments +are the point — `config.example.ini` is where this family documents what reads a key and +what an absent value does (`family::sourcehut-bench/config.example.ini::[bench.sr.ht]` is +the long-form example, with measured derivations for every ceiling). + +```ini +; myservice.sr.ht configuration. +; +; In production this section is merged into the single shared instance +; config.ini (the same file every *.sr.ht service reads). Only the keys in the +; [myservice.sr.ht] section below are ours; the rest are shared keys owned by +; other services and referenced (NOT duplicated) here — they must already be +; present and consistent across the instance. +; +; The section name must be spelled exactly "[myservice.sr.ht]": the ".sr.ht" +; suffix is what puts the service into the shared nav of every other service on +; the instance. Consequently the [myservice.sr.ht] origin= key has to be present +; in the config.ini of *all* services, not only this one — each of them builds +; its nav from its own copy of the file, once, at startup. + +[myservice.sr.ht] +; +; The URL this service is served at (protocol://domain). Used for absolute +; links, for the nav entry other services render, as the origin every mutating +; web request is checked against (same-origin CSRF check), and as the Host +; allowlist of /mcp if you mount one. +origin=https://myservice.srht.bigb.es +; +; The address a sibling daemon dials from inside the instance's network, when +; that is not the public one. Optional; falls back to origin. Never put this +; string in front of a browser. +;internal-origin=http://myservice:5097 +; +; The base a federation gateway appends "/query" to. Optional: omit it and +; nothing breaks except federation. /query rides the web listener, so this is +; normally just the origin. +;api-origin=https://myservice.srht.bigb.es +; +; PostgreSQL connection string. Required if the service has a database. +connection-string=postgresql://myservice@localhost/myservice.sr.ht?sslmode=disable +; +; Address the HTTP listener binds to. `bind-address` is the only spelling. +; Behind nginx on a plain host this is loopback; in a container it must be +; 0.0.0.0 or the reverse proxy cannot reach it across the docker network. +; 5090 diff, 5091 spec, 5092 bench, 5093 cov, 5094 tokens, 5095 artifacts, +; 5096 go are taken; this one takes 5097. +bind-address=127.0.0.1:5097 +; +; Set to "yes" to run brant migrations automatically on package upgrade. +; Honored by `myservicesrht-migrate -a up`; the daemon itself never migrates. +migrate-on-upgrade=yes +; +; Verbosity of the daemon's own log: debug, info, warn or error. The weakest of +; three sources — $LOG_LEVEL overrides it for one run and -d overrides both. A +; value none of them can read falls through rather than refusing to start. +log-level=info +; +; --- Object storage, only if you store blobs ------------------------------ +; Note the section: these are OURS. core-go's own S3 client reads [objects] +; instead, and this family does not use it. +;s3-upstream=garage:3900 +;s3-access-key= +;s3-secret-key= +;s3-bucket=myservice +; +; AES-256 key, base64 of 32 bytes, sealing credentials this service stores. +; `myservicesrht keygen` prints one. Only if you seal anything. +;key-seal= +; +; Ceilings. Every max-* key states its derivation in the comment, measured on +; the target machine (two cores), not on a laptop, and a refusal over one names +; the config key in its message. +;max-database-connections=19 +;max-upload-body-bytes=134217728 + +; --------------------------------------------------------------------------- +; Shared keys reused in place (owned by other services, listed for reference; +; do not duplicate their values here — they live in the shared config.ini): +; +; [sr.ht] network-key Fernet key; decrypts the unified-login cookie and +; seals Internal authorization. Absent, +; crypto.InitCrypto EXITS the process — the daemon +; checks for it first and refuses readably instead. +; [webhooks] private-key same exit rule, checked even earlier, and required +; by a service that emits no webhooks. It also +; derives the instance bearer HMAC key: every +; tokens.sr.ht working token is signed with it, and +; rotating it invalidates all of them at once. +; [sr.ht] site-name the nav brand +; [sr.ht] environment non-"production" adds a banner. The chrome reads +; an ABSENT value as "development", so an instance +; that means production has to say so. +; [sr.ht] owner-name / owner-email config.GetOwner panics without them +; [sr.ht] global-domain the domain meta sets the login cookie on; this +; service's host must be under it +; [sr.ht] internal-ipnet subnets Internal auth is accepted from. Optional: +; the default covers the docker networks. +; [meta.sr.ht] origin login/logout redirects, profile lookup +; [tokens.sr.ht] origin the only issuer of machine credentials; +; internal-origin preferred for the revocation +; check. Absent, the daemon still starts and serves +; every read, and refuses every bearer credential. +; [hub.sr.ht] origin optional; usernames link to hub profiles +; [git.sr.ht] repos / api-origin optional; bare repos on disk / GraphQL +; +; There is no static-dir key: CSS and JS are go:embed-ed into the binary. +; --------------------------------------------------------------------------- +``` + +--- + +## 3. Field reference, section by section + +### `[myservice.sr.ht]` — yours + +| field | required? | read by | notes | +|---|---|---|---| +| `origin` | **yes** | `family::sr-ht-ecore/chrome/chrome.go::BuildNav` (external only), `family::sr-ht-ecore/csrf/csrf.go::Require`, `family::sr-ht-ecore/instconf/instconf.go::ExternalOrigin`, `mirror::core-go/config/config.go::GetOrigin` | external `scheme://host`. Nav membership needs the **external** key specifically — a section with only `internal-origin` is skipped by `BuildNav`. | +| `connection-string` | yes if you have a DB | your `service.LoadConfig`; also `mirror::core-go/server/server.go::WithDefaultMiddleware` (which this family never calls) | `postgresql://user@host/db?sslmode=disable`. | +| `bind-address` | no — code default | your `service.LoadConfig`, as the `defaultAddr` argument to `family::sr-ht-core/server/server.go::New` | **The only spelling.** Compiled default is `127.0.0.1:<port>`; a container must set `0.0.0.0:<port>`. Two services (`spec`, `dolt`) pass `-b` from the entrypoint instead and carry no key — do not copy that, it puts the port in two files. | +| `internal-origin` | no | `family::sr-ht-ecore/instconf/instconf.go::InternalOrigin` | only when service-to-service traffic must skip the public proxy. | +| `api-origin` / `api-internal-origin` | no | `mirror::core-go/config/config.go::GetAPI`, `family::sr-ht-ecore/instconf/instconf.go::InternalAPIOrigin` | needed to be federated. `GetAPI` **panics** when the whole ladder is empty; `InternalAPIOrigin` returns a bool instead — use ecore's. | +| `migrate-on-upgrade` | no, default off | `<svc>srht-migrate -a` | a present-but-empty value is a boolean core-go panics on: parse it yourself and refuse at startup. | +| `log-level` | no, default `info` | `family::sr-ht-ecore/logging/logging.go::Defaults` (`LevelKey`) | read from **your own section**. Priority `-d` > `$LOG_LEVEL` > this > `info`. An unreadable value never fails startup. | +| `key-seal` | only if you seal credentials | your own sealing code (artifacts, curator) | base64 of 32 bytes. Daemon refuses to start once something needs sealing and it is empty. | +| `s3-upstream`, `s3-access-key`, `s3-secret-key`, `s3-bucket` | only if you store objects | your own S3 client (`family::sourcehut-curator/blobx/blobx.go`, `family::sourcehut-artifacts/core/config.go`) | in **your** section, not `[objects]`. artifacts additionally has `s3-bucket-oci` / `-mirror` / `-cache`. Bucket names need ≥3 characters — Garage refuses `go`. | +| `max-*` ceilings | no | your `service.LoadConfig` | house convention: every ceiling's comment carries a **measured** derivation, and every refusal over one names the key. | +| `repos`, `cache` | service-specific | dolt, spec | on-disk roots. | +| `static-dir` | **do not add one** | dolt only | every other service `go:embed`s its CSS. dolt is the reason the sentence "there is no static-dir key" appears in six sibling configs. | + +### `[sr.ht]` — global, must match instance-wide + +| field | required? | read by | notes | +|---|---|---|---| +| `network-key` | **yes** | `family::sr-ht-core/crypto/crypto.go::InitCrypto`, `family::sr-ht-ecore/internalauth/internalauth.go` | Fernet key. Decrypts the `sr.ht.unified-login.v1` cookie and seals `Authorization: Internal`. `InitCrypto` calls `log.Fatalf` without it — the process exits, it does not return an error, and `server.New` calls `InitCrypto` unconditionally. | +| `owner-name`, `owner-email` | **yes** | `mirror::core-go/config/config.go::GetOwner` — **panics** | also the committer identity for services that write git commits. | +| `site-name` | recommended | `family::sr-ht-ecore/chrome/chrome.go::NewService` | nav brand. Defaults to `sr.ht`. | +| `environment` | recommended | `family::sr-ht-ecore/chrome/chrome.go::NewService` | the chrome defaults an **absent** value to `development` and shows the banner. An instance that means production must say so. | +| `global-domain` | **yes in practice** | `mirror::core.sr.ht/srht/config.py::get_global_domain` | the `domain=` meta sets on the login cookie. Your host must be under it or the cookie is never sent and every viewer looks anonymous. Absent, meta derives it from its own origin's parent domain. | +| `internal-ipnet` | no, has a default | `mirror::core-go/config/config.go::LoadConfig` + `::IsInternalIP` | **read from `[sr.ht]`, nowhere else.** The `[<svc>::api] internal-ipnet` spelling that appears seven times in the deployed template belongs to the Python services. Default is loopback + RFC1918 + link-local, which covers the docker networks. A malformed CIDR **panics** at load. | +| `service-key` | not for you | `mirror::core.sr.ht/srht/app/flask.py::Flask` | Flask session secret. No Go service reads it. | +| `redis-host` | not for you | `mirror::core-go/server/server.go::WithDefaultMiddleware`, `mirror::core.sr.ht/srht/redis.py` | this family never calls `WithDefaultMiddleware` (it opens a second, uncapped Postgres pool and starts an email queue and a Redis client you do not want), so no custom service reads this. | +| `site-info`, `site-blurb` | not for you | upstream Python landing-page templates only | nothing in ecore or core-go reads them. | + +### `[webhooks]` + +| field | required? | read by | notes | +|---|---|---|---| +| `private-key` | **yes, unconditionally** | `family::sr-ht-core/crypto/crypto.go::InitCrypto` | base64 Ed25519 seed. It is checked **first**, before `network-key`, and `InitCrypto` exits without it — even for a service that emits no webhooks. It is also the seed of the instance bearer HMAC key: `bearerKey = HMAC-SHA256(ed25519 private key, "sr.ht HMAC key")`. See §6. | +| `queue-size` | no, default 512 | `mirror::core-go/webhooks/queue.go::NewQueue` | `config.DefaultQueueSize`. | + +### `[meta.sr.ht]` + +| field | required? | read by | notes | +|---|---|---|---| +| `origin` | **yes** | `family::sr-ht-ecore/chrome/chrome.go::NewService`, `::LoginURLFor` | login/logout redirects, register link, profile fetch. Without it your nav cannot send anyone to log in. | +| `api-origin` | yes if you validate meta PATs or mirror profiles | `family::sr-ht-ecore/metapat/metapat.go::CoreBackend` under Internal auth | resolved through the `GetAPI` ladder. | +| `oauth-client-id`, `oauth-client-secret` | no | upstream OAuth2 client flow | not needed to read the login cookie, and not used by this family — machine credentials come from tokens.sr.ht. | + +### `[tokens.sr.ht]` + +| field | required? | read by | notes | +|---|---|---|---| +| `origin` | no — but no bearer plane without it | `family::sr-ht-ecore/bearer/bearer.go::New` (`Options.Origin`) | `bearer.New` **returns an error on an empty Origin**, so building the token plane unconditionally makes the daemon fail to start on an instance that legitimately runs no token daemon. Build the plane only when the section is present, and log `tokens: disabled` when it is not. | +| `internal-origin` | no | `family::sr-ht-ecore/instconf/instconf.go::InternalOrigin` | preferred for the revocation check (container to container). The `/tokens` browser redirect uses the external origin. | + +### `[hub.sr.ht]`, `[git.sr.ht]` + +| field | required? | read by | notes | +|---|---|---|---| +| `[hub.sr.ht] origin` | no | `family::sr-ht-ecore/chrome/chrome.go::NewService` | if set, usernames link to hub profiles. `hub` is **excluded from the switcher**, so the brand link is the only route to it. | +| `[git.sr.ht] repos` | only if you read bare repos | your `gitx` package | `/var/lib/git`. Absent, a service that needs it degrades (503 on the affected endpoint) rather than failing to start. | +| `[git.sr.ht] api-origin` | only if you query git's GraphQL | `mirror::core-go/config/config.go::GetAPI` | Internal auth bypasses every `@access` scope check, so git hands over private repositories without objecting — applying visibility is entirely your job. | + +### `[objects]` and `[mail]` + +`[objects]` (`s3-upstream`, `s3-access-key`, `s3-secret-key`, `s3-region`, `s3-insecure`) is +core-go's own S3 client, `mirror::core-go/objects/middleware.go::NewClient`. It exists on this +instance for the upstream services. The custom family does not use it; put your S3 keys in +your own section instead. Note `s3-region` is load-bearing against Garage — a mismatch is an +`AuthorizationHeaderMalformed` sigv4 scope error. + +`[mail]` (`smtp-host`, `smtp-port`, `smtp-user`, `smtp-password`, `smtp-from`, +`smtp-encryption`, `smtp-auth`, `pgp-privkey`/`pgp-pubkey`/`pgp-key-id`, +`egress-queue-size`) is read by `mirror::core-go/email/send.go` and `::worker.go`. Nothing in +this family sends email. Skip the section entirely; do not copy it into your example. + +### `[<svc>.sr.ht::api]` + +`max-complexity` (`family::sr-ht-core/server/server.go::WithSchema`, default 250) and +`max-duration` (default 3s) are read only by services that go through core-go's own +`WithSchema`/`WithDefaultMiddleware`. A service mounting its own `/query` — which is every +one in this family — reads neither. The `internal-ipnet` seen in this subsection on the +deployed instance is the Python spelling; the Go side reads `[sr.ht] internal-ipnet`. + +### Keys upstream defines that this family never reads + +One line each, so you do not put them in your section by copying an upstream example: +`debug-host` / `debug-port` (upstream Python dev server, `mirror::core.sr.ht/srht/debug.py` +— the Go analogue is `bind-address`), `webhooks=redis://…` (the Celery broker for the Python +webhook worker), `s3-bucket` + `s3-prefix` paired with `[objects]` credentials (upstream +`pages.sr.ht` and `builds.sr.ht` only), `redis-host`, `service-key`, `site-info`, +`site-blurb`. + +--- + +## 4. Read config through the shared helpers + +There are now **two** Go layers, and for origins the newer one wins. + +| concept | core (`family::sr-ht-core/config/config.go`) | ecore (`family::sr-ht-ecore/instconf/instconf.go`) | Python (`mirror::core.sr.ht/srht/config.py`) | +|---|---|---|---| +| load the file | `LoadConfig` | — | `load_config` | +| string / int / bool | `GetString` / `GetInt` / `GetBool` | — | `cfg` / `cfgi` / `cfgb` | +| a service's external URL | `GetOrigin(conf, svc, true)` | **`ExternalOrigin`** | `get_origin(svc, external=True)` | +| a service's internal URL | `GetOrigin(conf, svc, false)` | **`InternalOrigin`** | `get_origin` | +| a service's API URL | `GetAPI` — **panics** when the ladder is empty | **`InternalAPIOrigin`** — returns `(string, bool)` | `get_api` | +| host to compare a `Host` header against | — | **`OriginHost`** / **`OriginAuthority`** | — | +| owner name/email | `GetOwner` — **panics** | — | — | +| internal-caller check | `IsInternalIP` | (via `internalauth`) | — | +| required-key check | — | **`Require` / `Need` / `NeedAny` / `Key.Because`** | — | +| cookie domain | — | — | `get_global_domain` | + +**Prefer ecore's for anything origin-shaped.** `instconf` exists because five services and +core-go each grew their own copy: one canonicalized with `TrimRight(o, "/")` and another with +`TrimSuffix`, so `https://x//` produced two different strings in two daemons that must agree +when one checks a browser `Origin` header against it; one host extractor answered +`"localhost"` for a malformed origin, on the path feeding an MCP DNS-rebinding guard where an +empty host means the guard is off. `instconf` returns canonical values (whitespace and +**every** trailing slash stripped), treats absent, missing-section and present-but-blank +alike as `""`, and splits `ExternalOrigin`/`InternalOrigin` into two named functions rather +than a bool argument, because a flipped flag is invisible at the call site. + +**Validate the whole config once, at startup, and report every gap together.** Use +`instconf.Require(conf, instconf.Need("sr.ht", "network-key").Because("crypto.InitCrypto +exits without it"), …)`, merged with your own `service.LoadConfig`'s problem list into one +error — `family::sourcehut-bench/cmd/benchsrht/main.go::validateConfig` is the pattern, and +`family::sourcehut-bench/service/service.go::CoreServerKeys` is the two-entry list of what +`server.New` fatals on. An operator editing `config.ini` wants the whole list, not one +restart per missing key. + +--- + +## 5. Canonical spellings + +| use this | not these | why | +|---|---|---| +| `bind-address` | `bind`, `listen`, `addr`, `host`/`port`, `debug-host`/`debug-port` | **The only spelling.** The tokens SPEC said `bind` and the daemon accepted it with a deprecation warning; the *spec* was corrected instead, because one instance-wide `config.ini` carrying two spellings of one idea is a trap for whoever edits it next (`family::sourcehut-tokens/config.example.ini::[tokens.sr.ht]`). `debug-host`/`debug-port` are real, but they are the upstream Python dev-server keys. | +| `origin` | `url`, `base-url`, `web-url` | `GetOrigin` / `ExternalOrigin` look up exactly `origin`. | +| `internal-origin` | `lan-url`, `private-origin` | the internal-preferred partner of `origin`. | +| `api-origin`, `api-internal-origin` | `graphql-url`, `api-url` | the four-key ladder is fixed: `api-internal-origin`, `internal-origin`, `api-origin`, `origin`. | +| `connection-string` | `db`, `dsn`, `database-url` | universal across upstream and the family. | +| `migrate-on-upgrade` | `auto-migrate` | packaging convention. | +| `log-level` | `verbosity`, `debug` | `logging.LevelKey`, read from your own section. | +| `<purpose>-listen` for a second listener | `bind-address-2` | dolt's `remotesapi-listen` / `credsapi-listen`. | +| `s3-upstream`, `s3-access-key`, `s3-secret-key`, `s3-bucket` | `s3-prefix`, `bucket`, `object-store` | `s3-prefix` belongs to upstream pages/builds against the shared `[objects]` credentials; this family addresses its own bucket instead. | + +Section names follow the same discipline: **one short prefix + `.sr.ht`** — `cov` not +`coverage`, `diff` not `compare`, `spec` not `specs`. The section name is also the apk +`pkgname` and the subdomain, and it is deliberately not the repository name, the module path +or the binary name. Renaming later moves everything the instance sees; **grants are compared +literally**, so a rename silently invalidates every minted token carrying the old prefix. + +--- + +## 6. Shared vs per-service, and what breaks + +Byte-identical instance-wide, no exceptions: + +| key | what breaks when it differs | +|---|---| +| `[sr.ht] network-key` | your service cannot decrypt the unified-login cookie: every viewer looks anonymous while being logged in everywhere else. Also breaks `Authorization: Internal` in both directions. | +| `[webhooks] private-key` | see below — the worst of the three. | +| `[sr.ht] global-domain` | the login cookie is set on a domain your host is not under, so it is never sent to you. Same visible symptom as a wrong `network-key`, different cause. | +| `[<svc>.sr.ht] origin` (**every** service's, in **every** copy) | see §8. | + +**`[webhooks] private-key` is not only about webhooks.** +`family::sr-ht-core/crypto/crypto.go::InitCrypto` derives the instance bearer HMAC key from +it — `bearerKey = HMAC-SHA256(webhookSk, "sr.ht HMAC key")`, where `webhookSk` is the +ed25519 private key expanded from the configured seed. Every tokens.sr.ht working token and +every meta PAT is a BARE-serialised `auth.BearerToken` signed with that key. Three +consequences: + +1. **Rotating it invalidates every working token on the instance at once**, silently — the + next request with a previously good credential answers 401 and nothing says why. Every CI + secret, every agent session, every registered token. +2. A service that emits no webhooks still **cannot start** without it: `InitCrypto` checks it + *before* `network-key` and calls `log.Fatalf`. `server.New` calls `InitCrypto` + unconditionally, so this is not opt-out. +3. The token payload is **authenticated, not encrypted** — anyone holding a token reads the + username, grants and expiry straight out of it. + +May differ per service: `bind-address`, `connection-string`, `log-level`, every `max-*`, +`key-seal`, the `s3-*` set, `migrate-on-upgrade`. Anything else, assume shared. + +--- + +## 7. `scopes` is not a config key, and getting it wrong takes down meta + +There is **no `scopes=` line** in any `config.example.ini` or in the deployed +`config.ini.tmpl` — verified by grep across both. Anything telling you to add `scopes=[]` to +your section is wrong. The scope list is a Go value published at `/query/api-meta.json`. + +`mirror::meta.sr.ht/metasrht/blueprints/oauth2.py` runs at **import time**: it walks every +config section ending in `.sr.ht`, GETs `{api-origin or origin}/query/api-meta.json`, and +stores `r.json()["scopes"]`. The personal-token template then does +`{% for scope in group['scopes'] %}` over every discovered service. A JSON **`null`** there +is not "a service with no scopes" — it is a Jinja iteration over `None`, i.e. a **500 on +meta's `/oauth2/personal-token` page for the whole instance**, every service's grants, not +just yours. Nobody would find it by testing the service that caused it. + +Two ways to emit `null`: `mirror::core-go/server/server.go::WithSchema(schema, nil)`, which +marshals the `[]string` it is handed; and hand-rolling the handler with a nil slice field. +The right answer is `family::sr-ht-ecore/apimeta/apimeta.go::Handler`, which takes scopes +variadically, normalizes `nil` to `[]string{}` before marshalling, and serves at +`apimeta.Path`. It also publishes `webhook-pubkey`, so `crypto.InitCrypto` must have run +first. + +**These are meta's scopes, not tokens.sr.ht's grants — two grammars sharing one token +format and nothing else.** A scope is the part **after** the service name, and meta prefixes +the service itself: a service checking `dolt.sr.ht/repos:RO` publishes `"repos"`. The +working-token vocabulary is separate and is never published here +(`family::sourcehut-artifacts/core/grants.go::Grants` = `artifacts:upload` and friends). +Derive the published list from the value the validator checks rather than spelling it twice +— `family::sourcehut-bench/graph/server.go::GrantScopes` is +`[]string{metapat.ScopeName(authn.ScopeRead)}`, with a test asserting it equals what +`api-meta.json` serves. A service whose `/query` accepts no meta PAT publishes an empty list, +deliberately. + +Because meta discovers at import time, your scopes appear on the token page only after meta +restarts. + +--- + +## 8. The section name is the nav membership, and every service needs your section + +`family::sr-ht-ecore/chrome/chrome.go::BuildNav` is the whole rule: + +```go +for section := range conf { + if !strings.HasSuffix(section, ".sr.ht") { continue } + short := strings.TrimSuffix(section, ".sr.ht") + if navExcluded[short] { continue } // paste, pages, hub + origin := config.GetOrigin(conf, section, true) // EXTERNAL only + if origin == "" { continue } + … +} +``` + +1. **The `.sr.ht` suffix is the membership test**, whatever host you are actually served + from. `[myservice]` or `[myservice.srht.bigb.es]` puts you in nobody's nav. +2. **A section with no external `origin` is skipped.** `internal-origin` alone is invisible. +3. **Every service builds its nav from its own copy of the file, at startup**, in + `chrome.NewService` — which reads the config exactly once. So your + `[myservice.sr.ht] origin=` must be in the config *every other service* reads (one shared + file makes this automatic), **and those services must be restarted** before you appear. + Adding it to your own config alone changes nothing anywhere else. +4. `paste`, `pages` and `hub` are excluded (upstream excludes only paste and pages), and the + switcher renders **only for an authenticated viewer** — a logged-out visitor sees no nav + entry for anyone, which is not evidence that your section is wrong. + +Your own pages go in `Service.Sections` (the in-service tab row), **not** `ExtraNav`. +`ExtraNav` is for a genuinely separate origin. Both artifacts and bench got this wrong; +putting your own pages in `ExtraNav` breaks active-tab highlighting and duplicates the +switcher entry the instance already renders — bench's navbar read "… spec tokens tokens". + +--- + +## 9. Generating keys + +The upstream tool is the script `sr.ht-keygen`, shipped by core.sr.ht +(`mirror::core.sr.ht/pyproject.toml::script-files` installs it under exactly that filename; +its own usage string spells itself `srht-keygen`, and `mirror::api.sr.ht/config.example.ini` +copies that spelling — the executable is `sr.ht-keygen`). Its three modes are the only +correct invocations: + +| key | invocation | shape | equivalent without the tool | +|---|---|---|---| +| `[sr.ht] network-key` | `sr.ht-keygen network` | Fernet key: url-safe base64 of 32 bytes | `python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"`, or `openssl rand -base64 32 \| tr '+/' '-_'` | +| `[webhooks] private-key` | `sr.ht-keygen webhook` | std base64 of a 32-byte ed25519 **seed**; prints the public half too | `openssl rand -base64 32` | +| `[sr.ht] service-key` | `sr.ht-keygen service` | hex of 32 bytes; Python tier only | `openssl rand -hex 32` | +| `[myservice.sr.ht] key-seal` | `myservicesrht keygen` | base64 of 32 bytes (AES-256) | `openssl rand -base64 32` | + +This instance uses the fallbacks rather than the tool: `phoebe-lab/srht/scripts/setup.sh` +generates both shared keys into `.env` on first run only. `<svc>srht keygen` is the family's +own subcommand for `key-seal` (`family::sourcehut-artifacts/cmd/artifactsrht/main.go`, +`family::sourcehut-curator/cmd/gosrht/keygen_test.go`) and prints a value `DecodeKey` accepts +verbatim, newline included. + +Never hand-roll `network-key`: it must be a valid Fernet key or `fernet.DecodeKey` fails and +`InitCrypto` exits. + +--- + +## 10. Ports on this instance + +| port | service | +|---|---| +| 5090 | diff.sr.ht (compare) | +| 5091 | spec.sr.ht (specs) — set with `-b` from the entrypoint, no config key | +| 5092 | bench.sr.ht | +| 5093 | cov.sr.ht (coverage) | +| 5094 | tokens.sr.ht | +| 5095 | artifacts.sr.ht (both hosts, one port — routes on `Host`) | +| 5096 | go.sr.ht (curator) — provisioned, currently commented out | +| **5097** | **next free for service #11** | +| 5116 / 5215 | fedgw — gateway `/query`, and a second listener for `/healthz` + `/metrics` | +| 5306 / 5307 / 5308 | dolt.sr.ht — `remotesapi-listen`, web (`-b`), `credsapi-listen` | +| 5000–5014, 5100–5114 | upstream Python services: `debug-port` and `api-origin` respectively | + +Each `config.example.ini` in the family names the ports its siblings hold, in the comment +above its own `bind-address`. Keep that convention — it is the only place the allocation is +written down. + +Two deployment-shape rules that go with the port: + +- In a container, `bind-address` must be `0.0.0.0:<port>`; the compiled default of + `127.0.0.1:<port>` is right for the nginx-on-one-host layout and wrong here. +- The entrypoint passes **no** `-b`, so `config.ini` is the single spelling of the port. A + unit and its config that both name the port are two things that drift. + +--- + +## 11. Startup checklist + +- [ ] Section named exactly `[myservice.sr.ht]`, with `origin`, in the **shared** file. +- [ ] Every other service restarted, so their nav picks it up. +- [ ] `[sr.ht] network-key` and `[webhooks] private-key` present — checked by your own + `validateConfig` **before** `server.New` reaches `InitCrypto`, so the operator gets a + readable refusal instead of `log.Fatalf("No webhook key configured")`. +- [ ] `[sr.ht] owner-name` / `owner-email` present — `GetOwner` panics. +- [ ] Your host under `[sr.ht] global-domain`, or no cookie ever arrives. +- [ ] `bind-address` = `0.0.0.0:<port>` in a container; no `-b` in the entrypoint. +- [ ] `/query/api-meta.json` serves `"scopes": []`, never `null`, via `apimeta.Handler`. +- [ ] Token plane built only when `[tokens.sr.ht]` is present — `bearer.New` errors on an + empty Origin. +- [ ] Every missing key reported in **one** error, not one restart per key. +- [ ] No `static-dir`, no `redis-host`, no `service-key`, no `[mail]` in your example file. diff --git a/skills/sourcehut-custom-service/references/deploy.md b/skills/sourcehut-custom-service/references/deploy.md new file mode 100644 index 0000000000000000000000000000000000000000..1b8100db08342f92e3a4377e200f6cdd1b16130a --- /dev/null +++ b/skills/sourcehut-custom-service/references/deploy.md @@ -0,0 +1,625 @@ +# Packaging, deployment and operations + +> Citations: `family::<repo>/<path>::<symbol>` — this instance's own repos; `mirror::<path>::<symbol>` +> — the upstream documentation mirror. Both roots are substituted at install time. `phoebe-lab/srht/…` +> is the deployment repo and is named without a prefix. Symbols, never line numbers. + +What actually runs the nine existing custom services, and what service #11 has to +produce to join them. This replaces the "nginx + DNS" wiring checklist in SKILL.md: +that shape exists, but it is the documented alternative, not the one in production. + +Citations: `family::<repo>/<path>::<symbol>` for the sibling service repos under +`~/data/home/sourcehut-*`, `mirror::<repo>/<path>` for the upstream documentation +mirror. The deployment repo is outside both roots and is referred to in plain text as +`phoebe-lab/srht/<file>` (READ-ONLY — never edit it from a service repo's session). + +--- + +## 1. What production actually is + +A single **Docker Compose stack** (`phoebe-lab/srht/docker-compose.yml`) behind an +**external Traefik** reverse proxy. Each service has a `Dockerfile.<svc>` that clones and +compiles nothing: it `apk add`s the service's own package, at a pinned version, from +**this instance's own artifacts.sr.ht apk channel** +(`https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22`). The Garage-backed +`repo.bigb.es` channel is the fallback for when artifacts.sr.ht itself is down. + +| Layer | Production | Documented alternative | +| --- | --- | --- | +| Process supervision | `docker compose` + `restart: unless-stopped`, tini as PID 1 | `contrib/<svc>.service` systemd unit | +| Routing / TLS | Traefik router labels on the compose service | `contrib/<svc>.conf` nginx server block | +| Migrations | container entrypoint | `ExecStartPre=` in the unit | +| Config | one shared `config.ini` in a read-only Docker volume | one shared `/etc/sr.ht/config.ini` on the host | +| Deploy | `labng push srht` → `post_push` hooks in `deploy.yml` | `apk upgrade` + `systemctl restart` | + +**`contrib/` is reference material, not files the package owns.** Every APKBUILD that +carries one says so in its `package()` comment — the directory is deliberately *not* +installed (`family::sourcehut-bench/APKBUILD::package`). Write it anyway: it is the only +place the plain-host operational reasoning has to live (why `RestartSec=5s`, which unix +user, who owns request timeouts), and the container deployment documents none of that. + +--- + +## 2. Claim your instance-wide resources first + +Two collide silently and are the cheapest thing to get right on day one. + +**Port.** Custom services occupy a contiguous band. Pick the next free slot and use it +in exactly three places: `bind-address` in `config.ini.tmpl`, the Traefik +`loadbalancer.server.port` label, and `EXPOSE` in the Dockerfile. + +| Port | Service | | Port | Service | +| --- | --- | --- | --- | --- | +| 5000 / 5100 | meta web / api | | 5090 | diff.sr.ht | +| 8080 / 5101 | git web / api | | 5091 | spec.sr.ht | +| 5002 / 5102 | builds web / api | | 5092 | bench.sr.ht | +| 5003 / 5103 | todo web / api | | 5093 | cov.sr.ht | +| 5011 / 5111 | paste web / api | | 5094 | tokens.sr.ht | +| 5012 / 5112 | pages web / api | | 5095 | artifacts.sr.ht | +| 5014 / 5114 | hub web / api | | 5096 | go.sr.ht | +| 5116 / 5215 | fedgw query / metrics | | 5306–5308 | dolt.sr.ht (grpc/web/creds) | + +**Next free custom slot: 5097.** Upstream services use `50NN` web / `51NN` api; do not +reach into that band. + +Also claim, without collision: + +- **Config section name** — `[<svc>.sr.ht]`, unique, and the `.sr.ht` suffix is what puts + you in the nav and in the federation loop (see SKILL.md §1/§4). +- **Postgres database + role** — `<svc>` / `"<svc>.sr.ht"` (§7). +- **Subdomain** — `<svc>.${SRHT_DOMAIN}`, which must equal your `origin` (§6). +- **`SRHT_<SVC>_VER`** in `phoebe-lab/srht/versions.env`. +- **apk `pkgname`** — the *service* name, not the repo name: repo `sourcehut-bench` ships + `pkgname=bench.sr.ht`; `sourcehut-compare` ships `diff.sr.ht`; `sourcehut-curator` + ships `go.sr.ht` (`family::sourcehut-tokens/APKBUILD::pkgname`). + +--- + +## 3. The APKBUILD skeleton + +Copy from the closest sibling by shape — stateful with Postgres → +`family::sourcehut-tokens/APKBUILD` (the most complete of the nine); stateless → +`family::sourcehut-compare/APKBUILD`. The annotated skeleton, with every trap named: + +```sh +# Contributor: Eugene Blikh <bigbes@gmail.com> +# Maintainer: Eugene Blikh <bigbes@gmail.com> + +pkgname=myservice.sr.ht # SERVICE name, not repo name +pkgver="${PKGVER:-0.0.0}" # (1) from the environment — never sed-ed in +pkgrel=0 +pkgdesc="..." +url="https://sourcecraft.dev/bigbes/sr-ht-myservice" +arch="x86_64" # one architecture; the instance runs one +license="MIT" +options="!check !tracedeps" # (2) +source="" # (3) CI's checkout IS the source +builddir="$startdir" +_version="${SRHT_VERSION:-$pkgver}" + +build() { + cd "$builddir" + export GOCACHE="$HOME/.cache/go-build" # (4) inside build(), not top-level + export GOMODCACHE="$HOME/go/pkg/mod" + make css ASSETS=/usr/share/sourcehut # (5) CSS strictly before the compiler + CGO_ENABLED=0 make build GOFLAGS="-trimpath -modcacherw" VERSION="$_version" + make check-css + make check-version # (6) fail before anything is staged +} + +package() { + cd "$builddir" + make install-files DESTDIR="$pkgdir" PREFIX=/usr ASSETS=/usr/share/sourcehut # (7) + make check-version CHECK_BIN="$pkgdir/usr/bin/myservicesrht" # (8) + make check-embedded-css CHECK_BIN="$pkgdir/usr/bin/myservicesrht" +} +``` + +**(1) `pkgver` from the environment.** Go records a VCS stamp in every binary compiled +inside a repository and decides `vcs.modified` from `git status --porcelain`. A CI task +that `sed`s the version into a *tracked* file — the APKBUILD itself — marks every binary +that run produces as built from a modified tree, for the life of the package (measured on +go1.26.5, `family::sourcehut-tokens/APKBUILD::header`). `PKGVER` and `SRHT_VERSION` come +from the manifest's `version` task (§9); a local `abuild` with neither builds `0.0.0`. + +**(2) `options="!check !tracedeps"`.** `!check`: the suite already ran as the manifest's +`test` task against a real Postgres before abuild was invoked, and re-running it under +fakeroot without a database silently skips every database-backed suite (coverage of `db/`: +4.5% without a Postgres, 77.2% with — `family::sourcehut-bench/APKBUILD::options`). +`!tracedeps`: `CGO_ENABLED=0` binaries are static and abuild's ELF scanner would pin +runtime deps that do not exist. + +**(4) Re-pin the caches *inside* `build()`.** abuild exports its own `GOCACHE` into a +wipe-per-build tmpdir *after* the file's top level is sourced, and an upstream `abuild.in` +typo makes `GOMODCACHE` follow `GOCACHE`'s value regardless of what the APKBUILD set. Pin +at file scope and the CI-restored module cache is invisible — the build re-downloads +everything it just downloaded (`family::sourcehut-tokens/APKBUILD::build`). `-modcacherw` +likewise: an extracted module cache is read-only by default, and a cache tarball made from +a read-only tree cannot be re-extracted next build. + +**(5) `make css` strictly before `go build`.** The shared partials are assembled by CI +into `/usr/share/sourcehut/scss` — **no apk ships them** — and `//go:embed` takes a +*directory*, so a `make css` that produced nothing still compiles and ships an unstyled +service. `check-css` counts stylesheets on disk; `check-embedded-css` reads the filename +`//go:embed` actually baked into the **staged** binary and requires it to be that same +file — the only gate that catches a binary compiled *before* `make css` ran +(`family::sourcehut-bench/Makefile::check-embedded-css`). + +**(6)/(8) `check-version` twice.** In `build()` it fails fast while the diagnostics about +the checkout are still true; in `package()` it interrogates the artifact that will ship. +It refuses a binary with no `vcs.revision` and one stamped `-dirty`, printing the +offending `git status --porcelain`, the ignored set and `GOTMPDIR` when the tree reads +clean after the fact (`family::sourcehut-bench/Makefile::check-version`). Strip runs after +`package()`, so the file in the apk is not byte-identical to what the gates read; the +verdict still transfers because `.go.buildinfo` and the embed data are allocated sections. +Re-measure if the package ever gains `options=!strip`. + +**(7) `install-files`, never `install`, and `package()` compiles nothing.** abuild runs +`package()` in a **fresh process under fakeroot** that re-sources the APKBUILD and never +calls `build()`, so nothing `build()` exported reaches it. `make install` depends on +`build`, whose binary targets are `.PHONY`, so it recompiles from a cold cache with none of +the flags or cache pins above — and *that* second copy is what ships, while +`check-version` was run against the first. Measured on the bench sibling (job 320): 0 lines +of `go: downloading` before `>>> Entering fakeroot`, 55 after it. + +### Install layout + +`make install-files DESTDIR=$pkgdir PREFIX=/usr ASSETS=/usr/share/sourcehut` +(`family::sourcehut-bench/Makefile::install-files`): + +| Path | Content | +| --- | --- | +| `/usr/bin/<svc>srht` | the daemon | +| `/usr/bin/<svc>srht-migrate` | migration binary, a thin wrapper around `git.sr.ht/~bitfehler/brant` | +| `/usr/share/sourcehut/migrations/<svc>.sr.ht/*.sql` | incremental migrations | +| `/usr/share/sourcehut/<svc>.sr.ht.sql` | full schema | +| `/usr/share/sourcehut/scss/` | shared partials — placed by CI, shipped by no apk | + +**`ASSETS` is a separate variable from `PREFIX` on purpose** +(`family::sourcehut-bench/Makefile::ASSETS`): deriving it would move +`/usr/local/share/sourcehut` under a non-`/usr` prefix and break the shared theme partial +path. The two agree only when `PREFIX=/usr`. + +**Do not install static assets as files.** CSS/JS are `go:embed`-ed, which is why a +correctly-wired service has no `static-dir` config key. `dolt.sr.ht` is the one exception +(a hashed `main.min.<sha>.css` under `/usr/share/sourcehut/static/dolt.sr.ht`, globbed at +startup, hence a `static-dir` key) — do not copy that unless you have its reason. + +--- + +## 4. Container and entrypoint + +`Dockerfile.<svc>` — copy `phoebe-lab/srht/Dockerfile.bench` for a plain Postgres-backed +service, `Dockerfile.artifacts` or `Dockerfile.dolt` if you need git-repo or Garage access: + +```dockerfile +# syntax=docker/dockerfile:1 +ARG SRHT_MYSVC_VER +FROM alpine:3.22 +ARG SRHT_MYSVC_VER + +# Signs the APKINDEX of our channel, which artifacts.sr.ht rebuilds and signs itself on +# every publish — CI's abuild key is throwaway and trusted by nothing. The filename is +# load-bearing: apk requires it to match the .SIGN.RSA256.<name> index segment exactly. +COPY keys/bigbes@artifacts.srht.bigb.es.rsa.pub /etc/apk/keys/ + +RUN echo "https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22" >> /etc/apk/repositories \ + && apk add --no-cache ca-certificates postgresql-client tini \ + myservice.sr.ht=${SRHT_MYSVC_VER} + +COPY scripts/entrypoint-mysvc.sh /entrypoint.sh +RUN chmod +x /entrypoint.sh +EXPOSE 5097 +ENTRYPOINT ["/sbin/tini", "-s", "--"] # tini is PID 1, for zombie reaping +CMD ["/entrypoint.sh"] +``` + +Add `git` only if the service reads git.sr.ht's bare repos (diff, cov); +`postgresql-client` is for the entrypoint's `pg_isready` wait and nothing else. + +`scripts/entrypoint-<svc>.sh` — three steps, nothing else: + +```sh +#!/bin/sh +set -e + +echo "==> Waiting for PostgreSQL..." +until pg_isready -h "${PGHOST:-postgres}" -U "${PGUSER:-postgres}" -q 2>/dev/null; do + sleep 1 +done + +# init applies schema.sql wholesale and stamps to head on a fresh database; +# up applies pending migrations on an existing one. +# Do NOT cd anywhere with a ./migrations in it: the migrate binary's resolvePaths +# prefers a dev checkout's ./migrations and only then falls back to the installed +# layout, which is where the apk actually put them. +echo "==> Running migrations..." +myservicesrht-migrate init 2>/dev/null || myservicesrht-migrate -a up + +# No -b flag: the listen address is [myservice.sr.ht] bind-address in the shared +# config.ini, set to 0.0.0.0:5097. Repeating the port here is how a unit and a +# config drift apart. +exec myservicesrht +``` + +`exec` matters: without it the daemon is not PID 1's direct child and signal delivery on +`docker stop` goes to the shell. The `init 2>/dev/null || … -a up` idiom deliberately does +not distinguish "already initialised" from a real init failure — both produce a Postgres +error on `init`, and a genuine failure then also fails `up` and stops the container under +`set -e`. Silence is safe, just noisy in `docker logs` on every normal boot. + +**Healthcheck.** Send an explicit `Host:` header if the service is Host-sensitive +(`phoebe-lab/srht/docker-compose.yml`, the artifacts service): + +```yaml +healthcheck: + test: ["CMD", "wget", "-q", "-O", "/dev/null", "--header", "Host: mysvc.${SRHT_DOMAIN}", + "http://127.0.0.1:5097/healthz"] + interval: 30s + timeout: 5s + retries: 5 + start_period: 30s +``` + +Traefik silently **unpublishes** a container whose healthcheck fails, so a probe that +forgets the Host header reads as "service down" and pulls the route. + +**No `USER` directive exists in any of the nine production Dockerfiles** — every custom +service container runs as root. The container boundary is the isolation mechanism here; +the dedicated-user story below belongs to the plain-host path only. If you add a `USER`, +you are ahead of the fleet, not behind it. + +--- + +## 5. The reference systemd unit + +Not used in production. Write it anyway (§1). The load-bearing settings, from +`family::sourcehut-bench/contrib/bench-srht.service`: + +```ini +[Unit] +After=network.target postgresql.service +Wants=postgresql.service # Wants, not Requires: the DB may be remote + +[Service] +Type=simple +User=myservice +Group=myservice +Restart=always +RestartSec=5s +ExecStartPre=/usr/bin/myservicesrht-migrate -a up +ExecStart=/usr/bin/myservicesrht +TimeoutStopSec=35 +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true + +[Install] +WantedBy=multi-user.target +``` + +- **`RestartSec=5s`, not systemd's 100 ms default.** `ExecStartPre` fails whenever + Postgres is not accepting connections yet. At 100 ms all of `StartLimitBurst` (5) burns + inside one second, inside `StartLimitIntervalSec` (10 s); the unit lands in `failed`, + stays there after the database returns, and needs a human's `systemctl reset-failed`. + Measured on systemd 257: with the default, still `failed` 20 s after the database came + back; with 5 s, `active (running)`, no operator action. +- **`KillSignal=SIGINT` — only if your `main` does not bridge.** core-go's `server.Run` + drains on SIGINT and on nothing else, so a service handing systemd's default SIGTERM + straight to core-go is hard-killed + (`family::sourcehut-compare/contrib/diff-srht.service::KillSignal`). Better: bridge + SIGTERM onto SIGINT in the daemon and omit the line — a `KillSignal` says the same thing + a second time, and stops being true the day the bridge is removed. +- **`TimeoutStopSec` = HTTP drain + any background phase.** core-go gives listeners a 30 s + deadline, so 35 s covers a service with nothing else running; bench uses 70 s because a + nightly retention sweep runs *after* the drain, sequentially. +- **The dedicated user.** `adduser -S -D -H -s /sbin/nologin <svc>` — no home, no shell, + no group membership. Its one privilege is **read on the shared `/etc/sr.ht/config.ini`** + (which carries `network-key` and the webhook private key), granted by group rather than + by widening the file. Postgres is reached over TCP as the role in `connection-string`, + so the unix user has nothing to do with the database identity. Run as the **`git` user** + instead only if the service reads git.sr.ht's bare repositories on disk — that is why + `diff.sr.ht` and `cov.sr.ht` do and bench does not. + +--- + +## 6. Routing + +**The `Host` must equal the service's configured `origin`.** Both the CSRF same-origin +check on every mutating web request and the `/mcp` DNS-rebinding guard compare the +incoming `Host` against `[<svc>.sr.ht] origin`. The MCP SDK's built-in guard rejects any +request arriving on a loopback listener with a non-loopback Host — exactly this deployment +— so these services replace it with an allowlist keyed on the configured origin. Forward +the wrong Host and every agent gets a 403 that reads like a client bug, while +bearer/API paths keep working: a confusing *partial* failure, and the most common way a +newly wired custom service breaks silently. + +**Traefik (production)** — labels on the compose service: + +```yaml +labels: + traefik.enable: true + traefik.docker.network: ${TRAEFIK_NETWORK:-traefik} + + traefik.http.routers.srht-mysvc-https.rule: Host(`mysvc.${SRHT_DOMAIN}`) + traefik.http.routers.srht-mysvc-https.entrypoints: websecure + traefik.http.routers.srht-mysvc-https.service: srht-mysvc-https-svc + traefik.http.routers.srht-mysvc-https.tls: true + traefik.http.routers.srht-mysvc-https.tls.certresolver: ${TRAEFIK_CERTRESOLVER:-letsencrypt} + traefik.http.services.srht-mysvc-https-svc.loadbalancer.server.port: 5097 + traefik.http.services.srht-mysvc-https-svc.loadbalancer.passHostHeader: true +``` + +`passHostHeader: true` is the Traefik spelling of the rule above. It is required, not +cosmetic. + +**nginx (`contrib/<svc>.conf`, plain-host path)** — sr.ht-nginx style +(`mirror::sr.ht-nginx/`), one origin per service +(`family::sourcehut-bench/contrib/bench.sr.ht.conf`): + +```nginx +server { + include sourcehut.conf; + include port80.conf; + server_name mysvc.srht.bigb.es; +} +server { + include sourcehut.conf; + include port443.conf; + include mysvc-ssl.conf; # per-service cert — mandatory, see below + server_name mysvc.srht.bigb.es; # == [myservice.sr.ht] origin + client_max_body_size 16m; # only where the service accepts uploads + client_header_timeout 60s; + client_body_timeout 60s; + send_timeout 60s; + keepalive_timeout 75s; + proxy_read_timeout 300s; + location = /mcp { + proxy_pass http://127.0.0.1:5097; + include headers.conf; + proxy_buffering off; # /mcp can answer a GET with SSE + include web.conf; + } + location / { + proxy_pass http://127.0.0.1:5097; + include headers.conf; + add_header Content-Security-Policy "default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always; + include web.conf; + } +} +``` + +- **The per-service `ssl_certificate` include is mandatory.** sr.ht-nginx's `nginx.conf` + carries no http-level `ssl_certificate`, so a `listen … ssl` block without one refuses + to load the **whole instance's** nginx config, not just this file. +- **Request timeouts live in the proxy and nowhere else.** core-go builds its + `http.Server` inline and exposes no field for `ReadHeaderTimeout`, `ReadTimeout`, + `WriteTimeout` or `IdleTimeout`. Whatever the nginx directives above (or Traefik's + entrypoint transport settings) say is the entire timeout story for these services, + unless one endpoint special-cases itself with an `http.NewResponseController`. +- **No `/static` location.** Assets are `go:embed`-ed and served off the same listener. + +--- + +## 7. Database provisioning and the migration contract + +One Postgres instance; **one database and one role per service**, created by an idempotent +one-shot init container before any service starts (`phoebe-lab/srht/scripts/init-db.sh`). +Add the service prefix to its `SERVICES` list: + +```sh +SERVICES="meta git todo man paste lists builds hub pages dolt spec cov bench tokens artifacts go" +``` + +For each it creates, guarded against existence: + +```sql +CREATE ROLE <svc> WITH LOGIN PASSWORD '<svc>'; +CREATE DATABASE "<svc>.sr.ht" OWNER <svc>; +GRANT ALL PRIVILEGES ON DATABASE "<svc>.sr.ht" TO <svc>; +``` + +Connection string convention: `postgresql://<svc>:<svc>@postgres/<svc>.sr.ht?sslmode=disable`. +Password equals role name — acceptable only because Postgres is not exposed outside the +docker network. **Skip this step entirely for a stateless service**: `diff.sr.ht` owns no +database and is absent from `SERVICES`; it authorizes every request against git.sr.ht's +own API and reads only the shared read-only git-repos volume. + +Migration contract: + +| | Fresh database | Existing database | +| --- | --- | --- | +| Command | `<svc>srht-migrate init` | `<svc>srht-migrate -a up` | +| Effect | applies `schema.sql` wholesale, stamps the version table to head so later `up` runs never replay it | applies pending migrations | +| `-a` | n/a | honours `migrate-on-upgrade` in the service's config section and exits early when it is off | + +Migrations are **forward-only**. `pg_dumpall` before bumping any version pin. + +--- + +## 8. Config distribution + +**One shared `config.ini` for the whole instance**, not per-service files +(`phoebe-lab/srht/CLAUDE.md::Configuration Flow`): + +1. `scripts/setup.sh` generates `.env` with random secrets, first time only. +2. A one-shot `config-init` container runs `envsubst` on `config/config.ini.tmpl` → + `/etc/sr.ht/config.ini`, written into the `srht-config` Docker volume that every other + container mounts **read-only** (`volumes: - srht-config:/etc/sr.ht:ro`). +3. Each service reads its own `[<svc>.sr.ht]` section plus the shared `[sr.ht]`, + `[webhooks]` and `[meta.sr.ht]` sections. **No upstream service's config is modified** — + integration is purely additive sections. + +Your section, appended to `config/config.ini.tmpl`: + +```ini +[myservice.sr.ht] +origin=https://mysvc.${SRHT_DOMAIN} +connection-string=postgresql://myservice:myservice@postgres/myservice.sr.ht?sslmode=disable +# 0.0.0.0 so Traefik reaches it across the docker network. The compiled-in default is +# 127.0.0.1:<port>, which is right for the nginx-on-one-host layout and wrong here. +bind-address=0.0.0.0:5097 +migrate-on-upgrade=yes +``` + +Keys that must agree instance-wide: + +| Key | Consequence of disagreement | +| --- | --- | +| `[sr.ht] network-key` | decrypts the unified-login cookie and signs internal service-to-service auth; wrong ⇒ every user looks logged out on your service alone | +| `[webhooks] private-key` | core-go also derives the instance-wide bearer-token HMAC key from it — rotating it invalidates **every** tokens.sr.ht working token at once | +| `[<svc>.sr.ht] origin` | must equal the Traefik router `Host()` / nginx `server_name` (§6) | +| `[<svc>.sr.ht] scopes` | must marshal as `[]`, **never `null`** — see below | +| CI's `CORE_VER` / `BOOTSTRAP_REV` | must track the deployment's `SRHT_CORE_VER` or your theme silently drifts from the rest of the instance; invisible from any single repo's CI log | + +**The `scopes: null` trap.** meta.sr.ht's token-minting page reads OAuth-scope discovery +from every service's `/query/api-meta.json`. A service with no scopes that passes a nil +slice to core-go's `coreserver.WithSchema(schema, nil)` marshals `"scopes": null` and +makes that page **500 for every service on the instance, not just the broken one** +(`phoebe-lab/srht/CLAUDE.md`, fixed in spec.sr.ht 0.0.111). Pass `[]string{}`. + +**A config-template edit does not propagate on its own.** `docker compose up -d` never +re-renders an already-exited `config-init`, and `labng push srht`'s `post_push` only +recreates containers whose *image* moved — a config-only change leaves every untouched +service serving the old answer indefinitely (Python services cache `config.ini` at +*import* time). Force-recreate `config-init`, then explicitly restart every consumer, and +verify by comparing container start time against the rendered file's mtime +(`docker exec … stat /etc/sr.ht/config.ini`) — not by asking a running process what it +believes. + +--- + +## 9. CI + +One linear builds.sr.ht manifest per repo, same shape in all nine +(`family::sourcehut-bench/.build.yml`): + +``` +cacher_install → cacher_init → scss → keygen → version → cache_restore → +postgres → test → build → publish → publish_artifacts → cache_save → +[coverage → bench self-upload] +``` + +| Task | What it does | +| --- | --- | +| `scss` | clones core.sr.ht at `CORE_VER` + Bootstrap at `BOOTSTRAP_REV`, assembles them into `/usr/share/sourcehut/scss`; cached by both pins so an upstream outage cannot fail the build | +| `keygen` | `abuild-keygen -a -n -i -q` — throwaway signing key; `-i` is not optional | +| `version` | produces the stamp (below) | +| `cache_restore` | S3-backed `cacher` keyed on a hash of `go.sum`; repairs a half-restored module cache via `go mod verify` and discards on failure | +| `postgres` | a real Postgres started in the CI VM (`initdb`/`pg_ctl`, `fsync=off`), because `!check` makes this the *only* place the database suites ever run | +| `test` | fails loudly if the test DSN is empty — that check is the only thing between a skipped suite and a green build | +| `build` | `REPODEST=$HOME/packages abuild -d` | +| `publish` | `rclone` the signed apk into the Garage `repo` bucket, **only if** `~/.apk-ci.env` exists; its absence is a clean `exit 0` | +| `publish_artifacts` | `PUT` the same apk into `~bigbes/main/apk/v3.22` on artifacts.sr.ht using the `~/.srht-token` bearer credential — the channel production installs from. A `409` (same version, different bytes; abuild stamps mtimes) is treated as success | + +**The version stamp:** + +```sh +cd "$REPO" +desc=$(git describe --tags --always --dirty) +base=${desc%-dirty} +case "$base" in + v*-g*) n=${base%-g*}; ver="${n%-*}"; ver="${ver#v}_git${n##*-}" ;; # X.Y.Z_git<n> + v*) ver="${base#v}" ;; # X.Y.Z + *) ver="0.0.$(git rev-list --count HEAD)" ;; # untagged repo +esac +echo "export PKGVER=$ver SRHT_VERSION=$desc" >> ~/.buildenv +echo "building $ver from $desc" +git status --porcelain # diagnostic, not a gate +``` + +`_git<n>` sorts *after* the plain release in Alpine's version grammar. The trailing +`git status --porcelain` is evidence for later debugging — the last moment the tree is +provably clean — not the gate; `make check-version` is the gate. + +**Traps that actually fired:** + +- **`go mod download all` dirties `go.sum`.** `all` walks the whole module graph and + appends test-only sums to the **tracked** `go.sum` — 228 lines, exit 0, no output — + dirtying the checkout for every later `go build`. `-mod=readonly` does not prevent it. + Killed `sourcehut-bench` build #359 at `check-version`. Use plain `go mod download`. +- **`GOTMPDIR` defaulting inside the checkout.** abuild's default `tmpdir` is + `$startdir/tmp`, so parallel `go build` invocations from one Makefile race: one binary's + transient work directory is untracked and visible to another's `git status` read. Hit + twice in `sourcehut-federation` (jobs #530/#531 — two binaries clean, one dirty, and the + after-the-fact diagnostic clean because Go had already removed the directory). Fixed by + pinning `GOTMPDIR="$HOME/.cache/go-tmp"` in the APKBUILD, on top of the family-standard + `/tmp/` `.gitignore` entry. **Pin it from day one for any multi-binary build** — only + `sourcehut-federation` does today, and whether the other eight are exposed is + *unverified*. +- **The manifest size cap.** builds.sr.ht stores the submitted manifest in a + `varchar(16384)`; a manifest over 16 KiB **cannot be submitted at all**, and the failure + is a branch with no CI, not a red build. Put the reasoning in `docs/ci.md` and a pointer + in the manifest. + +Secrets to request in `secrets:` (all FILE, mode 600): `~/.apk-ci.env` (apk-ci-s3), the +two `~/.s3-cache-key-{id,secret}` files for `cacher`, and `~/.srht-token` — a tokens.sr.ht +working token needing the `bench:upload`, `cov:upload` and `artifacts:upload` grants. + +--- + +## 10. Day-one runbook + +1. **Scaffold the repo.** Copy `family::sourcehut-tokens/APKBUILD` (stateful) or + `family::sourcehut-compare/APKBUILD` (stateless) and its Makefile. Wire + `check-version`, `check-css`, `check-embedded-css` and `install-files` in from the + start — four of the nine siblings skipped `check-version` and it is the single + highest-value gate in the family. +2. **Depend on `sourcecraft.dev/bigbes/sr-ht-core`** (the fork carrying the S3 checksum + patch, `patches/README.md`) as an ordinary pinned `go.mod` dependency — **not** a + `replace`. All nine services do this, whether or not they touch S3 today. +3. **Add `.build.yml`**, cloned from `family::sourcehut-bench/.build.yml`, with this + repo's own `sources:`/`environment:` and `CORE_VER`/`BOOTSTRAP_REV` pinned to the + deployment's current `SRHT_CORE_VER` (0.84.5 as of this writing). Keep it under 16 KiB. +4. **Push once with no secrets reachable** and confirm `test`/`build` go green while + `publish`/`publish_artifacts` no-op cleanly (`exit 0` on a missing credential file). +5. **Claim the port and the config section** (§2), then add the `[<svc>.sr.ht]` block to + `phoebe-lab/srht/config/config.ini.tmpl` (§8) — `origin`, `connection-string`, + `bind-address=0.0.0.0:<port>`, `migrate-on-upgrade=yes`, `scopes=[]` if none. +6. **Add `Dockerfile.<svc>` and `scripts/entrypoint-<svc>.sh`** (§4), plus a + `docker-compose.yml` service block with `<<: *srht-common`, the Traefik labels (§6), + `depends_on: config-init` + `postgres-init` (both `service_completed_successfully`) and + any peer service whose API you call at startup, and the `logging: driver: local` block. +7. **Add the service prefix to `scripts/init-db.sh`'s `SERVICES`** (§7). Skip for a + stateless service. +8. **Add every new file to `deploy.yml`'s `files:` list.** *This is the step whose + omission has broken the whole stack's deploys before* (`e69d730`/`12b1ab9`): anything a + `COPY` references — Dockerfile, entrypoint, keys — that is not listed there never + reaches phoebe, and `labng push srht` then fails for **every** service, not just the new + one. Note that `Dockerfile.gosrht` and `Dockerfile.fedgw` are currently absent from + that list, which is survivable only because both are profile-gated and never built. +9. **Add a `scrape.yml` entry using the container name and port**, never the public + domain — `/metrics` is commonly refused for a request naming a public host, by design. + If the service is profile-gated, leave the scrape entry commented out with a note: an + active target for a container that does not exist is a "no such host" line per scrape + interval, forever. +10. **Set `SRHT_<SVC>_VER` in `versions.env`** once CI has published, initially behind a + compose `profiles: ["<svc>"]` line exactly as `go.sr.ht` and `fedgw` are today. That + lets the block exist in `docker-compose.yml` before it is live and keeps + `just build`/`just up` from touching it. +11. **Deploy** — `labng push srht`, or locally `just build && just up`. Verify three + things: `docker logs srht-<svc>-1 | grep version=` matches the CI job's commit; the + route resolves under **both** the internal container name and the public Traefik router + (Host-sensitive, §6); and meta's personal-access-token page still renders (§8). +12. **Force-recreate `config-init` and restart every consumer** — the nav is built by each + service from *its own* copy of config, so your `[<svc>.sr.ht] origin` reaches the other + services' nav only after they re-read the file (§8). This is the step that makes the + service *appear*, and it is separate from the one that makes it *reachable*. +13. **Write `contrib/<svc>.service` and `contrib/<svc>.conf`** (§5, §6) even though + production consumes neither. + +--- + +## 11. Stated as unverified + +- Whether the eight repos other than `sourcehut-federation` are exposed to the `GOTMPDIR` + race — none of them pins it, and none has been observed failing. +- Whether the two version-stamping mechanisms in use (explicit `VERSION=` ldflags in + artifacts/coverage/tokens; Go's automatic VCS stamp alone in the rest) both produce the + `version=` startup log line that `phoebe-lab/srht/CLAUDE.md` documents as the universal + deploy check. Not traced into each `main.go`. +- `sourcehut-curator` is the only repo with an apk `install=` hook (`post-upgrade`). It + cannot fire under this deployment model — packages are installed fresh into a new image + layer and migrations run from the entrypoint, so no in-place `apk upgrade` ever happens. + Do not copy it. curator is also not deployed at all yet (`SRHT_GO_VER` commented out). diff --git a/skills/sourcehut-custom-service/references/ecore.md b/skills/sourcehut-custom-service/references/ecore.md new file mode 100644 index 0000000000000000000000000000000000000000..bcf3296ab948c84a5800c8372208459a08ec3835 --- /dev/null +++ b/skills/sourcehut-custom-service/references/ecore.md @@ -0,0 +1,724 @@ +# Shared libraries: `sr-ht-ecore` and the `sr-ht-core` fork + +> Citations: `family::<repo>/<path>::<symbol>` — this instance's own repos; `mirror::<path>::<symbol>` +> — the upstream documentation mirror. Both roots are substituted at install time. Symbols, never +> line numbers. + +Two Go modules stand behind every service on this instance. Neither is upstream `core-go`. + +| module | what it is | +| --- | --- | +| `sourcecraft.dev/bigbes/sr-ht-core` | fork of upstream `core-go` — module renamed, plus the S3 checksum patch and the `errors.OnPath` data-race fix. Nothing instance-specific added. | +| `sourcecraft.dev/bigbes/sr-ht-ecore` | a **new** library written after ten of these services: web chrome, unified-login decode, the two bearer planes, service-to-service auth, config reading, logging policy, test bootstrap. | + +ecore (`go 1.25.0`) depends on the fork, so you cannot mix ecore with upstream +`core-go` in one build without a `replace`. Take the fork. Import both: a service +on `sr-ht-core` alone re-derives nine services' worth of debugged web chrome by hand. + +The credential *model* behind `login`/`bearer`/`metapat`/`internalauth` — which +credential a surface accepts, and why — is `references/auth.md`. This page is the API. + +--- + +## Division of labour + +| concern | reach for | +| --- | --- | +| config **loading** (`config.ini` parse, daemon bootstrap) | **core** `config.LoadConfig`, `server.New` | +| config **reading** (origins, internal vs external, required keys) | **ecore** `instconf` | +| identity from the unified-login cookie | **ecore** `login` — never decode the cookie by hand | +| bearer / working tokens (tokens.sr.ht) | **ecore** `bearer` | +| meta.sr.ht personal access tokens | **ecore** `metapat` | +| internal service-to-service auth (mint **and** verify) | **ecore** `internalauth` | +| grant-string parsing | **ecore** `grants` | +| DB access | **core** `database` — no ecore equivalent | +| Redis | **core** `redis` — no ecore equivalent | +| GraphQL server, schema, resolvers, dataloaders | **core** `server` + gqlgen — ecore does not touch this | +| GraphQL error sentinels | **core** `errors` (the fork; upstream has the race) | +| webhooks | **core** `webhooks` — no ecore equivalent | +| HTTP middleware (private cache, panic recovery, 499) | **ecore** `middleware` | +| chi middleware (GET+HEAD, 404/405 rendering, request log) | **ecore** `chimw` | +| page chrome / nav / service switcher | **ecore** `chrome` | +| template set + safe rendering + error pages | **ecore** `pages` | +| CSRF | **ecore** `csrf` | +| static assets, content-addressed CSS | **ecore** `assets` | +| logging policy (level, colour, masking) | **ecore** `logging` (the handler is `auxilia/scribe`) | +| `api-meta.json` | **ecore** `apimeta` | +| MCP HTTP transport plumbing | **ecore** `mcphttp` | +| test bootstrap (fixture config, crypto init) | **ecore** `ecoretest` | +| S3 / object storage | **core** `objects` (fork — carries the checksum patch) | + +--- + +## ecore package reference + +### `instconf` — read the config + +Origin keys, read one way. Five services had grown five different +`TrimRight`/`TrimSuffix`/host-extraction helpers that disagreed on malformed input. + +```go +func ExternalOrigin(conf ini.File, section string) string // [section] origin +func InternalOrigin(conf ini.File, section string) string // internal-origin → origin +func InternalAPIOrigin(conf ini.File, section string) (string, bool) // 4-key ladder +func CanonicalOrigin(origin string) string +func OriginHost(origin string) string // host, no port +func OriginAuthority(origin string) string // host[:port] +func Need(section, name string) Key +func NeedAny(section string, names ...string) Key +func (k Key) Because(why string) Key +func Require(conf ini.File, keys ...Key) error // wraps ErrIncompleteConfig +``` + +The `InternalAPIOrigin` ladder: `api-internal-origin`, `internal-origin`, +`api-origin`, `origin`. `family::sourcehut-dolt/cmd/doltsrht/main.go::resolveSettings`: + +```go +if err := instconf.Require(conf, + instconf.Need(serviceName, "connection-string"), + instconf.Need(serviceName, "origin"), +); err != nil { + return settings{}, culpa.WithHint(culpa.Wrap(err, "reading the config"), + "origin is what places this service in every other service's nav") +} +``` + +- `OriginHost` returns `""` for anything malformed — **never a guessed + `"localhost"`**. A donor's older copy did guess, which silently made every + misconfigured origin agree with a local client on a DNS-rebinding guard. +- `OriginHost` vs `OriginAuthority` are two concepts, not spelling drift: a + Host-header check wants the bare host (the request's port is already stripped); + a JWT audience or sealed URL wants the authority, since `:8080` and `:9090` are + different audiences. +- `Require` reports **every** missing key in one `MissingKeysError`. Call it + before `config.GetAPI` or `crypto.InitCrypto`, both of which panic on missing + config with a message aimed at a programmer, not an operator. +- A present-but-blank value (`origin =`) counts as missing. + +### `logging` — slog policy, not a handler + +```go +func Defaults(conf ini.File, section string) Options // Options{Level, AddSource, Color, TimeFormat, MaskKeys, MaskPattern, MaskReplacement} +func DefaultsWithoutDebugFlag(conf ini.File, section string) Options +func Install(h slog.Handler) *slog.Logger // sets slog.SetDefault; call from main +func (o Options) ReplaceAttr() func(groups []string, a slog.Attr) slog.Attr +``` + +`family::sourcehut-bench/cmd/benchsrht/main.go::newHandler` feeds `Options` to +scribe (`scribe.WithLevel(opts.Level)`, `WithNoColor(!opts.Color)`, +`WithMaskKeys(opts.MaskKeys...)`, `WithMask(opts.MaskPattern, opts.MaskReplacement)`) +then `logging.Install(newHandler(conf))`. Without scribe, `Options.ReplaceAttr()` +plugs the same policy into a stdlib `slog.HandlerOptions`. + +- `middleware.RecoverPanics` and `chimw.RequestLogger` report through + `slog.Default()`. Skip `Install` and panics still log — unlevelled, unmasked, in + a format nothing else uses. `Install` is what makes ecore's middleware log right. +- Verbosity, strongest first: `-d` in argv → `$LOG_LEVEL` → `[section] log-level` + → `info`. `-d` is read straight off `os.Args` by a bare token scan (no getopt + clustering) because `server.New` parses argv *after* config load; for verbose + startup logging call `Defaults(nil, "")` before config exists. + `DefaultsWithoutDebugFlag` is for a binary whose `-d` means something else. +- `MaskPattern` matches the attribute **key path**, never the message or value: + `token`, `secret`, `api_key`/`apikey`, `password`, `pubkey`, `credential`, + `dsn`, `authorization`, `cookie`, plus an exact-key list. It masks `token_id` + too; for correlatable ids install `PartialMaskPattern`/`PartialMaskKeep` (keeps + 6 chars) **before** the blanket rule — first match wins. + +### `login` — the unified-login cookie + +The one decoder of `sr.ht.unified-login.v1` (see `mirror::core.sr.ht/srht/app/flask.py::get_session_cookie` +for the Python side that sets it). + +```go +const CookieName = "sr.ht.unified-login.v1" +const MaxUsernameLen = 64 +func Optional(opts ...Option) func(http.Handler) http.Handler +func Required(deny http.HandlerFunc, opts ...Option) func(http.Handler) http.Handler +func FromContext(ctx context.Context) string // "" == anonymous +func NewContext(ctx context.Context, username string) context.Context +func Username(value string, opts ...Option) string +func UsernameFromRequest(r *http.Request, opts ...Option) string +func WithValidator(valid func(string) bool) Option +func ValidName(name string) bool +``` + +`family::sourcehut-compare/cmd/comparesrht/main.go` installs `r.Use(login.Optional())` +and handlers read `login.FromContext(ctx)`. `family::sourcehut-artifacts/authn/cookie.go` +wraps both behind its own package (`return login.Optional()` / `return login.Required(deny)`). + +- Decrypts with `crypto.DecryptWithoutExpiration`, **never a TTL decrypt**. The + cookie has no service-side lifetime by design: its lifetime is the browser + cookie's `Expires` plus meta's key rotation, both instance-wide. A service that + adds its own TTL logs the viewer out of that one service on a schedule no + sibling shares — which reads as "this service is broken", not "I was logged out". +- **Every failure decodes to `""` — never an error, never a log line.** No + cookie, forged, truncated, wrong key, wrong JSON, invalid name: all identical. + The viewer cannot act on the distinction, and logging decrypt failures is a log + flood any attacker triggers by sending garbage cookies on every request. +- `Optional` vs `Required` is per-service *policy* about whether anonymous is a + valid viewer (compare/dolt: yes, public browsing must work; tokens.sr.ht: no, + every page is the viewer's own list). Install exactly one, never both. +- The name validator cannot be disabled — `WithValidator(nil)` restores the + default. A decoded name is attacker-influenced text headed for a path join, a + log line, a SQL parameter, possibly a GraphQL query to meta. +- Turning a username into a local DB row (`auth.LookupUser`) deliberately stayed + out: a name is instance-wide, a row is not. + +Needs `crypto.InitCrypto`. + +### `bearer` — tokens.sr.ht working tokens + +Four ordered steps, cheapest and local first: decode/verify, ownership, grant, +revocation (network, cached). + +```go +const TokensClientID = "tokens.sr.ht" +const DefaultCacheTTL = 60 * time.Second +func New(opts Options) (*Validator, error) // Options{Origin, ClientID, NodeID, HTTPClient, CacheTTL, Now} +func (v *Validator) Validate(ctx context.Context, presented, action string) (*Token, error) +func (v *Validator) Inspect(ctx context.Context, presented string) (*Token, error) +func (v *Validator) Forget(id int) +func (t *Token) Authorize(action string) error +func (t *Token) Registered() bool +func StatusFor(err error) int +func IsRefusal(err error) bool +func Challenge(realm string) string +``` + +`Token{Username string, Grants grants.Grants, TokenID int, Expires time.Time}`. +Sentinels: `ErrInvalid` (401), `ErrNotOurs`, `ErrForbidden` (403), `ErrRevoked` +(401), `ErrUnavailable` (503). +`family::sourcehut-artifacts/cmd/artifactsrht/main.go::instanceTokensWithHostname`: + +```go +origin := instconf.InternalOrigin(conf, tokensConfigSection) +nodeID, err := hostname() +validator, err := bearer.New(bearer.Options{ + Origin: origin, + ClientID: service.ConfigSection, + NodeID: nodeID, +}) +``` + +One `Validator` per process, held for the process lifetime. + +- `ErrUnavailable` maps to **503, never 401**. Treating an unreachable + tokens.sr.ht as "revoked" refuses every registered token on the instance for + the length of the outage. `StatusFor` is that switch — do not hand-write it + (one service wrote it three times before this was centralized). +- `ErrNotOurs` is deliberately **not** decided here: whether a foreign bearer + token is acceptable is per-service, per-surface policy. Route with + `metapat.PlaneOf` rather than catching this error. +- `Validate` vs `Inspect`: identity resolution happens in middleware, before + routing knows the action; authorization needs the action, known only in the + handler. Use `Inspect` in middleware and `Token.Authorize` in the handler — + forcing `Validate` into middleware means inventing an action there. +- Revocation cache: 60 s default, per-process, bounded at 4096 entries (sweep + expired, then drop-whole on overflow — never LRU). A revocation takes up to + `CacheTTL` to propagate; documented trade, not a bug. +- Needs `crypto.InitCrypto`, unchecked per request. Mints its own revocation-check + header via `internalauth.Authorization`, so `bearer` depends on `internalauth` + and `grants`. + +### `metapat` — meta.sr.ht personal access tokens + +The second bearer plane. api.sr.ht forwards one client credential (usually a meta +PAT) to every service a federated query touches, so a GraphQL endpoint that +refuses meta PATs cannot be federated at all. + +```go +func New(opts Options) (*Validator, error) // Options{Service, Backend, CacheTTL, Now} +func (v *Validator) Resolve(ctx context.Context, presented string) (*auth.AuthContext, error) +func (v *Validator) Forget(presented string) +func PlaneOf(presented string) Plane // PlaneUnknown | PlaneWorking | PlaneMeta +func Allows(ac *auth.AuthContext, scope, mode string) bool +func Scope(service, scope string) string +func ScopeName(scope string) string +func CoreBackend() Backend +``` + +`family::sourcehut-artifacts/cmd/artifactsrht/graphql.go`: +`validator, err := metapat.New(metapat.Options{Service: core.ServiceName})`. +Dispatch, `family::sourcehut-coverage/graph/server.go`: + +```go +if metapat.PlaneOf(presented) == metapat.PlaneMeta { + required = authn.ScopeRead + resolved, err = meta.VerifyToken(r.Context(), presented) +} else { + resolved, err = auth.Resolve(r.Context(), r) + if err == nil { + err = resolved.Authorize(authn.GrantRead) + } +} +``` + +- `Options.Service` is **required**, not convenience. `auth.DecodeGrants` needs a + service name in its context to expand a grant written without a service prefix, + and `config.ServiceName` **panics** rather than returning `""` when nothing set + it. Relying on the ambient config middleware works behind an HTTP router and + crashes anywhere else — background job, CLI, test. +- `PlaneOf` exists because routing by catching `bearer.ErrNotOurs` breaks on an + instance with no `[tokens.sr.ht]` section: there is no working-token validator + to fail first, so there is nothing to fall back *from*. `PlaneOf` decodes + locally and routes before either validator runs. +- `Allows` returns `true` unconditionally for a caller with no `BearerToken` at + all (cookie session, anonymous, or a resolved working token). Not a hole — + those callers were never scoped by meta's OAuth vocabulary and are gated by + their own mechanism. +- Only **successes** are cached (60 s). A failed lookup is one cheap local HMAC + to reproduce; caching failures would pin a transient meta outage to "revoked" + for the whole TTL. +- Scope enforcement is deliberately not part of `Resolve`: `Resolve` establishes + identity, the surface knows which scope and mode it is about to exercise. + +Needs `crypto.InitCrypto` and `bearer` importable (for `TokensClientID`). + +### `internalauth` — `Authorization: Internal <token>`, both ends + +Upstream implements only the receiving check, unexported, inside +`mirror::core-go/auth/middleware.go::internalAuth`. This package does both. + +```go +const Scheme = "Internal" +const Expiry = 30 * time.Second +func Guard(clientID, nodeID string, deny http.HandlerFunc) func(http.Handler) http.Handler +func Verify(r *http.Request, clientID, nodeID string) error +func Identify(r *http.Request, clientID, nodeID string) (Auth, error) +func Authorization(clientID, nodeID string) (string, error) +func AuthorizationAs(username, clientID, nodeID string) (string, error) +func FromContext(ctx context.Context) (Auth, bool) +func Reason(ctx context.Context) error +func Status(err error) int +func Deny(w http.ResponseWriter, r *http.Request) +``` + +Sentinels: `ErrSourceIP`, `ErrMissing`, `ErrToken`, `ErrPayload`, `ErrPeer`, +`ErrNetworkKey`. `family::sourcehut-dolt/web/router.go` guards and +`family::sourcehut-dolt/cmd/dolt-git-hook/main.go` mints against the same constants: + +```go +r.With(internalauth.Guard(core.InternalClientID, core.InternalNodeID, nil)). + Post("/internal/repos", a.handleInternalCreate) +// ... +authorization, err := internalauth.AuthorizationAs( + pc.Repo.OwnerName, core.InternalClientID, core.InternalNodeID) +``` + +- Both checks are required and neither alone suffices. The token proves the + caller holds the shared network key; the source-IP check (`config.IsInternalIP`) + is the weaker one, since behind a reverse proxy the address seen is the proxy's + and every forwarded request looks internal. It uses `RemoteAddr`, **never + `X-Forwarded-For`**, which a client sets trivially. +- `Guard("", "", nil)` means "accept any internal caller" (upstream's default). + Pinning to one sibling is ecore's addition and the right default for an + endpoint one specific service calls; if you pass empty strings, say why. +- `ErrPeer` (right instance, wrong service or node) and `ErrSourceIP` (not from + the instance at all) want different alerts — a provisioning bug or stale + deployment versus somebody knocking. +- Sealing recovers the panic `crypto.Encrypt`/`DecryptWithExpiration` raise when + `InitCrypto` has not run and turns it into `ErrNetworkKey`, the only 500 here. + It matters because the minting side runs in a git hook — a process that may + have "just enough of an instance to have loaded a config". + +### `grants` — the tokens.sr.ht grant grammar + +`<service>:<action>` members, space-separated, `*` universal, reserved `id:<n>`. +One parser for the minting daemon and every validator: two parsers disagreeing +about grant syntax is a security hole, not a style difference. + +```go +const Universal = "*"; const MaxGrantsLen = 4096 +func Parse(s string) (Grants, error) // stored/presented string: id: allowed +func ParseRequested(s string) (Grants, error) // caller-supplied string: id: refused +func All() Grants +func (g Grants) Has(grant string) bool +func (g Grants) IsSubsetOf(other Grants) bool +// also: All() bool, Empty() bool, TokenID() int, WithTokenID(int) Grants, Members() []string, String() string +``` + +Consumed inside `bearer`; a service usually reads `Token.Grants` instead of parsing. + +- `Parse` vs `ParseRequested` is a **privilege boundary**. A caller allowed to set + `id:` on a self-chosen grant string could name some *other* live token's id; + that token's owner revoking it would then not revoke this credential. +- An empty grant string parses to `All()` — intentional, for parent tokens minted + before a service existed. But the **zero value of `Grants` is the empty, + admit-nothing set**, deliberately different. "Forgot to call Parse" is safe; + "the token said nothing" is universal. +- Splits on **ASCII whitespace only**, not `strings.Fields`'s Unicode set. A + U+00A0 pasted into a mint form used to yield zero members — which, under the + empty-means-universal rule, turned "no grants stated" into "every grant". +- Members containing an uppercase letter are refused: grants compare literally, + so folding at parse time would move the drift to whichever validator forgot to fold. + +### `chrome` — nav, switcher, brand, section row + +```go +func NewService(conf ini.File, section string) *Service +func (s *Service) Page(r *http.Request, title, username string) Page +func (s *Service) SelfOrigin() string // also MetaOrigin, HubOrigin, SiteName, Environment +func (s *Service) LoginURLFor(r *http.Request) string +func Attach(t *template.Template) (*template.Template, error) // + MustAttach +func Funcs() template.FuncMap // dict, shortsha, reltime, abstime +const DefaultFaviconHref template.URL +``` + +Mutable `*Service` fields, set between `NewService` and the first `Page`: +`StyleHref`, `FaviconHref template.URL`, `Sections []Section`, `Assets map[string]string`. +`family::sourcehut-artifacts/web/server.go`: + +```go +chromeSvc := chrome.NewService(opts.Conf, core.ServiceName) +chromeSvc.StyleHref = cssHref +chromeSvc.Sections = []chrome.Section{ + {Name: "channels", Href: "/", Paths: []string{"/"}, Prefixes: []string{"/~"}}, + {Name: "cache", Href: "/cache", Paths: []string{"/cache"}, Prefixes: []string{"/cache/"}}, +} +``` + +- Switcher membership is `strings.HasSuffix(section, ".sr.ht")` plus a configured + origin — the Go mirror of `mirror::core.sr.ht/srht/app/flask.py::_network`. + **Your config section must literally be `"<name>.sr.ht"`**, whatever host you + serve from, or you never appear. +- `paste`, `pages` and `hub` are excluded from the switcher. Hub is reached only + through the brand, which here is two links (site name → hub, red label → self) + rather than upstream's one: chrome that does not name its own service is worse chrome. +- `Sections`/`Tabs` render only for `username != ""`, matching the switcher's own + rule. Not a permission check — public sections stay reachable by URL. +- `DefaultFaviconHref` is a `data:` URI, not a path: a `<link>` at a build artifact + the binary does not ship 404s on every page load. The `template.URL` typing is + load-bearing (`html/template` rewrites any non-http/https/mailto href to + `#ZgotmplZ`) and doubles as the guard against a request-derived value landing there. +- `chrome.Page` is meant to be embedded in a service's view struct; a same-named + field collides at compile time, not silently. +- `pages.Load` calls `chrome.Attach` internally — do not attach twice. + +### `pages` — template sets and safe rendering + +One `Set` = layout + chrome partials + shared error partial + service partials + +one content page. Two enforced invariants: every page must define a `"content"` +block (checked at `Load`), and every render buffers first. + +```go +func Load(fsys fs.FS, opts Options) (Set, error) // Options{Dir, Layout, PartialPrefix, ContentBlock, Funcs} +func (s Set) Render(w http.ResponseWriter, status int, name string, data any) error +func Error(status int, message string) ErrorData // + (ErrorData).BackTo(href, text) +func Message(status int) string // human-facing; APIMessage is the machine twin +func FormValues(w http.ResponseWriter, r *http.Request, max int64) (url.Values, error) +const ErrorPage = "error"; const ErrorPartial = "srht-error"; const DefaultMaxFormBytes = 1 << 16 +``` + +Zero-value `Options` is the family default (`templates/`, `layout.html`, `_` +prefix, `"content"`). Messages: `NotFoundMessage`, `UnauthorizedMessage`, +`ForbiddenMessage`, `MethodMessage`, `InternalMessage`, `UnavailableMessage`, plus +`API*` twins. Errors: `ErrNoContent`, `ErrUnknownPage`, `ErrInvalidForm`. +`family::sourcehut-artifacts/web/server.go`: `pages.Load(content, pages.Options{Funcs: templateFuncs()})`. + +- The content-block check parses each page file **alone**, without the layout, + because `Lookup("content")` against the assembled set would find the layout's + own default the moment the layout spells its hole `{{block "content" .}}` + instead of `{{template "content" .}}` — `block` *defines* the name it invokes. + "Make the layout's three holes consistent" is a tidying edit no reviewer would + question, and it disarms the guard for every page at once. +- `FormValues` returns `r.PostForm`, **never `r.Form`**. `r.Form` merges the query + string into the body's values, so a mutating handler reading it can be driven + entirely by a URL — and that GET-shaped request is exactly the one `csrf`'s + same-origin guard sees nothing wrong with, because a link really can be same-origin. +- `Render` answers the response in every case. A returned error means the response + is **already sent** (a bare 500, fixed body) and is for the log only; feeding it + to a service's own error path double-answers or recurses. +- That 500 body is the fixed string `"internal server error"`, never the raw + error — a donor used to write the template error to the browser, leaking + template names and payload internals. + +### `csrf` — Origin/Referer same-origin check + +The sole CSRF defence here: there is no per-service session cookie to hang a +synchronizer token on, since identity is meta's cookie on the parent domain and +no service can set its own `SameSite`. + +```go +func Require(selfOrigin string, deny http.HandlerFunc) func(http.Handler) http.Handler +func SameOrigin(r *http.Request, selfOrigin string) bool +func SafeMethod(method string) bool +const Message = "That request did not come from this site, so it was not carried out." +``` + +`family::sourcehut-tokens/web/router.go` installs it over the whole router with a +service-specific `deny` rendering its own error page; +`family::sourcehut-compare/web/router.go` uses `csrf.Require(s.chromeSvc.SelfOrigin(), nil)`. + +- **A request with neither `Origin` nor `Referer` is refused, not allowed.** A + request that will not say where it came from cannot be shown to have come from + us; waving those through reduces the guard to a header the attacker's page omits. +- Comparison is scheme + host **with port**, case-insensitive (RFC 6454 §4). + `:8443` is not the same origin as the bare host. +- Install on the **whole router, before routing** — not per handler. Two of five + original donors checked per handler and both later grew a form that shipped + unguarded. Router-wide also means an unrouted POST is refused rather than 404'd, + closing a route-enumeration side channel. +- A malformed `selfOrigin` fails closed: every mutating request refused. +- `deny` must answer **403, not a redirect**. Redirecting after a POST drops the + body and makes a refused mutation look like it worked. + +### `middleware` — router-agnostic (`net/http` only) + +```go +func PrivateCache(next http.Handler) http.Handler +func SetPrivateCache(w http.ResponseWriter) // non-middleware form, for a deny path +func RecoverPanics(render func(w http.ResponseWriter, r *http.Request, recovered any)) func(http.Handler) http.Handler +const StatusClientClosedRequest = 499 + +r.Use(middleware.RecoverPanics(func(w http.ResponseWriter, r *http.Request, _ any) { + s.renderError(w, r, http.StatusInternalServerError, internalMessage) +})) +r.Use(middleware.PrivateCache) +``` + +- `RecoverPanics` outermost, `PrivateCache` inside it, so the rendered error page + carries the same cache headers as everything else. `chimw.RequestLogger` goes + **outside** `RecoverPanics` — it must observe the final status a panicking + handler produced, not the unwinding stack. +- `http.ErrAbortHandler` is re-panicked unchanged; rendering over it would + resurrect a deliberately abandoned response. +- **A panic after the response has started drops the connection instead of + rendering** — this is where the package parts ways with its donors, which called + the renderer unconditionally. A second response over bytes already on the wire + is a truncated document behind a spurious 200. A second panic *while rendering + the error page* is logged and dropped, not retried. +- 499 is deliberately not any 5xx: it is what the request's own cancelled context + means, and paging an operator every time a tab closes mid-render is not useful. + +### `chimw` — the chi half + +Split from `middleware` so that package imports nothing but `net/http`. + +```go +func GetHead(r chi.Router, pattern string, h http.HandlerFunc) +func RenderRefusals(r chi.Router, render ErrorRenderer) +func RequestLogger(f SlogFormatter) func(http.Handler) http.Handler +func SkipPaths(paths ...string) func(r *http.Request) bool +type ErrorRenderer func(w http.ResponseWriter, r *http.Request, status int, message string) +type SlogFormatter struct{ Logger *slog.Logger; Message string; Skip func(*http.Request) bool } +``` + +`family::sourcehut-dolt/web/router.go`: `chimw.RenderRefusals(r, a.fail)` on the +router, then inside the group `r.Use(chimw.RequestLogger(chimw.SlogFormatter{Skip: +chimw.SkipPaths("/healthz")}))`. + +- **Import alias.** Every service already aliased chi's own middleware package as + `chimw`. Adopting this means renaming that to `chimiddleware` and reserving + `chimw` here. +- `chimw.GetHead` is **not** chi's `middleware.GetHead`. Chi's does not rewrite + `r.Method` — a widely-copied comment claims it does; it sets `RouteMethod` on + the route context and leaves the request alone. This one registers GET and HEAD + on the routing tree, sidestepping the question. +- `RenderRefusals`'s 405 cannot emit an `Allow` header: chi hands the matched-method + list only to its own unexported default handler. Accepted trade. + +### `assets` — content-addressed static files + +```go +const DefaultPrefix = "/static/" +func Resolve(fsys fs.FS, glob, urlPrefix string) (string, error) +func Handler(fsys fs.FS, urlPrefix string, notFound http.Handler) http.Handler +func Lookup(fsys fs.FS, urlPrefix, urlPath string) (string, bool) +func IsHashed(name string) bool; func CacheControl(name string) string +func NormalizePrefix(urlPrefix string) string; func DirFS(dir string) fs.FS + +// family::sourcehut-artifacts/web/server.go +cssHref, err := assets.Resolve(content, cssGlob, assets.DefaultPrefix) +chromeSvc.StyleHref = cssHref +s.static = assets.Handler(staticFS, assets.DefaultPrefix, http.HandlerFunc(s.handleNotFound)) +``` + +- An absent asset resolves to `""`, not an error; the caller decides whether that + is fatal (the shared answer is "not fatal, render unstyled"). `<link href="">` + re-requests the current page, so templates must `{{if}}`-guard it — + `chrome.Page` already does for `StyleHref` and `FaviconHref`. +- Hash floor is 8 hex chars (`\.[0-9a-f]{8,}\.(css|m?js)$`); every Makefile in the + family cuts sha256 to 8. A shorter hex run reads as a version number and gets + hour-cache, not immutable. +- `DirFS("")` returns an always-empty FS; `os.DirFS("")` serves the host filesystem root. +- The handler never publishes a directory listing and delegates non-assets to + `notFound`, so the page's own cache policy survives. + +### `apimeta` — `api-meta.json` + +For a service mounting its own `/query`, which bypasses core-go's authenticated +router auto-serving this file. + +```go +const Path = "/query/api-meta.json" +func Handler(scopes ...string) http.HandlerFunc +type Meta struct { + Scopes []string `json:"scopes"` + WebhookPubkey string `json:"webhook-pubkey"` +} +``` + +`family::sourcehut-artifacts/cmd/artifactsrht/main.go`: +`router.Get(apimeta.Path, apimeta.Handler(personalTokenScopes...))`. + +The document is marshalled **once**, at handler construction, so +`crypto.InitCrypto` must already have run or `WebhookPubkey` publishes as an empty +string — a boot-order bug, not reported per request. Scopes marshal as `[]`, never +`null`: meta.sr.ht iterates every service's scopes when rendering +`/oauth2/personal-token`, and a `null` there 500s **the whole instance's** grant page. + +### `mcphttp` — MCP HTTP plumbing + +The ~40 lines genuinely shared between MCP endpoints. Deliberately not a +framework: auth gates, tool sets and service interfaces stay per service, because +the two donors made genuinely different policy choices. + +```go +func StreamableOptions() *mcp.StreamableHTTPOptions // Stateless + DisableLocalhostProtection +func HostGuard(next http.Handler, origin string) (http.Handler, error) +func PrivateCache(next http.Handler) http.Handler +var ErrNoOriginHost + +h := mcp.NewStreamableHTTPHandler( + func(*http.Request) *mcp.Server { return srv }, mcphttp.StreamableOptions()) +guarded, err := mcphttp.HostGuard(myAuthGate(h), origin) +r.Handle("/mcp", mcphttp.PrivateCache(guarded)) +``` + +- `mcphttp.PrivateCache` **cannot be `middleware.PrivateCache`**: that one sets + headers on the way in, and the SDK's streamable transport calls + `Header().Set("Cache-Control", …)` from inside the handler, overwriting them. + This wrapper commits at `WriteHeader`/`Write`/`Flush` time. +- `DisableLocalhostProtection: true` is a matched pair with `HostGuard`, not a + regression on its own. Every daemon binds `127.0.0.1` behind nginx, which + forwards the instance's public Host, so the SDK's "loopback address + + non-loopback Host ⇒ reject" rule would 403 every genuine production request. + `HostGuard` replaces it with an allowlist keyed on the instance's origin, and + **fails closed** on an origin with no resolvable host — so adopting it changes + behaviour for a service that previously warned and served unguarded. +- `Stateless: true` is load-bearing for **identity**, not performance: in the + SDK's stateful mode a tool handler runs under the context of the request that + *opened* the session, not the one carrying the call — the session id becomes an + ambient credential, and a token revoked mid-session keeps working until the + client reconnects. Build options through `StreamableOptions()`, not by hand; + it returns a fresh value per call because the SDK takes a pointer. + +Needs `instconf.OriginHost`. + +### `ecoretest` — test bootstrap + +```go +func InitCrypto() // idempotent, from TestMain +func Config(section string, overrides ...func(ini.File)) ini.File // fresh maps every call +func Set(section, key, value string) func(ini.File) // + Delete(sections...), Section(name, values) +func Origin(section string) string +const NoOrigin = "ghost.sr.ht" +const NetworkKey, WebhookKey, SiteName, Environment, OwnerName, OwnerEmail + +func TestMain(m *testing.M) { ecoretest.InitCrypto(); os.Exit(m.Run()) } +conf := ecoretest.Config(configSection, ecoretest.Set(configSection, "origin", srv.URL)) +``` + +The keys are fixed constants, not generated: constancy is what lets two packages +of one service both initialise without the second rotating what the first sealed +with. `Config` allocates fresh section maps every call so one test's `Delete` +cannot leak into another's fixture. `NoOrigin` is a configured section with *no* +origin, present so every service tests the "must not appear in the switcher" rule +rather than only the services that remembered a fixture. + +--- + +## Construction order + +1. `logging.Defaults` + `logging.Install`, from `main`. Everything after logs + through it, including panic reports from `middleware`/`chimw`. +2. `config.LoadConfig` (core). +3. `instconf.Require(...)` — before anything that panics on missing config. +4. `crypto.InitCrypto(conf)` (core; `server.New` does it for you). +5. `bearer.New`, `metapat.New`, `internalauth.*` — all need step 4. +6. `pages.Load`, `assets.Resolve`, `chrome.NewService` — mutually independent; + only rendering a `Page` needs both chrome and pages. Set `chromeSvc.StyleHref` + and `.Sections` before the first `Page` call. +7. Router: `chimw.RequestLogger` → `middleware.RecoverPanics` → + `middleware.PrivateCache` → `csrf.Require` → `login.Optional`/`Required` → + routes; `chimw.RenderRefusals` on the router itself. + +Internal edges: `bearer` → `internalauth` + `grants`; `metapat` → `bearer`; +`pages` → `chrome`; `chimw` → `pages` + `middleware`; `mcphttp` → `instconf`. +Everything else is standalone. + +--- + +## The `sr-ht-core` fork + +**Why**: to pin the S3 checksum fix as an ordinary `go.mod` dependency rather than +a `replace` onto a local checkout. + +**Import path**: `sourcecraft.dev/bigbes/sr-ht-core/...` everywhere upstream docs +say `git.sr.ht/~sircmpwn/core-go/...`. + +Three differences from upstream: + +1. **Module rename** — for most packages that is the whole diff. +2. **`family::sr-ht-core/objects/middleware.go::NewClient`** — the checksum patch. + It swaps the AWS SDK v2 finalize-stage middleware `"ComputePayloadHash"` for + `v4.UnsignedPayload{}` unconditionally, and pins `s3.WithSigV4SigningRegion` so + the V1 endpoint resolver does not stomp the region back to `"default"`. The SDK + only uses `UnsignedPayload` over HTTPS; over HTTP (which `s3-insecure = true` + selects for an internal Garage) it falls through to `ComputePayloadSHA256`, + which seeks the body — and pages publish plus builds artifact upload pass + non-seekable bodies (`tar.Reader`, `io.LimitReader`) to `PutObject`. Note the + S3 keys live in `[objects]` (`s3-upstream`, `s3-access-key`, `s3-secret-key`, + `s3-region`, `s3-insecure`), not in your own section. +3. **`family::sr-ht-core/errors/errors.go::OnPath`** — fixes a live data race. + gqlgen's `graphql.ErrorOnPath` stamps the field path **in place** onto the + `*gqlerror.Error` a resolver returned. The package-level sentinels + (`ErrAccessDenied`, `ErrNotFound`, …) are shared pointers, so the first refused + field in a process permanently stamps its path onto every later refusal: a + wrong answer, a leak of which field another caller asked for, and a global + written from request goroutines. Root-caused live — a refused + `deleteUserWebhook` came back with `"path":["createUserWebhook"]`. The fix + gives the sentinels a non-nil empty `Path` (blocking gqlgen's guard) and + attaches the per-response path to a **copy**: + `func OnPath(err *gqlerror.Error, path ast.Path) *gqlerror.Error`. + No per-service change is needed for it to take effect — generated resolvers + keep calling gqlgen's own `ErrorOnPath`. **Any service built against real + upstream `core-go` still has this bug** if it returns the shared sentinels + under concurrent load. + +Upstream packages still used directly, un-superseded by ecore: + +| package | authoritative for | +| --- | --- | +| `auth` | `DecodeBearerToken`, `AuthCookie`, `LookupUser`, `DecodeGrants`, `AuthContext`. ecore's `bearer`/`metapat`/`login` build **on top of** these. | +| `config` | `LoadConfig`, `GetString`/`GetInt`/`GetBool`, `GetOrigin(conf, svc, external)`, `GetAPI`, `Middleware`, `IsInternalIP`, `ServiceName`. `instconf` wraps the origin reads, it does not replace loading. | +| `crypto` | `InitCrypto(conf)`, `Encrypt`/`Decrypt*`, `Sign`/`Verify`, `BearerHMAC`, `WebhookPubkey`. Every ecore auth package calls into it; none reimplements it. | +| `database` | sqlx/squirrel helpers. | +| `server` | `server.New(service, defaultAddr, conf, args)` — the GraphQL/HTTP bootstrap. ecore decorates the router it hands back. | +| `client` | GraphQL client for calling sibling services. | +| `errors` | gqlgen sentinels + `OnPath`. | +| `model` | cursor/RID/filter helpers for GraphQL pagination. | +| `webhooks` | legacy + GraphQL-native webhook plumbing. `apimeta` only publishes the same key this uses. | +| `objects` | S3, with the patch. | +| `redis`, `valid`, `feature`, `email` | present; no service in this family imports them directly. | + +--- + +## What ecore deliberately does not provide + +Beyond the "core" rows of the division table (GraphQL, webhooks, DB, Redis, S3): + +| you want | do this instead | +| --- | --- | +| an slog handler | `go.bigb.es/auxilia/scribe`, configured from `logging.Options` | +| username → local DB row | your own code; `auth.LookupUser` gives the meta profile, the row is yours | +| MCP tool sets and auth gates | your own package — `mcphttp` is transport plumbing only | +| an `Allow` header on a 405 | unavailable; chi does not expose the matched-method list | +| a per-service session cookie | there is none, which is why `csrf` is an Origin check | + +## Deprecated + +- **`chrome.Service.ExtraNav`** (and `chrome.Page.ExtraNav`). Every historical use + was the same mistake: putting one service's own page into the row that lists the + *instance's* services. Use `Sections` for a service's own nav row. No service + sets it today. diff --git a/skills/sourcehut-custom-service/references/federation.md b/skills/sourcehut-custom-service/references/federation.md new file mode 100644 index 0000000000000000000000000000000000000000..a76980585e4d6f0678cf8d3f54b8adebd2b60df9 --- /dev/null +++ b/skills/sourcehut-custom-service/references/federation.md @@ -0,0 +1,569 @@ +# Federating a custom service into this instance's GraphQL gateway + +Citations: `family::<repo>/<path>::<symbol>` for the sibling repos under the user's home +(`sourcehut-federation`, `sr-ht-api`, `thistle`, `sr-ht-core`, `sr-ht-ecore`, +`sourcehut-*`); `mirror::<path>::<symbol>` for the upstream documentation mirror. Both +prefixes are substituted at install time. Symbols, never line numbers. `phoebe-lab/srht/…` +is the deployment repo and is named without a prefix. + +## 1. State of play + +Upstream federation is dead here, measured, in every clause: + +| Claim | Measurement | +|---|---| +| `api.sr.ht` is deployed | It is not, anywhere. Upstream `git.sr.ht/~sircmpwn/api.sr.ht` is 27 commits, HEAD `d4406cc`, last touched 2023-03-01; the hostname is **NXDOMAIN**. | +| thistle is a dependency you can fetch | `git.sr.ht/~adnano/thistle` **404s**. The only surviving artifact is `proxy.golang.org`'s copy of `v0.0.0-20230301130259-6d6c6a2a22bb`. | +| services answer the federation handshake | **None do.** gqlgen federation is commented out in every upstream service at every tag on this mirror, and no generated federation file exists in any tree. `query { _service { sdl } }` gets "no such field". | +| the schemas merge | **Every pair of the seven upstream schemas fails.** Thirteen non-root type names are defined by several services with different field sets — `User` by all seven, seven distinct shapes — and `version` and `me` are root fields of all of them. | + +So `api-origin=` + SIGHUP federates nothing. Do not write that line and expect a result. + +**What is kept and why.** `family::sr-ht-api` is upstream `api.sr.ht` forked verbatim +(first commit is the upstream tree byte-identical at the upstream module path), plus ~6 +lines of fixes. It is kept as **the diffable reference** and **a fallback that still +builds** — not as the deployed gateway. Its rationale is in `family::sr-ht-api/FORK.md`. + +**What actually runs.** `family::sourcehut-federation`, deployed as the `fedgw` compose +service in `phoebe-lab/srht`, listening on **5116** with a second listener on **5215**. +Twelve services are in its schema today. + +| | `sr-ht-api` (reference/fallback) | `sourcehut-federation` (deployed) | +|---|---|---| +| binaries | `sr-ht-api` | `fedgw`, `fedshim`, `nsify` | +| config | instance `config.ini` | own `fedgw.json` + a `schemas/` dir | +| discovery | `[api]services`, else every `*.sr.ht` section with an API origin | explicit `services[]` array | +| access control | none — forwards whatever arrives | gated on a `federation:query` grant | +| namespacing | none, so it cannot work against a stock instance | `nsify` + per-service shim | +| crypto | needs `[sr.ht] network-key` + `[webhooks] private-key` | none for the handshake | + +## 2. `sourcehut-federation` = nsify + shim + fedgw + +``` +client ──> fedgw ──┬── shim ──> meta.sr.ht /query + ├── shim ──> git.sr.ht /query + └── … (in process, no socket between them) +``` + +**`nsify` — build time.** `family::sourcehut-federation/internal/nsify/nsify.go::Namespace` +parses a service's `schema.graphqls` and prefixes every type and every root field: +`User` → `MetaUser`, `me` → `metaMe`. It works on the gqlparser AST, not by regex, because +a rename must reach field types, argument types, union members, interface lists, list +wrappers and directive argument types (`::rewriter.renameType`, `::rewriter.typeRef`). +Output is two files per service: `<name>.graphqls` (namespaced SDL) and +`<name>.renames.json` (the manifest, `::Renames`). + +- `-reads-only` drops the whole `Mutation` root — `extend type Mutation` and the + `mutation:` entry of an explicit `schema {}` block included + (`::dropMutations`, `::withoutMutationOperation`). Every service on this instance is + namespaced with `-reads-only`. +- Names deliberately left alone (`::reserved`): `String Int Float Boolean ID`, + `Query Mutation Subscription`, gqlgen's `Upload Any Map`, thistle's `_FieldSet`. +- Descriptions are dropped by default: gqlparser's formatter re-emits a description + starting with a quote as `""""like this""""`, which its own parser then rejects. +- Input types left unreferenced by `-reads-only` are kept — legal SDL, they merge, and no + query can name one. + +**The shim — run time, one per service.** +`family::sourcehut-federation/internal/shim/shim.go::Service.Handler` does two jobs: +answers `query { _service { sdl } }` from the namespaced SDL handed to it at construction +(`::isHandshake`, on the *parsed* document, not a substring match), and rewrites an +incoming query by the manifest. **Root fields are renamed with an alias — `metaMe: me` — +not renamed outright**, so the response comes back under the key the gateway asked for and +nothing has to rewrite the answer (`::Service.rewriteDocument`, +`::Service.renameRootSelections`). + +Shim bounds apply to anything reaching your service through the gateway: + +| Bound | Symbol | Value | +|---|---|---| +| request body | `shim.DefaultMaxBodyBytes` | 1 MiB | +| selection depth | `shim.DefaultMaxDepth` | 32 | + +Depth matters because every SourceHut schema has cycles; without it one request is +arbitrary work. Also: `::Service.handle` turns a **non-JSON upstream body** into a +per-field GraphQL error, because a service answering 401 in plain text made the gateway +fail the *whole* federated query with "error decoding response", including every service +that answered correctly. + +**`fedgw` — the endpoint.** `family::sourcehut-federation/internal/gateway/gateway.go::Gateway` +holds the shims **in process**. thistle talks to a service over an `*http.Client`, so a +`RoundTripper` that dispatches to an `http.Handler` (`::localTransport.RoundTrip`) removes +the loopback hop, the second listener and the mount-path convention in one go. The URLs +registered with thistle (`http://<name>/query`) are routing keys and are never resolved. + +`fedshim` (`family::sourcehut-federation/cmd/fedshim/main.go`) is the same shims served +over HTTP at `/svc/<name>/query`, for a gateway that is not ours — a SourceHut gateway +appends `/query` itself, so such a service is configured +`api-origin=http://fedshim:5115/svc/meta.sr.ht`. Not deployed here. + +**Config** (`family::sourcehut-federation/internal/fedcfg/fedcfg.go::Config`), generated, +never hand-edited: + +```json +{ + "listen": "0.0.0.0:5116", + "metrics_listen": "0.0.0.0:5215", + "schema_dir": ".", + "upstream_timeout": "25s", + "auth": { "instance_config": "/etc/sr.ht/config.ini", + "grant": "federation:query", "node_id": "fedgw" }, + "services": [ + {"name": "meta.sr.ht", "upstream": "http://meta:5100/query", "namespaced": true} + ] +} +``` + +- `"namespaced": true` without a manifest is a **startup error**, and so is a manifest + that renames no root field — it would proxy every query unrewritten + (`::Config.buildService`). +- `metrics_listen` is a `*string` so "absent" and "empty" differ; default + `127.0.0.1:5215`, `""` disables, and a config putting it on the query listener is + refused at load. +- `gateway.max_requests` (`fedcfg.DefaultMaxRequests` = 50) bounds the service calls one + federated query may make. `gateway.debug` adds thistle's query-plan extensions and is + **off in production** — a plan names every service a query touched. + +**The gating grant.** Inbound, the caller presents a **tokens.sr.ht working token carrying +`federation:query`**, validated by `family::sr-ht-ecore/bearer/bearer.go::Validator.Validate` +through `family::sourcehut-federation/internal/shim/auth.go::Auth.Validate`. An empty +`auth.grant` is refused at load — it would admit every working token on the instance, +including one minted for another service — and the refusal reason is not relayed to the +caller. The **handshake is not gated**, deliberately: a gateway fetches schemas before any +client exists. `Authorization: Internal …` arriving at a shim is **stripped** +(`::Service.upstreamCredential`); a gateway's internal credential is never a client +credential. `upstream_token_file` (shim presents its own credential outbound) is +deliberately **unset** live, so the caller's token reaches each service the query touches. + +Reload is **SIGHUP** on both gateways (`::Gateway.Refresh` under a 60s context; +`family::sr-ht-api/main.go::updateSchema` for the fallback). Failure policy, worth copying: +a service that does not answer the handshake is logged with the reason and **left out**; a +merge that *fails* keeps the previously-served schema rather than replacing it, because a +schema that does not merge is not a degraded endpoint but a wrong one; building from +**zero** services is refused outright. + +## 3. What your service must do to be federatable + +This is the acting part. Everything else in this page is context for it. + +1. **Serve GraphQL at `/query` on a router that admits a bearer-bearing or anonymous + caller.** `mirror::core-go/server/server.go::WithSchema` puts `/query` on the + *authenticated* router, whose middleware speaks meta's OAuth-cookie vocabulary and 401s + a bearer token — the wrong router for a federatable endpoint. Mount it yourself. + Exemplar: `family::sourcehut-coverage/cmd/coversrht/main.go`. + + ```go + const queryRoute = "/query" // never inline; it is core-go's own path, + // and where hut and meta's PAT page already look + + router.Group(func(r chi.Router) { + r.Use(config.Middleware(conf, service.ConfigSection), database.Middleware(pool)) + r.Handle(queryRoute, gql) + r.Get(apimeta.Path, apimeta.Handler(graph.GrantScopes...)) + }) + ``` + + - `Handle`, not `Mount` — `Mount` rewrites the routing path and claims the `/query/*` + subtree. + - `Group`, not `Use` — chi refuses a `Use` once any route exists on a mux, and the + router already has some. The Group is a fresh inline mux over the same tree. + - **config + database middleware are required even for a read-only schema**: the bearer + plane's owner lookup is core-go's `auth.LookupUser`, which reads both off the request + context. Without them a perfectly good working token is answered **503**. + - `api-meta.json` is registered with `Get` and **outside** the config/database group — + it is a static document, not worth a transaction per poll from meta. + - Register `/query` before a `/`-mounted web tier, for the reader. + +2. **Serve `/query/api-meta.json`** with `family::sr-ht-ecore/apimeta/apimeta.go::Handler` + at `::Path`. Only a service that mounts its own `/query` owes the instance this file — + core-go serves it for schemas it hosts itself, and yours is not one. + **The scope list must marshal as `[]` and never `null`.** meta.sr.ht iterates it when + rendering `/oauth2/personal-token`, and a `null` is a **500 on that page for the whole + instance** — every service's grants, not just yours. Derive the list, never re-spell it: + + ```go + var GrantScopes = []string{metapat.ScopeName(authn.ScopeRead)} + ``` + +3. **Accept a meta.sr.ht personal access token on `/query`, beside your working token.** + This is the hardest requirement, and four services were changed for it in one pass. A + gateway forwards **one** client `Authorization` header to every service a federated + query touches; the only credential that works instance-wide is a meta PAT. An endpoint + that refuses them cannot be federated — the first authenticated query answers 401. + Use `family::sr-ht-ecore/metapat`. Exemplar: `family::sourcehut-specs/authn/meta.go::MetaAuth`. + + Route on the credential, never as a fall-through from a failed plane + (`family::sourcehut-specs/graph/server.go::resolveCaller`): + + ```go + presented := authn.BearerFromRequest(r) + if presented == "" { // overwrite with anonymous, do not pass through: + next.ServeHTTP(w, r.WithContext(authn.WithPrincipal(r.Context(), authn.Anonymous()))) + return + } + if metapat.PlaneOf(presented) == metapat.PlaneMeta { + p, err = meta.VerifyToken(r.Context(), presented) // meta PAT + } else { + p, err = rs.ResolveAgent(r.Context(), presented, ...) // tokens.sr.ht working token + } + ``` + + `metapat.PlaneOf` is one local HMAC — both planes are sealed with the same key and only + the ClientID (`bearer.TokensClientID`) tells them apart. `PlaneUnknown` goes to the + working-token arm so both planes owe an unreadable credential the same 401. + `family::sr-ht-ecore/metapat/metapat.go::Validator.Resolve` is local-first by design: + decode+expiry (no network) → plane check → `LookupUser` → `IsRevoked`, the last two + cached together for `metapat.DefaultCacheTTL` (60s), successes only, bounded at 4096. + Scope enforcement is separate: + `metapat.Allows(ac, metapat.Scope("myservice.sr.ht", "THINGS"), auth.RO)`. Sentinels map + to `ErrInvalid` 401, `ErrForbidden` 403, **`ErrRevoked` 401 not 403** (a client shown 403 + keeps presenting it), `ErrUnavailable` **503, never 401**. `metapat.Options.Service` is + required — `auth.DecodeGrants` reads the service name off the context and + `config.ServiceName` panics rather than returning `""`. + +4. **Keep that plane off the shared resolver.** Build it beside the `graph.New` call and + pass it only as `graph.Options{Meta: …}`. A plane on the `*authn.Resolver` is a plane on + **every** surface — `/mcp`, `/api`, push hooks — and those still owe a tokens.sr.ht + grant. This is structural, not a convention + (`family::sourcehut-specs/cmd/specsrht/main.go`). Make `graph.New` hard-fail on a nil + `Meta`: an endpoint with no PAT plane starts, serves every working token exactly as + before, and answers 401 only to federated queries — a failure visible only from the + gateway. + +5. **Publish the scope, then restart meta.sr.ht.** meta caches its service discovery at + import, so a new scope's checkbox never appears until it restarts. Name the object the + schema reads — `REPORTS`, `RESULTS`, `SPECS`, `ARTIFACTS`. Rename it *before* any token + exists; after that a rename is a credential that stops working. A scope published and + not checked admits what should be refused; one checked and not published cannot be + minted at all. + +6. **Let an insufficient scope degrade to an anonymous read, not a 403** — one credential + reaches every service of a federated query, and a 403 fails the whole query over a + service whose share was public anyway. Keep a forbidden *object* resolving to `null` + rather than to an authorization error, so existence cannot be probed by refusal shape. + +7. **Answer errors as JSON.** A plain-text 401 from your `/query` fails the entire + federated query at the shim, taking every correctly-answering service with it. + +Do **not** write any of these: + +| Not this | Why | +|---|---| +| a `_service { sdl }` resolver | the shim answers the handshake; no service on this instance does | +| `@key`, `_entities`, gqlgen's `federation:` block | this is deliberately not Apollo federation; the decision not to do cross-service joins is recorded and should not be reopened by accident | +| `version: Version!` | upstream's is exactly what collides; no custom service has one | +| `me: User!` non-null | four of six have `me` and it is **always nullable** — "anonymous is a normal caller on this schema, so a null here is an answer and not an error". Two have none and it costs a federated client the ability to ask who it is; add it | +| `@access`/`@internal`/`@private` directives | upstream's enforcement (`mirror::core-go/server/directives.go::Access`) calls `auth.ForContext`, which **panics** under anonymous-friendly plumbing. Not a paste job | +| a node interface / global RID | objects are addressed by natural keys here: `(owner, name)`, a path, an int id | + +Namespacing removes any need to hand-avoid collisions: across all twelve published schemas +the only names defined more than once are `type Query` (merged by thistle's +`mergeRootObjects`) and `scalar Upload` (identical, merges). The binding merge rules, in +`family::thistle/schema.go`, are: root fields disjoint (`mergeRootObjects`); a type defined +by several services must have the **same fields** (`mergeSharedObjects`, arguments merged +by intersection, a missing required argument is an error); `@key` marks an entity and +obliges `_entities` (`mergeEntities`). + +## 4. The gqlgen setup this family uses + +`gqlgen.yml` lives **inside `graph/`**, not at `api/` as upstream does: + +```yaml +schema: [schema.graphqls] +exec: { filename: api/generated.go, package: api } # exec/generated.go where a top-level api/ exists +model: { filename: model/models_gen.go, package: model } +resolver: { layout: follow-schema, dir: ., package: graph } +autobind: ["sourcecraft.dev/bigbes/sr-ht-<svc>/graph/model"] +omit_getters: true +``` + +Nobody sets `omit_slice_element_pointers`, `struct_tag`, `skip_validation`, a `directives:` +stanza, or a `federation:` block. Scalars: `scalar Time` in all six (gqlgen's built-in +binding; do not map it), `Cursor` bound to `family::sr-ht-core/model/cursor.go::Cursor` +where used, `Int64` → `graphql.Int64` where a column holds a bigserial or a byte total — +"Int is the 32-bit integer GraphQL says it is". + +**Generate directive** — `graph/generate.go`, `package graph`, **no build tag, no blank +import** (a blank import puts the whole codegen dependency tree into your go.mod +permanently): + +```go +//go:generate go run github.com/99designs/gqlgen@v0.17.94 generate +``` + +Run `go generate ./graph` from the repo root; no Makefile rule, no `dataloaden` anywhere. +**Check `git status` afterwards**: the generator loads the module through `go list`, and +that subprocess appends indirect requirements and hashes to `go.mod`/`go.sum` (one line and +77 in one measured run). `git checkout go.mod go.sum` if the build is green without them. + +**Layout**, package `graph`: `schema.graphqls`, `schema.resolvers.go` (generated stubs +only), `resolver.go` (hand-written `Resolver` + helpers), `ports.go` (the seams), +`server.go` (package doc + endpoint construction), `model/`, `api/`|`exec/`. Helpers go in +`resolver.go`, **never** at the tail of `schema.resolvers.go` — gqlgen moves anything there +that is not a resolver into a commented-out block at the end of the file, where it stops +compiling and starts rotting. + +The `Resolver` **carries its dependencies** (unlike upstream's `struct{}` + context), built +through an `Options` struct validated in `New` that fails fast on a nil seam. Seams are +declared consumer-side in `graph/` with the production type's own signatures, asserted in +the package rather than in a test: + +```go +var ( + _ Catalog = (*service.Modules)(nil) + _ MetaAuthenticator = (*authn.MetaAuth)(nil) +) +``` + +Read-only is then enforced structurally: adding a mutation means widening that file first, +which is a diff a reviewer sees. + +**Auth into a resolver.** No service uses `mirror::core-go/auth/middleware.go::Middleware` +or `WithSchema` for `/query`, and no resolver calls `auth.ForContext`. Each has a local +`authn` package with a **non-panicking** accessor — `authn.PrincipalFromContext(ctx)` +returning the anonymous zero value — because an unauthenticated request is an ordinary +state here, and a handler reached without the middleware installed must degrade to *less* +authority rather than to a panic. Contrast +`family::sr-ht-core/auth/middleware.go::ForContext`, which panics. Resolvers read it +through a one-line wrapper in `resolver.go` (`viewer`, `callerOf`). +`family::sourcehut-specs/coreauth/coreauth.go::Context` derives core-go's `AuthContext` +afterwards, because the webhook engine reads one out of context and `/query` no longer runs +behind core-go's middleware. + +**Transport: `transport.POST{}` and nothing else**, in all six. Never +`NewDefaultServer` — it installs GET, multipart upload and websockets. A GET query would be +a cross-origin-readable URL for data that is often private, and there is no cookie plane +here to make that safe. **Introspection is on** (`srv.Use(extension.Introspection{})`): a +client that cannot introspect cannot generate a typed client, and everything is already +gated per field. Pin both with tests named like +`family::sourcehut-coverage/graph/graph_test.go::TestGetIsNotATransport` and +`::TestIntrospectionAnswers`. + +**No loaders.** dataloaden is not used at all. N+1 is avoided at the schema/model level: +expensive fields are forced into field resolvers via `{resolver: true}` in `gqlgen.yml`, +models are **hand-written to omit** those fields so gqlgen generates resolvers for them, +and every field calls through the same service layer the web, REST and MCP surfaces use, so +batching decisions live in one place. Exemplar: +`family::sourcehut-dolt/graph/model/database.go::Database` — it carries a private +`*core.Repo` so the field resolvers have the row id and disk path, and +`{ databases { results { name visibility } } }` opens nothing on disk. + +**Error handling — the family has no shared convention; adopt this one.** Nobody calls +`SetErrorPresenter` or `SetRecoverFunc`; only specs uses `family::sr-ht-core/errors` codes +in `Extensions["code"]`, and `sr-ht-core/valid` is used by none. The one shared invariant is +worth keeping verbatim: *"'I could not check' reported as 'there is none' is how a client +learns a false fact about the instance and acts on it."* So a two-bucket policy in +`resolver.go` — `errUnavailable` vs `internalError`, `missingOrDenied` collapsing not-found +and forbidden. Errors are logged with the field name and never echoed; credential refusals +are plain-text HTTP written by middleware *before* gqlgen; 401 always sets +`WWW-Authenticate`; 503 is "could not check" and is never collapsed into 401. Construct a +**fresh** error rather than returning a shared package-level value — gqlgen's `ErrorOnPath` +mutates it in place (`family::sr-ht-core/errors/errors.go::OnPath`). + +**Complexity: nobody installs one, and a new service should.** No +`extension.ComplexityLimit`, no `FixedComplexityLimit`, against +`family::sr-ht-core/server/server.go::WithSchema` which reads `[<svc>::api] max-complexity` +(default 250). Per-field clamps (25/100, 50/200) do **not** compose, the shim's depth 32 is +the only bound through the gateway, and a **direct** caller of your `/query` is not behind +the shim at all. Add `srv.Use(extension.FixedComplexityLimit(n))` reading the same config +key. Note that if you run core-go's webhook worker, `srv.MaxComplexity` **must be non-zero** +— zero there does not mean "no limit", it means every delivery fails with "operation has +complexity 2, which exceeds the maximum of 0". + +**Pagination: three conventions exist; adopt core-go `Cursor`.** dolt and specs use it, +it is what every upstream service on the instance uses, what `hut` already knows, and its +cursor is **encrypted JSON, not base64** (`family::sr-ht-core/model/cursor.go`: +`UnmarshalGQL` = `crypto.DecryptWithoutExpiration` + `json.Unmarshal`), so it cannot be +forged or hand-edited into a page number the schema must then keep honest. + +```graphql +scalar Cursor +input Filter { count: Int = 25 } +type ThingCursor { results: [Thing!]!; cursor: Cursor } +things(cursor: Cursor, filter: Filter): ThingCursor! +``` + +Two rules from `family::sourcehut-dolt/graph/paging.go::page`: a filter given on this call +wins over what the cursor remembers, so a client can change page size mid-walk without +minting a new cursor; and a cursor whose row was deleted resumes at the next row instead of +failing. Changing a published field is a breaking change — pick this before the first +client exists. + +## 5. thistle and the gqlparser pin + +Both forks' first commit is the upstream tree **byte-identical, at the upstream module +path, with no edits**, so it can be diffed against what upstream published — with a +`.gitattributes` disabling line-ending translation added *before* the import, since this +machine's `core.autocrlf=input` would otherwise rewrite the very bytes the commit exists to +preserve. `family::thistle/FORK.md` records the proxy's own `go.sum` hashes, "because a +`go.sum` line is the only independent attestation left". + +What changed: the module path (`git.sr.ht/~adnano/thistle` → +`git.srht.bigb.es/~bigbes/thistle`, a rewritten path rather than a `replace`, so a fork +cannot be confused for upstream by a reader or by a build), and +`github.com/vektah/gqlparser/v2` **v2.5.1 → v2.5.36** for **GO-2024-2920** — a DoS in +`parseDirectives`, fixed in v2.5.14, reachable unauthenticated from any HTTP handler +through `parser.ParseQuery`, because a query is parsed before anything else happens to it. + +**Pin these, with plain `require` and no `replace` anywhere:** + +``` +git.srht.bigb.es/~bigbes/thistle v0.0.0-20260817155400-9563b055ab4c // gateways only +github.com/99designs/gqlgen v0.17.94 +github.com/vektah/gqlparser/v2 v2.5.36 +``` + +v2.5.36 is the family's security floor. Requiring the fixed gqlparser *directly* is what +MVS already selects; bumping the thistle pin as well is about the dependency graph saying +what the build is doing. Consequence of the bump, recorded because it outlives it: thistle +now **advertises `@defer` in introspection while implementing nothing for it** — measured +through a gateway, `{ ... on Query @defer { hello } }` was `Unknown directive "@defer"` at +v2.5.1 and a complete single-payload `{"data":{"hello":"world"}}` at v2.5.36. + +Do not build on `git.sr.ht/~adnano/thistle` directly: `GOFLAGS=-mod=mod GOPROXY=direct` +fails outright, and it works at all only while `proxy.golang.org` keeps answering. + +## 6. MCP is a first-class surface here + +Six of the seven custom services expose MCP as an `mcpsrv/` package. Transport is +**Streamable HTTP mounted at `/mcp`** on the service's own listener — no stdio server, no +SSE. SDK, identical everywhere including ecore: `github.com/modelcontextprotocol/go-sdk +v1.6.1`. + +`family::sr-ht-ecore/mcphttp` is deliberately three functions, not a framework — an audit +of two donors found the genuinely shared part to be about forty lines, and the auth gates +stay in the services because unifying them "would not be deduplication, it would be a +policy change smuggled in as one". + +| Symbol | What | +|---|---| +| `family::sr-ht-ecore/mcphttp/mcphttp.go::StreamableOptions` | `{Stateless: true, DisableLocalhostProtection: true}` — **a fresh value every call**; the SDK takes a pointer and a shared one lets any caller reconfigure every other endpoint | +| `family::sr-ht-ecore/mcphttp/hostguard.go::HostGuard` | Host-header allowlist replacing the SDK's DNS-rebinding guard, from the service origin's host plus loopback. **Fail-closed**: an origin with no host is `ErrNoOriginHost` and the surface refuses to build | +| `family::sr-ht-ecore/mcphttp/cache.go::PrivateCache` | writes `Cache-Control: private, no-store, no-transform` and `Vary: Cookie, Authorization` **at commit time**, because the SDK's transport `Set`s `Cache-Control` from inside the handler and a middleware setting it on the way in loses | + +**The stateless rule, with its reason.** Run the transport `Stateless: true`. The SDK's own +doc says a stateless server does not validate `Mcp-Session-Id` and uses a temporary session +with default initialization parameters. `family::sourcehut-dolt/mcpsrv/http_test.go::TestIdentityIsPerCallAndNotPerSession` +measures the consequence against this exact SDK version: **stateful — handshake anonymous, +call carrying the owner's token → 4 results; stateless → 6.** The token on the call is not +consulted at all. In a stateful configuration the session id becomes a bearer credential in +its own right, and **a token revoked mid-session keeps working until the client +reconnects**. Exemplar of the correct form: `family::sourcehut-dolt/mcpsrv/mcpsrv.go`, +where it is a named constant carrying that argument. Port that test. + +Wiring, outermost first: + +```go +h := mcp.NewStreamableHTTPHandler( + func(*http.Request) *mcp.Server { return srv }, + mcphttp.StreamableOptions(), +) +guarded, err := mcphttp.HostGuard(resolver.Middleware()(gate(h)), cfg.Origin) +router.Handle("/mcp", mcphttp.PrivateCache(guarded)) +``` + +- Register `/mcp` **exactly**, not `/mcp/*`, with `Handle` not `Mount` — one endpoint + speaking POST; GET is 405 in stateless mode. +- Mount it *before* any same-origin/CSRF group: bearer surface, no cookie. +- **Identity at the boundary, grant per tool.** The boundary check covers `initialize` and + `tools/list`, not just `tools/call`; but the tool name lives in the JSON-RPC body, not + the request, so a surface-wide `myservice:read` would refuse a token minted for + `myservice:propose` alone at `initialize`, before it ever named a tool. The reason the + grant check exists: the same credential reaches `/mcp` and `/api`, and a token restricted + to upload that could still read every private repository through an MCP client would make + the REST check decorative. +- Refusals are plain text + `WWW-Authenticate`, never a login redirect — every caller is a + machine. +- One Go struct per tool with `json:` + `jsonschema:` tags; the SDK derives the schema, so + no hand-written JSON schemas. Annotate reads + `&mcp.ToolAnnotations{ReadOnlyHint: true, IdempotentHint: true}`; leave writes unannotated + ("proposing twice opens two proposals"). Declare backend seams consumer-side in + `mcpsrv/ports.go`, naming no seam a write could reach if the surface is read-only. +- Deployment: the proxy must pass `Host` through (`proxy_set_header Host $host`, Traefik + `passHostHeader: true`) or `HostGuard` 403s everything; the daemon must have run + `crypto.InitCrypto` and `config.LoadConfig` first. +- A caller reaching the raw `*mcp.Server` over stdio is **anonymous** — identity comes off + an HTTP header and stdio has none. Expose it only with that caveat stated. + +## 7. Recipe: federate a new service + +**A. In the service** — items 1-7 of §3, plus: keep the schema at `graph/schema.graphqls`, +because the generator reads it **out of git at the pinned ref**, not from a working +checkout. + +**B. In the instance repo (`phoebe-lab/srht`)** + +1. Pin the version in `versions.env` (`SRHT_<SVC>_VER`). +2. Add a row to `table()` in `scripts/schemas.sh`: + `<svc>|SRHT_<SVC>_VER|<repo>|ours|graph/schema.graphqls|<prefix>|http://<host>:<port>/query`. + Use `-` for the schema path if the service genuinely has none — spelled out rather than + omitted, "so that 'not in the list' keeps meaning 'nobody has looked at it'". +3. `just schemas` — regenerates `schemas/` (namespaced SDL + rename manifest per service, + `MANIFEST`, and `fedgw.json` itself) from that one table, so the service list cannot be + right in one file and stale in another. It runs + `nsify -reads-only -prefix <p> -name <svc> -out <dir> <schema>`. + **This belongs in the same commit as the version bump.** +4. `just check-schemas` — regenerates to a temp dir and `diff -ru`s; fails on drift. This is + the gate between a version bump and a federated endpoint advertising fields the service + no longer has. +5. `just verify-schemas [svc]` — introspects each **running** service and diffs the live + schema's names, mapped back through `renames.json`, against what is published. Answers + what a pin cannot: is the running container the version the pin names. Needs `$SRHT_PAT` + (introspection on this instance is authenticated). + +**C. Deploy** + +6. Schemas are **baked into the image** (`COPY schemas/ /usr/share/fedgw/` in + `Dockerfile.fedgw`), not mounted — a container's binary and the schemas it publishes + must not be able to disagree. `just schemas` alone changes nothing on a running gateway. +7. Traefik router `api.${SRHT_DOMAIN}` → **5116**, `passHostHeader: true`. Healthcheck + probes **5215** (`/healthz`; the query listener 404s for it). +8. Add a `srht-fedgw` scrape job at `fedgw:5215` to `scrape.yml`. The compose profile lines, + the `versions.env` pin and this job move together — a scrape target for a container that + does not exist logs "no such host" every interval. +9. `just build && just up`. Without a rebuild, **SIGHUP** refetches every schema and rebuilds + the federated one. + +**D. Verify — check the pieces in order, gateway last** + +```sh +# 1. Your service answers the handshake THROUGH ITS SHIM (never on its own): +curl -s -X POST http://fedgw:5116/query \ + -H 'Content-Type: application/json' -H "Authorization: Bearer $FED_TOKEN" \ + -d '{"query":"query { __schema { queryType { fields { name } } } }"}' \ + | jq -r '.data.__schema.queryType.fields[].name' | grep '^myprefix' + +# 2. THE ONE THAT MATTERS — a real query across two services at once. It proves the +# merge AND the routing AND the alias rewrite in a single answer: +curl -s -X POST https://api.${SRHT_DOMAIN}/query \ + -H 'Content-Type: application/json' -H "Authorization: Bearer $FED_TOKEN" \ + -d '{"query":"query { metaMe { username } myprefixThings { name } }"}' + +# 3. Health and schema metrics (second listener, docker network only): +curl -s http://fedgw:5215/healthz # "OK 12 services" +curl -s http://fedgw:5215/metrics | grep fed_schema_services +``` + +`$FED_TOKEN` is a **tokens.sr.ht working token carrying `federation:query`**. tokens.sr.ht +validates no grant vocabulary, so that grant is mintable from its free-text field today and +is not on its checkbox list. + +**The metric to watch: `fed_schema_services_skipped`.** A schema built from fewer services +than are configured still answers `/healthz` **200** — degraded is not unhealthy, and that +counter is the only place a silently-missing service shows. `/healthz` answers 503 only +until the *first* schema is built. Metrics live in +`family::sourcehut-federation/internal/fedmetrics/fedmetrics.go`: +`fed_service_requests_total`, `fed_service_request_duration_seconds`, +`fed_service_errors_total` per service, plus `fed_query_*` and `fed_schema_*`. The +federation handshake is deliberately excluded from the per-service series. + +Second signal: `family::sourcehut-federation/internal/gateway/gateway.go::logQuery` writes +**one line per federated query** naming the services it touched, the call count and the +slowest — the query text is deliberately absent, since it carries variables and +identifiers. If your service is absent from a query that selected its fields, the **rename +manifest** is what to check; the shim's tests assert on *what upstream received*, precisely +because a rewriter that does nothing still returns a plausible answer. + +Known gap: **file uploads do not work through the gateway.** The shim `json.Unmarshal`s the +body while `fedgw`'s handler registers `transport.MultipartForm{}`, so an upload fails at +the shim. Fragment spreads are assumed consistent across a query. diff --git a/skills/sourcehut-custom-service/references/pitfalls.md b/skills/sourcehut-custom-service/references/pitfalls.md new file mode 100644 index 0000000000000000000000000000000000000000..aa3f74c9499a5d7c44cdd28b16bac420fa76acb3 --- /dev/null +++ b/skills/sourcehut-custom-service/references/pitfalls.md @@ -0,0 +1,639 @@ +# Pitfalls: what the family learned the expensive way + +Every rule below cost a sibling service a debugging session, a security hole, or a silently +green CI. They are grouped by **where you will be standing when it bites you**, not by which +repo taught it. The reason attached to each rule is the load-bearing half — a rule without it +gets discarded by the next reader as arbitrary. + +Citations: `family::<repo>/<path>::<symbol>` for the ten sibling services, `mirror::<path>::<symbol>` +for the upstream SourceHut documentation mirror. Symbols, never line numbers. + +--- + +## 1. Bootstrapping and process lifecycle + +**Pass the full `os.Args` to `server.New`, never `os.Args[1:]`.** core-go's `getopt` skips +`argv[0]` itself, so a pre-trimmed slice makes it swallow your first real flag and fall back to +the default bind address with no error — a `-b` that appears to be ignored. +(`family::sourcehut-bench/cmd/benchsrht/main.go::run`.) + +**Do not call `WithDefaultMiddleware`.** It opens a *second* Postgres pool from +`connection-string`, keeps the handle in an unexported field with no accessor, and never calls +`SetMaxOpenConns` — so your daemon serves from your capped pool while core-go holds an uncapped +one against the same database; it also starts an email queue and a Redis client you have no use +for. Wire the config/database/logging middleware yourself. +(`mirror::core-go/server/server.go::WithDefaultMiddleware`, refused in +`family::sourcehut-bench/cmd/benchsrht/main.go::mountRoutes`.) + +**Hang every route off one `AnonRouter().Group(...)`.** The anon router is frozen after +`server.New` returns, so a second group added later silently does not route. This is also why an +anonymous-capable read plane cannot use core-go's `auth.Middleware`, which 401s an un-cookied +request by design. + +**Bridge SIGTERM onto SIGINT inside the daemon, then call `Run()`.** core-go's `server.Run` traps +`os.Interrupt` only, and the instance runs services as containers where `docker stop` sends +SIGTERM to Go's default disposition — instant death, no drain. With the bridge the systemd unit +needs no `KillSignal=SIGINT`; without it, it does, and the container has no equivalent. Do not +roll your own `signal.Notify` + `http.Server.Shutdown` beside it: `Run` already owns a warm ~30 s +drain window and a second shutdown path racing it truncates in-flight requests. +(`family::sourcehut-bench/cmd/benchsrht/main.go::bridgeSIGTERM`.) + +**Accept that the four `http.Server` timeouts cannot be set, and put them in nginx instead.** +`server.Run` constructs `&http.Server{...}` inline per bind address into a local slice and +returns nothing — no field, accessor or option reaches one. Measured cost: a +one-byte-per-second client held a request body open for 15 s. The only in-process mitigation is +`http.NewResponseController(w).SetReadDeadline` per endpoint; the real fix is explicit bounds in +`family::sourcehut-bench/contrib/bench.sr.ht.conf`. + +**Validate the whole config once at startup and report every missing key together — including +`[sr.ht] network-key` *and* `[webhooks] private-key`.** One restart per missing key is how a +misconfigured instance eats an afternoon, and `crypto.InitCrypto` panics without either key, +which is not a readable error message. +(`family::sourcehut-bench/cmd/benchsrht/main.go::validateConfig`, +`mirror::core-go/crypto/crypto.go::InitCrypto`.) + +**Build an optional auth plane only when its config section is present.** `bearer.New` refuses +an empty Origin, so constructing the token plane unconditionally makes the daemon fail to boot +on a config that legitimately omits `[tokens.sr.ht]`. (`family::sr-ht-ecore/bearer/bearer.go`.) + +**A bearer-only mount still needs the config and database middleware.** Stripping "the identity +middleware" from an MCP or REST mount also strips what the bearer path's own user mirror +(`auth.LookupUser`) reads out of the request context — the result is a panic rendered as an +empty 500. Drop only the cookie middleware. +(`family::sourcehut-bench/cmd/benchsrht/main.go::mountRoutes`.) + +**Mount order and mount boundaries are three separate correctness properties.** `/query` and +`/mcp` go on **before** the web catch-all, which otherwise swallows them as document paths. The +REST surface is a wholly separate `chi.Mux`, so its CSRF exemption is structural and cannot +regress into a stringly-typed `strings.HasPrefix(path, "/api/v1")` check someone later re-orders. +And `/healthz` and `/static/*` sit **outside** the auth middleware group — inside it, a revoked +cookie turns a healthy process into a 401 the load balancer evicts. +(`family::sourcehut-bench/cmd/benchsrht/main.go::restRoutes`.) + +**Set up logging before core-go touches anything: parse `-d` out of `os.Args` yourself, then +`slog.SetDefault`.** core-go parses `-d` inside `server.New`, after config load, so a daemon that +waits for that parse is silent for exactly the window an operator passes `-d` to watch; and +`middleware.RecoverPanics` reports through `slog.Default()`, which no caller-supplied +`*slog.Logger` can intercept — skip the `SetDefault` and panics log unformatted and unmasked. +(`family::sr-ht-ecore/logging/logging.go::DebugRequested`, +`family::sr-ht-ecore/middleware/middleware.go::RecoverPanics`.) + +--- + +## 2. Building and packaging + +**`make css` must run before `go build`, and packaging must refuse a binary where they +disagree.** The hashed stylesheet is `//go:embed`-ed, so a `make css` after the build restyles +nothing and ships a binary whose CSS is a build behind. Two gates, not one: `check-css` (which +*counts* the matches — two stylesheets is as wrong as none, since the web layer takes the first) +and `check-embedded-css`. (`family::sourcehut-bench/Makefile::check-embedded-css`.) + +**Never `sed` the tracked `APKBUILD` to inject a version — export `PKGVER`.** Rewriting a +tracked file flips `git status --porcelain`, which is exactly what Go reads for `vcs.modified`, +so every binary built afterwards in that abuild run stamps `+dirty` forever. Independently +rediscovered in five repos before it was written down. + +**`.gitignore` must cover `/src/`, `/pkg/` *and* `/tmp/`.** abuild lends the toolchain a +`GOTMPDIR` inside the checkout and runs `make -j$(nproc)`, so two concurrent `go build`s each +create `tmp/go-buildNNN` there; whichever reads `git status --porcelain` second sees `?? tmp/` +and stamps dirty — then both remove their dirs, so the gate that refuses the build prints an +*empty* list of what was in the way. (`family::sourcehut-federation/.gitignore`.) + +**`go mod download`, never `go mod download all`.** The `all` pattern appends hashes for +packages the module never builds against (+92 lines in one repo), dirtying `go.sum` and every +later stamp. `-mod=readonly` does **not** prevent it — a comment claiming it does once sat +directly above the line disproving it. + +**Keep `-modcacherw` when you override `GOFLAGS`, and re-pin `GOCACHE`/`GOMODCACHE` inside +`build()`.** `GOFLAGS="-trimpath"` clobbers abuild's default; the module cache then extracts +read-only and the *next* build's cache tarball fails to unpack with a permission error that names +neither cause. abuild's own cache defaults point into a directory it later deletes. + +**`package()` must call `install-files` and never recompile — and the version/checksum gate must +run against the artefact that actually ships.** abuild runs `package()` in a fresh fakeroot +process where none of `build()`'s exported CGO/version environment applies, so a plain +`make install` there rebuilds from a cold cache and ships a second, uninspected binary. Gate both +`build()`'s output and the staged `$pkgdir` copy: the first proves what you compiled, the second +what the apk contains. (`family::sourcehut-bench/Makefile::install-files`.) + +**Never set `options="!check"` and call the pipeline tested.** One repo found every previously +published apk had been built from code CI compiled but never executed. Bring up a real Postgres +in the build VM instead. + +**Keep `.build.yml` under 16 KiB, comments included, and put the prose in `docs/ci.md`.** +builds.sr.ht stores the manifest in a `varchar(16384)`. Over the cap `hut builds submit` fails +loudly — but on a **push** the failure is silent: the push succeeds and *no build is ever +created*, so the branch has no CI and nothing is red. One repo measured its own manifest at +17747 bytes, 177 of 321 lines being comments, against siblings at 4.8–6.4 kB. +(`family::sourcehut-coverage/docs/ci.md`.) + +**One manifest, not a `.builds/` matrix.** builds.sr.ht has no matrix syntax, and git.sr.ht picks +at most four files out of a `.builds/` directory per push, at random. A second manifest is +legitimate only when it has a *different trigger* — e.g. a tag-only release pipeline +(`family::sourcehut-artifacts/.builds/release.yml`). + +**Route every `docker run` in CI through the cache, not just build artefacts.** Every builds.sr.ht +VM has an empty image store, so a `docker run` always pulls; one test's six-minute budget was +silently spent on a Docker Hub pull. + +**Give any build-tag-gated suite its own explicit CI task.** A tag excludes a file from every +default build, vet and test, so an `integration`-tagged test file that stopped compiling after a +logging refactor went unnoticed indefinitely — nothing ever said so. + +--- + +## 3. Config and keys + +**`[webhooks] private-key` is required unconditionally, even for a service that emits no +webhooks.** `crypto.InitCrypto` fatally requires it, *and* it derives the instance bearer HMAC +key via `HMAC-SHA256(webhook key, "sr.ht HMAC key")` — so rotating it invalidates every working +token on the instance at once. That blast radius is not obvious from the key's name. + +**A working token is authenticated, not encrypted.** It is `auth.BearerToken`, BARE-serialised + +HMAC-SHA256 — anyone holding one reads the username, grants and expiry straight out of it. Do +not describe it as "sealed" and do not put anything in it you would not hand the bearer. +(`mirror::core-go/auth/bearer.go::DecodeBearerToken`.) + +**Your section name must be spelled exactly `<prefix>.sr.ht`, and `origin=` must be in *every* +service's config, not only yours.** The `.sr.ht` suffix is what puts you into the shared nav; +each service builds its nav from its own copy of the file at startup, so the other services also +need restarting after the key lands. (`family::sr-ht-ecore/chrome/chrome.go::BuildNav`.) + +**The bind key is spelled `bind-address`. One instance-wide `config.ini` carrying two spellings +of one idea is a trap for whoever edits it next.** When a service's own spec disagreed with the +instance, the *spec* was corrected rather than an alias kept. + +**Do not add a `static-dir` key.** Every service embeds its assets, and every +`config.example.ini` says so explicitly; an on-disk static tree lets a version-skewed package +serve a stylesheet from a different build, which is what `check-embedded-css` exists to prevent. + +**Read foreign keys where they live; never copy them into your section.** `[sr.ht] network-key`, +`[sr.ht] owner-name`/`owner-email` (`config.GetOwner` **panics** if missing), `[sr.ht] +internal-ipnet` (this host must fall inside it or internal GraphQL calls are rejected), +`[meta.sr.ht] origin`, `[tokens.sr.ht] origin` + `internal-origin`. + +**Say `environment=production` out loud.** The shared chrome reads an *absent* value as +"development" and paints the banner accordingly. + +**Document, per foreign key, what an absent value does — and print it at startup.** Every +`config.example.ini` in the family ends in a block listing each foreign key, its absence +behaviour, and the boot line that reports it (`"tokens: disabled"`). + +**These keys are read by nobody in the family, whatever upstream docs suggest:** `redis-host`, +`service-key`, `debug-host`, `debug-port`, `s3-prefix`. The object-storage vocabulary is +`s3-upstream`, `s3-bucket` (+ `-oci`/`-mirror`/`-cache`), `s3-access-key`, `s3-secret-key`. + +**A refusal over a configured ceiling names the config key in its message.** `working-max-ttl`, +`max-upload-body-bytes` — so whoever hits it knows which line of `config.ini` to argue with. +(`family::sourcehut-tokens/CLAUDE.md`.) + +**Measure resource ceilings on the target platform.** A per-upload memory budget derived on a +many-core laptop was wrong for the two-core CI and production machine; the corrected derivation +(144 MiB, not 115) lives in `family::sourcehut-bench/config.example.ini`. + +**A malformed value on a security-relevant config key must refuse the boot, not warn.** An +unparseable Host-allowlist degraded to "a warning, then unguarded" — functionally +indistinguishable from a correctly guarded endpoint in every test. A logging *preference* is the +opposite case and must fall through to a default rather than block a boot. + +--- + +## 4. Auth and the request path + +**Resolve identity once per request, bearer before cookie, and leave it in the context.** Three +surfaces re-deriving identity is three chances for two of them to disagree about who a caller +is, and the visibility matrix is only as trustworthy as the identity it is applied to. +(`family::sourcehut-bench/authn/resolver.go::Resolver.Middleware`, rationale in +`family::sourcehut-bench/authn/doc.go`.) + +**A present bearer credential decides the whole request, valid or not.** A revoked token is +never rescued by a browser cookie sitting beside it; only the *absence* of an `Authorization` +header falls through to the cookie. + +**The error contract is deliberately asymmetric — implement all four arms:** +- missing/forged/expired **cookie** → **anonymous, never an error** (public browsing must survive + a key rotation, and meta being unreachable is a fact about another service's availability, not + a reason to break your public pages); +- present-but-unusable **bearer** → **401**, never a silent downgrade (a CI job that presented a + credential and got downgraded sails through its reads and fails incomprehensibly at its upload); +- valid token **missing the grant** → **403, not 401** (re-minting the same token forever will + not fix it); +- **backend unreachable** → **503**, distinct from all three (degrading it to 401 turns a daemon + restart into every upload in the fleet failing with "unauthorized"). + +**Check order on a mutation is itself the security property: visibility first (404 to a +stranger), then identity (401), then ownership (403).** Reorder any two and you reintroduce an +existence oracle. (`family::sourcehut-coverage/service`.) + +**Grants are compared literally.** Renaming a service silently invalidates every previously +minted token carrying the old prefix, so CI secrets must be re-minted in lockstep with the +rename commit — nothing on the service side notices the mismatch. + +**Declare your grant vocabulary once, as an array, and derive both the `api-meta.json` scope +list and local validation from it — with a test pinning that they agree.** Written twice, they +drift, and the drift is only visible to whoever mints a token. +(`family::sourcehut-artifacts/core/grants.go::Grants` + `family::sourcehut-artifacts/core/grants_test.go`.) + +**"Blank grants = everything" is right for a stored column and wrong for a request.** A mint +body omitting the field minted a token admitting everything while the caller believed it +admitted nothing. Absent or empty grants on a mint is a 400. + +**Every surface that reads data must check the read grant, including MCP.** One MCP surface +resolved principals through the same `authn` path as REST and then never checked the grant, so an +upload-only token could read everything. The spec had simply forgotten to list that surface among +the reads — enumerate surfaces, not endpoints. + +**Never mint credentials from a plane that checks no grant.** A leftover local token-minting form +let a read-only instance token mint a full-authority upload credential through the service's own +web surface. It was closed by deleting the plane, not by patching it. + +**Ask your own `Principal.IsOwner`, not core-go's authenticated-ness.** The shared `coreauth` +bridge collapses owner and agent alike into `AUTH_INTERNAL`, so moving `/query` to the anonymous +router to admit working tokens also admitted read-only agent tokens into mutation resolvers — a +read grant bought webhook create/delete. + +**Mint and check for any custom `Authorization` scheme must live in one package with one shared +type and one test.** Split across two programs, a payload change breaks provisioning at runtime +instead of at compile time; and a guard that checks only that *some* valid key holder called lets +any holder act for an arbitrary user. (`family::sr-ht-ecore/internalauth`.) + +**Use `r.PostForm`, never `r.Form`, when reading a mutation's fields.** `r.Form` merges the query +string into body values, letting a mutation be driven by a URL that the same-origin guard cannot +fault — it did come from "our own page". + +**Do not decode the unified-login cookie yourself, and do not impose your own expiry on it.** Six +hand-rolled copies existed; two had dropped username validation entirely, so an +attacker-controlled cookie field flowed into a GraphQL query, a path and a log line, and one +passed the name through with its leading `~` attached. Decrypt without an expiry check — a +service-side TTL logs a viewer out of *your* service on a schedule no sibling shares. +(`family::sr-ht-ecore/login/login.go::UsernameFromRequest`, `::ValidName`.) + +**The mirrored user row is insert-only, and `user.id` is meta's id.** Concurrent first visits +collapse via `ON CONFLICT DO NOTHING RETURNING`, which means the losing process's `RETURNING` +scans no row — handle that race explicitly with a re-select. + +**Browser mutations are authorized by the cookie plus a same-origin check, not by a CSRF token.** +No individual service can set `SameSite` on a cookie meta set on the parent domain, so the +defence is an Origin/Referer comparison against your own `origin`, and a request carrying neither +header is refused. (`family::sr-ht-ecore/csrf/csrf.go::Require`.) + +--- + +## 5. Data and SQL + +**Cap every `sql.DB`, including every test harness's.** Uncapped scratch pools peaked at 88–133 +connections under one `go test ./...` against a default `max_connections` near 100. + +**Sort multi-row upsert batches into a canonical order.** `INSERT … ON CONFLICT` locks rows in +VALUES-list order, so two concurrent uploads naming the same rows in different orders deadlock +with `40P01`. + +**Advisory locks, three rules in one.** Use the two-integer `(class, objid)` keyspace, not +`hashtext` into the single-bigint one — hashing collapses distinct lock kinds into one 2³² band +where unrelated locks hash alike and deadlock permanently. Take a *shared* lock on reads: an +exclusive lock on every GET/HEAD serialised 150 concurrent readers to 1.1 s against 273 ms +unlocked. And let Postgres queue your writers — a `pg_try_advisory_lock` poll loop never finds a +gap under a steady reader stream, so every writer times out. + +**Enforce identity invariants as schema constraints, not in query predicates.** Every visibility +predicate spells "anonymous" as the empty-string username, so without a `CHECK` an account whose +meta profile round-trips empty becomes *every anonymous caller's own account*. + +**Do not repurpose an existing timestamp for a new meaning.** A settings-change `updated` column +cannot stand in for "last upload" — it is never touched by an upload. Add the column. + +**Enumerate constraint violations exhaustively, never by the first one you hit.** Two unique +indexes guarded one name; the one with the lower Postgres OID was reported for every real +duplicate, so the "already handled" branch was unreachable and callers got raw 500s. Likewise +`23503` is not the only foreign-key SQLSTATE — PostgreSQL 18 reports `23001` for a row held by an +explicit `ON DELETE RESTRICT`. + +**A `text` column rejects any undecodable byte sequence, not only NUL.** A validator whose doc +comment claimed otherwise produced 70 unclassified 500s across 14 entry points. Pin such a +predicate against a real Postgres column, not a Go unit test restating the same belief. + +**Stream large `bytea` columns via chunked `substring()` reads.** `lib/pq` round-trips `bytea` +through hex (2×), and a read-then-decompress path measured 3.24× the payload in memory on an +anonymous, concurrently-requested endpoint. + +**Ship `schema.sql` *and* `migrations/`, with a test asserting they describe the same +database.** It is the only thing keeping the two from diverging. +(`family::sourcehut-tokens/db/migration_test.go::TestSchemaAndMigrationsAgree`.) + +**Watch the install paths: `schema.sql` and `migrations/` installed under `PREFIX`-derived paths +while the migrate binary looks under `[sr.ht] assets`-derived ones agree only at `PREFIX=/usr`.** +Hit twice in the family. + +--- + +## 6. Rendering and HTTP semantics + +**Never render an internal library's error string into a page — carry a bool on the view +struct.** A message field is a channel someone later assigns an error to; a bool cannot leak a +store path or a library's internals to a reader. + +**Distinguish "backend down" from "not found" in the web layer too.** Collapsing every lookup +error to 404 tells every user their data does not exist the moment Postgres hiccups — and looks +entirely correct until the dependency actually goes down. + +**Marshal JSON islands in Go and hand the bytes to `template.JS`.** `html/template` treats +`<script type="application/json">` as a JS context, so a naive string is JS-escaped and breaks +`JSON.parse`; the reflexive fix (`template.HTML`) disables escaping entirely and opens XSS via +user-controlled paths. `encoding/json` already escapes `<`, `>` and `&`. + +**Serve static assets through the shared handler, never a bare +`StripPrefix(http.FileServer(...))`.** The bare version answers `/static/` with a directory +listing — publishing the binary's vendored inventory as an hour-cacheable public page — and it +writes cache-control headers before delegating, so a later panic leaks `public, max-age=3600` +onto a viewer's error page. (`family::sr-ht-ecore/assets/assets.go::Handler`.) + +**Answer HEAD on every GET route.** GET-only routes 405 on HEAD, breaking uptime probes and cache +revalidation. (`family::sr-ht-ecore/chimw/chimw.go::GetHead`.) + +**Percent-encoding: route and refuse on the same string.** chi routes on `URL.RawPath` when it is +populated and `URL.Path` otherwise, and the two disagree on encoded URLs — so a refusal compared +against the decoded path is about a different route than the one that matched. Normalise RFC 3986 +**unreserved**-character escapes before routing (a real client encoding `~` broke every +`/~{owner}` route) and never reserved ones (decoding `%2F` splits one path segment into two). + +**Recover chi's allowed-method set on a custom 405.** chi computes it on the way to the 405 and +discards it once a custom `MethodNotAllowed` handler is installed — re-query the routing tree +rather than hand-maintaining a literal that drifts. + +**Challenge, don't refuse, an anonymous read of a private path.** pacman/libalpm can only present +a credential in response to a challenge, never pre-emptively, so a private path must answer +**401 + `WWW-Authenticate`** to a request carrying nothing (RFC 9110 §15.5.2) and 403 only to one +that already presented a credential (§15.5.4). Same resource, different verdict. + +**Match `*net.OpError`, not the `net.Error` interface, when classifying "backend unreachable".** +`context.deadlineExceededError` satisfies `net.Error`, so an `errors.As` arm meant for a dial +failure also catches a request that blew its own deadline — answering 503 + `Retry-After` to a +client whose retry hits the same wall. + +**Handle stored upstream URLs as raw strings on both the secret and the validation path.** +`url.URL.Redacted()` masks the password and leaves a bare username intact, so `https://<token>@host` +publishes its whole secret through "redaction" — strip userinfo entirely and declare secret fields +explicitly rather than trusting a generic redactor. And validate the raw stored string, not the +parsed form: `url.Parse` silently normalises away an empty query or fragment (`"…?"`, `"…#"`), +letting a hostile value past a check on the parsed result. + +**Mark `/mcp` uncacheable by hooking `WriteHeader`, `Write` *and* `Flush`.** The MCP SDK's +streamable transport sets its own `Cache-Control`/`Vary` and commits the response on any of the +three. Also re-authenticate **per call, not per session** — the SDK ties a session to the context +of the request that created it, so a stateful session keeps answering as whoever handshook. Proxy +the route **unbuffered** in nginx; the transport streams. + +**Replace the MCP SDK's DNS-rebinding guard with your own same-origin-or-loopback check.** The +SDK's guard rejects a loopback listener whose `Host` header is not loopback — exactly what nginx +forwarding to `127.0.0.1` produces — so it 403s only in production and passes every local test. + +**Use `Sections` for your own tab row and `ExtraNav` only for a genuinely separate origin.** +Putting several pages of one service in `ExtraNav` breaks active-tab highlighting and duplicates +the switcher entry the instance already renders. The shared nav is authoritative: when a feature +moves to another service, delete your copy. (`family::sr-ht-ecore/chrome/chrome.go::Section`.) + +**The shared sheet is Bootstrap 4** — `.text-right`, never `.text-end`. Hashed asset filenames +are discovered at startup, not baked at build time. + +**A service's nginx snippet that `include`s a nonexistent file fails `nginx -t` for the whole +instance.** Ship the file or drop the include. + +--- + +## 7. Federation and cross-service calls + +**Upstream `api.sr.ht` federation does not work, and no config flag turns it on.** Measured: +`api.sr.ht` is NXDOMAIN, its `thistle` dependency's repo answers 404 and survives only in the +module proxy, gqlgen federation is commented out in every upstream service at every pinned tag, +and every pair of the seven upstream schemas conflicts on `type Version` — `User` is defined by +all seven in seven distinct shapes. (`family::sourcehut-federation/README.md`.) + +**What a new service must do to be federatable on this instance is small and specific:** serve +`POST /query` normally, **accept a meta.sr.ht PAT there**, and publish a **non-null** +`api-meta.json` scope list. The gateway forwards one client `Authorization` header verbatim to +every service a query touches, so a `/query` that refuses meta PATs answers 401 to the first +authenticated federated query and cannot be federated at all. +(`family::sr-ht-ecore/metapat`.) + +**Marshal `[]`, never `null`, for `api-meta.json` scopes.** meta.sr.ht fetches that file from +every service it discovers when rendering `/oauth2/personal-token` and iterates each `scopes` +array — a JSON `null` is a nil iteration in meta, i.e. **a 500 on the personal-token page for the +entire instance**, every service's grants, not just yours. Nobody would find this by testing the +service that caused it. (`family::sr-ht-ecore/apimeta/apimeta.go::Handler`.) + +**Never mutate the caller's `*http.Request` in a `RoundTripper`.** It is forbidden by the +interface contract; clone first, and return an error rather than panicking. + +**Do not use package-level `*gqlerror.Error` sentinels.** `graphql.ErrorOnPath` mutates them in +place, stamping the first resolver's field path onto the shared value — every later refusal +anywhere in the daemon then reports that field. Wrong answer, cross-request leak, and a data +race, observed live. + +**Detect a GraphQL handshake on the parsed document, not by substring match on raw query text.** +Substring matching hands the schema to any query whose string literals happen to contain those +characters; and rewriting field names with a word-boundary regex fails on the compact JSON a real +gateway sends, where the character before a field name can be the `t` of an escaped `\t` — the +rewrite silently no-ops while reporting success. + +**Split a cross-service GraphQL enumeration into two documents when any field can fail per +row.** git.sr.ht's `path()` on an unresolvable revision fails the *whole* request, not just that +row, so one empty repo silently breaks a whole-owner enumeration. + +**Merging a schema built from zero services must be an error.** It merges silently into an +endpoint that 404s every field. + +--- + +## 8. Testing + +**`make test` green says nothing about the persistence layer.** DB-backed tests are gated on an +env DSN (`<SVC>SRHT_TEST_PG`) and skip without it: one service's `db/` covered **4.5 %** without +the DSN and **77.2 %** with it. Bring up Postgres in CI and make the task **fail** on an empty +variable rather than skip quietly. (`family::sourcehut-bench/README.md`.) + +**Machine-check the layering invariant; review does not hold it.** The guard test parses every +shipped source file at any depth, deliberately ignoring build tags ("there is no build tag under +which the invariant doesn't apply"), and does two things: refuses any `db` import from `api/` or +`web/`, and asserts nothing exported from `service/` *is, contains, or returns* a `db.Store` or +`db.Querier`. The second rule exists because the first is bypassable three ways — a DTO +embedding a store, a `.Store()` accessor, and a returned interface — all three of which were +found in the wild. (`family::sourcehut-bench/api/layering_test.go::TestInterfaceLayerDoesNotReachTheStore`.) + +**Test the refusal table itself.** Mapping `core.ErrNotFound` onto a 403 passed a whole suite +while being exactly the answer the spec forbids — nothing exercised the mapping. + +**A green suite over fixtures the author wrote is not proof.** Fourteen mutations of four +"well-tested" pages survived because every fixture happened to avoid the edge case: a delta of +exactly 0.0, a report with no commit time, a foreign pagination cursor. Worse for a format +parser, where an author-built fixture confirms one reading of the spec twice — baseline against +files the format's real producer emits (`dpkg-deb`, `abuild`, `rpmbuild`, `createrepo_c`) and run +the format's own conformance suite where one exists; an OCI conformance run surfaced 10 of 79 +real failures in three categories nothing else had found. + +**`httptest.ResponseRecorder` cannot observe *when* a header was set.** Its `Header()` hands back +the live map, so a recorder test passes for a wrapper that sets headers too late or never on the +path the test does not take. Test header timing against `httptest.NewServer` and a real client. + +**Do not inject a clock and assume you control time.** A working token's expiry is sealed into +the token and checked against the **real** wall clock by `DecodeBearerToken`, which no injection +reaches — a suite pinned to a fake afternoon passed at 09:00 UTC and failed at 14:02 the same +day. + +**Keep advisory locks and `pg_locks` assertions out of shared-cluster suites.** The advisory +keyspace is global to the cluster, not scoped per test schema, so two concurrent `go test ./...` +runs corrupt each other's "N locks held" assertions. + +**Silence the standard logger in benchmarks.** `auth.DecodeBearerToken` calls `log.Printf` on +every refused token; an unfiltered run produced 9.5 M lines / 901 MB, and `benchfmt` silently +skips lines it cannot parse — so the corrupted upload would have been accepted and reported as a +success. + +**Test a chi `Mount` through a real request path, not by driving `ServeHTTP` with +prefix-relative paths.** `Mount` sets `RoutePath` in the context and leaves `r.URL.Path` +untouched — correct for a chi sub-router, silently wrong for a stdlib `ServeMux` mounted the same +way. The endpoints 404'd every live request while every unit test stayed green. + +**Write verification commands that can fail.** `gofmt -l` prints offending files and exits 0 +(gate on `gofmt -s -l | tee` + `test -s`); `go build … | head && echo OK` reports the exit status +of `head`; a benchmark or coverage upload must be gated on a grep asserting that at least one +test or benchmark actually matched. + +**A claim copied from a donor's prose into three of your own documents is still one unverified +claim.** The family's stated reason for refusing `WithDefaultMiddleware` was repeated across a +SPEC, an ANALYSIS and a DESIGN, and was **backwards** when finally checked against the source. + +**Prefer making the bad state inexpressible over aligning conventions.** Two layers disagreed on +whether an empty filter list meant "matches nothing" or "matches everything", so a newly created +project silently widened a scoped search to the whole corpus. The fix was a three-state type, not +a convention memo. Likewise, when a blanket security rule breaks a legitimate use, add a second +**explicitly named** method rather than a boolean flag — "serving unreviewed content" should be +something a caller asks for by name and a reviewer can grep for. + +--- + +## Vocabulary and naming + +A service carries five to seven names and they deliberately do not match. **Derive each from the +layer it belongs to, never from another name.** + +| layer | pattern | examples | +|---|---|---| +| working directory | `sourcehut-<longer>` | `sourcehut-coverage`, `sourcehut-curator` | +| git repository | `sr-ht-<short>` | `sr-ht-cover`, `sr-ht-curator`, `sr-ht-compare` | +| Go module | `sourcecraft.dev/bigbes/sr-ht-<short>` | 8 of 9 services | +| **service / config section / apk `pkgname`** | `<prefix>.sr.ht`, one short prefix | `cov.sr.ht`, `diff.sr.ht`, `go.sr.ht`, `spec.sr.ht` | +| subdomain | `<prefix>.srht.bigb.es` | `cov.srht.bigb.es` | +| daemon binary | `<prefix>srht` | `coversrht`, `benchsrht`, `artifactsrht`, `gosrht` | +| migrate binary | `<prefix>srht-migrate` | `coversrht-migrate` | +| extra tools | plain short names | `art`, `fedgw`, `nsify`, `dolt-git-hook` | + +**Keep the prefix short** — `cov` not `coverage`, `diff` not `compare`, `spec` not `specs`. A +rename moves everything the *instance* sees (config section, origin, apk, container) and +deliberately moves nothing with a cost and no benefit (module path, repository name, binary name, +the routes people have in their history). + +Package names and what the suffixes mean: + +| package | meaning | +|---|---| +| `core/` | pure domain: validation, grammars, sentinel errors, the grant array. No I/O, stdlib only. | +| `db/` | Postgres. `Store` over a `Querier` (works on `*sql.DB` and `*sql.Tx`), `database/sql` + `lib/pq`, `$n` placeholders, typed sentinels. | +| `authn/` | **who is calling** — cookie / bearer / PAT decode producing a `Principal`. | +| `authz/` | **may they do it** — e.g. a git.sr.ht repo-access check with a TTL cache. | +| `service/` | orchestration. The only package `api`/`web`/`graph`/`mcpsrv` may call, and the only one allowed to call `db` and `authn`. | +| `web/` `api/` `graph/` `mcpsrv/` | the four surfaces: chi + `html/template`; REST under `/api/v1`; gqlgen read schema at `/query`; MCP at `/mcp`. | +| `ingest/` `jobs/` | payload parsing; background work. | +| `<name>x/` | **"read/write *this external thing*, thinly"** — `gitx`, `blobx`, `cachex`, `mirrorx`. | +| domain-named | everything else: `browse/`, `oci/`, `pkgrepo/`, `discover/`, `search/`. | + +**The two surfaces never import each other; shared helpers are copied deliberately**, so each +error message is written for the reader it will be shown to. + +Config keys in `[<svc>.sr.ht]`: `origin` (also the same-origin CSRF comparison target), +`connection-string`, `bind-address`, `migrate-on-upgrade`, `log-level`, `internal-origin` / +`api-origin` / `api-internal-origin`, `s3-*`, and `max-*` for every ceiling +(`max-concurrent-uploads`, `max-upload-body-bytes`, `max-database-connections`, …). + +Grants: space-separated `<service>:<action>`, split on ASCII whitespace, printable ASCII, no +uppercase; `*` (and, for a *stored* parent token only, the empty string) means everything. The +issuing daemon does not know the vocabulary and does not validate it — services declare it, so an +unknown grant simply admits nobody anywhere. An exchange may **drop** grants and never **add** +them. (`family::sr-ht-ecore/grants/grants.go::Parse`, `::IsSubsetOf`.) + +Error and log style: refusals name the config key they enforce; `ErrUnavailable` is 503 and never +401; mask attribute values by key path (keys ending `token`/`secret` to 6 chars; `cookie`, +`authorization`, `password`, `api_key` whole) but leave `token_id` unmasked — it is a correlation +id, not a credential. + +Commit subjects: **`<scope>: lowercase imperative sentence`**, scope being a package or one of +`ci`/`build`/`spec`/`docs` — e.g. `web: stop offering a ref tier that holds no reports`. A +minority of `fix(scope):`/`feat(scope):` subjects survive in older repos; do not start new ones, +do not rewrite published history. + +--- + +## Go conventions actually in force + +Measured across the family, not aspirational: + +- **`culpa` (`go.bigb.es/auxilia/culpa`) — yes, universally.** Direct `auxilia v0.7.0` require in + eight of nine services; ~560 imports family-wide. Use it for structured wrapping across layer + boundaries instead of bare `fmt.Errorf`. +- **`scribe` — yes.** ~277 imports; it is the slog handler paired with `sr-ht-ecore/logging`'s + *policy* (level resolution, colour, mask set). `scribe.Err` walks the whole error chain rather + than type-asserting the outermost error — which is exactly why a `culpa` chain under one + `fmt.Errorf` still logs its code, hint and stacktrace. +- **`steward` — no. Used nowhere.** Composition roots are hand-wired in `cmd/<svc>srht/main.go`. + Do not introduce DI here; the exemplar main is ~900 lines of explicit wiring and reads well. +- **testify — yes** (`require` for fatal checks, `assert` for the rest). +- **Context plumbing — yes**, `ctx` first argument everywhere from `service/` down through `db/`. +- **Graceful shutdown — yes, but you do not own it**: core-go's `server.Run` owns the drain + window; your job is `bridgeSIGTERM` before it. +- **Dependencies are direct requires on the forks** `sourcecraft.dev/bigbes/sr-ht-core` and + `sourcecraft.dev/bigbes/sr-ht-ecore`. **No `replace` directive anywhere in the family**, and + upstream `core-go` is not used. +- **Go version**: services on `go 1.26.4`, `sr-ht-ecore` on `1.25.0`. + +**Exemplar service: `sourcehut-bench`.** Its `cmd/benchsrht/main.go` package comment is ~120 +lines of design rationale for the daemon skeleton, its `authn/doc.go` is the canonical statement +of the three auth planes and the error contract, and its `api/layering_test.go` is the layering +guard worth copying verbatim. For the config-file house style read +`family::sourcehut-bench/config.example.ini`; for the agent-facing conventions read +`family::sourcehut-tokens/CLAUDE.md`. + +--- + +## The doc set a new repo must carry + +| file | what it is for | +|---|---| +| `README.md` | opens with "one module, one binary, pure Go"; the mount table; the name table (directory / repo / module / service / subdomain / binaries / apk). | +| `SPEC.md` | **normative**, Russian, fixed chapter skeleton: Введение · Архитектура · Модель данных · *domain chapters* · Аутентификация и видимость · Web UI · REST API · MCP · Ретеншен и фоновые задачи · Конфигурация · **Структура репозитория и карта форка** · Тестирование · Дорожная карта · Открытые вопросы. Behaviour changes belong here too, not only in the code. | +| `config.example.ini` | the shared-instance preamble, a prose comment per key, and a trailing block naming every foreign key read, what an absent value does, and the startup line reporting it. | +| `docs/ci.md` | exists because the manifest has a 16 KiB cap: prose goes here, with anchors matching the manifest's pointers byte-for-byte. | +| `contrib/` | `<svc>.conf` (nginx: `sourcehut.conf` + `port443.conf` + ssl include, `proxy_pass 127.0.0.1:<port>`, explicit CSP, unbuffered `/mcp`) and `<svc>-srht.service` (systemd: `KillSignal=SIGINT`, `TimeoutStopSec=35`, `ProtectSystem=strict`, `ProtectHome`, `PrivateTmp`, `NoNewPrivileges`). Ports 5090–5096 are taken. | +| `Makefile`, `APKBUILD`, `.build.yml` | per section 2. | +| `CLAUDE.md` | see below. | + +**The "карта форка" chapter is the most reusable artefact in the family**: a table of which +donor service each part was copied from, plus a "Грабли" (rakes) paragraph. Write it while the +copying is fresh — it is the only record of *why* a given file looks the way it does. + +**Write a real `CLAUDE.md`; do not ship the stub.** The family's `AGENTS.md` files are +byte-identical boilerplate, and all but one `CLAUDE.md` are the unedited `bd init` stub — still +carrying the literal `_Add your build and test commands here_`. That is why an agent working in +most of these repos gets none of the context on this page. The one hand-written exemplar +(`family::sourcehut-tokens/CLAUDE.md`) contains, and yours should contain: + +1. **The layering rule**, spelled as an arrow chain, with the note that the surfaces never import + each other and helpers are copied deliberately. +2. **`make test` vs `make test-pg`** — that the plain suite says nothing about `db/`, the env var + name that unlocks it, and the instruction to say plainly in a handoff if you could not run it. +3. **`make css` before `go build`** — and that `check-embedded-css` will refuse a mismatch at + packaging time. +4. **Doc-comment density** — comments here carry the *reasoning*: what else was considered, what + breaks under the alternative, which SPEC chapter decides it. A terser house style reads as + foreign; match the file being edited. +5. **The defaulting rule** — a missing field is never defaulted to the widest or longest-lived + answer, because failing towards the most dangerous credential leaves the caller holding one + they believe they did not ask for. diff --git a/skills/sourcehut-custom-service/references/upstream.md b/skills/sourcehut-custom-service/references/upstream.md new file mode 100644 index 0000000000000000000000000000000000000000..05085c79d7e8855e9f2fae000852c53d343c855b --- /dev/null +++ b/skills/sourcehut-custom-service/references/upstream.md @@ -0,0 +1,149 @@ +# The upstream mechanisms + +> Citations: `family::<repo>/<path>::<symbol>` — this instance's own repos; `mirror::<path>::<symbol>` +> — the upstream documentation mirror. Both roots are substituted at install time. Symbols, never +> line numbers. + +This page is about **upstream SourceHut** — the nine Python+Go services `~sircmpwn` publishes — and about which of their integration mechanisms a custom service on this instance actually reuses. Read it when the question is *"how does SourceHut itself do X"*, when you need to know whether a mechanism you are about to depend on is alive, or when you are debugging an interaction between a custom service and a stock one. + +For how to build a service here, read `references/anatomy.md` and the rest. This page is background, not a recipe. + +## What the custom services reuse, and what they don't + +Upstream offers five integration surfaces. The family here uses two of them as designed, replaced one, and ignores two. + +| Upstream surface | Status here | Where the real story is | +| --- | --- | --- | +| The unified-login cookie | **Reused as designed.** The single most important surface. | `references/auth.md` | +| The nav / service-switcher membership rule | **Reused as designed** — a config section ending in `.sr.ht` | `references/chrome.md` | +| The shared theme (SCSS) | **Reused as a build input**, not as a served asset | `references/chrome.md` | +| GraphQL federation via `api.sr.ht` | **Replaced.** Upstream's gateway does not work; `fedgw` does the job differently | `references/federation.md` | +| Webhooks | **Ignored.** No custom service emits or consumes upstream webhooks | — | + +The mechanisms below are the upstream half of the two that are reused, plus enough of the rest to recognise it when you see it. + +## There is no plugin system + +Nothing loads third-party code into a running SourceHut service. There is no registry, no entry point, no hook table, no versioned extension API. A custom service is a **separate process** that happens to agree with the others about a cookie format, a config file and a set of URLs. + +This is why every integration in this skill is config-driven, and why the coupling is to *internals* rather than to a published contract: `core.sr.ht` and `core-go` are libraries the upstream services share with each other, not an API offered to anyone else. Upstream is free to change them, and does. + +`dispatch.sr.ht` — the old "third-party integrations" service — was removed upstream and is not coming back. The `dispatch` package inside `sourcehut-ssh` is unrelated: it is SSH shell dispatch for git/hg. + +## The unified-login cookie + +Services do **not** each run a web-login OAuth dance. There is one shared cookie, `sr.ht.unified-login.v1`, set by meta.sr.ht on the **parent domain** (here `.srht.bigb.es`), httponly, carrying the viewer's profile as Fernet-sealed JSON. + +Read on every request (`mirror::core.sr.ht/srht/app/flask.py::get_session_cookie`): + +```python +cookie = request.cookies.get("sr.ht.unified-login.v1") +user_info = json.loads(fernet.decrypt(cookie.encode()).decode()) +user = self.oauth_service.lookup_user(user_info["name"]) +``` + +Written after login (`mirror::core.sr.ht/srht/app/flask.py::make_response`): + +```python +response.set_cookie("sr.ht.unified-login.v1", + fernet.encrypt(user_info.encode()).decode(), + domain=global_domain, httponly=True, max_age=...) +``` + +The Fernet key is `[sr.ht] network-key`, shared instance-wide (`mirror::core.sr.ht/srht/crypto.py::fernet`). Any service in any language that holds that key can decrypt the cookie and learn who is browsing. No per-service OAuth callback is needed to render pages as the logged-in viewer. + +Login and logout are plain redirects to meta (`mirror::core.sr.ht/srht/app/flask.py::login_url`, `::logout_url`): + +``` +{meta-origin}/login?return_to={your_url} +{meta-origin}/logout?return_to={your_url} +``` + +> Fernet is AES-128-CBC + HMAC-SHA256, base64url, with a version byte, timestamp and IV — the `cryptography` library's spec. Reimplementable anywhere; Go uses `github.com/fernet/fernet-go`, the same library `core-go` uses. + +**Do not write your own decoder.** `family::sr-ht-ecore/login` is the family's one copy and it settles five decisions that six independent decoders got wrong between them — see `references/auth.md`. This section exists so you recognise the format, not so you reimplement it. + +## The nav / service-switcher + +The service list in the top nav is computed from config: **every section name ending in `.sr.ht`** (`mirror::core.sr.ht/srht/app/flask.py::_network`): + +```python +_network = [ + s for s in config + if s.endswith(".sr.ht") and s not in ["paste.sr.ht", "pages.sr.ht"] +] +``` + +The template loops over `network` and links each through `get_origin()`, which reads `[service] origin=` (`mirror::core.sr.ht/srht/templates/nav.html::for _site in network`, `mirror::core.sr.ht/srht/config.py::get_origin`). + +Two consequences, both of which the family depends on: + +- **Adding `[myservice.sr.ht] origin=` to a service's config makes you appear in *that* service's nav.** Each service builds its nav from its own config copy, so appearing everywhere means the section is in every service's config — which is why this instance distributes one shared `config.ini`. +- The exclusion list is upstream's own; a custom service is not in it and will appear in the switcher. + +`family::sr-ht-ecore/chrome` implements the same membership rule for Go, so the two families agree about who is in the network. + +## The shared theme + +`mirror::core.sr.ht/scss/` is the Bootstrap-derived theme. Each upstream service has its own `scss/main.scss` importing it, and its Makefile compiles a hashed `main.min.<sha>.css` for cache busting. + +The custom services do the same thing rather than serving core.sr.ht's build output: CI assembles `core.sr.ht@$CORE_VER` plus Bootstrap into a shared SCSS tree and each service compiles its own hashed stylesheet from a one-file `main.scss`. See `references/chrome.md`. + +## GraphQL and the federation gateway + +Each upstream service publishes `api/graph/schema.graphqls` and serves `/query` from a Go binary built with `gqlgen`; the Python web tier of the same service consumes it through an `ariadne-codegen`-generated client under `<svc>srht/graphql/`. + +`api.sr.ht` was meant to merge those schemas into one endpoint, discovering services from config the same way the nav does (`mirror::api.sr.ht/main.go::main`): + +```go +for name := range conf { + if strings.HasSuffix(name, ".sr.ht") { + services = append(services, thistle.NewService(name, getOrigin(conf, name))) + } +} +``` + +It fetches each service's schema and merges with `thistle.BuildSchema` (`mirror::api.sr.ht/main.go::updateSchema`), elevating its own calls with `mirror::api.sr.ht/auth.go::InternalAuthTransport`. + +**This does not work, and the reason is not a configuration mistake.** `thistle` was deleted upstream, `api.sr.ht` is NXDOMAIN, no service answers `_service`, and every pair of upstream schemas fails to merge because all of them define `User` differently. `references/federation.md` has the measured detail and the replacement. Cite this section only to explain what upstream *intended*. + +## Internal service-to-service auth + +Upstream services call each other with an `Authorization: Internal` credential — a Fernet-sealed payload minted with `network-key`, accepted only from an address inside `internal-ipnet` (`mirror::core-go/auth/middleware.go::internalAuth`). `core-go` implements both ends unexported, which is why the family has its own copy in `family::sr-ht-ecore/internalauth`. + +The important property, and the reason `references/auth.md` treats this carefully: an internal call **bypasses the `@access` scope directives entirely**. It is not "an admin token", it is "no scope check at all". + +## Webhooks + +Two generations exist: GraphQL-native (`mirror::core-go/webhooks/queue.go::NewQueue` plus per-service `api/webhooks/`) and legacy HTTP (`mirror::core.sr.ht/srht/webhook/`). Smaller services (`paste`, `man`, `pages`) emit none. + +No custom service on this instance emits or consumes them today. If you add one, note that `[webhooks] private-key` is already load-bearing for a different reason — it derives the instance's bearer HMAC key (`references/config.md`), so it is not a key you may rotate to suit a webhook rollout. + +## The Python service shape, for reading upstream code + +Useful when tracing a stock service's behaviour. Every upstream user-facing service subclasses `srht.app.Flask` and inherits nav, unified login, theme, the GraphQL blueprint and error pages. paste's whole bootstrap is about 35 lines (`mirror::paste.sr.ht/pastesrht/app.py::PasteApp`): + +```python +from srht.app import Flask +from srht.config import cfg +from srht.database import DbSession + +db = DbSession(cfg("paste.sr.ht", "connection-string")); db.init() + +class PasteApp(Flask): + def __init__(self): + super().__init__("paste.sr.ht", __name__, user_class=User) + ... +``` + +Templates `{% extends "layout.html" %}`. This is a real option for a custom service — it is the shortest path to the integrated shell — but nothing in this family uses it, so you would be the first, without the shared Go libraries, on a Python toolchain the instance's packaging does not build. Choose it only if the service is genuinely Python-shaped. + +## Key upstream files + +- `mirror::core.sr.ht/srht/app/flask.py` — `::_network`, `::get_session_cookie`, `::make_response`, `::login_url`, `::logout_url`, `::Flask` +- `mirror::core.sr.ht/srht/templates/nav.html`, `mirror::core.sr.ht/srht/templates/layout.html` — the shared chrome +- `mirror::core.sr.ht/srht/config.py::get_origin` / `::get_api`; `mirror::core.sr.ht/srht/crypto.py::fernet` +- `mirror::core-go/config/config.go` — `::LoadConfig`, `::GetOwner`, `::GetOrigin`, `::GetAPI` +- `mirror::core-go/auth/middleware.go` — `::Middleware`, `::cookieAuth`, `::internalAuth` +- `mirror::api.sr.ht/main.go` — `::main`, `::updateSchema` +- `*/config.example.ini` — upstream's own field names per service