diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000000000000000000000000000000000000..f497583c193546ac756d37ef0b84e07fc96a253f --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,377 @@ +# dolt.sr.ht — "DoltLab for SourceHut" (self-hosted Dolt database hosting service) + +## Context + +bigbes wants a DoltLab-equivalent integrated into the self-hosted SourceHut instance +(`*.srht.bigb.es`): a new service `dolt.sr.ht` that hosts Dolt databases the way +git.sr.ht hosts git repos — `dolt clone/push/pull` over HTTPS plus an integrated web +UI (shared nav, unified login via the `sr.ht.unified-login.v1` cookie, shared +Bootstrap theme). Pure Go, single new repo at `/Users/blikh/data/home/sourcehut-dolt` +(currently empty). No upstream SourceHut modification — integration is config-driven +(a `[dolt.sr.ht]` config section puts us in every service's nav). DoltLab itself is +closed-source; everything we need is in Apache-2.0 `dolthub/dolt` packages. + +**v1 scope (user-confirmed):** create/list/browse databases + clone/push over the +remotesapi, with **both** auth flows: username + meta personal access token +(`--user` + `DOLT_REMOTE_PASSWORD`, Basic) **and** dolt's Ed25519 keypair flow +(`dolt creds`/`dolt login`, Bearer EdDSA JWT — the git-SSH-key-like UX). No GraphQL +federation, no SSH transport, no webhooks, no issues/PRs. + +All findings below were verified against dolthub/dolt `main` and the SourceHut mirror +at `/Users/blikh/data/home/sourcehut` (core-go/, core.sr.ht/, git.sr.ht/, sr.ht-nginx/). + +## Architecture summary + +One Go module **`go.bigb.es/sourcehut-dolt`**, two binaries: +- **`doltsrht`** — one process, three listeners: + - web UI (chi) on `-b localhost:5307` + - remotesapi (gRPC `ChunkStoreService` + HTTP chunk data plane, h2c-multiplexed + single port) on `127.0.0.1:5306` via importable + `github.com/dolthub/dolt/go/libraries/doltcore/remotesrv` + - a small separate gRPC server for `CredentialsService.WhoAmI` (keypair login) on + `127.0.0.1:5308` — nginx path-routes it, so no need to touch remotesrv internals +- **`doltsrht-migrate`** — brant migrations (`git.sr.ht/~bitfehler/brant`), modeled on + `sourcehut-migrate/cmd/sourcehut-migrate/main.go`. + +Storage: **bare NBS chunk-store dirs** (no `.dolt/`, no working set) at +`/~/` — exactly what `remotesrv` serves and `file://` remotes use. +Postgres holds metadata (user mirror + repository + access + dolt_key). +Clone URL: `dolt clone https://dolt.srht.bigb.es/~user/db`. + +Key verified facts that shape the design: +- `server.New` (core-go) works **without** `WithSchema` (gqlgen-only). But + `WithDefaultMiddleware` attaches `auth.Middleware`, which 401s requests with no + cookie/header — kills anonymous browsing. → don't call it; assemble our own + middleware group on `srv.AnonRouter()`. +- core-go token validation trio (mirrors `auth.OAuth2`): `auth.DecodeBearerToken` + (offline HMAC via `[webhooks]private-key`-derived key) + `auth.LookupUser` (local + `"user"` table, lazily populated by `FetchMetaProfile` internal-GraphQL to meta) + + `auth.LookupTokenRevocation` (GraphQL `tokenRevocationStatus` on meta). Requires + shared `[sr.ht]network-key`, `[webhooks]private-key`, and our host inside meta's + `internal-ipnet`. +- dolt CLI credential selection (`go/libraries/doltcore/env/grpc_dial_provider.go`): + `--user` ⇒ `authorization: Basic b64(user:pass)` with pass from + `DOLT_REMOTE_PASSWORD`; no `--user` but `dolt creds`/`dolt login` key present ⇒ + `authorization: Bearer ` (aud = our host, exp = now+30s, kid = + base32(SHA-512/224(pubkey)) custom alphabet `0123456789abcdefghijklmnopqrstuv`, + iss hardcoded `dolt-client.dolthub.com` — do NOT reject on iss); no creds at all ⇒ + no metadata ⇒ anonymous (must work for PUBLIC clones). Client is fully + host-agnostic: `dolt login --auth-endpoint --login-url `, defaults + settable via `dolt config --global` keys `remotes.default_host`, `creds.add_url`. +- Upstream `remotesrv.ServerInterceptor` drops the authenticated ctx and never sees + the repo path ⇒ per-repo ACLs impossible with it. → we write our own + unary+stream interceptors (keep upstream's method classification: write = + {GetUploadLocations, AddTableFiles, Commit}; read = the 8 others; unknown ⇒ deny). +- Data plane is collision-free for nginx: gRPC under + `/dolt.services.remotesapi.v1alpha1.*/`, sealed chunk URLs under + `/single_symmetric_key_sealed_request/` (AES-GCM sealed, 15-min expiry, possession + = authorization), web UI owns the rest. `getScheme()` honors + `x-forwarded-proto` — nginx must set it on grpc_pass. +- Empty-db init primitive: `doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, fs)` + + `ddb.WriteEmptyRepo(ctx, "main", name, email)`. +- Bare stores **cannot** be opened by the sqle engine / embedded driver (they expect + working sets). Browse UI uses low-level read-only APIs instead (see §Browse). +- dolt CLI v2.1.10 installed at `/opt/homebrew/bin/dolt` (integration tests). +- `github.com/dolthub/dolt/go` is a separate module pinned via pseudo-versions; + heavy deps, CGO (gozstd). Apache-2.0. + +## Repo layout + +``` +go.mod Makefile schema.sql migrations/ config.example.ini README.md LICENSE +contrib/dolt.sr.ht.conf # nginx +scss/main.scss # @import "base" against core.sr.ht shared scss +static/ # logo.svg + main.min..css build output +cmd/doltsrht/main.go +cmd/doltsrht-migrate/main.go +core/ # PURE domain: names.go (validate, ParseRepoPath), access.go (matrix), models.go +db/ # postgres: repos.go, access.go, keys.go (dolt_key CRUD) +authn/ # ctx.go (Caller), cookie.go (optional unified-login middleware), + # token.go (Basic: PAT trio + 60s cache), jwt.go (Bearer: EdDSA verify) +storage/ # init.go (InitStore/DeleteStore via WriteEmptyRepo), dbcache.go (remotesrv.DBCache) +remoteapi/ # server.go (remotesrv assembly), interceptors.go, credsvc.go (WhoAmI grpc server) +browse/ # open.go, log.go, tables.go, diff.go — read-only doltdb over bare stores +web/ # router.go, handlers_*.go, chrome.go, templates.go, templates/*.html +``` + +Dependency direction: `core` ← all; `db`/`authn`/`storage`/`browse` mutually +independent; `remoteapi` ← {core, db, authn, storage}; `web` ← {core, db, authn, +browse, storage}; `cmd` ← all. + +## Postgres schema (schema.sql, brant `-- +brant Up` migrations) + +```sql +CREATE TYPE visibility AS ENUM ('PUBLIC', 'PRIVATE', 'UNLISTED'); +CREATE TYPE user_type AS ENUM ('PENDING', 'USER', 'ADMIN', 'SUSPENDED'); +CREATE TYPE access_mode AS ENUM ('RO', 'RW'); + +CREATE TABLE "user" ( -- exact UserMixin shape; core-go FetchMetaProfile inserts rows + id integer PRIMARY KEY, -- meta's user id, inserted explicitly (not serial) + username varchar(256) UNIQUE, + created timestamp NOT NULL, updated timestamp NOT NULL, + email varchar(256) NOT NULL UNIQUE, + user_type user_type NOT NULL, + url varchar(256), location varchar(256), bio varchar(4096), + suspension_notice varchar(4096) +); +CREATE INDEX ix_user_username ON "user"(username); + +CREATE TABLE repository ( + id serial PRIMARY KEY, + created timestamp NOT NULL, updated timestamp NOT NULL, + name varchar(64) NOT NULL, + description varchar(1024), + owner_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE, + path varchar(1024) NOT NULL UNIQUE, -- absolute on-disk store dir + visibility visibility NOT NULL, + CONSTRAINT uq_repo_owner_id_name UNIQUE (owner_id, name) +); + +CREATE TABLE access ( + id serial PRIMARY KEY, + created timestamp NOT NULL, updated timestamp NOT NULL, + repo_id integer NOT NULL REFERENCES repository(id) ON DELETE CASCADE, + user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE, + mode access_mode NOT NULL, + CONSTRAINT uq_access_user_id_repo_id UNIQUE (user_id, repo_id) +); + +CREATE TABLE dolt_key ( -- Ed25519 pubkeys for dolt creds auth + id serial PRIMARY KEY, + created timestamp NOT NULL, + user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE, + kid varchar(64) NOT NULL UNIQUE, -- base32(SHA-512/224(pubkey)), dolt alphabet + pubkey bytea NOT NULL, -- 32-byte ed25519 public key + comment varchar(256), + last_used timestamp +); +``` + +DB name rule: `^[a-zA-Z0-9](?:[a-zA-Z0-9_-]*[a-zA-Z0-9])?$`, max 64 (names double as +SQL database identifiers; reject `.`/`..`). + +## Main wiring (`cmd/doltsrht/main.go`) + +```go +conf := config.LoadConfig() +srv := server.New("dolt.sr.ht", "localhost:5307", conf, os.Args[1:]) // runs crypto.InitCrypto +db := sql.Open("postgres", cfg[dolt.sr.ht]connection-string) +srv.AnonRouter().Group(func(r chi.Router) { + r.Use(chimw.RealIP, chimw.Recoverer) + r.Use(config.Middleware(conf, "dolt.sr.ht"), database.Middleware(db)) + r.Use(authn.OptionalCookieMiddleware()) // never 401s; anonymous stays anonymous + web.Register(r) +}) +rsrv := remoteapi.New(...) // remotesapi on 127.0.0.1:5306 (grpc+http one port) +csrv := remoteapi.NewCredSvc(...) // CredentialsService.WhoAmI on 127.0.0.1:5308 +go rsrv.Serve(); go csrv.Serve(); srv.Run() +``` + +No redis, no email, no `WithSchema` in v1. + +**Routes** (web/router.go): `GET /` dashboard; `GET,POST /create` (cookie required); +`GET /~{user}`; `GET /~{user}/{db}` overview (branches, latest commits, clone box +with both auth variants); `GET /~{user}/{db}/log?branch=&from=`; +`GET /~{user}/{db}/commit/{hash}`; `GET /~{user}/{db}/tree/{ref}`; +`GET /~{user}/{db}/table/{ref}/{table}?page=`; `GET,POST /~{user}/{db}/settings` +(owner only: description/visibility/ACLs/delete); `GET,POST /settings/keys` +(dolt-key association page — reads `#` URL fragment via a few lines of +inline JS to prefill the form, since `dolt login` appends the pubkey as a fragment); +`GET /static/*`. + +**Chrome** (web/chrome.go + templates): Go `html/template` port of core.sr.ht +`layout.html`/`nav.html` — `

`, brand `[sr.ht]site-name` + `dolt`, +network list = config sections ending `.sr.ht` (minus paste/pages) linking +`config.GetOrigin(conf, site, true)`, `active` on self; login block → +`{meta-origin}/login?return_to=...` / logout; environment banner when +`[sr.ht]environment != "production"`; stylesheet href globbed from +`static-dir/main.min.*.css` at startup. CSS: `sassc -I $(ASSETS)/scss scss/main.scss` +→ minify → `main.min..css` (Makefile cloned from git.sr.ht); logo.svg +copied from core.sr.ht. + +## Access matrix (core/access.go — pure, table-driven tests) + +Ops: OpBrowse, OpCloneRead, OpPush, OpAdmin. + +| Caller \ Visibility | PUBLIC | UNLISTED | PRIVATE | +|---|---|---|---| +| anonymous / any user | browse, clone | browse, clone (unlisted) | — | +| ACL RO | browse, clone | browse, clone | browse, clone | +| ACL RW | + push | + push | browse, clone, push | +| owner | all | all | all | + +PRIVATE + unauthorized ⇒ NotFound (don't leak existence). Suspended users: reads +allowed, push/admin denied. Token grants: `dolt.sr.ht/repos:RO` for reads, `:RW` for +push (empty-grant personal tokens pass via `HasAll`). + +## remotesapi subsystem + +- **Interceptors** (remoteapi/interceptors.go): our own + `grpc.ChainUnaryInterceptor`/`ChainStreamInterceptor` via `ServerArgs.Options`. + Per RPC: classify method → extract `authorization` metadata → + `authn.ResolveGRPC(ctx, header)`: + - `Basic` → username + pass; `auth.DecodeBearerToken(pass)` (offline HMAC) → + require token owner == presented username → parallel `LookupUser` + + `LookupTokenRevocation` (like `auth.OAuth2`) → positive results cached 60 s + in-process keyed by `sha512(pass)` (a push issues many RPCs; bounds revocation + lag), negatives uncached. + - `Bearer` → parse JWS (go-jose, EdDSA): kid header → `dolt_key` row → verify + signature with stored pubkey; check `aud == our host`, `exp`, + `sub == "doltClientCredentials/"`; ignore iss. Local-only, no cache needed + (30 s tokens); update `last_used`. Reuse `creds.PubKeyToKIDStr` from dolt. + - absent → anonymous. + Then repo path from `GetRepoPath()`/`RepoId` → `core.ParseRepoPath` (trim `/`, + optional `~`, reject `..`) → load repo row → `core.Allowed(caller, repo, op)` → + `handler(authedCtx, req)`. Stream wrapper overrides `Context()` and checks each + `RecvMsg`. Base ctx carries `config.Context` + `database.Context`. +- **DBCache** (storage/dbcache.go): `Get(ctx, path, nbfVer)` → require existing + repository row (no push-to-create in v1; creation is explicit via web) → + memoized `nbs.NewLocalStore(ctx, nbfVer, absPath, 128MiB, + nbs.NewUnlimitedMemQuotaProvider(), false)` (same as upstream LocalCSCache); + evict + close on repo delete. +- **Assembly** (remoteapi/server.go): `remotesrv.NewServer(ServerArgs{HttpHost: + "dolt.srht.bigb.es", HttpListenAddr == GrpcListenAddr: "127.0.0.1:5306", FS: + filesys.LocalFS, DBCache, Options: ourInterceptors, ConcurrencyControl: + PUSH_CONCURRENCY_CONTROL_IGNORE_WORKING_SET, ...})`. Sealed-URL data plane needs + no extra auth. Per-process sealer key ⇒ in-flight transfers break on restart + (client retries; acceptable). +- **CredentialsService** (remoteapi/credsvc.go): own `grpc.NewServer` on `:5308` + registering `remotesapi.RegisterCredentialsServiceServer`; `WhoAmI` verifies the + Bearer JWT exactly as above and returns Username/EmailAddress/DisplayName from the + user row. `dolt login` polls WhoAmI until the key is associated via the web page. +- **Create/delete flow** (storage/init.go + web): validate name → tx INSERT + repository → `doltdb.LoadDoltDB(ctx, types.Format_Default, + earl.FileUrlFromPath(absPath), filesys.LocalFS)` + `ddb.WriteEmptyRepo(ctx, + "main", [sr.ht]owner-name, owner-email)`; remove dir on tx failure. Delete: tx + delete row → `os.RemoveAll` → DBCache evict. + +**Keypair UX shipped in v1** (user-confirmed): users run +`dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url +https://dolt.srht.bigb.es/settings/keys` (or set `remotes.default_host` / +`creds.add_url` once via `dolt config --global`); the settings page associates the +posted pubkey with the logged-in SourceHut user; thereafter plain +`dolt clone/push` works with no `--user` and no env var — like git SSH keys. + +## Browse subsystem (browse/) — riskiest code, isolated + +No SQL engine (can't open bare stores). Per request, read-only, closed after: +`doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, filesys.LocalFS)` — NBS +manifest-based readers are safe alongside the push writer. +- Branches: `ddb.GetBranches(ctx)`; default `main`. +- Log: `ResolveCommitRef` + `commitwalk.GetTopologicalOrderIterator` (from + `go/libraries/doltcore/env/actions/commitwalk`), paginated by count + start hash. +- Tables/schema: commit → `RootValue` → `GetTableNames` / `GetTable(...).GetSchema()`. +- Rows: `table.GetRowData` → `durable.ProllyMapFromIndex` → + `prolly.Map.IterOrdinalRange(offset, offset+limit)`, rendered via schema types. +- Commit page: diff **summaries** only in v1 (per-table added/dropped/schema-changed + + row-count delta via the diff package); full row diffs deferred. + +Fallback if prolly iteration proves fragile across versions: keep a lightweight +materialized clone per db (real `.dolt`, refreshed post-push) opened via +`dolthub/driver` — swap contained inside `browse/`. + +## Config + nginx + +`config.example.ini` (goes into the single shared instance config.ini; existing +services need only the `[dolt.sr.ht] origin=` line + restart to show us in nav): + +```ini +[dolt.sr.ht] +origin=https://dolt.srht.bigb.es +connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable +repos=/var/lib/dolt +remotesapi-listen=127.0.0.1:5306 +credsapi-listen=127.0.0.1:5308 +static-dir=/usr/share/sourcehut/dolt.sr.ht/static +migrate-on-upgrade=yes +``` + +Reused in place: `[sr.ht]` network-key / internal-ipnet / owner-* / site-name / +environment; `[webhooks]private-key`; `[meta.sr.ht]origin`. Deployment checklist: +our host must be in **meta's** `internal-ipnet` (FetchMetaProfile + +LookupTokenRevocation use internal auth). + +nginx (`contrib/dolt.sr.ht.conf`, sr.ht-nginx style): + +```nginx +server_name dolt.srht.bigb.es; # TLS/http2 via sourcehut.conf + port443.conf +location /dolt.services.remotesapi.v1alpha1.ChunkStoreService/ { + grpc_pass grpc://127.0.0.1:5306; + grpc_set_header X-Forwarded-Proto https; + grpc_read_timeout 600s; grpc_send_timeout 600s; client_max_body_size 0; +} +location /dolt.services.remotesapi.v1alpha1.CredentialsService/ { + grpc_pass grpc://127.0.0.1:5308; + grpc_set_header X-Forwarded-Proto https; +} +location /single_symmetric_key_sealed_request/ { # chunk data plane GET+PUT + proxy_pass http://127.0.0.1:5306; + client_max_body_size 0; proxy_request_buffering off; proxy_read_timeout 600s; +} +location / { proxy_pass http://127.0.0.1:5307; } # web UI + /static +``` + +## Implementation phases (parallel-implementer convention from CLAUDE.md) + +**Phase 0 — foundation (serial, commit first):** go.mod; pin ALL deps via `go get` +(+ throwaway smoke-import build to populate go.sum): core-go@mirror-version, +`github.com/dolthub/dolt/go@latest`-resolved pseudo-version (record it), +go-jose, grpc, chi/v5, lib/pq, brant, logrus. Complete `core/` package + unit +tests; schema.sql + migrations/0001; config.example.ini; Makefile; nginx conf; +README. **Spike test** (`storage/spike_test.go`, build-tagged): WriteEmptyRepo → +remotesrv on localhost → `dolt clone http://127.0.0.1:PORT/x/y` with CLI v2.1.10 — +proves format/RPC compat before any parallel work. Gate: `go build ./... && go test +./core/...`. + +**Phase 1 — parallel wave A** (disjoint packages; each implementer builds/tests only +its own package, never `./...`): +- 1a `db/` — repository/access/dolt_key queries. +- 1b `authn/` — cookie + Basic-token + Bearer-JWT resolution; tests forge tokens via + `auth.BearerToken.Encode()` with synthesized ini (random fernet key + ed25519 + seed); revocation stubbed behind an interface; JWT tests sign with + ephemeral ed25519 keys. +- 1c `storage/` — InitStore/DeleteStore/Cache (repo lookup injected as func). +- 1d `browse/` — fixtures built programmatically via doltdb API. + +**Phase 2 — parallel wave B:** +- 2a `remoteapi/` — interceptors, server assembly, credsvc; integration test + (`-tags integration`) driving real dolt CLI: clone → sql insert → commit → push → + re-clone → verify; ACL denial; anonymous PUBLIC clone; Bearer-JWT clone after key + association; PRIVATE anonymous ⇒ NotFound. (needs 1b, 1c; 1a via interface) +- 2b `web/` — router/handlers/templates/chrome; httptest with stubbed interfaces. + (needs 1a, 1b, 1d) + +**Phase 3 — serial finish:** `cmd/doltsrht`, `cmd/doltsrht-migrate`, startup smoke +test (/healthz), **`go mod tidy` only now**, CSS build against instance scss, README +deployment section. + +## Verification (post-deploy, end-to-end) + +1. Add `[dolt.sr.ht]` section to shared config.ini, `doltsrht-migrate init`, deploy + nginx conf + DNS `dolt.srht.bigb.es`, start `doltsrht`. +2. Browser: log into meta → visit dolt.srht.bigb.es → nav shows all services + + "Logged in as bigbes"; nav of git/meta pages now shows "dolt". +3. Create db `demo` (PUBLIC) via /create. +4. Token flow: `export DOLT_REMOTE_PASSWORD=`; `dolt clone --user bigbes + https://dolt.srht.bigb.es/~bigbes/demo`; create table, insert, `dolt commit`, + `dolt push origin main`. +5. Web: overview shows branch + commits; /log, /table/main/t shows the row; + /commit/ shows table summary. +6. Keypair flow: `dolt creds new`; `dolt login --auth-endpoint + dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys`; + associate key in browser; `dolt clone https://dolt.srht.bigb.es/~bigbes/demo` + (no --user) + push succeed. +7. Negative: anonymous clone of PUBLIC works; flip to PRIVATE → anonymous clone ⇒ + not found; user2 token push ⇒ denied; grant RO → clone ok, push denied; revoke + PAT on meta → push blocked within ≤60 s. + +## Open risks + +- dolt CLI v2.1.10 ⟷ pinned `dolt/go` module compat — de-risked by Phase-0 spike; + worst case pin different module version or require newer CLI. +- prolly-level browse code is version-fragile — isolated in `browse/` with + materialized-clone fallback design. +- 60 s revocation-cache window for Basic tokens — accepted, documented. +- internal-ipnet / shared-key misconfig symptom: "Temporary error" on first + login/push — deployment checklist item.

~bigbes/sourcehut-dolt: b4d6a3e6

1 changed file(s)

~bigbes/sourcehut-dolt: `b4d6a3e6`