diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000000000000000000000000000000000000..f497583c193546ac756d37ef0b84e07fc96a253f --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,377 @@ +# dolt.sr.ht — "DoltLab for SourceHut" (self-hosted Dolt database hosting service) + +## Context + +bigbes wants a DoltLab-equivalent integrated into the self-hosted SourceHut instance +(`*.srht.bigb.es`): a new service `dolt.sr.ht` that hosts Dolt databases the way +git.sr.ht hosts git repos — `dolt clone/push/pull` over HTTPS plus an integrated web +UI (shared nav, unified login via the `sr.ht.unified-login.v1` cookie, shared +Bootstrap theme). Pure Go, single new repo at `/Users/blikh/data/home/sourcehut-dolt` +(currently empty). No upstream SourceHut modification — integration is config-driven +(a `[dolt.sr.ht]` config section puts us in every service's nav). DoltLab itself is +closed-source; everything we need is in Apache-2.0 `dolthub/dolt` packages. + +**v1 scope (user-confirmed):** create/list/browse databases + clone/push over the +remotesapi, with **both** auth flows: username + meta personal access token +(`--user` + `DOLT_REMOTE_PASSWORD`, Basic) **and** dolt's Ed25519 keypair flow +(`dolt creds`/`dolt login`, Bearer EdDSA JWT — the git-SSH-key-like UX). No GraphQL +federation, no SSH transport, no webhooks, no issues/PRs. + +All findings below were verified against dolthub/dolt `main` and the SourceHut mirror +at `/Users/blikh/data/home/sourcehut` (core-go/, core.sr.ht/, git.sr.ht/, sr.ht-nginx/). + +## Architecture summary + +One Go module **`go.bigb.es/sourcehut-dolt`**, two binaries: +- **`doltsrht`** — one process, three listeners: + - web UI (chi) on `-b localhost:5307` + - remotesapi (gRPC `ChunkStoreService` + HTTP chunk data plane, h2c-multiplexed + single port) on `127.0.0.1:5306` via importable + `github.com/dolthub/dolt/go/libraries/doltcore/remotesrv` + - a small separate gRPC server for `CredentialsService.WhoAmI` (keypair login) on + `127.0.0.1:5308` — nginx path-routes it, so no need to touch remotesrv internals +- **`doltsrht-migrate`** — brant migrations (`git.sr.ht/~bitfehler/brant`), modeled on + `sourcehut-migrate/cmd/sourcehut-migrate/main.go`. + +Storage: **bare NBS chunk-store dirs** (no `.dolt/`, no working set) at +`/~/` — exactly what `remotesrv` serves and `file://` remotes use. +Postgres holds metadata (user mirror + repository + access + dolt_key). +Clone URL: `dolt clone https://dolt.srht.bigb.es/~user/db`. + +Key verified facts that shape the design: +- `server.New` (core-go) works **without** `WithSchema` (gqlgen-only). But + `WithDefaultMiddleware` attaches `auth.Middleware`, which 401s requests with no + cookie/header — kills anonymous browsing. → don't call it; assemble our own + middleware group on `srv.AnonRouter()`. +- core-go token validation trio (mirrors `auth.OAuth2`): `auth.DecodeBearerToken` + (offline HMAC via `[webhooks]private-key`-derived key) + `auth.LookupUser` (local + `"user"` table, lazily populated by `FetchMetaProfile` internal-GraphQL to meta) + + `auth.LookupTokenRevocation` (GraphQL `tokenRevocationStatus` on meta). Requires + shared `[sr.ht]network-key`, `[webhooks]private-key`, and our host inside meta's + `internal-ipnet`. +- dolt CLI credential selection (`go/libraries/doltcore/env/grpc_dial_provider.go`): + `--user` ⇒ `authorization: Basic b64(user:pass)` with pass from + `DOLT_REMOTE_PASSWORD`; no `--user` but `dolt creds`/`dolt login` key present ⇒ + `authorization: Bearer ` (aud = our host, exp = now+30s, kid = + base32(SHA-512/224(pubkey)) custom alphabet `0123456789abcdefghijklmnopqrstuv`, + iss hardcoded `dolt-client.dolthub.com` — do NOT reject on iss); no creds at all ⇒ + no metadata ⇒ anonymous (must work for PUBLIC clones). Client is fully + host-agnostic: `dolt login --auth-endpoint --login-url `, defaults + settable via `dolt config --global` keys `remotes.default_host`, `creds.add_url`. +- Upstream `remotesrv.ServerInterceptor` drops the authenticated ctx and never sees + the repo path ⇒ per-repo ACLs impossible with it. → we write our own + unary+stream interceptors (keep upstream's method classification: write = + {GetUploadLocations, AddTableFiles, Commit}; read = the 8 others; unknown ⇒ deny). +- Data plane is collision-free for nginx: gRPC under + `/dolt.services.remotesapi.v1alpha1.*/`, sealed chunk URLs under + `/single_symmetric_key_sealed_request/` (AES-GCM sealed, 15-min expiry, possession + = authorization), web UI owns the rest. `getScheme()` honors + `x-forwarded-proto` — nginx must set it on grpc_pass. +- Empty-db init primitive: `doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, fs)` + + `ddb.WriteEmptyRepo(ctx, "main", name, email)`. +- Bare stores **cannot** be opened by the sqle engine / embedded driver (they expect + working sets). Browse UI uses low-level read-only APIs instead (see §Browse). +- dolt CLI v2.1.10 installed at `/opt/homebrew/bin/dolt` (integration tests). +- `github.com/dolthub/dolt/go` is a separate module pinned via pseudo-versions; + heavy deps, CGO (gozstd). Apache-2.0. + +## Repo layout + +``` +go.mod Makefile schema.sql migrations/ config.example.ini README.md LICENSE +contrib/dolt.sr.ht.conf # nginx +scss/main.scss # @import "base" against core.sr.ht shared scss +static/ # logo.svg + main.min..css build output +cmd/doltsrht/main.go +cmd/doltsrht-migrate/main.go +core/ # PURE domain: names.go (validate, ParseRepoPath), access.go (matrix), models.go +db/ # postgres: repos.go, access.go, keys.go (dolt_key CRUD) +authn/ # ctx.go (Caller), cookie.go (optional unified-login middleware), + # token.go (Basic: PAT trio + 60s cache), jwt.go (Bearer: EdDSA verify) +storage/ # init.go (InitStore/DeleteStore via WriteEmptyRepo), dbcache.go (remotesrv.DBCache) +remoteapi/ # server.go (remotesrv assembly), interceptors.go, credsvc.go (WhoAmI grpc server) +browse/ # open.go, log.go, tables.go, diff.go — read-only doltdb over bare stores +web/ # router.go, handlers_*.go, chrome.go, templates.go, templates/*.html +``` + +Dependency direction: `core` ← all; `db`/`authn`/`storage`/`browse` mutually +independent; `remoteapi` ← {core, db, authn, storage}; `web` ← {core, db, authn, +browse, storage}; `cmd` ← all. + +## Postgres schema (schema.sql, brant `-- +brant Up` migrations) + +```sql +CREATE TYPE visibility AS ENUM ('PUBLIC', 'PRIVATE', 'UNLISTED'); +CREATE TYPE user_type AS ENUM ('PENDING', 'USER', 'ADMIN', 'SUSPENDED'); +CREATE TYPE access_mode AS ENUM ('RO', 'RW'); + +CREATE TABLE "user" ( -- exact UserMixin shape; core-go FetchMetaProfile inserts rows + id integer PRIMARY KEY, -- meta's user id, inserted explicitly (not serial) + username varchar(256) UNIQUE, + created timestamp NOT NULL, updated timestamp NOT NULL, + email varchar(256) NOT NULL UNIQUE, + user_type user_type NOT NULL, + url varchar(256), location varchar(256), bio varchar(4096), + suspension_notice varchar(4096) +); +CREATE INDEX ix_user_username ON "user"(username); + +CREATE TABLE repository ( + id serial PRIMARY KEY, + created timestamp NOT NULL, updated timestamp NOT NULL, + name varchar(64) NOT NULL, + description varchar(1024), + owner_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE, + path varchar(1024) NOT NULL UNIQUE, -- absolute on-disk store dir + visibility visibility NOT NULL, + CONSTRAINT uq_repo_owner_id_name UNIQUE (owner_id, name) +); + +CREATE TABLE access ( + id serial PRIMARY KEY, + created timestamp NOT NULL, updated timestamp NOT NULL, + repo_id integer NOT NULL REFERENCES repository(id) ON DELETE CASCADE, + user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE, + mode access_mode NOT NULL, + CONSTRAINT uq_access_user_id_repo_id UNIQUE (user_id, repo_id) +); + +CREATE TABLE dolt_key ( -- Ed25519 pubkeys for dolt creds auth + id serial PRIMARY KEY, + created timestamp NOT NULL, + user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE, + kid varchar(64) NOT NULL UNIQUE, -- base32(SHA-512/224(pubkey)), dolt alphabet + pubkey bytea NOT NULL, -- 32-byte ed25519 public key + comment varchar(256), + last_used timestamp +); +``` + +DB name rule: `^[a-zA-Z0-9](?:[a-zA-Z0-9_-]*[a-zA-Z0-9])?$`, max 64 (names double as +SQL database identifiers; reject `.`/`..`). + +## Main wiring (`cmd/doltsrht/main.go`) + +```go +conf := config.LoadConfig() +srv := server.New("dolt.sr.ht", "localhost:5307", conf, os.Args[1:]) // runs crypto.InitCrypto +db := sql.Open("postgres", cfg[dolt.sr.ht]connection-string) +srv.AnonRouter().Group(func(r chi.Router) { + r.Use(chimw.RealIP, chimw.Recoverer) + r.Use(config.Middleware(conf, "dolt.sr.ht"), database.Middleware(db)) + r.Use(authn.OptionalCookieMiddleware()) // never 401s; anonymous stays anonymous + web.Register(r) +}) +rsrv := remoteapi.New(...) // remotesapi on 127.0.0.1:5306 (grpc+http one port) +csrv := remoteapi.NewCredSvc(...) // CredentialsService.WhoAmI on 127.0.0.1:5308 +go rsrv.Serve(); go csrv.Serve(); srv.Run() +``` + +No redis, no email, no `WithSchema` in v1. + +**Routes** (web/router.go): `GET /` dashboard; `GET,POST /create` (cookie required); +`GET /~{user}`; `GET /~{user}/{db}` overview (branches, latest commits, clone box +with both auth variants); `GET /~{user}/{db}/log?branch=&from=`; +`GET /~{user}/{db}/commit/{hash}`; `GET /~{user}/{db}/tree/{ref}`; +`GET /~{user}/{db}/table/{ref}/{table}?page=`; `GET,POST /~{user}/{db}/settings` +(owner only: description/visibility/ACLs/delete); `GET,POST /settings/keys` +(dolt-key association page — reads `#` URL fragment via a few lines of +inline JS to prefill the form, since `dolt login` appends the pubkey as a fragment); +`GET /static/*`. + +**Chrome** (web/chrome.go + templates): Go `html/template` port of core.sr.ht +`layout.html`/`nav.html` — `