diff --git a/docs/tasks/lethe-collector-claude-code.md b/docs/tasks/lethe-collector-claude-code.md new file mode 100644 index 0000000000000000000000000000000000000000..908ac70910dcbea58c491d4120e32e414708e72b --- /dev/null +++ b/docs/tasks/lethe-collector-claude-code.md @@ -0,0 +1,176 @@ +# lethe-collector-claude-code + +**Status:** Design (hands-off) +**Module:** `sourcecraft.dev/bigbes/lethe` +**Depends on:** `lethe-server.md` (#1) — locks the wire format and ingest semantics this task targets. +**Sibling tasks (deferred):** `lethe-search-and-opencode.md` (#3) and per-tool follow-ups (`lethe-collector-crush.md`, etc.) when the time comes. + +## Design + +### Purpose + +Stand up the `lethe-collector` binary and the first parser (Claude Code). End state: a systemd user service on the laptop watches `~/.claude/projects/`, parses new turns, ships them to the running `lethe` server over Tailscale, and survives offline periods via a local outbox. Re-runs are safe and resumable. + +A successful end state for this task: the collector has been running on the laptop for an hour against real Claude Code activity, and the server's HTML timeline shows my actual recent sessions, with turns matching what's in the `.jsonl` files. + +### Scope + +**In:** +- Single Go binary `lethe-collector` (`cmd/lethe-collector/main.go`), cobra-based: + - `lethe-collector daemon` — long-running, watches all configured sources. + - `lethe-collector backfill ` — one-shot, walks all source files from offset 0, ships everything; resumable via the same offset state. + - `lethe-collector status` — prints per-source ingestion lag, outbox depth, last error. +- Parser interface in `internal/collector/parser/` (the Parser type from RFC §6.2, populated against the locked `internal/shared/wire/` types). +- Claude Code parser in `internal/collector/parser/claudecode/` with golden-file fixtures. +- Local state DB in `~/.local/state/lethe/state.db` (SQLite, one file): tables `ingestion_state` (per source file offset) and `outbox` (buffered events when server is unreachable). +- Polling-based discovery and ingestion loop (no fsnotify); per-source goroutines orchestrated via `auxilia/async`. +- HTTPS POST to the server's `/api/v1/ingest` over Tailscale; relies on `tailscale serve` injecting `Tailscale-User-Login` for the authenticated daemon. +- Outbox replay with exponential backoff, bounded size (default 100 MB), oldest-drop on overflow with WARN. +- systemd user unit shipped at `deploy/lethe-collector.service` with `Restart=always`, `WantedBy=default.target`, journald logging. +- Configuration via YAML at `~/.config/lethe/collector.yaml`, loaded with the same Viper strict-mode pattern as the server. +- Logging via `scribe`, errors via `culpa`. + +**Out:** +- Other parsers (opencode → #3; crush, pi, kimi → their own task files later). +- Any server-side change. Server is locked from #1; if a wire-format gap is discovered, it gets a separate amendment task. +- macOS launchd unit (Linux only for v1; trivially added later — same binary, different unit file). +- TUI / curses status. `status` prints plain text. +- File-watcher backend (fsnotify or inotify directly). +- HTTP/2 push, gRPC, or any non-NDJSON-over-HTTPS transport. +- Multi-user / multi-account configurations (still single-tenant). + +### Chosen approach + +**CLI: cobra.** Three subcommands. `daemon` is the default deployed mode; `backfill` is the bootstrap and disaster-recovery tool; `status` is the operator's quick-look. Cobra is overkill for one command but right for three and pulls its weight from this task onward. + +**Discovery: polling.** Every source has a configurable `poll_interval` (default 30s). On each tick, the source walks its root, lists candidate files (e.g. `**/*.jsonl` under `~/.claude/projects/`), and processes each one independently. Polling beats fsnotify here because: +- Source tools may write via `rename(tmp, final)` — fsnotify fires on a path that immediately doesn't exist at handle-open time. +- Long-running sessions append continuously; a single file gets touched many times — polling coalesces naturally. +- Cross-machine, cross-FS, cross-tool: polling has no edge cases. fsnotify has many. + +**Per-source ingestion loop.** +1. Walk the root, list source files. +2. For each file: load `last_offset` from `ingestion_state` keyed by `(tool, source_file)`. +3. Open file read-only, seek to `last_offset`, scan to EOF using `bufio.Scanner` with a sufficiently-large buffer (Claude Code lines can be hundreds of KB). +4. For each complete line, hand to the parser, accumulate `wire.TurnEvent`s. +5. Batch up to N events (default 500) or M bytes (default 8 MiB), whichever first; serialize to NDJSON; POST to `/api/v1/ingest`. +6. On `200 {accepted: K, errors: [...]}`: persist `last_offset = offset_at_line(K)` and continue from line K+1. If `K < N`, log the errors at WARN and skip the bad lines (their offset is also persisted past them so they don't loop forever). +7. On 5xx or network error: serialize the unsent events into the `outbox` table and break to next file. Replay attempted on next tick. +8. Sleep `poll_interval`, loop. + +**Outbox.** A `outbox` table in the state DB: `(id INTEGER PK AUTOINCREMENT, tool TEXT, host TEXT, source_file TEXT, payload BLOB, created_at INTEGER)`. On every tick, before processing fresh files, the loop tries to replay outbox rows oldest-first in chunks. Each successful POST deletes the rows it committed. Bounded by `outbox.max_bytes` config (default 100 MiB); when exceeded, oldest rows are dropped and a WARN is logged. The "happy path" (server reachable) never writes to the outbox at all — it's a strict overflow buffer. + +**Parser interface.** +```go +package parser + +import "sourcecraft.dev/bigbes/lethe/internal/shared/wire" + +type Parser interface { + Tool() string + Discover(root string) ([]SourceFile, error) + Parse(path string, since int64) (events []wire.TurnEvent, newOffset int64, err error) +} + +type SourceFile struct { + Path string + Size int64 +} +``` + +`Parse` returns events in source order with monotonically-increasing `seq`. If a line is malformed, the parser returns it as a `system`-role turn with the raw line in `metadata` (so it shows up in the archive but doesn't poison search) and continues. `newOffset` is the byte position immediately after the last fully-parsed line — never mid-line, so a partial trailing write is left for the next poll. + +**Claude Code parser specifics.** +- Source root: `~/.claude/projects/`. File pattern: `*/.jsonl` (one file per session). +- `session_id`: the UUID from the filename. The directory name (``) goes into `session_meta.metadata` for project attribution. +- One `.jsonl` line = one event, parsed into a permissive struct that uses `json.RawMessage` for any ambiguous field. +- Event-type mapping: + - `type: "user"` → `role: "user"`, `content` = the user message text. + - `type: "assistant"` → `role: "assistant"`, `content` = joined assistant text parts; `model` from the event; `tokens_in/out` from `usage.input_tokens/output_tokens` when present. + - `type: "tool_use"` and `type: "tool_result"` → `role: "tool"`, `content` = a short rendered summary (e.g. `""`), full payload into `tool_calls` JSON. + - `type: "summary"` and unknown types → `role: "system"`, content from event, full event into `metadata`. +- `cwd` field → `session_meta.working_dir`. The path of the file → `session_meta.source_file`. +- `cost_usd` left null (Max-billed sessions don't reliably report cost). +- `turn_id`: prefer the event's `uuid` field. When missing, synthesize `sha256(session_id || seq || timestamp || content[:64])` truncated to 16 bytes hex. +- `parentUuid` (resume chaining): stored in turn `metadata` for now. Chaining sessions across files is a #3-or-later UI concern — every `.jsonl` file is a session in this task. +- Slash commands and sub-agent invocations: their event subtypes go into `metadata` opaquely. The UI in #1 already renders `metadata` as JSON-collapsed; surfacing them properly is a later refinement. + +**Auth.** The collector POSTs to `https://.tailnet.ts.net/api/v1/ingest`. `tailscale serve` on phoebe terminates HTTPS and injects `Tailscale-User-Login` from the connecting node's owner. The server validates that header against its allowlist. If `tailscale serve` doesn't inject the header for non-browser clients (the open question from #1), the deploy step fixes it — the collector code itself is unchanged. + +**Configuration.** YAML at `~/.config/lethe/collector.yaml`: + +```yaml +server_url: "https://phoebe..ts.net" +host: "laptop" # required; identifies this machine in the archive +state_dir: "~/.local/state/lethe" + +http: + timeout: "30s" + retry_max: 5 + +outbox: + max_bytes: 104857600 # 100 MiB + +sources: + - tool: "claude-code" + path: "~/.claude/projects" + poll_interval: "30s" + batch_max_lines: 500 + batch_max_bytes: 8388608 # 8 MiB + +log: + level: info + format: human +``` + +`host` is required and has no default. The host string is the user's choice; the server stores it verbatim. + +**Tradeoffs that settled it.** +- *Polling vs fsnotify:* polling is correct in every case; fsnotify isn't. The wasted CPU of one `os.ReadDir` per minute is irrelevant. +- *Outbox in SQLite vs flat-file queue:* one file (`state.db`) for both offsets and outbox, atomic transactions, no separate format to debug. Cost is one extra dependency that was already required. +- *Parse-then-batch vs streaming POST:* batching keeps the wire protocol simple (NDJSON body, one HTTP call) and lets the server commit chunks atomically. Streaming would force the server to handle interrupted bodies — the RFC's chunked-commit response shape works because the body is bounded. +- *Synthesize missing turn_ids vs require source IDs:* Claude Code always provides UUIDs in current versions, but the parser can't assume that holds for older fixture files or future regressions. Synthesis preserves idempotency; the rare case of a `content[:64]` collision within one session at one timestamp is acceptable. + +**Unknowns that remain.** +- Whether `tailscale serve` injects `Tailscale-User-Login` for daemon HTTP clients (vs only browsers). If not, I add a `lethe-token` shared-secret fallback header in the deploy step — a 5-line server change. Confirmed empirically before declaring this task done. +- True line-size distribution of Claude Code `.jsonl` events. If it exceeds `bufio.Scanner`'s default 64 KiB token buffer, the parser uses `Scanner.Buffer(buf, maxSize)` with maxSize = 16 MiB. Captured here so the test fixtures cover the long-line case. +- Whether the laptop's `~/.claude/projects/` ever contains files concurrent-written from multiple Claude Code processes. If yes, the parser still works (append-only, monotonic offset), but the test plan should cover it. + +### Backwards-compatibility check + +Greenfield collector. The only interface contract this task can break is the wire format with the server, which is locked into `internal/shared/wire/` and cannot drift unilaterally. + +### Hands-off decisions + +- udesign: parser interface placed at `internal/collector/parser/` rather than `internal/parsers/` (RFC §6.2 used `internal/parsers//`) — keeps "collector-internal" code in one subtree, mirrors `internal/server/` from #1. +- udesign: outbox lives in the same state DB as offsets — single SQLite file is simpler than two stores; transaction guarantees are useful when an offset bump and an outbox dequeue happen together. +- udesign: bad-line handling skips with WARN rather than halting the file — one corrupt line in `.jsonl` shouldn't pause ingestion of the rest. Risk: silent data loss if many lines are wrong; mitigated by counting WARNs in `status`. +- udesign: `host` is required config with no default — auto-detecting via `os.Hostname()` produces noise on machines whose hostname is `myname-mbp.local`. Forcing the user to choose `laptop` / `workpc` keeps the archive's host column meaningful. +- udesign: TOML → YAML for collector config — consistent with the server's config format from #1; one parser, one mental model. +- udesign: `parentUuid` chaining of resumed Claude Code sessions deferred — every `.jsonl` is one session in this task. Surfacing chains is a UI concern for later. +- udesign: synthesized `turn_id` uses `sha256(session_id || seq || timestamp || content[:64])[:16]` — `content[:64]` is enough to disambiguate within a single timestamp; full-content hash would balloon for large turns. + +TDD: yes (reason: parser behavior on golden fixture `.jsonl` files, offset persistence/resume semantics, outbox replay, and idempotent re-POST behavior are exactly the deterministic regression-prone surfaces TDD is good for. CLI scaffolding and systemd unit are exempt.) + +### Invariants + +- The collector opens source files **read-only**. No code path writes, renames, or deletes anything under any source root. +- `ingestion_state.last_offset` is persisted **only after** the server returns `accepted: N` and the offset has been advanced past line N. +- Offsets are byte positions immediately after a fully-parsed `\n`. Never mid-line; partial trailing lines are left for the next poll. +- A re-run after crash, kill -9, power loss, or container restart resumes from the last persisted offset. Server-side idempotency handles any duplicates the offset miss generates. +- The outbox is bounded by `outbox.max_bytes`. Overflow drops oldest entries with a WARN; never blocks the loop. +- One source file's failure (parse error, permission error, deleted file mid-loop) does not stop ingestion of any other source file. +- `wire.TurnEvent.host` on every emitted event equals the `host` from config. The collector does not infer host from `os.Hostname()` or any environment. +- The Parser interface is the only point of tool-specific knowledge in the collector. The ingestion loop knows nothing about Claude Code's JSON shape. +- All HTTP requests to the server include the configured `server_url` and the path `/api/v1/ingest`; no other endpoints are called from `daemon` mode (`status` may call read endpoints). +- `daemon` mode handles SIGTERM by stopping new polls, draining in-flight POSTs (bounded), persisting any in-memory offset advances, and exiting with code 0. + +### Principles + +- Polling beats watching. The cost is bounded; the correctness is total. +- Each source's loop is independent. Nothing in the architecture forces cross-source coordination. +- The parser is the only file that knows a tool's format. Adding a tool is one new directory under `internal/collector/parser/`, plus a `register.go` import. +- The outbox is a safety net, not the primary path. The happy path skips it entirely. If a bug forces all traffic through the outbox, that's a regression worth alerting on. +- Permissive parsing: unknown fields → `metadata`, malformed lines → system-role turn with raw payload. Never panic, never stall. +- No background goroutines without a `context.Context` tied to shutdown. +- Test against real fixture files (anonymized snippets from `~/.claude/projects/` checked into `testdata/`), not hand-crafted minimal JSON. diff --git a/docs/tasks/lethe-search-and-opencode.md b/docs/tasks/lethe-search-and-opencode.md new file mode 100644 index 0000000000000000000000000000000000000000..27eac9360d64161605fe4bd02c9b27b0ff67e916 --- /dev/null +++ b/docs/tasks/lethe-search-and-opencode.md @@ -0,0 +1,146 @@ +# lethe-search-and-opencode + +**Status:** Design (hands-off) +**Module:** `sourcecraft.dev/bigbes/lethe` +**Depends on:** `lethe-server.md` (#1) — FTS5 tables and triggers were created in #1; this task only adds query code. `lethe-collector-claude-code.md` (#2) — the collector framework and Parser interface this task extends. +**Sibling tasks (deferred):** per-tool parsers (`lethe-collector-crush.md`, `lethe-collector-pi.md`, `lethe-collector-kimi.md`); RFC backlog items (cost rollups for tools that report it, tagging, JSON/Markdown export). + +## Design + +### Purpose + +Make the archive answer "what did I ask any robot last Tuesday about X" in under five seconds, and add the second tool to prove the parser interface scales. End state: a search box on the timeline that returns ranked turn snippets with click-through to session, a stats page showing token rollups, and a working `opencode` parser registered in the collector. + +A successful end state for this task: I can type "tt bundle lockfile" into the search box, see ranked snippets across both Claude Code and opencode sessions on either machine, click into any result and land on the matching turn. The stats page shows a per-tool, per-day token bar for the last month. + +### Scope + +**In:** +- Server-side: + - `GET /api/v1/search?q=&tool=&host=&since=&until=&include_tool_outputs=&limit=&cursor=` — FTS5 query against `turns_fts`. With `include_tool_outputs=1`, also union `tool_outputs_fts` and merge by BM25 rank, deduped on `(tool, host, session_id, turn_id)`. + - `GET /api/v1/stats?group_by=tool|host|day&since=&until=` — token rollups (sum `tokens_in`, `tokens_out`, count of turns, count of sessions). Cost is summed only where non-NULL. + - HTML `GET /search?q=...` — server-rendered results page with snippet/highlight, filters in URL. + - HTML `GET /stats` — table view of the stats endpoint, simple and readable. + - Search-as-you-type debouncing (vanilla JS, ~300 ms) on the timeline and search pages. + - Session view gains an anchor-jump on `#turn-` and a "find on page" hint. +- Collector-side: + - `internal/collector/parser/opencode/` — new parser implementing the same Parser interface from #2. + - Format-discovery spike captured under `docs/spikes/opencode-format.md` before the parser is written; spike output is checked in. + - One config-entry change to register opencode in the dispatcher; otherwise the collector framework is untouched. +- Golden fixtures for the opencode parser (anonymized snippets in `testdata/opencode/`). + +**Out:** +- Cost rollups for tools that don't report cost. Stats endpoint sums what's there and reports nulls as null. +- crush, pi, kimi parsers — separate task files when the time comes. +- Tag system for manual session annotation (RFC backlog). +- JSON / Markdown export endpoints (RFC backlog). +- Faceted search UI (e.g. histogram-driven date range pickers). Filters stay as form fields and URL params. +- Saved searches, alerts, RSS, anything subscription-shaped. +- A second machine's deploy (the goal is to prove the parser interface with #2; running on the work PC is a deployment exercise, not a code change). + +### Chosen approach + +**Search query (default — `include_tool_outputs=0`).** One FTS5 `MATCH` against `turns_fts`, filtered by the optional `tool`, `host`, `since`, `until` UNINDEXED columns: + +```sql +SELECT + f.tool, f.host, f.session_id, f.turn_id, f.timestamp, f.role, + snippet(turns_fts, 0, '', '', '…', 32) AS snippet, + bm25(turns_fts) AS rank +FROM turns_fts AS f +WHERE turns_fts MATCH ? + AND (? IS NULL OR f.tool = ?) + AND (? IS NULL OR f.host = ?) + AND (? IS NULL OR f.timestamp >= ?) + AND (? IS NULL OR f.timestamp < ?) +ORDER BY rank LIMIT ?; +``` + +Pagination: cursor encodes `(rank, turn_id)` of the last row; next page applies `(rank, turn_id) > cursor` in lexicographic order. Offset-based pagination over FTS5 is correct but gets pathological at depth; a tuple cursor is barely more code. For the personal scale, offset+limit would also be fine — defaulting to cursor anyway because changing pagination later is an API break. + +**Search query (`include_tool_outputs=1`).** Two `MATCH` queries (one per FTS table), `UNION ALL`, then a window dedupe on `(tool, host, session_id, turn_id)` keeping the better-ranked match (a turn might appear in both indexes). The frontend sets `include_tool_outputs=1` only when the user toggles the "include tool output" checkbox; the default URL stays clean and fast. + +**Stats query.** Standard SQL aggregation over `turns`, parameterized by `group_by`: +- `group_by=tool` → `GROUP BY tool` returning per-tool totals. +- `group_by=host` → `GROUP BY host`. +- `group_by=day` → `GROUP BY date(timestamp, 'unixepoch')` returning a daily series. + +The `idx_turns_timestamp` index from #1 covers the date-range filter; for the small data scale this stays sub-second without further tuning. + +**Search HTML.** A new `search.html` template sharing `base.html` with the timeline and session views. The result list shows: tool icon, host, working_dir, timestamp, snippet (with ``), session-link to `/session/{tool}/{host}/{session_id}#turn-`. Snippets pass through `bluemonday` UGC policy before insertion. + +**JS (vanilla, no framework).** One small script (`internal/ui/static/lethe.js`): +- Debounced search-as-you-type: 300 ms after last keystroke, fetch `/search?q=...&format=fragment`, replace the results region's `innerHTML`. The `format=fragment` query param tells the server to return only the results partial template, no `` chrome. +- Anchor-jump highlight on session view (`#turn-` → scroll, briefly add a `.highlighted` CSS class). +- Tool-call expand/collapse uses native `

` from #1; no JS needed. + +Total JS budget: under 50 lines. CSP-safe; no `eval`. + +**opencode parser — discovery first.** + +The format is unknown until the spike runs. Step 0 of this task is `cmd/lethe-spike-opencode/main.go` (deleted before the task closes), which: +1. Walks `~/.local/share/opencode/` and `~/.config/opencode/` and `~/.cache/opencode/`. +2. Reports file types, sizes, sample contents. +3. Identifies whether sessions are stored as JSONL, JSON-per-session, SQLite, or something else. + +Spike output goes to `docs/spikes/opencode-format.md`. **Until that spike runs, the parser implementation below is provisional.** Two likely shapes: + +- *If JSONL or JSON-per-session:* implementation mirrors the Claude Code parser. `Parse()` reads from a byte offset, returns `wire.TurnEvent`s, persists offset. +- *If SQLite-backed:* `Parse()` opens the source DB read-only with `_busy_timeout` set, queries rows newer than a stored marker (rowid or `(session, sequence)`), maps to `wire.TurnEvent`. The `ingestion_state.last_offset` column gets reused for the marker (it's an INTEGER — works for either rowid or byte offset). + +In both cases the Parser interface from #2 is unchanged. The opencode-specific code is contained in `internal/collector/parser/opencode/`. + +If the spike reveals an opaque or encrypted format, opencode is dropped from this task and an issue is filed against opencode upstream. The task still ships search. + +**Tradeoffs that settled it.** +- *Single-table FTS query vs always-union:* unioning costs nothing on small data, but it muddies BM25 scoring (tool outputs have wildly different term distributions than prose). Default to clean prose search; let the user opt in. +- *Cursor vs offset pagination:* offset+limit is fine at this scale, but changing pagination later is an API break. Cursor cost is one struct and a base64 helper. +- *Server-side fragment rendering vs client-side JSON+template:* fragment rendering keeps JS at 50 lines and templates as the single source of truth. Client-side rendering would force a parallel JS template path. +- *Discovery spike vs guess-and-iterate on opencode:* the spike is 30 minutes; iterating against a wrong assumption costs an evening. The spike's output is also documentation worth keeping. + +**Unknowns that remain.** +- opencode's actual on-disk format. Resolved by the spike before parser code is written. +- BM25 tuning. SQLite's default BM25 (`k1=1.2`, `b=0.75`) is fine for prose. If empirical search quality is bad against real data, tune via `bm25(turns_fts, 5.0, 1.0, ...)` per-column weights — that's a #3-and-a-half follow-up, not a blocker. +- Stats-page chart. v1 is a table. Adding a sparkline is a one-CSS-grid afternoon if I want it; not in scope here. + +### Backwards-compatibility check + +- Server: only **adds** routes. `/api/v1/sessions/...` and the timeline/session HTML are unchanged. The `internal/shared/wire/` types are untouched. No schema changes. +- Collector: only adds a new parser package and one config entry under `sources:`. The Parser interface is unchanged. Existing claude-code config keeps working. +- Database: zero migrations. The FTS tables and triggers were created in #1 and are merely queried here for the first time. + +### Hands-off decisions + +- udesign: cursor pagination for `/api/v1/search` rather than offset — costs one helper, prevents an API break later. Stats keeps `since`/`until` only (no pagination — the result set is intrinsically small). +- udesign: `include_tool_outputs` defaults to false — keeps default search latency low and result lists uncluttered. UI surfaces a checkbox; URL param drives behavior. +- udesign: server-side fragment rendering for live-search — single template path beats a parallel JS template engine. `?format=fragment` returns the partial. +- udesign: BM25 default weights — no per-column tuning until empirical evidence forces it. +- udesign: opencode discovery spike is mandatory before parser code — protects against an evening of guessing. +- udesign: spike binary at `cmd/lethe-spike-opencode/` deleted before task closes; spike findings preserved in `docs/spikes/opencode-format.md`. Spike code is throwaway; the writeup is the artifact. +- udesign: stats UI is a table for v1 — sparkline / chart is YAGNI until I actually use the page. +- udesign: cost rollups null-pass — sum where present, ignore otherwise. No fake "estimated cost" math. Honest data. +- udesign: anchor-jump uses `#turn-` rather than scroll-into-view-via-JS — anchors are stable across renders, work with browser back/forward, and survive the page being reloaded from a bookmark. + +TDD: yes (reason: FTS query result shape is exactly the kind of thing that silently regresses on schema or trigger changes; stats aggregation has off-by-one risks around timezone/date boundaries; opencode parser determinism mirrors the claude-code parser's TDD justification.) + +### Invariants + +- All schema migrations remain in #1's `migrations/`. This task adds **no** new migration files. +- `internal/shared/wire/` types are not modified by this task. If a need arises, it's flagged and split into a separate amendment task. +- `GET /api/v1/search` and `GET /api/v1/stats` are read-only: they execute `SELECT` only, with no transaction lower than `BEGIN DEFERRED`. +- BM25 ranking is the default for `/api/v1/search`; time-ordered results are explicitly opt-in via a future `?order=time` parameter (not implemented in this task; reserved). +- Search and stats endpoints honor the same `Tailscale-User-Login` allowlist middleware as #1's existing routes. No exemptions. +- Snippets and any user-supplied query reflected back into HTML pass through `bluemonday` before insertion. +- The opencode parser implements `parser.Parser` from #2 verbatim — no special interface, no opencode-specific hooks in the ingestion loop. +- Source-format identification for opencode is documented in `docs/spikes/opencode-format.md` and committed before the parser code lands. +- `tool_outputs_fts` is queried only when `include_tool_outputs=1`; default search path never touches it. +- All client-side JavaScript stays under 50 lines, lives in one file, and works without any build step. + +### Principles + +- Search defaults are the prose-only path. Tool-output search is a power-user toggle, not the headline. +- One template tree, server-rendered. Live updates go through `?format=fragment` partials, not a parallel JS render path. +- A new parser is one directory under `internal/collector/parser//` plus a single `register.go` import — opencode's job is to prove this and leave the path obvious for crush/pi/kimi. +- Spike before commit when the format is unknown. The spike's writeup is the artifact; the spike's code is throwaway. +- Stats are honest. Null-pass on missing fields. No estimates, no synthesis. +- Vanilla JS, no framework, no build step, no transpiler, no bundler. The day this stops working is the day the project has changed shape enough to merit its own task. diff --git a/docs/tasks/lethe-server.md b/docs/tasks/lethe-server.md new file mode 100644 index 0000000000000000000000000000000000000000..c9180ca1deb1c5951d608200937b278a91671ca5 --- /dev/null +++ b/docs/tasks/lethe-server.md @@ -0,0 +1,397 @@ +# lethe-server + +**Status:** Plan +**Module:** `sourcecraft.dev/bigbes/lethe` +**Branch:** master +**Worktree:** none +**Parent RFC:** Personal AI Assistant Log Aggregator (2026-04-25) +**Sibling tasks (deferred):** `lethe-collector-claude-code.md` (#2), `lethe-search-and-opencode.md` (#3) + +## Design + +### Purpose + +Stand up the `lethe` server binary: SQLite-backed ingestion endpoint, session list/detail JSON API, forward-auth header trust against an Authelia-protected reverse proxy, ready to deploy on phoebe behind Caddy/Traefik+Authelia. Search, stats, any HTML/UI, and any collector code are explicitly deferred to siblings or a later UI task. + +A successful end state for this task: I can `curl -X POST` a fixture NDJSON file at `/api/v1/ingest`, then `curl` the sessions list and a single session detail through the reverse proxy (with either an Authelia session forwarding `Remote-User` or an Authelia-issued OIDC bearer) and get the expected JSON back. + +### Scope + +**In:** +- Single Go binary `lethe` (`cmd/lethe/main.go`) — JSON API server only. +- Full SQLite schema, including the FTS5 tables and triggers from the start (so #3 only adds query code, no migrations). +- Embedded migrations via `embed.FS` + `golang-migrate/v4`, applied on startup. +- `POST /api/v1/ingest` — NDJSON, turn-only protocol (server upserts session rows from turn data). +- `GET /api/v1/sessions` — paginated list with filters (`tool`, `host`, `since`, `until`). +- `GET /api/v1/sessions/{tool}/{host}/{session_id}` — full session with turns inline. +- Two auth paths, both gated by the same `auth.allowed_users` allowlist: + - **Forward-auth**: trust `Remote-User` from an upstream reverse proxy (Caddy/Traefik) gated by Authelia. + - **OIDC bearer**: validate `Authorization: Bearer ` against Authelia's OIDC issuer (JWKS lookup, signature, `iss`/`aud`/`exp`); take user from `preferred_username` (fallback `sub`). +- Each mode independently enable-able; if both enabled, bearer is checked first then header. +- **Per-user data isolation**: every session and turn is owned by the user that ingested it. List and detail endpoints filter by current user. Members of the configurable `auth.admins` list can see all owners' data and override via `?owner=` (or `?owner=*`) on read endpoints. +- `/healthz`, `/readyz`, `/metrics` (Prometheus). +- Graceful shutdown, structured logging via `scribe`, structured errors via `culpa` rendered as RFC 7807 `application/problem+json`. +- `Justfile`, `.air.toml`, `Dockerfile`, `docker-compose.yml`, `config.example.yaml`, `.golangci.yml` per `go-selfhosted-backend` skill conventions. +- Wire types live in `internal/shared/wire/` so #2's collector imports them directly. +- Daily `.backup` is **documented** in the README (cron + `sqlite3 .backup`); no code in this task. + +**Out:** +- Any HTML / web UI — timeline view, session view, templates, CSS, static assets, markdown rendering (`goldmark`/`bluemonday`). Deferred to a later UI task; the JSON API is the only consumer surface in this task. +- Collector binary, parsers, ingestion loop, outbox, backfill — all #2. +- `/api/v1/search` + search UI + `tool_outputs_fts` query path — #3 (table + triggers exist; queries don't). +- `/api/v1/stats` + rollups view — #3. +- `/debug/pprof` — defer until something forces the issue. +- Sub-agent / `parentUuid` session chaining (Claude Code resume semantics) — parser concern, lands with #2. +- Backup automation in code. + +### Chosen approach + +**Storage: option B (locked).** SQLite (`modernc.org/sqlite`, no CGO) with WAL, single DB file. Schema includes `turns_fts` (FTS5 over prose `content`) and `tool_outputs_fts` (FTS5 over `tool_calls` text). Both indexes populated by INSERT/UPDATE/DELETE triggers from the start — #3 only wires queries. + +**Wire protocol: option B (locked).** Collector emits turn-only NDJSON. Server upserts the session row on first-seen turn from `session_meta` carried on the turn; subsequent turns extend the session's `ended_at`. No separate session events on the wire. Reduces collector state and makes outbox replay trivially idempotent. + +**Repo layout: option A (locked).** Monorepo. `cmd/lethe/` (this task) and `cmd/lethe-collector/` (placeholder dir for #2) under one `go.mod`. Shared types in `internal/shared/wire/`. + +**Auth model.** Server binds `127.0.0.1` only. A reverse proxy on phoebe (Caddy or Traefik) terminates TLS and forwards to localhost. Two independent auth paths inside lethe, each enable-able via config: + +1. **Forward-auth (header trust).** Reverse proxy runs Authelia forward-auth on the lethe vhost, and on success injects `Remote-User` (and `Remote-Email`, `Remote-Groups`) headers. Middleware reads `Remote-User` (header name configurable for non-Authelia setups), checks allowlist, 403 on miss. Used by browsers and any other tool that already has an Authelia session cookie. + +2. **OIDC bearer.** Lethe is registered as an OIDC client of Authelia (`client_id`/`client_secret` configured at the Authelia side, `client_id` only at lethe — the server validates tokens, never issues them). Middleware accepts `Authorization: Bearer `, validates against Authelia's published JWKS (discovered via `/.well-known/openid-configuration`), enforces `iss`, `aud`, `exp`, then resolves user from `preferred_username` falling back to `sub`. Used by the collector and any scripted client. No code-flow / callback / cookie machinery in lethe — the bearer must already be obtained out-of-band (Authelia issues it via its OIDC flow to whatever client got it). + +Both paths drop the user identity into the request context so handlers see the same shape regardless of how the user was authenticated. Both paths are gated by the same `auth.allowed_users` allowlist as a defense-in-depth check. If both are enabled and a request carries both an `Authorization` header and a `Remote-User` header, the bearer is validated first; the proxy header is ignored unless bearer validation fails open (configurable). Health and metrics endpoints are mounted *outside* the auth middleware so phoebe can scrape them locally without going through the proxy. The implicit assumption — that nothing else on phoebe binds 127.0.0.1 and forges the header — is the whole trust model for path 1; documented in the README. + +**Layered design** (from `go-selfhosted-backend`): +- `internal/config` — Viper, strict mode, validator tags, fails on unknown keys. Top-level `Config` struct exposes substructs (`Server`, `Database`, `Auth`, …) via `config-section:""` tags so steward can inject them by type into individual services. +- `internal/platform/database` — sqlx connection, migration runner over `embed.FS`, transaction helper. Implements steward `Init` (open + migrate) and `Destroy` (close). +- `internal/platform/observability` — scribe logger service (`Init` sets `slog.SetDefault`), Prometheus registry singleton. +- `internal/platform/health` — `Checker` interface, `Set` aggregator service with steward multi-inject (`[]Checker \`inject:""\``), DB check service registered as a `Checker`. Adding a new check = registering a new asset; no edits to the set. +- `internal/server` — chi router, middleware stack (request-id, logging, metrics, recovery, auth), `Start` (listen) and `Stop` (graceful shutdown). Marked `steward.Root()` so it's always started. +- `internal/domain/ingest` — handler + service for `POST /api/v1/ingest`, both as steward services. Service owns the upsert-session-then-upsert-turn logic in a single transaction per batch. +- `internal/domain/session` — handler + repository for the list/detail JSON API, both as steward services. +- `internal/shared/wire` — `TurnEvent`, `SessionMeta` types. Imported by both server and (eventually) collector. +- `internal/pkg/httputil` + `internal/pkg/apierror` — JSON helpers, RFC 7807 problem rendering with culpa code → status mapping. Pure libraries, not steward components. + +**Wiring & lifecycle.** All wiring goes through `steward.Manager`. `cmd/lethe/main.go` is a thin shell: parse `-config`, load `Config`, register configuration + service assets, conditionally register `OIDCVerifier` only when `cfg.Auth.OIDC.Enabled`, call `Inject → Init → Start`, wait on signal, call `Stop → Destroy`. Dependency direction is enforced by struct tags (`config:""`, `inject:""`, `inject:"" optional:"true"`); the manager solves the topological order. No global state outside `slog.Default` (set by the logger service in `Init`). + +**Wire type (locked contract for #2):** +```go +package wire + +type TurnEvent struct { + Tool string `json:"tool"` + Host string `json:"host"` + SessionID string `json:"session_id"` + TurnID string `json:"turn_id"` + Seq int64 `json:"seq"` + Role string `json:"role"` // user | assistant | tool | system + Timestamp int64 `json:"timestamp"` // unix epoch seconds + Content string `json:"content"` + Model *string `json:"model,omitempty"` + TokensIn *int64 `json:"tokens_in,omitempty"` + TokensOut *int64 `json:"tokens_out,omitempty"` + CostUSD *float64 `json:"cost_usd,omitempty"` + ToolCalls json.RawMessage `json:"tool_calls,omitempty"` + SessionMeta SessionMeta `json:"session_meta"` + Metadata json.RawMessage `json:"metadata,omitempty"` +} + +type SessionMeta struct { + WorkingDir *string `json:"working_dir,omitempty"` + SourceFile string `json:"source_file"` + StartedAt *int64 `json:"started_at,omitempty"` // optional; server falls back to MIN(turn.timestamp) + Metadata json.RawMessage `json:"metadata,omitempty"` +} +``` + +`tool_calls` is `json.RawMessage`: the server doesn't interpret it, just persists it and feeds its serialized text to `tool_outputs_fts`. Carrying `session_meta` on every turn is redundant (~100 bytes/turn) and intentional: collector replay never has to ask "did I already send the session header?" + +**Ingest semantics.** +- Request body is NDJSON; one `TurnEvent` per line. Hard cap on body size (configurable, default 16 MiB). Per-turn `content` soft-capped at 4 MiB (configurable); a single oversize turn is a `LineError`, not a 413. +- **Required wire fields**: `tool`, `host`, `session_id`, `turn_id`, `seq`, `role`, `timestamp`, `content`. `role ∈ {user, assistant, tool, system}`. `source_file` (in `SessionMeta`) capped at 1024 bytes. Validation runs per-line before DB; failure → `LineError`. +- `owner` is **server-derived** from the authenticated user (request context), never read from the wire. The wire format in `internal/shared/wire/` deliberately has no `owner` field — collectors cannot impersonate other owners. +- All timestamps stored as SQLite `INTEGER` (unix epoch seconds). No TEXT timestamps anywhere in the schema. +- Server processes lines **in order** within a single SQLite transaction. +- For each turn: + 1. `INSERT INTO sessions ... ON CONFLICT (owner, tool, host, session_id) DO UPDATE SET ended_at = MAX(ended_at, excluded.ended_at)`. `started_at`, `working_dir`, `source_file`, `metadata` are **first-write-wins** (preserved on conflict); only `ended_at` extends. + 2. `INSERT INTO turns ... ON CONFLICT (owner, tool, host, session_id, turn_id) DO UPDATE SET `. Last-write-wins on the turn row. Triggers keep both FTS tables in sync. +- On any per-line error (malformed JSON, missing required field, FK violation), the transaction is rolled back **for the failing line and beyond**, and the response is `200 {"accepted": N, "errors": [{"line": N, "error": "..."}]}` where `accepted` is the count of lines successfully committed in a previous chunk. Practically, the server processes and commits in chunks (e.g. every 500 lines) so a single bad line near the end of a batch doesn't lose 499 good ones. Collector advances its offset by exactly `accepted`. +- Hard server errors (DB down, OOM): 5xx with empty `accepted`, collector retries the whole batch from the same offset. + +**Schema (initial migration).** As specified in the RFC §4.2, with these adjustments: +- Add `owner TEXT NOT NULL` to `sessions` and `turns`. Composite PKs become `sessions(owner, tool, host, session_id)` and `turns(owner, tool, host, session_id, turn_id)` with FK `(owner, tool, host, session_id)` → sessions. Owner-leading PK gives a free index for "list my sessions" and makes per-user isolation a schema property. +- Add `owner UNINDEXED` column to both FTS tables so #3's search can `WHERE turns_fts MATCH ? AND owner = ?` cheaply; triggers carry it from the source row. +- Add `turns_fts_update` trigger so an UPSERT on `turns` keeps the FTS row current (the RFC has insert/delete only). +- Add the parallel `tool_outputs_fts` table with insert/update/delete triggers, indexing the `tool_calls` column when non-NULL. +- Add `schema_migrations` (managed by golang-migrate) — replaces the RFC's hand-rolled `schema_version`. +- Index `sessions(owner, started_at DESC)` for the timeline/list query. + +**CLI.** This binary has one mode (run server). `flag` package, no cobra. Single arg: `-config `. (Cobra will land with the collector binary in #2.) + +**Tradeoffs that settled it.** +- *SQLite vs Postgres:* Phoebe doesn't need another service, FTS5's `snippet`/`highlight` and tokenizer are better than tsvector for this workload, single-user means no write contention. Migrating to PG later is mechanical if I'm wrong; the cost of being wrong is an afternoon. +- *Turn-only wire vs explicit session events:* Outbox replay correctness wins over "cleaner" two-event schema. The redundant `session_meta` per turn costs nothing at this scale. +- *Inline tool-call payloads vs blob/dedupe:* RFC's expected scale doesn't justify the extra code. Revisit in #3 if FTS index size or query latency actually bites. +- *Defer all UI:* Skipped HTML views in this task to land the JSON API + ingest pipeline first. The collector (#2) only needs the API; the UI lands once there's real data to look at. + +**Unknowns that remain.** +- True size distribution of tool-call payloads in real Claude Code transcripts — won't know until #2 runs against `~/.claude/projects/`. If the FTS index for `tool_outputs_fts` grows pathologically, #3 has the option to add a size cap or move that table to a separate attached DB. +- JWKS rotation cadence at Authelia and the right cache TTL on the lethe side (`go-oidc` defaults usually fine; revisit if validation latency or 401-storms appear). Not blocking #1. + +### Backwards-compatibility check + +Greenfield. Empty repo, no consumers, nothing to break. The only forward-compat concern is the wire format, which is locked into `internal/shared/wire/` and versioned implicitly via the `/api/v1/` path prefix. Future breaking changes get `/api/v2/`. + +TDD: yes (reason: ingest idempotency, the upsert-session-from-turn semantics, the chunked-commit-with-partial-accept response shape, the auth middleware allowlist, and migration application on startup are all deterministic, regression-prone surfaces.) + +### Invariants + +- Server never opens, reads, or writes any file outside its own data directory and config path. `source_file` from incoming turns is stored as opaque string only. +- The host identifier on a turn is whatever the collector says it is. Server does not derive or validate it against the authenticated user (whether resolved via forward-auth header or OIDC bearer). +- All schema changes go through `embed.FS` migrations applied on startup. No ad-hoc DDL, no startup-time conditional `CREATE TABLE`. +- Composite primary keys: `sessions` keyed on `(owner, tool, host, session_id)`; `turns` keyed on `(owner, tool, host, session_id, turn_id)`. No surrogate IDs anywhere. +- `POST /api/v1/ingest` is idempotent at the turn level **per owner**: re-POST of identical `(tool, host, session_id, turn_id)` by the same authenticated user produces the same final state regardless of how many times it's sent. Two different users posting the same `(tool, host, session_id, turn_id)` produce two distinct rows. +- `owner` is set from the authenticated user on every ingest write. The wire format has no `owner` field; the server never reads owner from the request body. +- Read endpoints (`GET /api/v1/sessions`, `GET /api/v1/sessions/{tool}/{host}/{session_id}`) return only rows where `owner = `, except when the current user is in `auth.admins` and supplies `?owner=` (specific owner) or `?owner=*` (all owners). Non-admin requests with `?owner=` are 403. +- Every route under `/api/v1/*` validates the configured user header (default `Remote-User`) against `auth.allowed_users`. Only `/healthz`, `/readyz`, `/metrics` are unauthenticated. +- The HTTP listener binds `127.0.0.1` only. Binding any other interface is a config error and fails fast at startup. +- SQLite is opened in WAL mode with `_busy_timeout` configured. Foreign keys are enforced (`PRAGMA foreign_keys = ON`). +- A turn insert/update fires both FTS triggers as appropriate; the `turns_fts` and `tool_outputs_fts` tables are never written to directly outside triggers. +- Errors leaving any HTTP handler are rendered as RFC 7807 `application/problem+json` with the culpa code mapped to status; internal (5xx) errors are logged with full stacktrace via `scribe.Err` before being sanitized for the response. + +### Principles + +- Greenfield — no backwards-compat shims, no deprecation paths, no `_unused` parameters. If something turns out wrong, rewrite the file. +- Schema additions over schema changes. Tool-specific fields go into the `metadata` JSON column on the relevant row; new SQL columns require justification. +- Fail fast on config and migration errors. The auth allowlist has no default — empty list means the server refuses to start. +- Stdlib + `chi` + `sqlx` + `golang-migrate` + `modernc.org/sqlite` + `go-oidc/v3` (JWT/JWKS validation only — no auth-code flow) + `go.bigb.es/auxilia` (`steward` for DI/lifecycle, `culpa` for errors, `scribe` for logs, `async` only if a background task surfaces). No ORM, no template engine, no UI dependencies in this task. +- Lifecycle and dependency wiring go through `steward.Manager`. Adding a new component is registering an asset; `main.go` does not grow. +- Each layer is a steward service that declares its deps via struct tags (`config:""`, `inject:""`, optional + multi-injection where it earns its keep). Constructors are the zero value; setup happens in `Init`. +- Unit tests construct services with hand-built deps (no manager). Integration / e2e tests use `steward.Manager` to assemble a real graph against a `:memory:` DB. +- Errors propagate as `culpa` errors with codes; HTTP layer translates once at the boundary. +- Every authenticated route and every ingest semantic has a regression test. +- `internal/shared/wire/` is treated as a published API even though it isn't published — changes ripple into the collector and need to be obvious in diff. + +## Plan + +Approach: build bottom-up — wire types → config → DB+schema → platform → HTTP foundation → auth → ingest → read API → main. Each phase is one commit; tests land with the phase that introduces the behavior. Greenfield, so no compat shims. + +### Phase 1 — Bootstrap & wire contract + +- **1.1** `go.mod` (create) — `module sourcecraft.dev/bigbes/lethe`, Go 1.22+. Direct deps stub: `chi/v5`, `sqlx`, `modernc.org/sqlite`, `golang-migrate/v4`, `viper`, `validator/v10`, `prometheus/client_golang`, `coreos/go-oidc/v3`, `go.bigb.es/auxilia/{steward,culpa,scribe}`. +- **1.2** `Justfile`, `.air.toml`, `Dockerfile`, `docker-compose.yml`, `.golangci.yml`, `.gitignore`, `config.example.yaml` (create) — per `go-selfhosted-backend` skill conventions; SQLite volume mount, no CGO. +- **1.3** `README.md` (create) — purpose, quickstart, **trust model section** documenting both auth paths: (a) the `127.0.0.1` + reverse-proxy + Authelia forward-auth + `Remote-User` chain (with a sample Caddy `forward_auth` snippet), and (b) the OIDC bearer flow against Authelia (sample Authelia `identity_providers.oidc.clients` entry + sample lethe `auth.oidc` config). **Backup section** with the `sqlite3 .backup` cron snippet. +- **1.4** `cmd/lethe/main.go` (create, ~30 lines) — `flag.String("config", ...)`, prints version and exits. Real wiring in Phase 9. +- **1.5** `internal/shared/wire/wire.go` (create, ~40 lines) — `TurnEvent`, `SessionMeta` exactly as specified in Design. No methods; pure data. Locked contract for #2. +- **Invariant:** `internal/shared/wire/` published-API-discipline (Principles). +- Commit: `feat: bootstrap lethe server skeleton + wire contract` + +### Phase 2 — Config + +- **2.1** `internal/config/config.go` (create, ~190 lines) — `Config` struct with `Server`, `Database`, `Auth`, `Logging`, `Ingest` substructs. Each substruct has `mapstructure`, `validate`, **and `config-section:""`** tags so steward can `inject` them by type into individual services. + - `Database` substruct: `Path string` (sqlite file path), `BusyTimeout time.Duration` (default 5s). + - `Auth` substruct: `AllowedUsers []string`, `Admins []string` (subset of allowed users; may be empty), `ForwardAuth ForwardAuthConfig{ Enabled bool; UserHeader string (default "Remote-User") }`, `OIDC OIDCConfig{ Enabled bool; Issuer string (URL); Audience string; UsernameClaim string (default "preferred_username") }`. + - `Ingest` substruct: `MaxBodyBytes int64` (default 16 MiB), `MaxTurnContentBytes int64` (default 4 MiB), `ChunkSize int` (default 500). + - `Server` substruct: `Bind string`, `ShutdownGrace time.Duration` (default 10s). + - `func Load(path string) (*Config, error)` — viper strict mode, validator, env-var overrides via `viper.SetEnvPrefix("LETHE")` + `viper.AutomaticEnv()` + `viper.SetEnvKeyReplacer(NewReplacer(".", "_"))`, returns `culpa` error on failure. + - `func MustLoad(path string) *Config` — wraps `Load`, panics on error. Used by `main.go`. + - Validation: `Server.Bind` must equal `127.0.0.1` or `127.0.0.1:` (regex); `Database.Path` `required`; at least one of `Auth.ForwardAuth.Enabled` or `Auth.OIDC.Enabled` must be true (custom validator: `auth_at_least_one`); `Auth.AllowedUsers` `min=1`; every entry in `Auth.Admins` must also appear in `Auth.AllowedUsers` (custom validator: `admins_subset_of_allowed`); if OIDC enabled, `Auth.OIDC.Issuer` `url` and `Auth.OIDC.Audience` `required`; `Ingest.MaxBodyBytes` `gt=0`, `Ingest.MaxTurnContentBytes` `gt=0,ltefield=MaxBodyBytes`, `Ingest.ChunkSize` `gt=0`. +- **2.2** `internal/config/config_test.go` (create, TDD) — tests for: empty allowlist rejected; non-loopback bind rejected; both auth modes disabled rejected; OIDC enabled without issuer rejected; OIDC enabled with non-URL issuer rejected; admin not in allowed_users rejected; empty admins list accepted; missing `Database.Path` rejected; `MaxTurnContentBytes > MaxBodyBytes` rejected; unknown YAML key rejected (strict mode); env override works (`LETHE_AUTH_ALLOWED_USERS` overrides YAML); valid forward-auth-only config loads; valid OIDC-only config loads; valid both-enabled config loads; defaults applied (`UserHeader="Remote-User"`, `UsernameClaim="preferred_username"`, `MaxBodyBytes=16MiB`, `MaxTurnContentBytes=4MiB`, `ChunkSize=500`, `BusyTimeout=5s`, `ShutdownGrace=10s`). +- **Invariants:** auth allowlist has no default (Principles); listener binds 127.0.0.1 only. +- Commit: `feat(config): viper-loaded config with fail-fast validation` + +### Phase 3 — Database & schema + +- **3.1** `internal/platform/database/database.go` (create, ~110 lines) — `Database` is a steward service. + - `type Database struct { Cfg config.DatabaseConfig \`config:""\`; DB *sqlx.DB }` — `DB` populated in `Init`. + - `func (d *Database) Init(ctx context.Context) error` — opens via `modernc.org/sqlite` with `_journal_mode=WAL`, `_busy_timeout=5000`, `_foreign_keys=on`, `_synchronous=NORMAL`, `cache=shared`; then runs `Migrate(d.DB)`. + - `func (d *Database) Destroy(ctx context.Context) error` — closes the DB. + - `func Migrate(db *sqlx.DB) error` — runs `embed.FS` migrations via `golang-migrate/v4` `iofs` source + `sqlite` driver. Pure function so tests can call it directly. + - `func InTx(ctx, db, fn func(*sqlx.Tx) error) error` — transaction helper, rollback on error. Pure function. + - Other services depend on this via `inject:""` and read `.DB`. +- **3.2** `internal/platform/database/migrations/0001_init.up.sql` + `.down.sql` (create) — sessions, turns, turns_fts (FTS5 over `content` + `owner UNINDEXED`), tool_outputs_fts (FTS5 over `tool_calls` + `owner UNINDEXED`), insert/update/delete triggers for both FTS tables (triggers carry `owner` from source row). Composite PKs: `sessions(owner, tool, host, session_id)`; `turns(owner, tool, host, session_id, turn_id)` with FK `(owner, tool, host, session_id)` → sessions. All timestamps (`started_at`, `ended_at`, `turns.timestamp`) are `INTEGER NOT NULL` (unix epoch seconds). `tool_calls`, `metadata` are `TEXT` storing JSON. Index on `sessions(owner, started_at DESC)` for timeline. +- **3.3** `internal/platform/database/migrations.go` (create, ~10 lines) — `//go:embed migrations/*.sql` `var FS embed.FS`. +- **3.4** `internal/platform/database/database_test.go` (create, TDD) — tests with `:memory:` DB: migrate is idempotent on second run; turn insert populates `turns_fts` with correct `owner`; turn update updates `turns_fts`; turn delete removes from `turns_fts`; same for `tool_outputs_fts` when `tool_calls` non-NULL; FK rejects orphan turn; two owners with same `(tool, host, session_id)` coexist as distinct sessions; FTS query with `owner = ?` filter returns only that owner's rows. +- **Invariants:** WAL + busy_timeout + FKs on; FTS tables only via triggers; embed.FS migrations only. +- Commit: `feat(db): SQLite schema with FTS5 + migration runner` + +### Phase 4 — Observability & health + +- **4.1** `internal/platform/observability/logger.go` (create, ~110 lines) — `Logger` steward service. + - `type Logger struct { Cfg config.LoggingConfig \`config:""\`; L *slog.Logger }` + - `func (l *Logger) Init(ctx) error` — builds `scribe.NewTintHandler` (or JSON handler per cfg), applies `WithLevel`, `WithMaskKeys("password","token","authorization","secret","cookie")`, wraps with a small `contextHandler` that pulls `request_id` and `user` from `r.Context()` and adds them to every record. Sets `slog.SetDefault(l.L)`. + - `func WithRequestID(ctx, id string) context.Context`, `func RequestIDFrom(ctx) string` — context helpers used by the request-id middleware in Phase 5. +- **4.2** `internal/platform/observability/metrics.go` (create, ~80 lines) — `Metrics` steward service. + - `type Metrics struct { Registry *prometheus.Registry; HTTPRequests *prometheus.CounterVec; HTTPDuration *prometheus.HistogramVec; IngestLinesAccepted, IngestLinesErrored, IngestChunksCommitted prometheus.Counter }` + - `func (m *Metrics) Init(ctx) error` — `prometheus.NewRegistry()`; register `collectors.NewProcessCollector` + `collectors.NewGoCollector`; register HTTP histograms with labels `{method, route, status}` (route from `chi.RouteContext(r.Context()).RoutePattern()` — never raw path, to keep cardinality bounded); register ingest counters. + - HTTP middleware in Phase 5 reads from `Metrics`; ingest service in Phase 7 increments the ingest counters. +- **4.3** `internal/platform/health/health.go` (create, ~90 lines) + - `type Checker interface { Name() string; Check(ctx context.Context) error }` + - `type DBCheck struct { DB *database.Database \`inject:""\` }` — implements `Checker`; registered as a steward service tagged for multi-injection. `Check` runs `SELECT 1`. + - `type Set struct { Checks []Checker \`inject:""\` }` — steward multi-injects every registered `Checker`. + - `func (s *Set) Run(ctx) (results map[string]error, allOK bool)` — applies a per-check 2s timeout via `context.WithTimeout`. Empty `Checks` slice → returns `allOK = true` (intentional: no checks means nothing has declared a readiness signal yet, not an error). + - Adding new checks later = registering a new asset that implements `Checker`. No edits to `Set`. +- **4.4** `internal/platform/health/health_test.go` (create, TDD) — `Set` returns aggregate failure when any check errors; passes when all OK; empty `Checks` returns `allOK=true`; per-check timeout enforced. Uses fake `Checker` implementations (no steward needed for unit test). +- **4.5** `internal/platform/steward_unwind_test.go` (create, TDD, throwaway after Phase 4) — confirms steward calls `Destroy` on already-init'd siblings when a later component's `Init` errors; if it doesn't, `Database.Destroy` won't run on partial-init failures and we need to add an explicit guard in `main`. Verifies the assumption underpinning the lifecycle design. +- Commit: `feat(platform): scribe logger, prometheus registry, health checker set` + +### Phase 5 — HTTP foundation + +- **5.1** `internal/pkg/apierror/apierror.go` (create, ~80 lines) + - `type Problem struct { Type, Title, Status, Detail, Code, Instance, Errors }` — RFC 7807 shape. + - `func Render(w, r, err error)` — extracts `culpa.Code` from `err`, maps to HTTP status (NotFound→404, Invalid→400, Unauthorized→401, Forbidden→403, Conflict→409, Internal→500), writes `application/problem+json`. 5xx logs full stacktrace via `scribe.Err` before sanitizing. +- **5.2** `internal/pkg/httputil/httputil.go` (create, ~50 lines) — `ReadJSON`, `WriteJSON`, `ReadNDJSONLines(r io.Reader, maxBytes int64) iter.Seq2[[]byte, error]`. +- **5.3** `internal/server/server.go` (create, ~150 lines) — `Server` is the steward root service. + - `type Server struct { Cfg config.ServerConfig \`config:""\`; Log *observability.Logger \`inject:""\`; Metrics *observability.Metrics \`inject:""\`; Health *health.Set \`inject:""\`; Auth *auth.Authenticator \`inject:""\`; Ingest *ingest.Handler \`inject:""\`; Sessions *session.Handler \`inject:""\`; httpSrv *http.Server }` + - `func (s *Server) Init(ctx) error` — builds chi router, mounts middleware stack: + - **request-id**: generate ULID, set on context via `observability.WithRequestID`, echo as `X-Request-ID` response header. + - **logging**: structured access log per request, picks up request-id automatically via the `contextHandler` from Phase 4.1. Body never logged. + - **metrics**: increments `Metrics.HTTPRequests` and observes `Metrics.HTTPDuration` using `chi.RouteContext(r.Context()).RoutePattern()` as the `route` label. + - **recovery**: panics → 500 problem. + - Unauthenticated routes: `GET /healthz` (process up), `GET /readyz` (calls `s.Health.Run` with 5s timeout, 503 on any failure), `GET /metrics` (`promhttp.HandlerFor(s.Metrics.Registry, promhttp.HandlerOpts{})`). + - Authed `/api/v1/*` group with `s.Auth.Middleware` then `s.Ingest.Mount(r)` and `s.Sessions.Mount(r)` (paths inside `Mount` are relative to the `/api/v1` group). + - Validates `Cfg.Bind` resolves to a loopback IP — error otherwise. + - `func (s *Server) Start(ctx) error` — spawns `http.Server.ListenAndServe` in a goroutine; returns nil immediately. Errors propagate via stop channel. + - `func (s *Server) Stop(ctx) error` — `httpSrv.Shutdown(ctx)` with `Cfg.ShutdownGrace` (default 10s) drain budget. In-flight ingest chunks finish their commit; partially-processed batches return their `Accepted` count truthfully. + - `steward.Root()` so it's always started even if no other component injects it. +- **5.4** `internal/pkg/apierror/apierror_test.go` (create, TDD) — each culpa code maps to expected status; problem JSON has all required fields; internal-error response detail is sanitized (no stack trace in body). +- **5.5** `internal/server/server_test.go` (create, TDD) — non-loopback bind returns error from `Server.Init`; recovery middleware turns panic into 500 problem; request-id propagates to log lines. Tests construct `Server` directly with hand-built deps (skip steward; unit test of router behavior). +- **Invariants:** errors rendered as RFC 7807; 5xx logged with stack; bind 127.0.0.1 enforced. +- Commit: `feat(http): chi server with middleware stack + RFC 7807 problem renderer` + +### Phase 6 — Auth middleware (forward-auth + OIDC bearer) + +- **6.1** `internal/server/auth/oidc.go` (create, ~140 lines) — `OIDCVerifier` is a steward service, registered conditionally in `main` only when `cfg.Auth.OIDC.Enabled`. + - `type OIDCVerifier struct { Cfg config.OIDCConfig \`config:""\`; verifier *oidc.IDTokenVerifier; usernameClaim string }` + - `func (v *OIDCVerifier) Init(ctx) error` — builds `oidc.NewProvider(ctx, Cfg.Issuer)` (which fetches `/.well-known/openid-configuration` + JWKS) and `provider.Verifier(&oidc.Config{ClientID: Cfg.Audience})`. Accepts `go-oidc` default clock skew (no explicit option). Hard-fails at startup if Authelia unreachable; that's the chosen tradeoff (see Risks). + - `func (v *OIDCVerifier) Verify(ctx, raw string) (user string, err error)` — validates JWT, extracts username via `usernameClaim`, falls back to `sub`. Returns `culpa.Unauthorized`-coded error on any validation failure. +- **6.2** `internal/server/auth/middleware.go` (create, ~150 lines) — `Authenticator` is a steward service. + - `type Authenticator struct { Cfg config.AuthConfig \`config:""\`; Log *observability.Logger \`inject:""\`; Verifier *OIDCVerifier \`inject:"" optional:"true"\`; allowed, admins map[string]struct{} }` + - `func (a *Authenticator) Init(ctx) error` — builds `allowed` and `admins` lowercase sets from `Cfg.AllowedUsers` / `Cfg.Admins`; if `Cfg.OIDC.Enabled && Verifier == nil` → hard error (config invariant breach). + - `func (a *Authenticator) Middleware(next http.Handler) http.Handler` — resolution order: (1) if OIDC enabled and `Authorization: Bearer ` present, call `Verifier.Verify`; (2) if forward-auth enabled and `` non-empty, take it; (3) else 401 problem. After resolving user: lowercase, check `allowed`, 403 problem on miss, otherwise put `Identity{User, IsAdmin}` into request context via `WithIdentity` and call `next`. + - `type Identity struct { User string; IsAdmin bool }` + - `func WithIdentity(ctx, Identity) context.Context`, `func IdentityFrom(ctx) (Identity, bool)`, `func MustIdentity(ctx) Identity` — context helpers used by handlers. + - Caller (`Server.Init`) mounts middleware on `/api/v1/*` only; `/healthz`, `/readyz`, `/metrics` unmounted (Phase 5/9). +- **6.3** `internal/server/auth/middleware_test.go` (create, TDD) — table-driven against an in-memory router: + - **Forward-auth path** (OIDC disabled): missing header → 401; header set, user not in allowlist → 403; header set, allowed → 200; case-insensitive allowlist match → 200; configurable header name (`X-Forwarded-User`) honored → 200. + - **OIDC path** (forward-auth disabled): missing `Authorization` → 401; malformed bearer → 401; valid JWT signed by test JWKS, allowed user → 200; valid JWT, user not in allowlist → 403; expired JWT → 401; wrong-audience JWT → 401; `preferred_username` claim used; falls back to `sub` when `preferred_username` absent. + - **Both enabled**: bearer present and valid → user resolved from JWT (header ignored); bearer invalid + header present → 401 (do not silently fall back to header — fail closed); both absent → 401. + - **Admin flag**: user in `Auth.Admins` → `IdentityFrom(ctx).IsAdmin == true`; user not in admins → `false`; admin not in `AllowedUsers` rejected at config load (covered in Phase 2 tests). + - Test helper sets up a local `httptest.Server` serving JWKS + OIDC discovery + signs JWTs with a generated RSA key; pointed at by the verifier under test. + - Problem JSON shape verified for 401 and 403. +- **Invariant:** every `/api/v1/*` route validates auth; only `/healthz`, `/readyz`, `/metrics` exempt (enforced by mount point in Phase 9). Same `auth.allowed_users` allowlist applied regardless of which auth path resolved the user. +- Commit: `feat(auth): forward-auth + OIDC bearer middleware with shared allowlist` + +### Phase 7 — Ingest domain + +- **7.1** `internal/domain/ingest/repository.go` (create, ~140 lines) — `Repository` is a steward service. + - `type Repository struct { Database *database.Database \`inject:""\` }` + - `func (r *Repository) UpsertChunk(ctx, tx *sqlx.Tx, owner string, turns []wire.TurnEvent) error` — single tx: per turn, `INSERT … ON CONFLICT (owner,tool,host,session_id) DO UPDATE SET ended_at = MAX(ended_at, excluded.ended_at)` for sessions (first-write-wins on metadata); `INSERT … ON CONFLICT (owner,tool,host,session_id,turn_id) DO UPDATE SET ` for turns. `owner` is bound from the parameter on every row — never sourced from the wire payload. `started_at` falls back to `MIN(turn.timestamp)` when `SessionMeta.StartedAt` is nil. +- **7.2** `internal/domain/ingest/service.go` (create, ~160 lines) — `Service` is a steward service. + - `type Service struct { Cfg config.IngestConfig \`config:""\`; Repo *Repository \`inject:""\`; Log *observability.Logger \`inject:""\`; Metrics *observability.Metrics \`inject:""\` }` + - `type Result struct { Accepted int; Errors []LineError }` + - `func validateTurn(t wire.TurnEvent, maxContentBytes int64) error` — required fields (`tool`, `host`, `session_id`, `turn_id`, `seq`, `role`, `timestamp`, `content` non-empty); `role ∈ {user, assistant, tool, system}`; `len(content) ≤ maxContentBytes`; `len(SessionMeta.SourceFile) ≤ 1024`. Returns `culpa.Invalid`-coded error on failure. + - `func (s *Service) Ingest(ctx, owner string, body io.Reader, maxBytes int64) (Result, error)` — reads NDJSON via `httputil.ReadNDJSONLines`; per line: JSON-unmarshal then `validateTurn`; buffers up to `Cfg.ChunkSize` lines; on full chunk, opens tx and calls `Repo.UpsertChunk(ctx, tx, owner, chunk)`; on commit success, increments `Metrics.IngestChunksCommitted` and `IngestLinesAccepted` by chunk len, adds chunk len to `Accepted`; on parse/validation/DB error mid-chunk, rolls back chunk, increments `IngestLinesErrored`, logs at WARN with line number, owner, `tool/host/session_id`, returns with `Accepted` reflecting prior chunks plus a `LineError` for the failing line; subsequent lines are not processed. +- **7.3** `internal/domain/ingest/handler.go` (create, ~70 lines) — `Handler` is a steward service. + - `type Handler struct { Cfg config.IngestConfig \`config:""\`; Service *Service \`inject:""\` }` + - `func (h *Handler) Post(w, r)` — extract `auth.MustIdentity(r.Context()).User` as owner, `Content-Type` check, body limit reader at `Cfg.MaxBodyBytes`, calls `Service.Ingest(ctx, owner, ...)`, returns `200 {"accepted": N, "errors": [...]}`. Hard server errors (DB down) → 5xx problem with empty `accepted`. + - `func (h *Handler) Mount(r chi.Router)` — `r.Post("/ingest", h.Post)`. Called by `Server.Init`. +- **7.4** `internal/domain/ingest/service_test.go` + `repository_test.go` (create, TDD) — tests against `:memory:` DB: + - **Wire validation**: each required field missing → `LineError`; bad `role` value → `LineError`; `content` over `MaxTurnContentBytes` → `LineError` (not 413, body still under cap); `source_file` over 1024 → `LineError`. All other lines in the same chunk that came before the bad line still commit. + - Idempotency: posting same NDJSON twice as the same owner produces identical row counts and identical row contents. + - First-write-wins on session: posting turn-1 with WorkingDir=A then turn-2 with WorkingDir=B leaves session.WorkingDir=A. + - Last-write-wins on turn: re-posting same turn_id with new content updates the row. + - `ended_at` extends: `MAX(existing, incoming)`. + - `started_at` fallback to MIN turn timestamp when SessionMeta.StartedAt is nil. + - Chunked partial accept: 3-chunk batch with bad line in chunk 3 returns `Accepted = 2*chunkSize`, error references correct line number. + - FK violation produces a LineError, not a 5xx. + - DB-down → 5xx, no Accepted. + - Body over `MaxBodyBytes` → 413 problem. + - FTS rows present after ingest with correct `owner` (`turns_fts` contains turn content; `tool_outputs_fts` contains tool_calls JSON when present). + - **Per-user isolation**: same NDJSON ingested by user A then by user B produces two distinct sessions and two distinct sets of turns; rows have correct `owner`. Modifying user B's session's `ended_at` does not affect user A's row. + - **Wire `owner` is ignored**: a TurnEvent JSON line that includes a stray `"owner": "evil"` field (not in the wire struct) does not change the stored owner; ingest still attributes to the authenticated user. (Negative test against the trust model.) +- **Invariants:** ingest is idempotent at turn level per owner; owner is server-derived; chunked-commit-with-partial-accept semantics; first-write-wins session, last-write-wins turn. +- Commit: `feat(ingest): NDJSON ingest with chunked transactions and partial-accept` + +### Phase 8 — Sessions read API + +- **8.1** `internal/domain/session/repository.go` (create, ~150 lines) — `Repository` is a steward service. + - `type Repository struct { Database *database.Database \`inject:""\` }` + - `type OwnerScope struct { User string; AllOwners bool; SpecificOwner *string }` — resolved by handler from identity + `?owner=` param. + - `type ListFilter struct { Owner OwnerScope; Tool, Host *string; Since, Until *int64; Limit, Offset int }` + - `func (r *Repository) List(ctx, f ListFilter) ([]Session, error)` — dynamic WHERE built from non-nil filters; owner clause: if `AllOwners` no clause, if `SpecificOwner` `WHERE owner = ?`, else `WHERE owner = User`; `ORDER BY started_at DESC`; `LIMIT/OFFSET`. + - `func (r *Repository) Get(ctx, scope OwnerScope, tool, host, sessionID string) (*SessionWithTurns, error)` — owner clause same as above; joined select; returns `culpa.NotFound` when session missing **or owner mismatch** (do not leak existence by returning 403 vs 404 — a non-admin asking for someone else's session must see the same 404 as if the session didn't exist). +- **8.2** `internal/domain/session/handler.go` (create, ~120 lines) — `Handler` is a steward service. + - `type Handler struct { Repo *Repository \`inject:""\` }` + - Pagination defaults: `limit=50` default, `limit` capped at 200, negative `limit`/`offset` clamped to default/0. `since > until` → 400 problem. + - `func (h *Handler) resolveScope(r) (OwnerScope, error)` — read `auth.MustIdentity(r.Context())`, parse `?owner=`; if param empty → scope is current user; if param non-empty and identity is admin → scope is `SpecificOwner` (or `AllOwners` for `?owner=*`); if param non-empty and identity is **not** admin → 403 problem. + - `func (h *Handler) List(w, r)` — `resolveScope`, parse other query params (`tool`, `host`, `since`, `until`, `limit`, `offset`), calls `Repo.List`, writes JSON. + - `func (h *Handler) Get(w, r)` — `resolveScope`, chi URL params, calls `Repo.Get`, writes JSON; 404 problem when not found (auto via apierror). + - `func (h *Handler) Mount(r chi.Router)` — `r.Get("/sessions", h.List)`; `r.Get("/sessions/{tool}/{host}/{session_id}", h.Get)`. Called by `Server.Init`. +- **8.3** `internal/domain/session/repository_test.go` + `handler_test.go` (create, TDD) — tests against `:memory:` DB: + - Filters compose correctly (tool only, host only, time range, all combined); pagination caps `limit` (e.g. max 200) and clamps negatives; ordering by `started_at` desc; turns inline in correct order by `seq`. + - **Per-user isolation (List)**: with rows owned by A and B, a request authenticated as A returns only A's rows. B's rows do not appear regardless of filter combination. + - **Per-user isolation (Get)**: A asking for B's session by URL → 404 problem (not 403 — must not leak existence). A asking for own session → 200. + - **Admin override**: admin with `?owner=B` lists/gets B's rows. Admin with `?owner=*` lists across all owners. Admin without `?owner=` defaults to admin's own rows (no implicit cross-tenant view). + - **Non-admin `?owner=` rejected**: non-admin user A passing `?owner=A` (their own user!) or `?owner=B` → 403 problem. (Param is admin-only; non-admins must not pass it at all.) +- **Invariants:** errors rendered as RFC 7807; composite PKs throughout; per-user isolation enforced; existence never leaked across owners. +- Commit: `feat(session): list and detail JSON API with filters` + +### Phase 9 — Main wiring + ops surface + +- **9.1** `cmd/lethe/main.go` (modify, ~70 lines) — thin shell. No business logic; everything is a steward asset. + ```go + cfg := config.MustLoad(*configPath) + mgr := steward.NewManager() + mgr.AddComponent(ctx, + steward.MustConfigurationAsset(cfg), + steward.MustServiceAsset(&observability.Logger{}), + steward.MustServiceAsset(&observability.Metrics{}), + steward.MustServiceAsset(&database.Database{}), + steward.MustServiceAsset(&health.DBCheck{}), // registers as Checker + steward.MustServiceAsset(&health.Set{}), + steward.MustServiceAsset(&auth.Authenticator{}), + steward.MustServiceAsset(&ingest.Repository{}), + steward.MustServiceAsset(&ingest.Service{}), + steward.MustServiceAsset(&ingest.Handler{}), + steward.MustServiceAsset(&session.Repository{}), + steward.MustServiceAsset(&session.Handler{}), + steward.MustServiceAsset(&server.Server{}, steward.Root()), + ) + if cfg.Auth.OIDC.Enabled { + mgr.AddComponent(ctx, steward.MustServiceAsset(&auth.OIDCVerifier{})) + } + must(mgr.Inject(ctx)); must(mgr.Init(ctx)); must(mgr.Start(ctx)) + // wait on SIGINT/SIGTERM + must(mgr.Stop(stopCtx)); must(mgr.Destroy(ctx)) + ``` + - All routes mounted by `Server.Init` (Phase 5.3): `/healthz`, `/readyz`, `/metrics` outside the auth group; `/api/v1/*` inside `Authenticator.Middleware`. Main does **not** touch chi. + - Signal handling: `signal.NotifyContext(ctx, SIGINT, SIGTERM)` → on cancel, `mgr.Stop(ctx with 15s deadline)` → `mgr.Destroy(ctx)` → exit. +- **9.2** `cmd/lethe/main_test.go` (create, TDD, light) — end-to-end smoke via `steward.Manager` assembled in-test with a `:memory:` DB and a random-port `Server`: POST a fixture NDJSON as user A, GET sessions list as A (sees own row), GET session detail as A. Then POST a fresh batch as user B with the same `(tool, host, session_id)` and confirm both rows coexist; A still only sees A's. Confirms wiring + isolation reaches all the way through the steward graph. +- **9.3** `README.md` (modify) — fill in real config example with all keys (including both `auth.forward_auth` and `auth.oidc` blocks), add curl commands for ingest + list + detail (one variant with `Remote-User` for testing forward-auth, one variant with `Authorization: Bearer …` for OIDC), finalize the dual-auth trust-model section and backup section. +- **Invariant:** only `/healthz`, `/readyz`, `/metrics` are unauthenticated; mounting layout enforces this. +- Commit: `feat(cmd): wire server with /healthz /readyz /metrics + authed /api/v1` + +### Order & dependencies + +Linear: each phase depends on all prior phases. Phase 4 (observability/health) and Phase 5 (HTTP foundation) could parallelize but commit-coupling makes it not worth it. No phase can land before its dependencies (config → db → platform → http → auth → handlers → main). + +### Open questions / risks / rollback + +- **Risk:** `golang-migrate/v4` + `modernc.org/sqlite` driver compatibility — multiple-statement migrations (FTS triggers) may need `;` handling. **Mitigation:** Phase 3 test asserts migration applies; if it fails, swap to per-statement execution or `goose`. +- **Risk:** FTS5 trigger SQL on UPSERT — SQLite doesn't fire UPDATE triggers from `INSERT … ON CONFLICT … DO UPDATE` in older versions. **Mitigation:** Phase 3 test specifically covers UPSERT path; if triggers don't fire, replace UPSERT with explicit `SELECT then INSERT/UPDATE`. +- **Risk:** OIDC discovery on startup blocks if Authelia is down. **Mitigation chosen:** fail fast — `NewOIDCVerifier` errors propagate from `main`, lethe refuses to start. Acceptable because Authelia is on the same host; if Authelia is down lethe is unusable anyway. Forward-auth-only deployments are unaffected. +- **Open:** Authelia OIDC client registration is a manual step on the Authelia side — not in this repo. Documented in Phase 9.3 README only. +- **Rollback:** greenfield, single `git revert` per phase commit. + +### Backwards-compat check + +Greenfield — no compat surface. Wire format is the only forward-compat concern; pinned in `internal/shared/wire/` and `/api/v1/` URL prefix per Design. No further checks needed. + +### Acknowledged out-of-scope (won't surface in this task, listed so they aren't re-discovered later) + +- Rate limiting on `/api/v1/ingest` — body cap + per-turn content cap are the only safeguards in v1. +- OpenAPI spec / generated client — collector (#2) hand-writes against `internal/shared/wire/`. +- CORS — JSON API behind reverse proxy on the same origin; not needed. +- Cursor-based pagination / total-count on `/api/v1/sessions` — offset+limit is enough for the expected volume. +- Per-route per-user rate limits (would need `Authenticator` to expose a per-identity bucket). +- Pluggable auth backends beyond Authelia (the OIDC verifier is generic enough that pointing at a different IdP works, but only Authelia is documented).

File	Status	+
`docs/tasks/lethe-collector-claude-code.md`	A	+176
`docs/tasks/lethe-search-and-opencode.md`	A	+146
`docs/tasks/lethe-server.md`	A	+397

~bigbes/lethe: 8a744799

3 changed file(s)

~bigbes/lethe: `8a744799`