diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 7f4b98fb4b908b987a353387a95efe5292b926af..61fc06628bd36462c53daeb80738c1f7b1c7d408 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -263,7 +263,11 @@ link resolution and search, and is far cheaper to reject at push time than to find weeks later. The escape hatch exists so that a hook bug or a bad schema can never lock you out of your own repository. - **`post-receive`** (after the fact, cannot reject) — notify the daemon so it - reindexes the changed documents and updates the space's index rev stamp. + rebuilds that **space** at its new revision and updates the index rev stamp. + Note the unit: a space at a revision, not a set of changed documents. An + earlier draft said "reindexes the changed documents", which contradicts the + decision to absorb the batch rebuilder — and nothing tracks per-document change, + because `index_stamp` records exactly a space and a rev. #### The hook is service code, not a shell script @@ -513,6 +517,59 @@ **Refs outside the two namespaces are rejected for both principals.** The design covers the approved branch and `proposals/*` and was silent on tags and other branches. Silence would mean "allowed", and a ref nothing ever reads is a place for content to rot unnoticed. Loosening this later is one branch in a switch. + +## Bilingual search: per-line, not per-document + +Specs here are written in Russian and English, often within one document (cf. the +`ru-spec-style` skill). The design's suggested fix — detect each document's +language and route it to a `ru`- or `en`-analyzed field — was implemented, +measured, and **found insufficient**. The resolution is per-*line* routing into +`title_ru`/`title_en` and `body_ru`/`body_en`, queried across all four. Each +document still carries one dominant-language label, used for reporting and +snippet selection, never for placing text. + +The measurements that settled it, each a committed test: + +1. **A single analyzer does not *break* the other language — it silently stops + stemming it.** The `ru` analyzer leaves `"indexes rebuild all the documents"` + entirely unstemmed (and keeps English stop words); the `en` analyzer does the + same to Russian. Foreign-script terms still match *literally*, which is why + this looks fine in casual testing. The failure is subtler and worse: singular + stops finding plural. +2. **Per-document routing loses the minority language outright.** A dominantly + Russian spec quoting English requirements verbatim, indexed whole into + `body_ru`: searching `attachments` finds it, searching `attachment` does not. + That is exactly the document shape this corpus has. +3. **Per-line routing finds both halves**, with neighbouring single-language + documents unaffected. + +**Dual-writing the whole text into both fields was rejected**, although it also +fixes stemming: it doubles the index and makes every mixed document match in two +fields at once, summing both scores in the disjunction — a systematic ranking +bias toward mixed documents that has nothing to do with relevance. + +**The threshold is 35% Cyrillic, not 50%**, requiring at least 12 letters to +decide at all, falling back to the document's dominant language otherwise. The +error directions are not symmetric: English prose has essentially no Cyrillic, +whereas Russian technical prose here routinely runs a third Latin characters — +identifiers, product names, untranslated terms. + +Residual limitation, stated plainly: a single *line* mixing both languages goes +wholly to one field. Fixing that needs token-level routing and is not worth it. + +**Filters must be exact-match, and warren's were not.** warren filtered by +section using a match query on an *analyzed* field. At this scale that is a +correctness bug rather than a wart: a project **is** a space filter, so an +analyzed `~bigbes/home-ops` tokenizes and a filter for one space can return +another's documents. `space`, `section` and `lang` are keyword-analyzed and +matched as terms. + +**Measured rebuild cost** (1000 synthetic specs across 5 spaces, ~3 KB each): +full rebuild **284 ms**, single space of 200 documents **58 ms**, queries +**0.4 ms**. At roughly 30× the confirmed daily volume, the "absorb the batch +rebuild, do not build incremental indexing" decision holds by a wide margin — +and rebuild duration is instrumented, so the decision to revisit it will be +measured rather than guessed. ## Consistency and recovery @@ -1210,14 +1267,9 @@ is what the config block above assumes. - **MCP transport.** Streamable HTTP on the same chi router (`/mcp`) keeps it to one listener and one nginx block; a second port is only needed if MCP ends up wanting different timeouts than the web UI. -- **Mixed Russian/English search — partially solved, not finished.** warren's - `search/keyword.go` already wires both the `lang/en` and `lang/ru` analyzers - with per-index selection, so the starting point is better than an earlier draft - claimed. What is still missing is **per-document** language routing: specs here - are written in both (cf. the `ru-spec-style` skill), and one analyzer per index - mangles whichever language it was not chosen for. Options: detect language at - index time and write to `ru`/`en` fields, querying both; or accept degraded - stemming on the minority language. +- ~~**Mixed Russian/English search.**~~ **Resolved — see "Bilingual search" + below.** Per-*document* routing was measured and found insufficient; the answer + is per-*line* routing. - **Attachments and binaries — now a concrete gap, not a hypothetical.** The Space section says a space holds "markdown documents, plus attachments", but the git layer enumerates and accepts **`.md` only**, so `![[diagram.png]]` has