diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 51489b02fa12a7f62195d12a05f7e47b22d8158d..0000000000000000000000000000000000000000 --- a/PLAN.md +++ /dev/null @@ -1,724 +0,0 @@ -# tarantool-protobuf — Roadmap & Test Plan - -This document tracks the long-form design + testing strategy for -`protoc-gen-tarantool` and the `pb` Lua runtime. Short-term task tracking -lives in the in-session task list; this file is the slow-changing record. - -## 1. Vision - -A first-class Protocol Buffers + gRPC stack for Tarantool that: - -- Generates idiomatic Lua from `.proto` files with **two modes** (full inline - vs descriptor-driven runtime) so users can pick speed-vs-flexibility. -- Interoperates byte-for-byte with mainline protoc implementations (Go, - Python, C++) — passes Google's protobuf conformance suite for proto3. -- Ships with a small, FFI-aware runtime that respects Tarantool conventions - (LuaJIT cdata for 64-bit ints, no global state, no monkey-patching). -- Is comfortable to use from real Tarantool apps: clean integration with - `net.box`, `fiber`, `box.session`, and the 3.x config framework. - -## 2. Current state (post-M7) - -See `README.md` for the user-visible summary. Internally: - -- **Plugin**: Go, two emission modes (`mode=full` inline + `mode=runtime` - descriptor-delegating wrappers). Sibling `protoc-gen-tarantool-doc` - emits Markdown reference per `.proto`. -- **Runtime**: pure Lua + LuaJIT FFI. `wire.lua` + `codec.lua` - + `lazy.lua` (zero-copy view) + `text.lua` (encode + decode) + - `json.lua` (strict proto3 JSON) + `wkt.lua` (all 9 well-known types) + - `grpc.lua` (transport + loopback) + `parser.lua` / `dynamic.lua` / - `fileset.lua` (three runtime descriptor producers — `.proto` source, - AST, FileDescriptorSet bytes). -- **Tests**: 639 luatest tests across 15 test files (39 groups), parametrized over - both codegen modes where applicable. -- **Conformance**: proto3 binary+JSON suite **1493 ✓ / 0 failures**; - proto3 text-format suite **416 ✓ / 0 failures** (Google's - `conformance_test_runner` v34.1). -- **Bench**: `bench/baseline.json` tracks allocation/op (regression - gate at 5%); `just jit-trace` pins LuaJIT trace stability. - -The remaining unfinished bullets in section 3 are mostly M8 (release -engineering) and a handful of optional perf items in M6. - -## 3. Phased roadmap - -Each milestone ends with a green CI run, an updated README, and a tagged -release on sourcecraft.dev. - -### M1 — Two codegen modes + luatest harness *(done)* - -- [x] Promote scalar encode/decode to typed helpers in `pb.wire`. - `wire.encode_int32(v)`, `wire.decode_string(buf, pos)`, etc., for all - 15 scalar proto types. Both modes consume the same primitives. -- [x] **Full (inline) codegen**: emit per-message `_encode` / `_decode` - functions with no descriptor lookup. Tag bytes precomputed at gen - time as Lua string literals. This is the JIT-friendly hot path. -- [x] **Runtime codegen**: keep current behavior. Useful for introspection, - schema registries, and forward-compat with descriptor-only consumers. -- [x] Plugin parameter `mode=full|runtime` (default: `full`). -- [x] Migrate tests to luatest groups. Every behavior tested against **both** - generated modules to prove parity. -- [x] Generate side-by-side outputs in `examples/expected/{full,runtime}/` - for visual diffing. - -### M2 — Composite types *(done)* - -- [x] **`map`**: emit as repeated synthetic `*Entry` messages with - `key`/`value` fields per spec. Decode merges into a Lua table; encode - iterates with `pairs`. Key types limited to scalars + string per spec. -- [x] **`oneof`**: descriptor includes `oneof_index` per field. Encode - emits at most one branch (last assignment wins). Decode clears prior - oneof siblings on assignment. Hot-path lookup goes through - `desc.oneofs_list` (flat array) to keep the trace JIT-stable. -- [x] **proto3 explicit `optional`**: respect presence — emit field even - when value equals scalar default. Generated descriptor exposes - `has_(t)` / `clear_(t)` helpers. - -### M3 — Well-known types *(done)* - -- [x] `google.protobuf.Timestamp` ↔ Tarantool `datetime` module - (epoch + nsec mapping). Out-of-spec inputs (negative nanos, - year > 9999) keep the raw `{seconds, nanos}` table so JSON - serialization can reject them with `serialize_error` instead of - crashing on decode. -- [x] `google.protobuf.Duration` ↔ `{seconds, nanos}` table (interval - proved a poor fit — it carries months/days that don't map - cleanly). -- [x] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar: - pass plain Lua value → auto-wrap; decode → auto-unwrap. -- [x] FieldMask: `repeated string`. Strict round-trip validation — - snake_case paths must use only `[a-z0-9_]`, no leading/trailing - `_`, no `__`, and `_` must precede a lowercase letter (not a - digit). JSON form rejects any `_` (must be lowerCamelCase). -- [x] `Empty`. -- [x] `Any`: opaque `{type_url, value}` form by default; `pb.register(desc)` - + `pb.any.pack(desc, t)` / `pb.any.unpack(any_t)` for typed - round-trips. JSON canonical mapping emits the flat `{"@type": ...}` - object when the type is registered, falls back to base64 opaque form - otherwise. -- [x] `Struct`, `Value`, `ListValue` ↔ idiomatic Lua tables. - `box.NULL` is the null_value sentinel (re-exported as `pb.NULL`). - `pb.wkt.struct(t)` / `pb.wkt.list(t)` tag tables when the auto-detect - heuristic (`t[1] ~= nil` → list, else struct) needs to be overridden, - and decode preserves the tag for byte-stable round trips. - -**Done when**: WKT-using protos round-trip and integrate visibly with -`datetime` (e.g. `os.date(...)`-comparable timestamps). - -### M4 — gRPC services (transport-agnostic) *(done)* - -- [x] Per-service descriptor: methods with full path `/pkg.Svc/Method`, - input/output type refs, `client_streaming` / `server_streaming` flags. -- [x] **Client stub**: `MyService_client(transport)` returns a table with - one function per method. Transport interface, all served by the - shipped loopback + multiplex implementations: - ```lua - transport:unary(path, req_bytes, ctx) -> resp_bytes - transport:server_stream(path, req_bytes, ctx) -> stream - transport:client_stream(path, ctx) -> stream - transport:bidi(path, ctx) -> stream - ``` - Streaming returns a `stream` object with `send` / `close_send` / - `recv() -> msg, err` / `cancel`. End-of-stream is `(nil, nil)`; - handler errors surface as `(nil, err_string)`. -- [x] **Server side**: `MyService_server(impl)` returns - `{service, methods, streams}` consumable by the loopback / - multiplex transports. Handler signatures by RPC kind: - ```lua - unary: function(req, ctx) -> resp - server_stream: function(req, stream, ctx) - client_stream: function(stream, ctx) -> resp - bidi: function(stream, ctx) - ``` -- [ ] Reference network transports (separate projects, deferred). Full - design in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): - - `pb.grpc.transport.http_server` — Connect-JSON over HTTP/1.1 via - `tarantool/http`. Default external transport; works with browsers - and `curl` without an HTTP/2 proxy. - - `pb.grpc.transport.netbox` — gRPC tunneled over `net.box` calls. - First-class in-cluster path. - - `pb.grpc.transport.http_client_unary` — outbound, unary only, - via `http_client`. - - HTTP/2 termination — explicitly *not* shipped. Recommend Envoy in - front; the transport contract is HTTP/2-shaped so the same - generated code works behind it. - - Conformance anchor: `connectrpc/conformance` (same framed-runner - shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect - from one harness). - -**Done.** The loopback transport runs each streaming handler on its own -fiber and bridges client ↔ handler via `fiber.channel`. All four flavors -(unary + 3 streaming) are exercised by parameterized luatest groups. - -### M5 — Conformance + interop *(proto2 + proto3 both closed; CI wire-up still pending)* - -- [x] Wire up Google's [protobuf conformance test runner][conformance]. - `cmd/conformance-runner.lua` is the testee: reads length-prefixed - `ConformanceRequest` on stdin, runs it through our codec / JSON / - text codecs, writes a length-prefixed `ConformanceResponse` on - stdout. Loops to EOF. Core dispatch lives in - `cmd/conformance/core.lua` and is exercised directly by - `test/conformance_test.lua` so the inner dev loop doesn't need - Docker. -- [x] Run the canonical `conformance_test_runner` binary locally and - track the pass-rate as a numeric metric (regression gate). - `docker/conformance.Dockerfile` builds `conformance_test_runner` - from upstream protobuf v34.1 source (matching the host's - `libprotoc 34.1`) and bundles Tarantool 3 from the official deb; - `just conformance` regenerates Lua then runs the harness against - `cmd/conformance-runner.lua` with the repo mounted as a volume. - Watchlists at `test/conformance/known_failures.txt` (binary + - JSON suite) and `test/conformance/known_failures_text.txt` - (text-format suite); **both are empty for the proto3 suites as - of 2026-05-16.** - - Current baseline: - - Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures** - - Text-format suite: **416 ✓ / 18 skipped / 0 failures** - - Both proto2 and proto3 schemas now run through the harness: - Binary + JSON 2806 ✓ / 0 failures; Text-format 434 ✓ / 0 - failures (was 1493 + 416 ✓ with proto2 fully skipped). The four - MessageSet-flavored nested messages remain stripped from our - vendored copy of `test_messages_proto2.proto` because - protobuf-go's protoreflect rejects `message_set_wire_format`; - see [docs/codegen.md#proto2-support](../docs/codegen.md). - - Strict-validation closures landed across three commits on the - `text-conformance-output` branch: - - `pb.text.decode` — full grammar coverage (recursive-descent - parser, ~580 LOC). - - `codec` -0.0 preservation — float/double `is_default_scalar` - and the inline-codegen elision both gained a sign-bit guard - (`1/v == math.huge`). - - `pb.json` strict-validation pass — duplicate-key rejection - (literal + camel/snake alias detection via a byte-walking - pre-scan), null-in-container rejection, unknown-enum-name - rejection (with `ignore_unknown_fields` opt for the - `JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category), - `google.protobuf.NullValue` round-trip as JSON `null`, - strict FieldMask round-trip validation. - - The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to - short-circuit JSON output if a future encoder bug starts - crashing jsoncpp. -- [ ] CI wire-up. The Docker image build is the long pole (~10–15 min - on a clean cache); a registry push from a scheduled job would - let CI runs reuse a warm cache. -- [x] Cross-impl interop: 18-fixture corpus in `test/interop/fixtures/` - produced by mainline `protoc --encode`; tests assert byte-for-byte - equality. - -[conformance]: https://github.com/protocolbuffers/protobuf/tree/main/conformance - -### M6 — Performance + production polish *(delayed; bench harness shipped, further perf work parked)* - -- [x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for - messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped - as `bench/bench.lua`, run via `just bench`. Throughput is stderr-only - (varies with CPU load); JSON document on stdout. -- [x] Allocation profiling — bytes per encode/decode op, measured via GC - delta with `collectgarbage('stop')` framing. Committed as - `bench/baseline.json`. Regression gate: `just bench-compare` exits - non-zero if alloc/op grows >5% vs baseline. Allocations are - deterministic to ~10 bytes regardless of hardware. -- [x] Trace stability — `just jit-trace` (`bench/jit_trace.lua`) - attaches a `jit.attach('trace')` listener over the hot - encode/decode paths and asserts no aborts in our source files - fall into the fatal set (NYI bytecode, blacklisting, persistent - type instability). Pass: 13/13 scenarios. Two fixes shipped: - decode_varint grew a 1-byte fast path so callers no longer drag - an inner loop into the root trace; `pb.finalize_message` now - precomputes `desc.oneofs_list` so runtime-mode oneof encoding - uses `ipairs` instead of `pairs` (the latter compiles to bytecode - ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports - interpreter-bridge counts as a benchmark-quality metric (decoders - have 0–4 per run depending on JIT timing — caused by side traces - returning from inlined `decode_varint` calls, which LuaJIT 2.1 - can't stitch back cleanly; small per-call overhead, structural - to the engine). Scope caveat: map fields still encode via - `pairs()` and remain off-trace — pinned by the gate's last - scenario so we notice if upstream lifts the restriction. -- [ ] Optional output: ibuf-based encoder that writes into a caller-owned - `ffi.cdata` byte buffer instead of building a string list. Targets - hot RPC paths where allocation cost dominates. -- [x] Decoder fast path that returns an `msgpack.object`-like lazy view - for nested messages; only materializes touched fields. - Shipped as `runtime/pb/lazy.lua` + `M._decode_lazy` codegen - stubs in both modes. Surface: `:get / :has / :which / :iter / - :names` on MessageView; `:len / :at / :iter / :tolist` on - ArrayView; `:get / :has / :keys / :iter / :totable` on MapView. - Mutation via `:set` is supported and propagates sub-view edits - transparently (sub-MessageViews tracked on a flat array for - JIT-stable `is_dirty` — see lazy.lua's `_sub_msg_views`). - Re-encode is passthrough: untouched views return their original - bytes verbatim; partially-dirty views walk fields in id order, - splicing clean segments and re-emitting dirty ones. WKT - descriptors (those with `desc.decode`) are eager-wrapped so the - API stays uniform. - - Conformance: every interop fixture round-trips byte-equal through - `decode_lazy(b):encode()`. Trace stability: gated by `make - jit-trace` — index pass, sparse `:get` x2, and passthrough - `:encode` all compile with no fatal aborts. - - Workload characteristics (from `tarantool bench/lazy_bench.lua`, - Person at 1KB / 10KB / 100KB, after the SoA index refactor): - - **Passthrough re-encode** is the headline win: **1.6–1.9×** - faster than eager decode→encode across all sizes and both modes. - Untouched views never re-walk the wire. - - **Sparse read** (`:get` two top-level fields) is **0.90–1.16× - of eager**: break-even at small sizes, slight win at 100KB - (especially in runtime mode where eager pays more dispatch - cost). Earlier 0.60–0.77× regression came from one Lua table - per wire entry; replacing with parallel int arrays closed the - allocation gap. - - **Mutate-then-reencode** is **1.09–1.26× of eager**; the - per-field splice path now consistently beats full re-encode. - The honest framing: lazy is a *byte-passthrough* optimization - that also handles sparse reads at parity. Best fit: proxy / - router shapes that decode, touch a few fields, and re-encode. - -### M7 — Developer ergonomics *(done)* - -- [x] Generated EmmyLua / lua-language-server type annotations so - `t:Person_encode({name=...})` autocompletes in editors. - Codegen emits `---@class ` per message (with one - `---@field` per field), `---@alias integer` per - enum, and `---@param` / `---@return` on every `_new`, `_encode`, - `_decode`, `_decode_lazy`, `_has_*`, `_clear_*` wrapper. Class - identifiers use proto full names verbatim so cross-file - references resolve. Lazy view types (`pb.MessageView`, - `pb.ArrayView`, `pb.MapView`) are declared inline in - `runtime/pb/lazy.lua` so the LSP sees them. Pure comment - addition — no runtime impact, 300/300 luatest + 19/19 - jit-trace gate stay green. -- [x] `pb.from_pb(file_descriptor_set)` — accepts binary `FileDescriptorSet` - bytes (output of `protoc --descriptor_set_out=...`) and returns - `{files = {[name] = module}, order, lookup}`. Each per-file module - has the same surface as `pb.parse` output (statically-generated - runtime mode). Translation pipeline: hand-built `descriptor.proto` - descriptors decode the wire bytes via `pb.codec`, then a translator - converts each `FileDescriptorProto` to the AST shape `pb.parser` - emits, which `pb.dynamic.build` consumes. Map fields are - reconstructed from synthetic entry messages (skipped from - `nested_messages`); `proto3_optional` is rehydrated as - `optional=true` instead of being modelled as a synthetic oneof. -- [x] JSON encoding per the [proto3 JSON spec][proto3json] - (`pb.json.encode` / `pb.json.decode`). -- [x] Text-format printer. `pb.text.encode(desc, t, opts)` returns the - `protoc --decode` form (one field per line, 2-space indent, octal - byte escapes); `opts.single_line=true` collapses to a space-separated - one-liner for log lines and inline goldens. Codegen emits - `M._text(t, opts)` in both modes. WKT types know their - idiomatic Lua shapes — `Timestamp`/`Duration` accept datetime cdata - or `{seconds,nanos}`, wrappers print their unwrapped scalar as - `value: ...`, `Struct`/`Value`/`ListValue` walk the tagged-table - form, `FieldMask` prints `paths: ...` per entry, `Any` stays opaque. -- [x] Text-format parser. `pb.text.decode(desc, text, opts)` is the - recursive-descent counterpart: handles every grammar bucket the - proto3 conformance suite exercises — decimal/hex/octal int literals, - float specials (`inf`/`infinity`/`nan` any case, oversize exponents - saturating to ±inf, underflow to ±0), C-style + `\u`/`\U` string - escapes with adjacent-literal concat and surrogate rejection, - aggregate `{}` / `<>` bodies, repeated short-form `[a, b, c]`, - `key: K value: V` map entries, the `[type.googleapis.com/...]` - inline Any form, enum-by-name-or-number, reserved-name drop, and - numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects - duplicate singular fields, and threads through the conformance - runner — `cmd/conformance/core.lua` no longer skips `text_payload`. - Plugin gained a small `reserved_names` emitter so the parser can - match mainline TextFormat::Parser's "silently drop reserved" rule. - Proto3 text-format conformance suite: **8 ✓ / 426 skipped → 416 - ✓ / 18 skipped / 0 failures** (the 18 are the proto2 message-type - bucket; everything in scope passes). -- [x] JSON strict-validation pass. Six classes of relaxation that the - proto3 JSON conformance corpus flagged are now enforced — together - with the -0 codec fix this empties `known_failures.txt`: - 1. Duplicate JSON keys (`{"foo":1,"foo":2}`) rejected via a - byte-walking pre-scan that runs before `json.decode`. - 2. camelCase / snake_case aliases of the same proto field - rejected via a per-message `field_seen` set. - 3. JSON `null` inside repeated arrays and map values rejected. - 4. Unknown enum *names* rejected by default; the - `ignore_unknown_fields=true` opt silently drops them (and - the conformance dispatch forwards this flag when - `req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST`). - 5. `google.protobuf.NullValue` JSON canonical form: literal - `null`, not the string `"NULL_VALUE"`. Null on a - NullValue-typed oneof member marks the oneof active. - 6. Strict FieldMask round-trip (see M3 entry). -- [x] **Preserve descriptor options.** Every populated `*Options` message - surfaces on the generated descriptor as a plain Lua sub-table named - `options` (or `oneof_options` / `value_options` for oneof / enum value). - Standard fields use their proto name; extensions use their - fully-qualified name (bracket-quoted). The walker is generic — - consumers pull `(google.api.http)`, `(versionpb.etcd_version_*)`, - `[deprecated = true]`, and any in-house extension without pb knowing - about them. Option-free protos emit no `options` keys (byte-identical - output to before). Resolver re-links *Options* messages so in-file - extensions surface via the protoreflect walker (protogen builds - `f.Desc` before in-file extensions are registered; we rebuild - manually). See [docs/codegen.md](docs/codegen.md) for the per-descriptor - shape and the README's "Descriptor options" section for examples. -- [x] `protoc-gen-tarantool-doc`: sibling Go plugin under - `cmd/protoc-gen-tarantool-doc/` that emits one Markdown file per - input `.proto`. Sections: header (package + imports), messages - (per-message description + field table with `# | Field | Type | - Label | Description`), enums (value table), services (method table - with `unary` / `client` / `server` / `bidi` streaming label). - Field type cells render scalar names, full type names for - message/enum references, and `map` for maps; synthetic - map-entry messages are skipped. Leading comments are preserved via - SourceCodeInfo (squashed to a single line inside table cells). - Build with `just build-doc`; generate sample docs into - `examples/docs/` with `just gen-docs`. - -[proto3json]: https://protobuf.dev/programming-guides/proto3/#json - -### M8 — Release engineering - -- [ ] Sourcecraft.dev project + CI pipeline (matrix: Tarantool 2.11 / 3.x - EE + CE, Linux + macOS). -- [ ] Tagged releases; rockspec for the Lua runtime; pre-built binaries - for the Go plugin. -- [ ] Migration guide from Tarantool's built-in `require('protobuf')` to - `require('pb')`. -- [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via - protobuf-encoded events on a queue. - -### M9 — Proto2 baseline *(done; 100% conformance on the patched proto2 suite)* - -- [x] Plugin accepts `syntax = "proto2"`. Field descriptors now carry - `required = true`, `default_value = …`, and `optional = true` for - every singular field. Existing `IsPacked()`-based packing logic - already produced the correct proto2 unpacked-by-default. -- [x] Runtime parser captures `required`, `[default = X]`, and threads - them through `pb.dynamic`. The dynamic builder now calls - `codec.compile_writers/compile_readers` so source-parsed schemas - get the same per-field specializations build-time codegen does. -- [x] Codec: `required` writer errors on missing-on-encode in both - inline and runtime modes. -- [x] JSON + text codecs: presence-tracked elision (`required`, - `optional`, oneof) bypasses the proto3 implicit-zero rules. -- [x] **`group` fields** end-to-end. Wire layer carries - `WIRE_SGROUP=3` / `WIRE_EGROUP=4`; the codec's - `decode_group(desc, buf, pos, stop_id)` mirrors decode_message - but stops on EGROUP. Inline + runtime codegen both emit start/end - tag pairs around the body (no LEN prefix). Text format renders - groups under the capitalized submessage name. Runtime parser - desugars `optional|required|repeated group Name = id { … }` into - a nested message plus a `kind='group'` field whose lowercased - name is `name` — matches build-time codegen for the dynamic path. -- [x] **`extend` blocks / `extensions` ranges.** Plugin iterates - `file.Extensions` + per-message `m.Extensions`, emitting - `pb.register_extension(extendee_desc, {…})` calls. The codec - stores set extensions under `data._extensions[full_name]` and - walks them after the regular field loop on encode; decode routes - tags not in `field_by_id` through `decode_extension` when - `extensions_by_id` matches. Text + JSON encoders/decoders speak - the bracketed `[pkg.ext_name]` syntax. Meta-extensions on - `google.protobuf.*` descriptors (file/message/field options) are - skipped at codegen time — the WKT module doesn't surface those - descriptors at runtime. -- [x] **Proto2 closed enums.** Plugin emits `closed = true` on every - proto2 enum descriptor (driven by protoreflect's `IsClosed()`). - Text format's `parse_enum_value` rejects integer literals that - don't map to a declared value for closed enums; proto3 stays - open to preserve forward-compatibility on the wire. -- [x] 50+ luatest cases (`proto2_basic.*` groups) — full, runtime, and - dynamic-from-source byte parity, plus group wire-byte pinning - and JSON/text presence rules. -- [ ] **`MessageSet`** wire format. `protobuf-go`'s protoreflect - refuses to load a FileDescriptor declaring - `option message_set_wire_format = true;`, calling it a removed - proto1 feature. We strip the four MessageSet-flavored nested - messages from our vendored - `test/conformance/proto/test_messages_proto2.proto` so the rest - of the schema compiles. Real support would require forking - protoreflect; deferred until a real Tarantool consumer asks. -- [x] **Conformance numbers.** Full suite, strict mode (protobuf v34.1): - Binary + JSON 2806 ✓ / 0 failures (was 1493 / 1313 skipped); - Text-format 434 ✓ / 0 failures (was 416 / 18 skipped). - -## 4. Per-feature design notes - -### 4.1 Inline (full) codegen — wire bytes - -For each message we emit one `_encode` and one `_decode` function. Tag -bytes are precomputed string literals; field-presence checks are inlined. -The runtime is reduced to wire-format primitives. - -```lua -function M.Address_encode(t) - local out, n = {}, 0 - local v - v = t.street - if v ~= nil and v ~= '' then - n = n + 1; out[n] = '\x0a' -- tag(1, LEN) - n = n + 1; out[n] = wire.encode_string(v) - end - -- ... - return table.concat(out) -end -``` - -Decoder uses an `if-elseif` chain on field id (LuaJIT compiles this -well for small chain lengths; for >16 fields we may want a numeric jump -table or tag-byte switching). - -### 4.2 Map fields - -`map` is wire-format-equivalent to: - -```proto -message FooEntry { K key = 1; V value = 2; } -repeated FooEntry foos = N; -``` - -Codegen synthesizes the entry message internally but exposes the field as a -Lua table (hash, not array). Encode iterates with `pairs`; decode merges -on duplicate keys (last wins, per spec). - -### 4.3 Oneofs - -Descriptor gains `oneofs = {[oneof_name] = {field_names...}}`. Encode walks -fields in declaration order and emits the first non-nil branch. Decode -clears sibling fields on assignment so callers see exactly one set. - -In inline mode, the encode walk becomes an explicit `if`-chain; the decode -clears are emitted alongside each `elseif id == N then` arm. - -### 4.4 64-bit integer ergonomics - -Locked: `int64`/`uint64`/`fixed64`/`sfixed64`/`sint64` are LuaJIT -`int64_t` / `uint64_t` cdata. Reasons: -- Lossless beyond 2^53. -- Same convention as Tarantool `msgpackffi`, `net.box`, `box.tuple`. -- Compares cleanly against `0` (numeric coercion in LuaJIT). - -Two coercion helpers — `pb.to_uint64(v)` and `pb.to_int64(v)` (re-exported -from `pb.wire`) — accept Lua number, cdata, or a numeric string and return -the matching 64-bit cdata. Use them at the boundary when values come from -JSON, text format, or net.box arguments where the type isn't already cdata. - -### 4.5 Unknown fields *(implemented)* - -`t._unknown_fields` is a single Lua string holding the verbatim -concatenation of tag+value bytes for fields the decoder didn't recognize. -Captured during decode in encounter order; re-emitted at the tail of -`_encode`. Mirrors Tarantool's built-in `protobuf` module convention. - -Map entries don't preserve unknowns (per spec — synthetic Entry messages). -WKT types bypass this too, since they have custom `desc.encode/decode`. - -Both codegen modes implement it: runtime mode in `pb.codec`, full (inline) -mode emits per-message capture/re-emit blocks. Tests live in -`test/unknown_test.lua` and run against both modes. - -### 4.6 Protobuf editions *(not implemented; tracking)* - -We advertise `CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL` only, not -`FEATURE_SUPPORTS_EDITIONS`. If protoc is invoked against an -`edition = "2023";` file with our plugin, it errors out — mainline -refuses to call an editions-unaware plugin. - -**What editions changes.** Editions replaces the proto2-vs-proto3 split -with one language whose behavior is controlled by per-file / per-field -`FeatureSet` annotations. The knobs that matter for us: - -- `field_presence = EXPLICIT | IMPLICIT | LEGACY_REQUIRED` — presence - becomes an opt-in/opt-out per field instead of a syntax-wide default. -- `repeated_field_encoding = PACKED | EXPANDED` — per-field opt-out - from packed encoding for scalar `repeated` fields. -- `enum_type = OPEN | CLOSED` — closed enums route unknown values to - unknown fields (proto2 semantics) instead of round-tripping the int. -- `utf8_validation = VERIFY | NONE` — per-field strictness on `string`. -- `json_format = ALLOW | LEGACY_BEST_EFFORT` — JSON behavior overrides. - -**What we'd actually gain.** Honestly modest, ranked by user-visible -value: - -1. **Per-field packed opt-out** — the only knob proto3 doesn't expose - today. Real interop scenarios (talking to legacy proto2 services, - certain gRPC gateways) sometimes need expanded encoding on a - specific field. Right now our users have no escape hatch. -2. **Migration path off proto3.** Proto3 syntax is being phased out in - favor of edition 2023. This is a forcing function, not a feature - win — at some point users will write `edition = "2023";` and we'll - need to read it. -3. **Explicit presence by default** without the `optional` keyword (and - the synthetic-oneof wart it creates today). -4. **Per-field UTF-8 toggle** for users carrying not-quite-UTF-8 bytes - in `string` fields for legacy compat. -5. **Closed enums** for users who want proto2-style unknown-value - routing. -6. **`LEGACY_REQUIRED` field presence** — niche; only way to express - "required" without writing proto2. - -**What it would cost.** Setting the bit is one line; the work is: - -- Plugin (`cmd/protoc-gen-tarantool/internal/gen`) needs to read each - field's resolved `FeatureSet` and drive codegen off it (packed, - presence, UTF-8 strictness, enum closedness) instead of the proto3 - defaults baked in. -- Runtime parser (`runtime/pb/parser.lua`) needs to accept - `edition = "2023";` and the `features = { ... }` option syntax. -- `runtime/pb/dynamic.lua` and `runtime/pb/fileset.lua` need to - propagate resolved features onto the descriptor. The descriptor - contract in `docs/codegen.md` would gain a `features` field per - message/field/enum. -- `CodeGeneratorResponse` needs `minimum_edition` / `maximum_edition` - set, else protoc rejects an editions-supporting plugin. -- The conformance suite has editions-specific buckets we'd start - encountering. - -**Current verdict.** Deferred. No user pull today, proto3 will keep -working for years, and the practical-win list above is mostly small -quality-of-life items rather than blocking gaps. Revisit when (a) -someone needs per-field packed control, (b) the conformance suite -starts gating on editions, or (c) protoc deprecates proto3 syntax -hard enough that real users hit the error. - -### 4.7 gRPC service descriptors - -```lua -M.Greeter_service = { - name = 'hello.Greeter', - methods = { - SayHello = { - full_name = '/hello.Greeter/SayHello', - input = M.HelloRequest_descriptor, - output = M.HelloReply_descriptor, - client_streaming = false, - server_streaming = false, - }, - -- ... - }, -} -``` - -Client: `M.Greeter_client(transport)` returns -`{SayHello = function(req) ... end, ...}`. - -Server: `M.Greeter_server(impl)` returns a table compatible with the -`server:register(svc)` interface. - -## 5. Testing strategy - -Tests live as a flat `test/*.lua` set (15 files, 39 luatest groups, 639 -assertions as of 2026-05-16). Each behavior file is parametrized over -both codegen modes via the pattern: - -```lua -for _, mode in ipairs({'full', 'runtime'}) do - local g = t.group(name .. '.' .. mode) - local hello = require(mode .. '.hello.hello_pb') - g.test_x = function() ... end -end -``` - -Categories — see file names under `test/`: - -- **Wire + codec primitives** — exercised indirectly through `protobuf_test` - (scalars, repeated, nested, optional, oneof) and `interop_test` - (the byte-for-byte fixture corpus under `test/interop/fixtures/`, - 10 `.txtpb`/`.bin` pairs from mainline `protoc --encode`). -- **Composites** — `protobuf_test` for map/oneof/explicit-optional. -- **WKT** — `struct_value_test`, `any_fieldmask_test`. -- **Lazy** — `lazy_test` covers MessageView/ArrayView/MapView + the - byte-splice re-encode path. -- **Codec dialects** — `json_test`, `text_test`, `text_decode_test`, - `unknown_test`. -- **Dynamic descriptors** — `dynamic_test` (.proto source via - `pb.parse`), `fileset_test` (FileDescriptorSet via `pb.from_pb`), - with parity assertions against the generated modules. -- **Codegen surface** — `codegen_doc_comments_test`, - `doc_test` (the `protoc-gen-tarantool-doc` plugin), `conformance_test` - (self-test of the runner against crafted requests). -- **gRPC** — `grpc_streaming_test` against the loopback transport. - -Cross-mode parity is pinned by `parity.full_vs_runtime.*` groups -inside each behavior file. - -### Conformance and interop - -The Google protobuf conformance suite is driven by -`cmd/conformance-runner.lua` (stdin/stdout framing) and run via -`just conformance` (Docker-bundled `conformance_test_runner`). Known -failures live in `test/conformance/known_failures.txt` (binary+JSON) -and `test/conformance/known_failures_text.txt` (text-format) — both -empty for proto3 as of 2026-05-16. - -Interop fixtures (`test/interop/fixtures/*.{txtpb,bin}`) are produced -by mainline `protoc --encode` via `just goldens` and asserted in -`interop_test.lua` for round-trip equality across both codegen modes. -Map fixtures use a single key to lock byte-for-byte equality (Lua -`pairs` ordering won't match protoc's text-proto-order output); -multi-key map behavior is covered via decode-then-compare-Lua-table -rather than byte equality. - -### Performance + JIT-trace stability - -`bench/baseline.json` tracks allocation per op (a deterministic -metric; throughput swings 30%+ across machines). `just bench-compare` -fails on >5% alloc regression. `just jit-trace` (`bench/jit_trace.lua`) -asserts the hot encode/decode paths stay on the JIT trace — a -companion safety net for the "no `pairs()` on hot paths" rule. - -### Not (yet) automated - -- Fuzz harnesses (malformed-input + random-valid) — out of band. -- Go-side `go test`. Codegen correctness is asserted end-to-end via - Lua tests against committed expected outputs in - `examples/expected/{full,runtime}/`. -- CI pipeline. Local-only today; sourcecraft.dev wire-up is M8. - -## 6. Tooling roadmap - -- **Makefile + Justfile** — both shipped. Makefile covers `build`, `gen`, - `test`, `goldens`, `bench`, `bench-baseline`, `bench-compare`, - `jit-trace`, `clean`. Justfile adds `just conformance` (Docker-bundled - conformance runner). -- **golangci-lint** for the Go side; **luacheck** for the Lua side. -- **gofumpt** + **stylua** for formatting. -- **Coverage**: `go test -cover` for plugin; `luacov` for runtime. -- **Doc generation**: `pkgsite` for Go; manually maintained Markdown for - Lua until a tool emerges. - -## 7. Open questions / future decisions - -| Question | Notes | -|---|---| -| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add `pb.encode_deterministic` flag later. | -| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. | -| Should we ship a stub HTTP/2 transport for gRPC? | Answered in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. | -| msgpack-flavored encoder for proto schemas? | Design sketched in [`docs/specs/msgpack_encoding.md`](docs/specs/msgpack_encoding.md). Open: map-keyed-by-int (default) vs name; ARRAY layout for `box.space` feeders; MP_TUPLE ext opt-in. | -| How to handle the existing Tarantool-builtin `require('protobuf')`? | Document migration; do not override the loader. | -| Lua module path mapping when no `lua_package` and no proto package? | Currently uses bare filename; consider erroring out instead. | -| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. | -| Integration with Tarantool 3.x declarative config? | A `pb.types.` config role would be nice; defer until use case appears. | - -## 8. Non-goals (for now) - -- **proto2 `extend` / `extensions` / `group` / `MessageSet`** — the - conformance-only edges. Core proto2 (required, optional, custom - defaults, packed semantics) is in; see M9. -- **Protobuf editions** beyond what proto3 enables — see §4.6 for - the per-feature analysis (what we'd gain, what it would cost, the - conditions that would force a revisit). -- **HTTP/2 termination** for gRPC. Push it to Envoy; see - [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md). - Connect-JSON over HTTP/1.1 is the default external transport. -- **Reflection service** (gRPC server reflection) — implement after M5. -- **gRPC-Web** — covered by the same Connect conformance harness if we - add it, but not on the v1 path. - -## 9. How to update this plan - -Edit this file directly. After any milestone closes: -1. Move the milestone section under a `## Done` heading. -2. Cross-link the closing PR. -3. Update `README.md` status table. -4. Cut a release on sourcecraft.dev. diff --git a/README.md b/README.md index 24f5edfe741c688af9eb1d39a71fba780a22aa24..2a5df5f44adaaae8e42a2574882345f5f0fb10af 100644 --- a/README.md +++ b/README.md @@ -335,8 +335,8 @@ - **[docs/codegen.md](docs/codegen.md)** — plugin internals, descriptor shape contract, how to add a new wire type or scalar, the LuaJIT hot-path rules generated code observes. -Roadmap is in [PLAN.md](PLAN.md); cross-codebase invariants in -[CLAUDE.md](CLAUDE.md). +Open roadmap work is tracked in Beads (`bd ready`, `bd show `); +cross-codebase invariants are in [CLAUDE.md](CLAUDE.md). ## Conformance diff --git a/docs/index.md b/docs/index.md index ae563700127f397acbf71c699d5994b7cd9275e0..09d9f8ee1b16aba2bcfb525a77ada0eace197f32 100644 --- a/docs/index.md +++ b/docs/index.md @@ -80,8 +80,8 @@ Not implemented; design sketch for picking up later. ## Internals -- **[../PLAN.md](../PLAN.md)** — phased roadmap (M1–M8) with - per-milestone status and per-feature design notes. +- **Beads issue tracker** — durable roadmap and task state. Run + `bd ready` for available work and `bd show ` for context. - **[../CLAUDE.md](../CLAUDE.md)** — invariants and conventions enforced across the codebase: no `pairs()` on hot paths, 64-bit ints as cdata, SoA over AoS for large index structures, keep hot diff --git a/test/unknown_test.lua b/test/unknown_test.lua index 198fff8059cd71d3a9930c40b115e34373f74b07..bf334b60022a7a8e541ec353d7acae416c570ae8 100644 --- a/test/unknown_test.lua +++ b/test/unknown_test.lua @@ -1,7 +1,7 @@ -- Unknown-field passthrough: a decoder that meets fields not in its schema -- must capture their raw bytes into result._unknown_fields and a subsequent -- encode must re-emit them verbatim. Mirrors Tarantool's built-in `protobuf` --- module convention (see PLAN.md §4.5). +-- module convention. local t = require('luatest') local function hex(s)