diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 0bf4cfdbc835c5dc513b98d7c290e51f7b537532..ff908731299f20354ba72d562d9065a94acc01d8 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -58,7 +58,7 @@ {"_type":"issue","id":"tarantool-protobuf-3e6","title":"Encoder: fiber-local recycled output buffer (Tarantool-specific)","description":"The 136 B/op encode floor is the 'local out, n = {}, 0' workspace table allocated per top-level encode call. Tarantool ships a per-fiber ibuf via require('buffer'); could grab one from a pool, reset it on each top-level encode, write into it, then ffi.string(ibuf.rpos, ibuf:size()) at the end.\n\nTradeoffs:\n - Saves the workspace alloc (~136 B per top-level call + ~136 B per nested message).\n - Adds fiber-local state — non-Tarantool LuaJIT runs would need a different path or none.\n - Subsumed by lkz (single-pass two-phase) which goes directly to a single buffer without intermediate strings. Worth filing as a smaller alternative path in case lkz proves too invasive to land.\n - drm notes record two prior ibuf prototypes (stashed) that regressed by ~2x because per-byte b:alloc(1) was 20x interpreter-dispatch-bound. This proposal avoids that pitfall only because it composes with h8v (codegen FFI direct writes that bypass alloc(1)).\n\nConcrete path: only useful in combination with h8v. Without h8v this would just shift the alloc from 'out table' to 'output string' without removing any per-string varint allocation in the field bodies — net neutral on alloc and worse on speed.","notes":"Sketch: shared scratch buffer per encoding, capacity exposed as ffi cdata pointer + length. Codegen emits 'local _buf = pb.codec.acquire_buf(); local _off = 0; ...; return pb.codec.finalize_buf(_buf, _off)'. acquire_buf returns a pre-allocated buffer of growing capacity; finalize_buf returns a string and recycles. Care needed for recursive encode calls (nested message encoding into the same buffer).","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:15:08Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:15:08Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-0gg","title":"Decoder: investigate field-id dispatch as binary search or jump table","description":"Generated Type_decode bodies use linear 'if id == 1 elseif id == 2 ...' chains. For Person (~15 fields) the chain is short and the comparisons cheap. For messages with 30+ fields the linear walk to a high-id tag costs N comparisons per occurrence.\n\nOpen question: does LuaJIT's IR already lower this to a switch/jump? If yes, no work needed. If no, three options:\n\n 1. Codegen-emit a balanced if-elseif tree (binary search) when field count exceeds threshold.\n 2. For messages where field IDs are dense and small, emit a numeric branch table: 'local _f = _dispatch[id]; if _f then return _f(buf, pos, result) end'. Closure-per-field has setup cost but avoids the dispatch cost on every tag.\n 3. Sort by frequency (impossible to know at codegen time without profiling input). Skip this option.\n\nPre-work: jit.dump on Person_decode to check whether LuaJIT collapses the if-elseif. If it does, close as won't-fix; if it doesn't, decide between (1) and (2).\n\nLowest priority of the perf items because: (a) we don't know it's a problem yet, (b) Person profile shows decode_string + decode_tag + utf8 dominate, not the if-elseif walk.","notes":"Recent profile (Person 1KB): line 1007 (decode_tag call) is 14% of generated body time, and line 1021 (list append) is 24%. Linear if-elseif walk doesn't show up as a hotspot for Person, but Person has only 15 fields. Needs a bigger fixture to manifest.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:15:07Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:15:07Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-1bo","title":"Codegen: chunk oversized generated decode bodies into per-range helpers","description":"test_messages_proto3_pb.lua is 4212 lines — the conformance fixture has so many fields that a single Type_decode body exceeds LuaJIT's inline budget and trace size limits. Symptom (to verify): bench/jit_trace.lua trace aborts on the big conformance message, despite no NYI bytecodes in the body.\n\nFix: when a message has \u003eN fields (threshold to find empirically; LuaJIT defaults LJ_TRACE_MAX_BC=8000), codegen splits the if-elseif chain into per-range helpers: _decode_fields_1_to_15, _decode_fields_16_to_31, etc. The top-level Type_decode dispatches to the right chunk by id range:\n\n if id \u003c= 15 then\n pos = _decode_fields_1_to_15(buf, pos, id, wt, result)\n elseif id \u003c= 31 then\n pos = _decode_fields_16_to_31(buf, pos, id, wt, result)\n ...\n end\n\nEach chunk is small enough to JIT-compile cleanly. Adds one function call per tag, but only on messages large enough to need it — small messages stay inline.\n\nThe range size and split threshold need measurement. Start with: split when message has \u003e32 fields, into chunks of 16. Measure throughput on conformance message before/after to validate the trade.","notes":"Pre-work: confirm the abort is real by running bench/jit_trace.lua against a conformance-like fixture and checking the FATAL traceerr codes. If LuaJIT compiles the 4212-line body fine, this is a non-issue.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:39Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-gi0","title":"Codegen: emit wire.encode_varint directly for int32/int64/uint32/uint64 instead of typed alias","description":"runtime/pb/wire.lua aliases M.encode_int32 = encode_varint (and same for int64/uint32/uint64). Generated code calls wire.encode_int32(v) which resolves to encode_varint through two table lookups — wire.encode_int32 (one hash lookup), then the alias resolution. LuaJIT may collapse this when the trace stays hot, but each break re-incurs both.\n\nCodegen can emit wire.encode_varint directly for the four unsigned-varint scalar types (int32/int64/uint32/uint64) since they're literally encode_varint with different names. Skips one alias indirection per varint encode. Pairs naturally with kot (localize wire.* upvalues) — together they reduce the call to a direct LJ_FUNCC dispatch with no name lookup.\n\nSame applies to bool (encode_bool = encode_varint with v and 1 or 0 wrapper) and sint32/sint64 (zigzag wrapper) — codegen could inline the wrapper logic at the call site for sint, but that overlaps with h8v (FFI direct writes) which subsumes the question.","notes":"Codegen-side change in protoc-gen-tarantool. Constraint: the alias provides the typed encoder slot in TYPE_INFO that runtime-mode encoders walk — those aliases must stay. Only the generated mode=full code changes.","status":"closed","priority":3,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:38Z","created_by":"Eugene Blikh","updated_at":"2026-05-24T20:14:49Z","started_at":"2026-05-24T20:11:24Z","closed_at":"2026-05-24T20:14:49Z","close_reason":"Premise invalid. wire.encode_int32 / encode_int64 / encode_uint32 / encode_uint64 ARE literally encode_varint at module load (one assignment, same function value). kot's localize already captures wire.encode_int32 once per function, so the in-body call is a direct local read with no alias resolution. Verified: wire.encode_int32 == wire.encode_varint -\u003e true. No extra TGETS at call time. The only residual is kot creating separate locals when a function uses both encode_int32 and encode_varint (could dedupe to one closure upvalue) — marginal closure-size win, not the 'two table lookups' premise. Closing as won't-fix; the work was already done by Lua alias semantics + kot localization.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-5y9","title":"Decoder: int64/uint64 return Lua number when value fits in 2^53","description":"decode_int64/decode_uint64 always return ffi cdata, forcing a fresh cdata allocation per call. For values in [-2^53, 2^53) (the common case for IDs, sequence numbers, timestamps fitting in 53 bits, byte counts, etc.) a Lua number is exactly representable and skips the cdata header allocation.\n\nProposed shape: opt-in variant decoder, since changing the default breaks any caller that does type(v)=='cdata' or relies on cdata-only operators. Two API options:\n\n 1. Per-descriptor flag (desc.int64_as_number = true) wired in via a codegen option or generator flag. Generated code emits a different decoder fn.\n 2. Separate typed decoders (wire.decode_int64_n / decode_uint64_n) that callers opt into explicitly.\n\nWatch case: values exceeding 2^53 must still return cdata (with a runtime branch). The branch cost only pays off if cdata allocation cost \u003e one comparison, which it is on hot paths.","notes":"Bench impact bounded by how many int64 fields the workload has. For hello.Person, user_id (fixed64) is the only one — so impact on this fixture would be ~1-2%. Bigger win on protobuf workloads dominated by timestamps and sequence numbers (datastore RPCs, log streams).","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:37Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-5y9","title":"Decoder: int64/uint64 return Lua number when value fits in 2^53","description":"decode_int64/decode_uint64 always return ffi cdata, forcing a fresh cdata allocation per call. For values in [-2^53, 2^53) (the common case for IDs, sequence numbers, timestamps fitting in 53 bits, byte counts, etc.) a Lua number is exactly representable and skips the cdata header allocation.\n\nProposed shape: opt-in variant decoder, since changing the default breaks any caller that does type(v)=='cdata' or relies on cdata-only operators. Two API options:\n\n 1. Per-descriptor flag (desc.int64_as_number = true) wired in via a codegen option or generator flag. Generated code emits a different decoder fn.\n 2. Separate typed decoders (wire.decode_int64_n / decode_uint64_n) that callers opt into explicitly.\n\nWatch case: values exceeding 2^53 must still return cdata (with a runtime branch). The branch cost only pays off if cdata allocation cost \u003e one comparison, which it is on hot paths.","notes":"Bench impact bounded by how many int64 fields the workload has. For hello.Person, user_id (fixed64) is the only one — so impact on this fixture would be ~1-2%. Bigger win on protobuf workloads dominated by timestamps and sequence numbers (datastore RPCs, log streams).","status":"closed","priority":3,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:37Z","created_by":"Eugene Blikh","updated_at":"2026-05-24T21:22:52Z","started_at":"2026-05-24T21:04:46Z","closed_at":"2026-05-24T21:22:52Z","close_reason":"Implemented as opt-in via plugin flag --tarantool_opt=int64_as_number=true (mode=full only; default off). Adds wire.decode_int64_n / decode_uint64_n / decode_sint64_n / decode_fixed64_n / decode_sfixed64_n that return Lua number when the decoded value fits [-2^53, 2^53] (inclusive — both endpoints are powers of two and exact as doubles), cdata otherwise. LL/ULL literals are int64_t/uint64_t cdata; comparisons compile to plain 64-bit integer compares on trace.\n\nCodegen plumbs cfg.Int64AsNumber through writer.int64AsNumber (avoids threading through 10+ function signatures). decodeFnSuffix() returns \"_n\" for the affected scalar types only when the flag is set. All wire.decode_\u003cst\u003e emit sites updated.\n\nPlugin guards: --int64_as_number=true with mode=runtime errors out (would require descriptor flag wired through pb.codec — out of scope). Under PB_ENABLE_C=1 the option is a no-op since the C runtime makes its own number-vs-cdata decision via luaL_pushint64; test gates accordingly with t.skip.\n\nMeasured tradeoff on c_int64.Wide decode (full mode, no PB_ENABLE_C, 5 fields):\n tiny (all 1-byte vars) 2050 -\u003e 1700 ns/op (-17%, cdata avoided entirely)\n medium (3-byte vars) 3220 -\u003e 3600 ns/op (+11%, extra cmp+tonumber)\n huge (past 2^53) 7575 -\u003e 7750 ns/op (+2%, noise)\n\nWorkload-specific: enable for fields dominated by small IDs/counters/small timestamps (where 1-byte varint hits), leave off for large values. Documented in the plugin flag help.\n\nTest fixture: examples/expected/full_n/c_int64/c_int64_pb.lua regenerated by 'just gen-int64-as-number'. test/int64_as_number_test.lua covers byte-identical encoding, small-value Lua number returns, default cdata returns, 2^53 boundary inclusive, past-2^53 cdata fallback, Lua-number-input round-trip. Suites: test 771/771, test-c 1057/1057 (with 5 skipped under PB_ENABLE_C=1).","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-8k2","title":"Docs: remove PLAN.md after Beads migration","description":"PLAN.md has been converted from active roadmap to duplicate design-history prose now that all actionable work is tracked in Beads. Remove the file or replace remaining references with Beads/README pointers so project state has a single durable task source of truth.","status":"closed","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T18:33:09Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T18:33:48Z","closed_at":"2026-05-17T18:33:48Z","close_reason":"Removed PLAN.md and retargeted remaining references to Beads or concrete docs.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-c0i","title":"C accel: generated C codec backend for mode=full","description":"Extend protoc-gen-tarantool with an optional generated-C backend. Emit C source plus Lua wrappers for each .proto package; generated functions know tag bytes, field names, defaults, oneofs, maps, proto2 required/defaults/groups/extensions, and nested message calls. The Lua API remains M.Type_encode(t) -\u003e string and M.Type_decode(bytes) -\u003e table. Goal: one Lua-\u003eC call per top-level message, no Lua table.concat, no Lua decode_tag ladder, no per-field FFI boundary.","notes":"DEFERRED per docs/c-accel.md. Spike (04c) showed S4 hand-written C had ≤15% headroom over S3 (generic C runtime, ra6) and went the wrong way at scale (S4 *slower* than S3 at 10KB and 100KB encode). Codegen complexity not justified by current numbers. REVIVAL CRITERIA: a measured real-workload shape where ra6's per-field dispatch costs ≥25% over hand-written for that shape, demonstrated with a microbenchmark, AND the affected workload is on a hot path for a real user. Likely trigger shapes: wide messages with many optionals, heavy oneof use, complex maps, deeply nested (5+ levels) hierarchies. When triggered: write 2-3 paragraphs documenting shape and numbers, re-open this issue, scope narrowly via desc.encode/desc.decode override (same mechanism as WKT). If 6 months after ra6 ships no trigger fires, close as 'not justified'. Now blocked by ra6 (so it surfaces in ready list only after ra6 implementation lands).","status":"open","priority":3,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:32Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:41Z","dependencies":[{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-18T23:19:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":4,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-bnj","title":"Tooling: add formatting targets for Go and Lua","description":"PLAN.md section 6 lists gofumpt and stylua. Add formatter configuration and Justfile targets, with generated code excluded unless the generator itself is intended to emit stylua-compliant output.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:28Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/Justfile b/Justfile index b1b6eab73c9a32d190e64c5a97f72a2e8b48023f..ba5b44dd03e0735a7859e0c4b904edae1535202d 100644 --- a/Justfile +++ b/Justfile @@ -86,7 +86,7 @@ # Codegen # --------------------------------------------------------------------------- # Regenerate examples/expected/{full,runtime}/* + conformance protos. -gen: gen-full gen-runtime gen-conformance gen-proto2-tests +gen: gen-full gen-runtime gen-conformance gen-proto2-tests gen-int64-as-number # Generate full-mode Lua (inline encode/decode bodies). gen-full: build @@ -141,6 +141,20 @@ --tarantool_out={{gen_dir}} \ --tarantool_opt=mode=runtime,prefix=runtime \ -I {{proto2_test_dir}} -I options \ {{proto2_test_dir}}/*.proto + +# Generate the c_int64 fixture into examples/expected/full_n/ with the +# `int64_as_number=true` codegen flag set, so the opt-in 5y9 path has a +# parallel fixture for round-trip + type-stability tests next to the +# default full/ output. Single fixture is enough to cover the codegen +# surface; no need to double the entire examples/expected/full/ tree. +gen-int64-as-number: build + mkdir -p {{gen_dir}} + protoc \ + --plugin=./{{plugin}} \ + --tarantool_out={{gen_dir}} \ + --tarantool_opt=mode=full,prefix=full_n,int64_as_number=true \ + -I {{proto2_test_dir}} -I options \ + {{proto2_test_dir}}/c_int64.proto # Regenerate Markdown reference docs (examples/docs/*.md) — committed output. gen-docs: build-doc diff --git a/cmd/protoc-gen-tarantool/internal/gen/gen.go b/cmd/protoc-gen-tarantool/internal/gen/gen.go index bfd8c8a41c9aa5bc25c947741855890528211cf2..e14ee94345068cb75aec0de38f80a6453fad1076 100644 --- a/cmd/protoc-gen-tarantool/internal/gen/gen.go +++ b/cmd/protoc-gen-tarantool/internal/gen/gen.go @@ -51,6 +51,12 @@ // require path (and its on-disk subpath). Lets the same .proto be // generated under multiple namespaces in one project — e.g. for // side-by-side full vs runtime mode comparison in tests. Prefix string + // Int64AsNumber: when true (mode=full only), 64-bit decoders return a + // Lua number for values that fit in [-2^53, 2^53] (or [0, 2^53] for + // unsigned) and cdata otherwise. Skips the per-call cdata allocation + // on the dominant small-value case (IDs, timestamps, byte counts). + // Default false — decoded 64-bit fields are always cdata. (5y9) + Int64AsNumber bool } // GenerateFile emits one `.lua` file per input `.proto`. @@ -65,7 +71,11 @@ allMsgs := flattenMessagesSkippingMapEntries(file.Messages, nil) allEnums := flattenEnums(file.Enums, file.Messages) out := plug.NewGeneratedFile(outputFilename(file.Desc, cfg.Prefix), "") - w := &writer{GeneratedFile: out, opts: newOptionsResolver(plug)} + w := &writer{ + GeneratedFile: out, + opts: newOptionsResolver(plug), + int64AsNumber: cfg.Int64AsNumber, + } emitHeader(w, file) imports := collectImports(file, allMsgs, cfg.Prefix) @@ -157,6 +167,10 @@ // straight to the GeneratedFile. captureLines sets/restores it; used by // emitInlineEncode/Decode to scan a function body for wire.* refs and // rewrite them to bare locals. buf *[]string + // int64AsNumber mirrors Config.Int64AsNumber (5y9). Attached to the + // writer rather than threaded through every emit-* signature so the + // option is one field-lookup away wherever a decode call is emitted. + int64AsNumber bool } func (w *writer) line(format string, args ...any) { diff --git a/cmd/protoc-gen-tarantool/internal/gen/inline.go b/cmd/protoc-gen-tarantool/internal/gen/inline.go index 77c51b2aaa125639a3b6fce1887c82aa7f4b297b..e346e38194b89bd8c0a544cef462b49ee57b3a77 100644 --- a/cmd/protoc-gen-tarantool/internal/gen/inline.go +++ b/cmd/protoc-gen-tarantool/internal/gen/inline.go @@ -10,6 +10,21 @@ "google.golang.org/protobuf/compiler/protogen" "google.golang.org/protobuf/reflect/protoreflect" ) +// decodeFnSuffix returns "_n" when 5y9's int64_as_number option is set +// and `st` is one of the cdata-returning 64-bit scalar kinds, "" otherwise. +// Used to swap wire.decode_int64 -> wire.decode_int64_n at codegen sites +// where the value type is statically known. (5y9) +func decodeFnSuffix(w *writer, st string) string { + if !w.int64AsNumber { + return "" + } + switch st { + case "int64", "uint64", "sint64", "fixed64", "sfixed64": + return "_n" + } + return "" +} + // encodeCallExpr returns the Lua expression that encodes a single value // of the given scalar type `st`. For types where mode=full can elide // wire.to_int64 / wire.to_uint64's runtime type dispatch (sint64, fixed64, @@ -260,10 +275,10 @@ w.line("%s if _b2 == 0 then error(\"overlong varint at offset \" .. p2, 0) end", indent) w.line("%s %s = _b - 128 + _b2 * 128", indent, valExpr) w.line("%s p2 = p2 + 2", indent) w.line("%s else", indent) - w.line("%s %s, p2 = wire.decode_%s(payload, p2)", indent, valExpr, st) + w.line("%s %s, p2 = wire.decode_%s%s(payload, p2)", indent, valExpr, st, decodeFnSuffix(w, st)) w.line("%s end", indent) w.line("%selse", indent) - w.line("%s %s, p2 = wire.decode_%s(payload, p2)", indent, valExpr, st) + w.line("%s %s, p2 = wire.decode_%s%s(payload, p2)", indent, valExpr, st, decodeFnSuffix(w, st)) w.line("%send", indent) case "bool": // Bool values are spec-valid only as 0 or 1, always 1-byte on the wire. @@ -277,7 +292,7 @@ w.line("%selse", indent) w.line("%s %s, p2 = wire.decode_bool(payload, p2)", indent, valExpr) w.line("%send", indent) default: - w.line("%s%s, p2 = wire.decode_%s(payload, p2)", indent, valExpr, st) + w.line("%s%s, p2 = wire.decode_%s%s(payload, p2)", indent, valExpr, st, decodeFnSuffix(w, st)) } } @@ -824,7 +839,7 @@ if st == "string" && !validateUTF8 { st = "bytes" } w.line(" local _val") - w.line(" _val, pos = wire.decode_%s(buf, pos)", st) + w.line(" _val, pos = wire.decode_%s%s(buf, pos)", st, decodeFnSuffix(w, st)) w.line(" local _e = result._extensions") w.line(" if _e == nil then _e = {}; result._extensions = _e end") w.line(" _e[%q] = _val", full) @@ -886,17 +901,17 @@ w.line(" _payload, pos = wire.decode_len(buf, pos)") w.line(" local p2, lim = 1, #_payload") w.line(" while p2 <= lim do") w.line(" local _val") - w.line(" _val, p2 = wire.decode_%s(_payload, p2)", st) + w.line(" _val, p2 = wire.decode_%s%s(_payload, p2)", st, decodeFnSuffix(w, st)) w.line(" _list[#_list + 1] = _val") w.line(" end") w.line(" else") w.line(" local _val") - w.line(" _val, pos = wire.decode_%s(buf, pos)", st) + w.line(" _val, pos = wire.decode_%s%s(buf, pos)", st, decodeFnSuffix(w, st)) w.line(" _list[#_list + 1] = _val") w.line(" end") } else { w.line(" local _val") - w.line(" _val, pos = wire.decode_%s(buf, pos)", st) + w.line(" _val, pos = wire.decode_%s%s(buf, pos)", st, decodeFnSuffix(w, st)) w.line(" _list[#_list + 1] = _val") } } @@ -1094,7 +1109,7 @@ dst := luaFieldAccess("result", fname) emitInlineStringBytesScalar(w, st, dst, validateUTF8) } else { w.line(" local val") - w.line(" val, pos = wire.decode_%s(buf, pos)", st) + w.line(" val, pos = wire.decode_%s%s(buf, pos)", st, decodeFnSuffix(w, st)) w.line(" %s = val", luaFieldAccess("result", fname)) } } @@ -1209,7 +1224,7 @@ w.line(" else") w.line(" local list = %s", dst) w.line(" if list == nil then list = {}; %s = list end", dst) w.line(" local val") - w.line(" val, pos = wire.decode_%s(buf, pos)", st) + w.line(" val, pos = wire.decode_%s%s(buf, pos)", st, decodeFnSuffix(w, st)) w.line(" %s = %s + 1; list[%s] = val", cnt, cnt, cnt) w.line(" end") } else { @@ -1546,7 +1561,7 @@ // by calling wire.decode_bytes (identical wire shape, no utf8_len). if st == "string" && !validateUTF8 { st = "bytes" } - w.line(" %s, _ep = wire.decode_%s(payload, _ep)", dst, st) + w.line(" %s, _ep = wire.decode_%s%s(payload, _ep)", dst, st, decodeFnSuffix(w, st)) } } diff --git a/cmd/protoc-gen-tarantool/main.go b/cmd/protoc-gen-tarantool/main.go index 6362197803de7a82c8e2b0925f9a8ca69268b8cc..0fa89f95078a454ffa7a6314ab60813790fb504f 100644 --- a/cmd/protoc-gen-tarantool/main.go +++ b/cmd/protoc-gen-tarantool/main.go @@ -52,6 +52,14 @@ modeFlag := flags.String("mode", "full", "codegen mode: full | runtime") prefixFlag := flags.String("prefix", "", "prefix prepended to every generated module's Lua require path "+ "(useful for side-by-side generation in tests)") + int64AsNumberFlag := flags.Bool("int64_as_number", false, + "opt-in (mode=full only): emit decoders that return a Lua number "+ + "for int64/uint64/sint64/fixed64/sfixed64 values that fit in "+ + "[-2^53, 2^53), falling back to cdata for values outside that "+ + "range. Skips the per-call cdata allocation on the common case "+ + "(IDs, timestamps in seconds/ms, byte counts). Decoded type is "+ + "unstable (number-or-cdata); arithmetic works transparently. "+ + "Default false (always cdata).") plugin, err := protogen.Options{ParamFunc: flags.Set}.New(req) if err != nil { fail("init protogen: %v", err) @@ -61,7 +69,11 @@ mode, err := gen.ParseMode(*modeFlag) if err != nil { fail("%v", err) } - cfg := gen.Config{Mode: mode, Prefix: *prefixFlag} + if *int64AsNumberFlag && mode != gen.ModeFull { + fail("int64_as_number is currently mode=full only " + + "(runtime mode would require a descriptor flag wired through pb.codec)") + } + cfg := gen.Config{Mode: mode, Prefix: *prefixFlag, Int64AsNumber: *int64AsNumberFlag} // Advertise proto3 optional support so protoc lets us see those fields. plugin.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL) diff --git a/examples/expected/full_n/c_int64/c_int64_pb.lua b/examples/expected/full_n/c_int64/c_int64_pb.lua new file mode 100644 index 0000000000000000000000000000000000000000..7182ff84e46e40833cee5035fdef71fddabf686d --- /dev/null +++ b/examples/expected/full_n/c_int64/c_int64_pb.lua @@ -0,0 +1,264 @@ +-- Code generated by protoc-gen-tarantool. DO NOT EDIT. +-- source: c_int64.proto +-- syntax: proto3 +-- package: c_int64 + +local pb = require("pb") +local wire = pb.wire +local string_byte = string.byte +local utf8_len = require('utf8').len +local band = bit.band +local rshift = bit.rshift +local CHARS = wire.CHARS +local table_new = require('table.new') +local ffi = require('ffi') +local INT64 = ffi.typeof('int64_t') +local UINT64 = ffi.typeof('uint64_t') + +local M = {} + +M.options = {go_package = "tarantoolpb_synthetic/c_int64"} + +-- Pre-declare message descriptors so cross-references resolve. +M.Wide_descriptor = {name = "c_int64.Wide"} + +-- Message: c_int64.Wide +M.Wide_descriptor.fields = { + {name="a_int64", id=1, kind='scalar', proto_type="int64"}, + {name="a_uint64", id=2, kind='scalar', proto_type="uint64"}, + {name="a_sint64", id=3, kind='scalar', proto_type="sint64"}, + {name="a_fixed64", id=4, kind='scalar', proto_type="fixed64"}, + {name="a_sfixed64", id=5, kind='scalar', proto_type="sfixed64"}, +} +pb.finalize_message(M.Wide_descriptor) +M.Wide_fields = pb.field_names({ + a_int64 = "a_int64", + a_uint64 = "a_uint64", + a_sint64 = "a_sint64", + a_fixed64 = "a_fixed64", + a_sfixed64 = "a_sfixed64", +}) + +-- EmmyLua / lua-language-server type annotations. +-- These are comments — no runtime effect. They give editors +-- autocomplete and type-checking for the generated wrappers. +---@class c_int64.Wide +---@field a_int64 integer +---@field a_uint64 integer +---@field a_sint64 integer +---@field a_fixed64 integer +---@field a_sfixed64 integer + +---@param t? c_int64.Wide +---@return c_int64.Wide +function M.Wide_new(t) return t or {} end + +---@param t c_int64.Wide +---@return string +function M.Wide_encode(t) + local _d = M.Wide_descriptor + if pb.c_runtime ~= nil then + local _p = _d.c_plan or pb.c_runtime.compile_plan(_d) + return pb.c_runtime.encode(_p, t) + end + if type(t) ~= 'table' then + error("expected table for c_int64.Wide, got " .. type(t), 0) + end + local out, n = {}, 0 + local v + -- field 1: a_int64 + v = t.a_int64 + if v ~= nil and v ~= 0 then + n = n + 1; out[n] = "\x08" + n = n + 1; out[n] = wire.encode_int64(v) + end + -- field 2: a_uint64 + v = t.a_uint64 + if v ~= nil and v ~= 0 then + n = n + 1; out[n] = "\x10" + n = n + 1; out[n] = wire.encode_uint64(v) + end + -- field 3: a_sint64 + v = t.a_sint64 + if v ~= nil and v ~= 0 then + n = n + 1; out[n] = "\x18" + n = n + 1; out[n] = wire.encode_sint64_i(INT64(v)) + end + -- field 4: a_fixed64 + v = t.a_fixed64 + if v ~= nil and v ~= 0 then + n = n + 1; out[n] = "\x21" + n = n + 1; out[n] = wire.encode_fixed64_u(UINT64(v)) + end + -- field 5: a_sfixed64 + v = t.a_sfixed64 + if v ~= nil and v ~= 0 then + n = n + 1; out[n] = "\x29" + n = n + 1; out[n] = wire.encode_sfixed64_u(UINT64(v)) + end + local _uf = t._unknown_fields + if _uf ~= nil and _uf ~= '' then n = n + 1; out[n] = _uf end + return table.concat(out) +end + +---@param b string +---@return c_int64.Wide +function M.Wide_decode(buf) + local decode_tag = wire.decode_tag + local _d = M.Wide_descriptor + if pb.c_runtime ~= nil then + local _p = _d.c_plan or pb.c_runtime.compile_plan(_d) + return pb.c_runtime.decode(_p, buf) + end + if type(buf) ~= 'string' then + error("expected string for c_int64.Wide decode, got " .. type(buf), 0) + end + local result = {} + local pos, len = 1, #buf + local _uf + while pos <= len do + local _tag_start = pos + local id, wt + local _b = string_byte(buf, pos) + if _b ~= nil and _b < 0x80 then + wt = band(_b, 7) + if wt >= 6 then error("illegal wire type " .. wt, 0) end + id = rshift(_b, 3) + if id == 0 then error("illegal field number 0", 0) end + pos = pos + 1 + elseif _b ~= nil and pos < len then + local _b2 = string_byte(buf, pos + 1) + if _b2 < 0x80 then + if _b2 == 0 then error("overlong tag varint at offset " .. pos, 0) end + local _v = _b - 128 + _b2 * 128 + wt = band(_v, 7) + if wt >= 6 then error("illegal wire type " .. wt, 0) end + id = rshift(_v, 3) + pos = pos + 2 + else + id, wt, pos = decode_tag(buf, pos) + end + else + id, wt, pos = decode_tag(buf, pos) + end + if id == 1 then + local val + val, pos = wire.decode_int64_n(buf, pos) + result.a_int64 = val + elseif id == 2 then + local val + val, pos = wire.decode_uint64_n(buf, pos) + result.a_uint64 = val + elseif id == 3 then + local val + val, pos = wire.decode_sint64_n(buf, pos) + result.a_sint64 = val + elseif id == 4 then + local val + val, pos = wire.decode_fixed64_n(buf, pos) + result.a_fixed64 = val + elseif id == 5 then + local val + val, pos = wire.decode_sfixed64_n(buf, pos) + result.a_sfixed64 = val + else + local _ebid = M.Wide_descriptor.extensions_by_id + local _ext = _ebid and _ebid[id] or nil + if _ext ~= nil then + pos = pb.codec.decode_extension(_ext, buf, pos, wt, result) + else + pos = wire.skip_field(buf, pos, wt, id) + if _uf == nil then _uf = {} end + _uf[#_uf + 1] = buf:sub(_tag_start, pos - 1) + end + end + end + if _uf ~= nil then result._unknown_fields = table.concat(_uf) end + return result +end + +---@param b string +---@return c_int64.Wide +function M.Wide_decode_unsafe(buf) + local decode_tag = wire.decode_tag + local _d = M.Wide_descriptor + if pb.c_runtime ~= nil then + local _p = _d.c_plan or pb.c_runtime.compile_plan(_d) + return pb.c_runtime.decode_unsafe(_p, buf) + end + if type(buf) ~= 'string' then + error("expected string for c_int64.Wide decode, got " .. type(buf), 0) + end + local result = {} + local pos, len = 1, #buf + local _uf + while pos <= len do + local _tag_start = pos + local id, wt + local _b = string_byte(buf, pos) + if _b ~= nil and _b < 0x80 then + wt = band(_b, 7) + if wt >= 6 then error("illegal wire type " .. wt, 0) end + id = rshift(_b, 3) + if id == 0 then error("illegal field number 0", 0) end + pos = pos + 1 + elseif _b ~= nil and pos < len then + local _b2 = string_byte(buf, pos + 1) + if _b2 < 0x80 then + if _b2 == 0 then error("overlong tag varint at offset " .. pos, 0) end + local _v = _b - 128 + _b2 * 128 + wt = band(_v, 7) + if wt >= 6 then error("illegal wire type " .. wt, 0) end + id = rshift(_v, 3) + pos = pos + 2 + else + id, wt, pos = decode_tag(buf, pos) + end + else + id, wt, pos = decode_tag(buf, pos) + end + if id == 1 then + local val + val, pos = wire.decode_int64_n(buf, pos) + result.a_int64 = val + elseif id == 2 then + local val + val, pos = wire.decode_uint64_n(buf, pos) + result.a_uint64 = val + elseif id == 3 then + local val + val, pos = wire.decode_sint64_n(buf, pos) + result.a_sint64 = val + elseif id == 4 then + local val + val, pos = wire.decode_fixed64_n(buf, pos) + result.a_fixed64 = val + elseif id == 5 then + local val + val, pos = wire.decode_sfixed64_n(buf, pos) + result.a_sfixed64 = val + else + local _ebid = M.Wide_descriptor.extensions_by_id + local _ext = _ebid and _ebid[id] or nil + if _ext ~= nil then + pos = pb.codec.decode_extension(_ext, buf, pos, wt, result) + else + pos = wire.skip_field(buf, pos, wt, id) + if _uf == nil then _uf = {} end + _uf[#_uf + 1] = buf:sub(_tag_start, pos - 1) + end + end + end + if _uf ~= nil then result._unknown_fields = table.concat(_uf) end + return result +end + +---@param b string +---@return pb.MessageView +function M.Wide_decode_lazy(b) return pb.decode_lazy(M.Wide_descriptor, b) end +---@param t c_int64.Wide +---@param opts? {single_line: boolean?, indent: string?} +---@return string +function M.Wide_text(t, opts) return pb.text.encode(M.Wide_descriptor, t, opts) end + +return M diff --git a/runtime/pb/wire.lua b/runtime/pb/wire.lua index beb9b2920dcdcce7637005fb36ce728a6afa88a2..aab1fd091c202571b48549ecdb1713945df20e1b 100644 --- a/runtime/pb/wire.lua +++ b/runtime/pb/wire.lua @@ -31,6 +31,19 @@ local INT64 = ffi.typeof('int64_t') local UINT64_ZERO = UINT64(0) local CONT_MASK = UINT64(bit.bnot(0x7f)) -- 0xFFFFFFFFFFFFFF80 +-- 2^53 thresholds used by the opt-in decode__n variants (5y9). +-- A Lua double exactly represents every integer in [-2^53, 2^53] (bounds +-- inclusive — both endpoints are themselves powers of two and fit a +-- double's 53-bit mantissa exactly). Outside that range tonumber() would +-- silently lose precision on odd values. The _n decoders return a Lua +-- number when the decoded value fits the bounds, otherwise the cdata. +-- LL/ULL literals are already int64_t/uint64_t cdata constants; cdata +-- comparisons against them compile to plain 64-bit integer compares on +-- a hot trace. +local FITS_MAX_I = 0x20000000000000LL -- 2^53 +local FITS_MIN_I = -0x20000000000000LL -- -2^53 +local FITS_MAX_U = 0x20000000000000ULL -- 2^53 + -- FFI scratch unions used by the fixed-width decoders. Allocated once -- and reused — `ffi.copy` from a `uint8_t*` cast over the Lua string body -- writes into the union's byte view; reading back via `.u` / `.d` / @@ -565,6 +578,63 @@ M.decode_sint64 = decode_sint64 M.decode_bool = decode_bool M.decode_sfixed32 = decode_sfixed32 M.decode_sfixed64 = decode_sfixed64 + +-- Opt-in decode variants that return a Lua number when the decoded value +-- fits in [-2^53, 2^53) (or [0, 2^53) for the unsigned ones), else the +-- usual cdata. Wired in by mode=full codegen when the plugin is invoked +-- with `int64_as_number=true` — see cmd/protoc-gen-tarantool docs. The +-- type at decode time is unstable (number vs cdata) under this option, +-- but Lua's `+`/`-`/`*`/`==` work transparently on both and the saved +-- cdata allocation pays off on workloads dominated by small IDs, +-- timestamps that fit 2^53, byte counts, etc. (5y9) +local function decode_int64_n(buf, pos) + local b = buf:byte(pos) + if b == nil then error("truncated varint at offset " .. pos, 0) end + if b < 0x80 then return b, pos + 1 end -- 0..127, Lua number + local u, np = decode_varint(buf, pos) + local i = INT64(u) -- reinterpret bits + if i <= FITS_MAX_I and i >= FITS_MIN_I then + return tonumber(i), np + end + return i, np +end +local function decode_uint64_n(buf, pos) + local b = buf:byte(pos) + if b == nil then error("truncated varint at offset " .. pos, 0) end + if b < 0x80 then return b, pos + 1 end + local u, np = decode_varint(buf, pos) + if u <= FITS_MAX_U then return tonumber(u), np end + return u, np +end +local function decode_sint64_n(buf, pos) + local b = buf:byte(pos) + if b == nil then error("truncated varint at offset " .. pos, 0) end + if b < 0x80 then return zigzag_decode32(b), pos + 1 end + local u, np = decode_varint(buf, pos) + local i = zigzag_decode64(u) + if i <= FITS_MAX_I and i >= FITS_MIN_I then + return tonumber(i), np + end + return i, np +end +local function decode_fixed64_n(buf, pos) + local u, np = decode_fixed64(buf, pos) + if u <= FITS_MAX_U then return tonumber(u), np end + return u, np +end +local function decode_sfixed64_n(buf, pos) + local u, np = decode_fixed64(buf, pos) + local i = INT64(u) + if i <= FITS_MAX_I and i >= FITS_MIN_I then + return tonumber(i), np + end + return i, np +end +M.decode_int64_n = decode_int64_n +M.decode_uint64_n = decode_uint64_n +M.decode_sint64_n = decode_sint64_n +M.decode_fixed64_n = decode_fixed64_n +M.decode_sfixed64_n = decode_sfixed64_n -- RFC 3629 UTF-8 validator. Rejects: out-of-range continuation bytes, -- truncated multi-byte sequences, overlong encodings, UTF-16 surrogate -- code points (U+D800..U+DFFF), and code points above U+10FFFF. diff --git a/test/int64_as_number_test.lua b/test/int64_as_number_test.lua new file mode 100644 index 0000000000000000000000000000000000000000..34cec3a0eb57cbab63e6d8fffe2ac8597491cca2 --- /dev/null +++ b/test/int64_as_number_test.lua @@ -0,0 +1,143 @@ +-- 5y9: int64_as_number codegen option. Verifies that the opt-in `_n` +-- decoders return Lua number when the decoded value fits in [-2^53, 2^53] +-- and fall back to cdata otherwise. Encodes via the default (cdata-returning) +-- module and decodes via the `_n` module; assertions cover both type and +-- value across every 64-bit kind in c_int64.Wide. +-- +-- examples/expected/full_n/c_int64/c_int64_pb.lua is regenerated by +-- `just gen-int64-as-number`, which runs the plugin with +-- `int64_as_number=true` for this single fixture so the default tree +-- stays cdata-typed and existing tests are unaffected. (5y9) +local t = require('luatest') +local ffi = require('ffi') + +local INT64 = ffi.typeof('int64_t') +local UINT64 = ffi.typeof('uint64_t') + +local wide_default = require('full.c_int64.c_int64_pb') +local wide_n = require('full_n.c_int64.c_int64_pb') + +local g = t.group('int64_as_number') + +-- The opt-in `_n` decoder lives in the pure-Lua codegen path. With +-- PB_ENABLE_C=1 the generated _decode prologue short-circuits to the +-- C runtime, which makes its own number-vs-cdata choice independent +-- of this codegen flag (luaL_pushint64 returns Lua number for values +-- that fit in double and cdata otherwise — the Tarantool convention). +-- Skip the type-stability assertions in that mode; the value-equality +-- behavior is still covered by the existing c_runtime_int64_test suite. +g.before_each(function() + if os.getenv('PB_ENABLE_C') == '1' then + t.skip('int64_as_number flag is a no-op under PB_ENABLE_C=1 — ' .. + 'the C runtime decides number-vs-cdata on its own') + end +end) + +-- Confirm the two fixtures share a wire format — the codegen flag only +-- changes the Lua return type, never the bytes that come off the wire. +g.test_encode_bytes_identical = function() + local p = { + a_int64 = INT64(-1234567), + a_uint64 = UINT64(8765432), + a_sint64 = INT64(-987654), + a_fixed64 = UINT64(42), + a_sfixed64 = INT64(-42), + } + t.assert_equals(wide_n.Wide_encode(p), wide_default.Wide_encode(p)) +end + +-- Values inside the 2^53 window should come back as plain Lua numbers +-- under the _n decoder, but stay cdata under the default decoder. +g.test_small_values_return_number = function() + local p = { + a_int64 = INT64(-123), + a_uint64 = UINT64(456), + a_sint64 = INT64(-789), + a_fixed64 = UINT64(1011), + a_sfixed64 = INT64(-1213), + } + local bytes = wide_default.Wide_encode(p) + + local d_n = wide_n.Wide_decode(bytes) + t.assert_equals(type(d_n.a_int64), 'number') + t.assert_equals(type(d_n.a_uint64), 'number') + t.assert_equals(type(d_n.a_sint64), 'number') + t.assert_equals(type(d_n.a_fixed64), 'number') + t.assert_equals(type(d_n.a_sfixed64), 'number') + + t.assert_equals(d_n.a_int64, -123) + t.assert_equals(d_n.a_uint64, 456) + t.assert_equals(d_n.a_sint64, -789) + t.assert_equals(d_n.a_fixed64, 1011) + t.assert_equals(d_n.a_sfixed64, -1213) + + -- Default decoder stays cdata for everything 64-bit. + local d = wide_default.Wide_decode(bytes) + t.assert_equals(type(d.a_int64), 'cdata') + t.assert_equals(type(d.a_uint64), 'cdata') + t.assert_equals(type(d.a_sint64), 'cdata') + t.assert_equals(type(d.a_fixed64), 'cdata') + t.assert_equals(type(d.a_sfixed64), 'cdata') +end + +-- Exactly 2^53 / -2^53 sit on the inclusive boundary (both are powers of +-- two and round-trip a Lua double exactly). Anything one notch past the +-- boundary on the magnitude side falls back to cdata. +g.test_boundary_2p53_returns_number = function() + local p = { + a_int64 = INT64( 0x20000000000000LL), -- 2^53 + a_sint64 = INT64(-0x20000000000000LL), -- -2^53 + a_uint64 = UINT64(0x20000000000000ULL), -- 2^53 + a_fixed64 = UINT64(0x20000000000000ULL), -- 2^53 + a_sfixed64 = INT64(-0x20000000000000LL), -- -2^53 + } + local d_n = wide_n.Wide_decode(wide_default.Wide_encode(p)) + t.assert_equals(type(d_n.a_int64), 'number') + t.assert_equals(type(d_n.a_sint64), 'number') + t.assert_equals(type(d_n.a_uint64), 'number') + t.assert_equals(type(d_n.a_fixed64), 'number') + t.assert_equals(type(d_n.a_sfixed64), 'number') + -- 2^53 is exact as a double; the value comparison passes. + t.assert_equals(d_n.a_int64, 2 ^ 53) + t.assert_equals(d_n.a_uint64, 2 ^ 53) + t.assert_equals(d_n.a_fixed64, 2 ^ 53) + t.assert_equals(d_n.a_sint64, -(2 ^ 53)) + t.assert_equals(d_n.a_sfixed64, -(2 ^ 53)) +end + +g.test_past_2p53_returns_cdata = function() + local p = { + a_int64 = INT64( 0x40000000000000LL), -- 2^54 + a_uint64 = UINT64(0x40000000000000ULL), + a_sint64 = INT64(-0x40000000000000LL), + a_fixed64 = UINT64(0x40000000000000ULL), + a_sfixed64 = INT64(-0x40000000000000LL), + } + local d_n = wide_n.Wide_decode(wide_default.Wide_encode(p)) + t.assert_equals(type(d_n.a_int64), 'cdata') + t.assert_equals(type(d_n.a_uint64), 'cdata') + t.assert_equals(type(d_n.a_sint64), 'cdata') + t.assert_equals(type(d_n.a_fixed64), 'cdata') + t.assert_equals(type(d_n.a_sfixed64), 'cdata') + -- Value still round-trips correctly (just as cdata, not number). + t.assert_equals(d_n.a_int64, INT64( 0x40000000000000LL)) + t.assert_equals(d_n.a_uint64, UINT64(0x40000000000000ULL)) + t.assert_equals(d_n.a_sint64, INT64(-0x40000000000000LL)) + t.assert_equals(d_n.a_fixed64, UINT64(0x40000000000000ULL)) + t.assert_equals(d_n.a_sfixed64, INT64(-0x40000000000000LL)) +end + +-- Lua-number inputs round-trip clean too: encode accepts numbers (via the +-- existing encode wrappers), decode returns numbers under _n. +g.test_lua_number_inputs_round_trip = function() + local p = { + a_int64 = 100, a_uint64 = 200, a_sint64 = -300, + a_fixed64 = 400, a_sfixed64 = -500, + } + local d_n = wide_n.Wide_decode(wide_default.Wide_encode(p)) + t.assert_equals(d_n.a_int64, 100) + t.assert_equals(d_n.a_uint64, 200) + t.assert_equals(d_n.a_sint64, -300) + t.assert_equals(d_n.a_fixed64, 400) + t.assert_equals(d_n.a_sfixed64, -500) +end