Repository navigation
Conversation
…2762) platform/tools/translator/ holds the translator that writes the C++ twin of a library/kernels/common builder module, and its C header, from the Python: tr/ (the translator), tools/ (the declaration scanner and the code-map miner it reads the tree with), designs/ (the header rules, tagged values, host clones) and HOWTO.md. platform/cpp/include/rocke/rocke_py.hpp is the prelude every generated source includes. `python -m tr.regen <module>...` regenerates and writes into the enclosing tree; `--check` writes nothing and exits 1 when the checked-in generated files are not current. No module is generated in this change and no existing file changes. The translator was built and checked on the attention backward port; the generated backward modules, their data and the package adapters header land with that port. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
platform/dsl_docs/architecture/python_cpp_translator_rfc.md describes the translator: the pipeline, type recording, the generated C header, holes and fills, the prelude, packages, the regen contract, design decisions, risks, the landing order and the test plan. Status Planned. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…K-2762) platform/dsl_docs/architecture/python_cpp_translator_architecture.html: the pipeline (click a stage, or play the flow), one real function in Python and the C++ the translator wrote for it, and the mechanical share per module. Self-contained; open it in a browser. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…est (AICK-2762) Facts: tr.record keyed each function's recorded types by "<qualname>@<firstlineno>", and the translator looked them up by the same line. Any edit above a function silently dropped its types and degraded the translation. Functions are now keyed by qualname alone (tr/factkeys.py). Generator expressions, lambdas and comprehensions carry a #<n> ordinal in source order, and so does a named function whose qualname repeats in the module. The AST walker reproduces CPython's co_qualname and co_firstlineno for every code object in library/kernels/common. Each entry also stores src, a hash of the function's AST without docstrings. When the body changes after the recording, the translator prints "facts stale for <module>.<function>; re-run tr.record" and does not change its output. Old line-keyed files are still read, and tr.rekey converts them while their lines match. Secondary modules in one recording are keyed "<module>:<qualname>" so equal names stay apart. Self-test: tools/translator/tests is a pytest suite run by hand (python -m pytest tools/translator/tests from platform/). There is no CMake, ctest or CI change. TR_DATA, TR_HOLES and TR_LIBRARY point the translator at a fixture root: pinned copies of the attention backward modules, their facts, names and cut-down code maps. The C headers still come from the enclosing tree. The tests check: - the addr, valu and counted headers and the valu and counted sources against goldens; - byte determinism; - facts keys and ordinals; - line-shift invariance; - the stale warning; - old-key compatibility. Goldens change only through --update-goldens. The goldens equal the translator's output before this change, byte for byte. tr.regen --check still reproduces the four generated attention backward modules exactly, with both old and re-keyed facts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…p/include (AICK-2762) What: - Move rocke_py.hpp from platform/cpp/include/rocke/ to tools/translator/tr/prelude/. The translator change now adds nothing under platform/cpp, and nothing installs the prelude. - Add the attention backward adapters header as package data, tr/packages/attention_bwd/rocke_py_adapters.hpp. It includes the backward module headers, so it is kept out of the generic prelude; it includes the prelude by a quoted same-directory name. - Generated sources include "rocke_py.hpp" and "rocke_py_adapters.hpp" by quoted same-directory names. - tr.regen copies every same-directory header a written source includes from its master copy (tr/prelude/, then tr/packages/<package>/) into platform/cpp/instances/common/, and never over an existing copy (it prints "skip <path> (exists)"). Files there are compiled into the engine and are not installed. This replaces the glob of tr/prelude/rocke_py*.hpp, which matched nothing because the directory did not exist, so regen copied no prelude at all. - package.py: Package.data_dir and support_file(). - HOWTO and RFC: the new prelude location and copy rule. Why: the review asked that the translator not add C++ to the installed headers or own files under platform/cpp. Once copied, the prelude is maintained by hand like the sources next to it. Tests: test_prelude.py compiles the prelude alone with g++ and clang++ (-std=c++17 -Wall -Wextra -Werror, skipped without a compiler), checks that no rocke_py*.hpp is left in cpp/include and that every quoted include of the goldens has a master copy. Goldens re-baselined: only the two include lines of each .cpp changed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…CK-2762) The C++ twins are edited by hand as a matter of course, so a translated file must become ordinary hand code the moment it lands. Until now tr.regen overwrote every target unconditionally and the banners said "do not edit". - Banners: the .cpp says "first draft from <py>:<sha> by tr/translate.py; hand-maintained from here", the header "first draft by tr/abi.py ... hand-maintained from here". - tr.regen writes only targets that do not exist yet in --into: the .cpp, the module header and the support headers. Each existing one prints "skip <path> (exists)" and the exit code stays 0. --overwrite writes over them; --pretend lists what would be written or skipped and writes nothing. - --check is removed: with every file hand-owned there is nothing for a currency check to enforce. - ABI: when every module named already has its header in the --rocke tree (hand-written or an earlier draft; they share one name), the sources are translated against it (--hand-abi) so a new draft fits existing callers. --generated-abi overrides; a partial set is an error that asks for one of the two flags. - tests/test_regen_scaffold.py: first regen writes all six files, a second regen after hand edits skips all six and keeps them, --overwrite, --pretend (with and without --overwrite), auto ABI on an existing header of either origin, the partial-set error, and --check rejected. Goldens re-baselined: banner lines only. - HOWTO and RFC: the regen contract, ownership of translated files and the --check references updated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tant (AICK-2762) The translator now writes a first draft that is hand code from its first commit, and never writes over an existing file. The RFC, HOWTO and the one-page architecture still described generator-owned files kept current by a regen --check step, which is no longer true. RFC: experimental porting assistant in the title, status notes and summary; translated files are hand code and hand and agent edits are normal, with the twin edit history as evidence; the byte-identity gate is the correctness guard; a new section 4.8 lists what a first port adds to the tree (and that the translator change adds nothing under platform/cpp); facts keyed by qualified name with the stale warning; the self-test is described and is not run by CMake or CI; risks 3, 4 and 6 restated (the translator owns no file outside its directory and nothing in the build or CI depends on it), new risk 7; generator-owned output with a CI check listed as a rejected alternative; per-function updates and a drift report listed as future work on a trigger. HOWTO: the ownership paragraph and wording from generated to translated. Architecture page: the regen --check node becomes the hand-code stage, and the emit stage no longer says the output is never edited. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-2762) Appendix A lists the design changes from the first draft of the RFC and tool (generator-owned C++ with a CI currency check) to the porting assistant, and compares the risks one by one: which stayed, which changed and why, which version 1.0 did not list, and which this version accepts. It is for the review only and says to remove it before merge, as the RFC convention keeps version history in git. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… backward (AICK-2762) Every path the translator used was fixed to the attention backward package: modules under library/kernels/common, sources written to platform/cpp/instances/common, headers named helper_rocke.instances.common.<m>.h, parity drivers in library/tests/parity, and code maps mined from the *attention*bwd* files. The package entry now carries them (py_root, py_package, cpp_dir, header, parity_dir and the mining globs), and every reader takes them from there. platform_instance() builds the entry for a kernel instance of the platform (platform/python/rocke/instances/<arch>/<module>.py, instance_<module>.h, platform/tests/instances/parity). Entries for batched_transpose, transpose_bc, batched_contraction and the four gfx1250 Qwen3 instances are added. tr.record now names the missing driver instead of failing on import, and a module that is not in the selected package is an error that names the package. Attention backward is unchanged: the self-test passes (37) and the mined maps (sigmap, twin_map, owners) are byte-identical to the previous commit's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… instances (AICK-2762) A platform instance imports its helpers from a parent package (from ...helpers.spec import kernel_name_join). The translator kept that as "helpers.spec" and the map miner skipped it, so every such name was unknown. Both now read it by its absolute name (rocke.helpers.spec); the maps list the helpers a platform instance imports and their C twins. Attention backward has no such imports: its mined maps are byte-identical. $TR_PARITY points tr.record at a driver drafted outside the tree. The transpose instance (ported by hand) is a package, and the self-test records it from the real tree to check the platform paths (6 new tests, 43 pass). Checked by hand: the seven new instance packages record (every driver config builds except 1 of 5 for batched_transpose and 3 of 5 for qwen3_kv_cache, which the Python rejects), generate a header and analyze; 20 of 47 functions translate with no fill. The holes are translation rules, not paths. HOWTO.md gains a Packages section; the RFC's section on packages says where files live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
afagaj
added
documentation
organization: ROCm
project: hip-kernel-provider
rocke: platform
… not its successor (AICK-2762) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
✅ All Checks Passed — Ready for Review
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
…d strings (AICK-2762) RT-R1 now accepts a handler that binds the exception (`except KeyError as e:`). The name is bound for its handler only, as Python unbinds it, and str(e) lowers to the ckc exception's message, which is Python's str() of the exception it stands for (a KeyError's message is the repr of its argument, as every translated raise and adapter writes it). Other uses of the name (repr(e), e.args, ...) still refuse. py_adapt::arch_target now throws the KeyError with the exact Python text (rocke_set_unknown_arch_reason), so `return False, str(e)` around ArchTarget.from_gfx reports what Python reports. A std::string a function returns as const char* (an f-string or str(e) reason in a (bool, str) result) was returned as .c_str() of a local that dies with the function. A return value now goes through py::keep, a prelude function that interns the text (one copy per distinct value). No attention backward output changes (no golden returns such a string). Platform instances: _require_gfx1250 and the three _require_supported now translate (all.line_pct 12.4 -> 16.4, held 18.6 -> 25.4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on backward adapters (AICK-2762) Every translated source that used an adapter included rocke_py_adapters.hpp, package data of attention backward that includes _attention_bwd_frag.h and calls rocke_attn_bwd_eval_layout. So no platform instance source could compile as the translator writes it: it needs headers that only the backward port has. The adapters are split: - tr/prelude/rocke_py_adapters.hpp: the generic adapters, for any package. It needs only the installed rocKE headers and the prelude; the self-test now compiles it alone with g++ and clang++. - tr/packages/attention_bwd/rocke_py_adapters_attention_bwd.hpp: eval_layout and eval_layout_scratch, the two that need the backward headers. A source includes a package's own adapters header only when it calls one of its functions (Package.adapters_header / adapter_names). - py_adapt::iv_of names its int | Value type (iv_of<rocke_<prefix>_iv_t>); it defaulted to the backward union. - tr.regen also copies the support headers a support header includes. Attention backward: the valu golden gains one line (#include "rocke_py_adapters_attention_bwd.hpp", valu calls eval_layout); nothing else changes. Self-test: 51 passed. Checked by hand on the seven platform instances (no fills): none of the translated sources refers to attention backward, and the four Qwen3 sources compile (g++ -fsyntax-only) except for the #error lines of their holes. The other three fail only on their generated headers, which do not define the spec struct (a Literal dtype alias and a TileSpec field have no C type yet). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (AICK-2762)
Defect: the generated header used a struct it never declared. In
batched_transpose and transpose_bc, `dtype: DType` (DType = Literal["f16",
"bf16"] at module level) parsed as ('obj', 'DType'), so tr.abi refused the
spec struct ("no C type for ('obj', 'DType')") but still wrote
rocke_<p>_spec_t into the exported is_valid_spec and build prototypes. In
batched_contraction a field typed TileSpec (a class of
rocke.instances.common.gemm_universal) had the same effect.
How it showed: measure.py headers_compile 4/7 and cpp_compile 4/7; g++
"'rocke_batched_transpose_spec_t' does not name a type" (same for
transpose_bc and batched_contraction).
Fix:
- tr.abi: a Literal whose members are all str, written out or through a
module-level alias, is str (const char*), as the hand headers declare
dtype.
- tr.kernel: a class imported from another module whose header in the tree
has its C twin struct (the header's "class TileSpec rocke_gemm_tile_spec_t"
table, else its one rocke_*_<snake>_t struct) maps to that struct, held by
value. A generated source that names such a type includes its header.
- tr.abi guard: a struct, typedef or declaration that uses a rocke_*_t the
header neither defines nor finds in the tree (or as a package sibling's
name) is refused, not written; tr.translate --analyze reports the
function as a hole ("not exported: ... defined nowhere") and keeps it as
a unit so its callees are still listed.
- designs/generated_abi.md lists the two type rules and the guard.
Evidence: measure.py headers_compile 4/7 -> 7/7 and cpp_compile 4/7 -> 7/7,
all.line_pct 16.4 -> 16.4; batched_transpose and transpose_bc now export
the spec struct (const char* dtype), spec_default and block_size;
batched_contraction's spec struct is still refused (Tuple[int, ...] field)
and its is_valid_spec/build declarations are refused with it. New
tests/test_abi_types.py (fails 4/4 before, passes after); self-test 55
passed, attention backward goldens unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-2762) Defect: a translated internal function (anonymous namespace) whose public entry was not written, or a private helper only such a function calls, was still emitted. A static function with no caller fails -Wunused-function under -Wall -Werror, the flags the build uses. How it showed: the qwen3 instances' sources, built to an object with g++ -std=c++20 -Wall -Werror (hole #error lines stripped), failed with "defined but not used": qwen3_kv_cache py__quantize_for_kv_store and py__require_gfx1250, qwen3_qk_norm_rope py__require_supported, py__dtype_ir and py_Qwen3QkNormRopeSpec__pair, qwen3_sampler and qwen3_token_embedding py__require_supported and py__dtype_ir (their callers are holes). Before the previous commit, transpose_bc's py_TransposeBcSpec_block_size was the same (its struct, so its entry, was not exported). measure.py's -fsyntax-only check does not run the unused-function pass, so it did not see these. Fix: house style exports what the hand ABI exports (spec_default and properties such as block_size are now exported once the spec struct is, previous commit); what is left unused is not emitted: used_internals() keeps an internal only when an exported entry, a fill, a struct or another kept internal names it, until nothing changes. The record still says translated; a fill that calls the helper keeps it. Evidence: all seven instance sources now build to an object with -Wall -Werror (holes stripped); measure.py headers_compile 7/7, cpp_compile 7/7, all.line_pct 16.4 (unchanged). New tests/test_unused_internal.py (a dropped chain, a probe module whose hole's helper is not emitted, and an object build with -Werror); self-test passes, attention backward goldens unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nclude once (AICK-2762) Defect: a generated .cpp included rocke/helper_helper_rocke.helpers.spec.h next to rocke/helper_rocke.helpers.spec.h. Both declare rocke_kernel_name_join; the include scan mapped each called function to the first header (alphabetically) that declares it, so kernel_name_join pulled in helper_helper_rocke.helpers.spec.h while rocke_validate_io pulled in helper_rocke.helpers.spec.h: the same declaration came in twice, and the group1 hand ports had to drop the extra include by hand. Nothing kept a scanned header from repeating a fixed include either (error.hpp). How it showed: the group1 first drafts (ports/group1/*/regen_original) and a re-run now with their hand fills (TR_HOLES=ports/group1/<m>/holes): the transpose_bc and batched_transpose sources list both spec headers. Fix: tr.env Headers records every header declaring a function (func_files). The include scan takes the headers of types first, then of functions declared once, and a function several headers declare adds a header only when none of them is already included; error.hpp is never re-added and an include line already written is not written again. Evidence: the group1 re-run includes helper_rocke.helpers.spec.h only. New tests/test_includes_once.py (a probe whose fill calls both functions: fails before, passes after); self-test 61 passed, attention backward goldens unchanged; measure.py headers_compile 7/7, cpp_compile 7/7, all.line_pct 16.4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ort headers (AICK-2762) Defect: tr.regen --overwrite wrote over every existing target, including the shared support headers (rocke_py.hpp, rocke_py_adapters.hpp, rocke_py_adapters_<package>.hpp) in the package's C++ directory. Those headers serve every translated source there and may carry hand fixes, so regenerating one module silently dropped a hand fix to them. How it showed: the self-test's test_overwrite appended a hand edit to every target, ran --overwrite and asserted the whole tree came back equal to the first draft, support headers included (it passed on the old code: the hand edits to the three support headers were lost). Fix: --overwrite replaces only the named modules' sources and headers; an existing support header is written over only with the new --overwrite-support flag (alone, or with --overwrite for everything). The early "every target exists" exit is skipped when either flag is given. HOWTO.md and the tr.regen docstring say so. Evidence: test_overwrite now checks the support headers keep their hand edit and the sources and headers are replaced; test_overwrite_support checks the new flag replaces only the support headers, and both flags together restore the first draft; the pretend test counts the new set. Self-test 62 passed; measure.py unchanged (all line_pct 16.4, headers 7/7, cpp 7/7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…AICK-2762)
Defect: the tr.translate --analyze summary counted every hand-filled
function as a hole ("N/M functions mechanical, K holes" with K = holes +
fills), and each filled line printed the failed rule's reason as if it were
a hole's. tr.regen counted the same functions its own way ("x/n
mechanical, f hand fills, h holes"), so the two reports disagreed, and
neither said that derived C-only companions (_free, _default) are part of
the mechanical count.
How it showed: batched_transpose with the group1 hand fills
(TR_HOLES = a copy of ports/group1/batched_transpose/holes) printed three
"filled" lines and "2/5 functions mechanical, 3 holes" from --analyze,
while tr.regen said "2/5 mechanical, 3 hand fills, 0 holes".
Fix: one tally of the four statuses (translated, derived, filled, hole) and
one summary line in tr/translate.py, used by --analyze and tr.regen:
"<m>/<n> functions mechanical (<t> translated, <d> derived), <f> hand
fills, <h> holes". A filled line says "(hand fill for: <reason> at line
<n>)". The per-function line format is otherwise unchanged. HOWTO.md
documents the statuses and the line.
Evidence: batched_transpose with the fills now reports "2/5 functions
mechanical (1 translated, 1 derived), 3 hand fills, 0 holes" from both
tools. tests/test_reports.py translates a probe module with one
translated, one derived and one filled function and checks the
per-function lines and that --analyze and tr.regen print the same summary
(on the old code it fails: --analyze counts the fill as "1 holes").
Self-test 64 passed; goldens unchanged; measure.py unchanged (all
line_pct 16.4, headers 7/7, cpp 7/7).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…K-2762) Defect: when tr.abi refuses a struct, function or constant (no C type for it), the generated header lacks it, but no report said so. The refusals were printed only as "not exported: ..." lines on stderr by tr.translate's install_abi (also for every other generated module it loaded), tr.abi itself printed nothing, and the --analyze and tr.regen summaries counted only functions: a module whose header lacks its grid and signature entries reported "0 holes" and tr.regen exited 0, so a user could not tell the header was incomplete. How it showed: batched_transpose with the group1 hand fills: --analyze ended "2/5 functions mechanical, 3 holes" and tr.regen "2/5 mechanical, 3 hand fills, 0 holes", with batched_transpose2d_grid and batched_transpose2d_signature (no type for the return) missing from the header and named only on stderr; batched_contraction had 14 refusals (BatchedContractionSpec twice) and no summary mention. Fix: translate() keeps its own module's refusals in Result.abi_refused (one entry per name, reasons joined), install_abi no longer prints. The summary line shared by --analyze and tr.regen ends ", <r> ABI refusals"; --analyze lists each as "refused <name> (not exported: <why>)" (a line measure-style parsers of translated|derived|filled|hole do not count); tr.regen lists them under "ABI refusals: the generated headers lack these"; tr.abi prints "not exported: ..." and "<r> ABI refusals" on stderr. Exit codes are unchanged (a refused function that is called is still a hole). HOWTO.md documents it. Evidence: batched_transpose now ends "2/5 functions mechanical (1 translated, 1 derived), 3 hand fills, 0 holes, 2 ABI refusals" in both tools and lists both. tests/test_reports.py adds a probe module with a function whose list[str] | None parameter has no C type and checks tr.abi, --analyze and tr.regen all report the one refusal (on the old code tr.abi prints nothing); test_fully_mechanical now also requires "0 ABI refusals" for valu and counted. Self-test 66 passed; goldens unchanged; measure.py unchanged (all line_pct 16.4, headers 7/7, cpp 7/7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ABI (AICK-2762)
Defect: for a platform instance, tr.abi mapped is_valid_spec's
Tuple[bool, str] return to a translator tuple struct
{bool f0; const char* f1} and, because the function may raise, added a
char* reason / size_t reason_cap buffer as well. The owner of f1 was
unclear (the translated body returns a py::keep'd string or a literal),
a raise returned a zeroed struct indistinguishable from (False, NULL), and
no other instance_*.h has this shape: every one returns bool and fills the
caller's reason buffer (instance_batched_transpose.h of the reviewed hand
port is the model).
How it showed: the generated instance_batched_transpose.h and
instance_transpose_bc.h declared rocke_<p>_tuple_bool_str_t
rocke_<p>_is_valid_spec(..., char* reason, size_t reason_cap)
(instance_batched_contraction.h carried the struct, its is_valid_spec
being refused); the group-1 hand fill had to invent where f1 points.
Fix: a package flag, ok_reason_abi (on for platform_instance() packages,
off for attention backward). With it, a function returning
tuple[bool, str] (is_valid_spec and any function of the same shape) is
declared bool f(..., char* reason, size_t reason_cap). Its generated entry
calls the internal, copies the str into reason (bounded, NUL-terminated)
and returns the bool; a raise is false plus its text through
py_adapt::with_reason, as for the other reason entries. The rule is
added to designs/generated_abi.md.
Evidence: the generated batched_transpose and transpose_bc headers now
declare bool rocke_<p>_is_valid_spec(const spec_t*, const char* arch,
char* reason, size_t reason_cap), matching the hand port. New
tests/test_ok_reason_abi.py (a probe appended to the real
batched_transpose module through a symlink overlay, helper in trtest.py)
checks the declaration, the entry body and that the source compiles with
-Wall -Werror. Self-test 70 passed, goldens unchanged (attention backward
does not change). measure.py: all line_pct 16.4 (unchanged), headers 7/7,
cpp 7/7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry (AICK-2762) Defect: a function that calls a function filled by hand (holes/<module>/<f>.cpp) was still refused whole with "call to X, which is not translated" (for a method: "method C.m is not translated"). The lowering only knew the internal (py_*) version of a module function, and a filled function has none; but the fill defines the function's C entry, declared in the module's header with a known signature, so the caller can call that. How it showed: with TR_HOLES at a copy of the group-1 batched_transpose holes minus the build fill, --analyze printed "hole build_batched_transpose2d (call to is_valid_spec, which is not translated at line 170)" although is_valid_spec (and kernel_name) were listed as filled; transpose_bc the same at line 304. Fix: lower.call_filled. When a module function, or a method of a non-family class, has no translation but has a fill and a C entry in the module's own header, the call goes through that entry the way a call into another module's generated C twin does (c_args + gen_call, the receiver as self). gen_call learns the (ok, reason) validity-gate ABI of the previous commit: role reason_str gives back the Python tuple (bool, std::string(reason)) and does not raise; a plain reason entry still raises its text. Without a fill nothing changes (the caller stays a hole). HOWTO.md (Holes and hand fills) says so. Evidence: with the group-1 fills, build_batched_transpose2d and build_transpose_bc now get past is_valid_spec and kernel_name and are refused for a different, real reason: helper io:io_ir_type has no sigmap entry (line 174 / 308). New tests/test_call_filled.py: a probe (probe_build calls the filled probe_valid and the filled BatchedTranspose2DSpec.kernel_name) translates; its C++ follows the Python line by line (the validity call into reason_1, the tuple (ok, why), the raise "invalid probe spec: " + why, then kernel_name into a buffer, then the return) and compiles with -Wall -Werror; without the fill the caller is still a hole. Self-test 74 passed, goldens unchanged. measure.py (no fills): all line_pct 16.4 (unchanged), headers 7/7, cpp 7/7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…AICK-2762) Rule: a call of a function imported from another rocke module (rocke.helpers.spec, rocke.helpers.reduction, ...) that the package's own helper map does not list now falls back to the sigmap's external_helpers entry, when it has a C function twin. The C parameters no Python argument feeds are re-tagged from the C declaration alone: - `char* X` then `size_t X_cap` (or out_cap) for a Python `-> str`: the result through a caller buffer (the existing str_out / str_cap roles), and a following `size_t* X_len` is passed NULL; - a sequence argument passed as None (flags=None): the C pointers and the count that go with it are NULL / 0; - *parts of both str spellings collect as owned strings, passed as a C string array of pointers into that (named) list. mine_maps records each external helper's return annotation. Evidence: kernel_name_join(self.name, self.dtype, f"..", ...) in all seven instance modules becomes rocke_kernel_name_join(self->name, parts, n, nullptr, nullptr, 0, buf, sizeof(buf), nullptr), as the hand port of batched_transpose calls it; the C twin skips NULL / empty parts as Python drops falsy ones. kernel_name translates in all seven modules. test_call_filled's filled method keeps its purpose through a flags dict (still untranslatable); new tests/test_platform_helpers.py. Attention backward goldens unchanged. Score (all / tune / held line_pct): 16.4 / 3.0 / 25.4 -> 22.3 / 7.5 / 32.1; headers 7/7, cpp 7/7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r helpers (AICK-2762) Rule: a helper whose C twin returns its Python tuple through out parameters may also return a success flag (int / bool, 1 on success, 0 with the builder's sticky error set) when it takes the builder: the call becomes `if(!f(..., &a, &b)) py_adapt::raise_builder(b);`, so the builder's error is raised where Python raises. A `Tuple[...]` return annotation counts as a tuple like `tuple[...]`. Evidence: block_lds_reduce_with_index(b, ..., block_size=bs, combine="argmax") in qwen3_sampler becomes rocke_block_lds_reduce_with_index(b, ..., bs, ROCKE_INDEX_ARGMAX, &val, &idx) with the check (the C twin returns 0 on a bad combine, non-f32 value, non-i32 index or non-power-of-two block). New test in test_platform_helpers.py. Attention backward goldens unchanged. Score: unchanged this commit (22.3 / 7.5 / 32.1 all / tune / held line_pct); the sampler build still stops at b.param keywords, the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…_t (AICK-2762) Rule: b.param(name, t, noalias=True, readonly=True, align=16) (explicit keywords, no **attrs) declares a rocke_param_opts_t, sets the field of each keyword's name in Python's order and, when the struct has one, its `<name>_set` flag, then passes &opts to rocke_b_param. The C builder adds only the flagged attributes, as Python's dict(**attrs) holds only the keywords passed; the IR printer sorts attributes, so the field order does not change the text. A keyword with no field (or a `_set` name), a non-bool value for a bool field, or a temporary string is refused. Evidence: every instance build function declares its pointer params this way; the output matches the hand port of batched_transpose (opts.noalias = true; opts.noalias_set = true; ... align_set = true). qwen3_token_embedding and qwen3_sampler now translate whole (their build functions compared line by line with the Python). New test in test_platform_helpers.py. Attention backward goldens unchanged. Score (all / tune / held line_pct): 22.3 / 7.5 / 32.1 -> 32.7 / 7.5 / 49.3; headers 7/7, cpp 7/7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to its C singleton (AICK-2762)
Rule: an IR scalar type constant imported from rocke.core.ir that the
fixed table does not list (FP8E4M3, BF8E5M2, TF32, ...) is
`NAME = Type("n")` in the Python; it lowers to rocke_n() when a header
declares that function with no parameters and a const rocke_type_t*
return. The same lookup names a Type value folded from the module.
Evidence: qwen3_kv_cache's _dtype_ir (FP8E4M3 / BF8E5M2) now translates
to rocke_fp8e4m3() / rocke_bf8e5m2(), the interned singletons ir.h
declares; its two build functions now stop at attention helpers with no
C twin rule (dequant_bf8x8_to_dtype, PagedKvDescriptor) instead. New test
in test_platform_helpers.py. Attention backward goldens unchanged.
Score (all / tune / held line_pct): 32.7 / 7.5 / 49.3 -> 33.7 / 7.5 / 51.1;
headers 7/7, cpp 7/7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CK-2762) Rule: +, -, * and / with a float operand and a float or (non-bool) int operand are double arithmetic; the int converts to double first, as Python converts it to float (both IEEE doubles, round to nearest). / is py::truediv (new in the prelude), which raises Python's "float division by zero" text (as ValueError, the prelude's ZeroDivisionError convention). Floor division and modulo of floats stay refused. Evidence: qwen3_qk_norm_rope's `b.const_f32(1.0 / H)` becomes rocke_b_const_f32(b, py::truediv(1.0, (double)H)); that build now stops later, at `out = [None] * H` filled with Values. New tests in test_platform_helpers.py (the lowering, and the prelude helper built and run with g++). Attention backward goldens unchanged. Score: unchanged this commit (33.7 / 7.5 / 51.1 all / tune / held line_pct); headers 7/7, cpp 7/7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (AICK-2762)
Rule: obj.attr / obj.method(...) on a platform struct that has no such C
field now also finds the C twin a header documents for it (PM-R1, widened):
- the prefix may come from the header's Python -> C table row
"<struct>_t + <prefix>_* () getters" (rocke_arch_target_t + rocke_arch_*),
besides rocke_<stem>_ and "-> <struct>_t and <prefix>_*";
- the doc comment above it may start "<Class>.<attr>" ("ArchTarget.fits_lds.",
"ArchTarget.max_threads_per_block property.");
- an attribute is a call of that getter (self its only parameter) when the
Python member, found through the class the doc names and the module's
(module or function level) imports, is a property;
- a bool return with no builder parameter is the result (not an error flag)
when the Python method is annotated -> bool.
Evidence: transpose_bc.is_valid_spec translates: target.max_threads_per_block
becomes rocke_arch_max_threads_per_block(target), target.wave_size stays the
field; it matches the hand port (rocke-transpose-twins transpose_bc.cpp) and
keeps every reason string. batched_transpose's target.fits_lds(lds_bytes)
becomes rocke_arch_fits_lds(target, (long)lds_bytes). New probe_arch test in
test_platform_helpers.py. Attention backward goldens unchanged.
Score (all / tune / held line_pct): 33.7 / 7.5 / 51.1 -> 38.7 / 19.9 / 51.1;
headers 7/7, cpp 7/7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d takes Tuple[bool, ...] helper results (AICK-2762) Rules (the validate_io(IOSpecRule(...)) chain every small-op is_valid_spec uses; they share the sigmap external_helpers path): - an imported class of another rocke module whose sigmap entry has a C struct twin (IOSpecRule -> rocke_io_spec_rule_t) is constructed like a platform class. When fields left out have no literal default (None, a tuple), the C init is used if its parameters after the struct are exactly the dataclass's required fields in order (rocke_io_spec_rule_init(rule, dtype, block_size, vec): "the dataclass defaults and the three required fields"); each other field passed is then set, only a scalar C field of the same name, an Optional one through its <field>_set flag (None leaves the init's unset). A sequence field (allowed_vecs) is refused: it would point at a temporary. - a builder-free helper annotated Tuple[bool, ...] whose C twin returns int / bool and the other items through out parameters: the C return is the bool, the out parameters the rest; a C string result is copied into an owned string at once. A `rocke_arena_t*` parameter of such a helper is a py_adapt::scratch_arena of the caller's scope (new in rocke_py_adapters.hpp, freed by RAII; a failed init is Python's MemoryError). - typing.Tuple[...] is read as tuple[...] in annotations (tr.ctypes and tr.abi), so Tuple[bool, str] / Tuple[int, int, int] returns and Tuple[int, ...] fields get their C types without recorded facts. Evidence: batched_transpose.is_valid_spec translates: IOSpecRule(dtype=, block_size=spec.block_size, vec=) becomes rocke_io_spec_rule_t + rocke_io_spec_rule_init after spec.block_size is evaluated (Python order), then rocke_validate_io(&arena.a, &rule, &reason) != 0 with the reason copied, as the hand port (rocke-transpose-twins batched_transpose.cpp) does with its own arena. The typing.Tuple reading exports the *_grid functions (batched_transpose2d_grid translates: floor division, the batch count passed through) and BatchedContractionSpec, so the function and line counts grow. test_call_filled's probe_valid stays untranslatable through a sequence field (allowed_vecs); new probe_io tests. Attention backward goldens unchanged. Score (all / tune / held line_pct): 38.7 / 19.9 / 51.1 -> 44.1 / 32.2 / 52.0 (functions 49 -> 58); headers 7/7, cpp 7/7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… field (AICK-2762) Rule: <struct>_default() of a dataclass left every default_factory field zeroed. That is right only for an empty list / tuple / dict; for a factory that builds a value (trait: TraitSpec = field(default_factory=TraitSpec)) the field is now set through its struct's own C _default() (no parameters, returning the field's struct), and with none the class gets no derived default: a hole to fill rather than zeros the Python never produces. Evidence: the previous commit's typing.Tuple reading exports BatchedContractionSpec, and its derived default left trait all zeros (TraitSpec's pipeline, split_k = 1, ... lost). rocke_gemm_trait_spec_t has no C default, so rocke_batched_contraction_spec_default is now a hole. New tests/test_derived_default.py. Attention backward goldens unchanged. Score (all / tune / held line_pct): 44.1 / 32.2 / 52.0, unchanged (the default is a C-only companion with no Python lines; held functions mechanical 33 -> 32); headers 7/7, cpp 7/7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…structs (AICK-2762)
Rule: a C struct that holds a Python int | Value is recognised from its
header alone: the struct has exactly one by-value constructor from an int
and one from a rocke_value_t* (rocke_stride_t with rocke_stride_imm /
rocke_stride_value in helper_rocke.helpers.tensor_view.h). An int, a Value
or an int | Value converts into it through those constructors, and a fixed
tuple of them into a `const S*` parameter becomes a local C array of S. A
tuple display of mixed ints and Values keeps its items, so each one is
converted from its own type instead of through the package's int | Value
union (which no platform-instance header declares).
Evidence: make_global_view(X, shape=.., strides=(batch_stride_x, W, 1)) in
build_batched_transpose2d now lowers to
const rocke_stride_t strides_N[3] = {rocke_stride_value(batch_stride_x),
rocke_stride_value(W), rocke_stride_imm(1)};
as the hand port writes it. New probe test probe_strides
(test_platform_helpers.py) covers a mixed tuple and an all-int tuple.
Attention backward goldens unchanged; self-test 95 passed.
Score: unchanged (all 44.1, tune 32.2; 7/7 headers and objects). The
function's next hole is TensorView.tile, which the next commit covers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r C twins (AICK-2762)
Rule: obj.method(...) on the C struct of a rocke.* class (rocke_<stem>_t,
no class of this module) goes through rocke_<stem>_<method> when that
function is declared in helper_<python module>.h, has exactly one
parameter pointing to the struct (self, in any position:
rocke_tensor_view_tile(out, view, ...), rocke_tile_window_load_vec(b, w,
...)) and its doc comment names <Class>.<method>. The Python method is
read from that module; its parameters (minus self) are paired with the C
ones by mine_maps' slot_map, as a helper's are: by name, then position;
out, count and builder parameters by role. So *local_indices becomes the
value array and its count, an out struct the result, a rocke_status_t
return the ValueError. call_helper's body is now call_c_twin, shared by
both; c_args takes the caller-supplied parameters (self) anywhere in the
list.
Also, from the same shared path: a Python argument the C twin has no
parameter for (make_lds_view(exclusive=...), rocke_make_lds_view has none)
was dropped silently; it now translates only when it is the default, else
the function is a hole.
Evidence: build_batched_transpose2d translates whole. Compared with the
Python and with the hand port (rocke-transpose-twins batched_transpose.cpp
bt_build): same C calls (rocke_make_global_view, rocke_tensor_view_tile,
rocke_make_lds_view, rocke_tile_window_load_vec / store_vec /
load_scalar, rocke_b_vec_pack), every IR-emitting call its own statement
in Python's order (the lds origin constants before the tile, the index
constants before each load/store, the scalar loads inside the vec loop).
New tests in test_rocke_methods.py (methods, the exclusive refusal, the
probe compiles with -Wall -Werror). Attention backward goldens unchanged;
self-test 101 passed.
Known difference on error paths only (the convention call_helper already
had): a failing C status raises ValueError("<method> failed") where Python
raises its own rank-mismatch text.
Score: all 44.1 -> 54.4, tune 32.2 -> 58.0, held 52.0 -> 52.0;
headers 7/7, objects 7/7 (c3_impl.json).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e differently (AICK-2762) Defect: the include picker took a type's header from typedef_file, the first header (alphabetically) that defines it. rocke_tensor_descriptor_t is two different structs, in helper_rocke.helpers.tensor_view.h and in helper_rocke.helpers.transforms.h; a source that names the type and calls transforms.h functions got both headers. How it showed: the e2e port of transpose_bc against the PR #14538 headers. A build_transpose_bc fill that spells its descriptors rocke_tensor_descriptor_t* (and calls rocke_tensor_descriptor_naive / _transform / _unmerge_lower of transforms.h) regenerates with both includes; g++: "redefinition of 'struct rocke_tensor_descriptor'". The fill had to write auto* to dodge it. Fix: Headers records every header that typedefs a name and the bodies of the complete definitions. A type several headers define comes from a header the source already needs (the one of the functions it calls with it), else from the first that adds no conflicting definition. If two headers the source needs still define a name differently, that is an include conflict: tr.translate lists "conflict <type> (...)", writes an #error in the source and exits 1, and tr.regen writes nothing. A type defined in one header is picked as before (goldens unchanged). Evidence: tests/test_include_conflicts.py (4 tests; all 4 fail before the fix): a transforms.h call picks transforms.h only and the source compiles; calls into both headers are a reported conflict and regen writes nothing. The e2e transpose_bc regen with the typed fill now includes transforms.h only. Self-test 105 pass; measure.py unchanged (all 54.4%, 7/7, 7/7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eturns ok (AICK-2762) Defect: in the hand-ABI shim (translating against an existing header), a Python (ok, reason) function behind a C `bool f(..., char* reason, size_t reason_cap)` became `return (bool)py_is_valid_spec(...);` inside py_adapt::with_reason: a std::tuple cast to bool, which does not compile, and the reason is never written. The generated-ABI shim already had the right body. How it showed: the e2e regen of batched_transpose and transpose_bc against the PR #14538 headers (rocke_*_is_valid_spec); g++: "invalid cast from type 'std::tuple<bool, const char*>' to type 'bool'". Hand edit needed in both. Fix: one routine, ok_reason_body(), builds the gate's body (the reason copied into the buffer, truncated and NUL-terminated, ok returned) and both make_shim (hand ABI) and make_gen_shim (generated ABI) use it. Any other tuple result behind a reason entry is refused as an ABI hole instead of a cast that cannot compile. The generated-ABI output is unchanged. Tests: tests/test_hand_abi_shims.py translates batched_transpose against the reviewed instance_batched_transpose.h of PR #14538 (fixtures/hand_abi/, put into a symlink copy of the tree by trtest.overlay_root): the gate writes the reason and returns ok, both paths share the routine, the source compiles (-Wall -Werror). All 3 fail before the fix. trtest.strip_holes also strips an #error that clang-format wrapped after padding. Evidence: self-test 108 pass (goldens unchanged); measure.py unchanged (all 54.4%, headers 7/7, cpp 7/7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… counts ABI holes (AICK-2762) Defect: translating against an existing header (hand ABI), a public function whose Python int (int64_t inside) the header returns as C int got '#error "abi hole: <c>: shim: Python return int64_t as C int"'. --analyze reported it as translated (0 holes) and tr.regen wrote the file anyway, a source that cannot compile. How it showed: the e2e regen of batched_transpose and transpose_bc against the PR #14538 headers (rocke_batched_transpose2d_block_size, rocke_transpose_bc_block_size): "0 holes", files written, #error in both; a hand edit was needed in both. Fix: (a) make_shim narrows an int64_t result to a C int / int32_t with a range check. A builder entry raises on the builder when out of range (its existing convention, -1). Any other entry takes the failure value its header documents (documented_fail reads the comment above the declaration, e.g. "Returns 0 for a NULL spec or vec == 0"): a NULL pointer argument the comment names, a Python raise (ckc::Error) and an out-of-range result all return it. A header that documents no failure value leaves an ABI hole. (b) An ABI hole is a hole of the report: status "hole", reason "abi hole: ...", counted by --analyze and the summary; tr.translate exits 1 and tr.regen writes nothing while one is open. A hand fill of the function replaces its C entry (status "filled"), as for any refused function. Tests (tests/test_hand_abi_shims.py, against the reviewed header in fixtures/hand_abi/): the narrowing follows the header (NULL guard, range check, 0 on a raise); with the comment removed the entry is a hole that --analyze counts, translate exits 1 and regen writes nothing; a fill replaces it; documented_fail takes only failure values ("Returns 1 if the spec is valid" is not one). 5 of these fail before the fix. Evidence: e2e regen (scratch copy) of both modules: block_size entries generated, both sources pass g++ -std=c++20 -Wall -Werror -fsyntax-only. Self-test 113 pass (goldens unchanged); measure.py unchanged (all 54.4%, headers 7/7, cpp 7/7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ICK-2762) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cgmillette
pushed a commit
that referenced
this pull request
Oct 30, 2026
…-2858) (#14706) ## rocKE: buffer address space in the shared C++ tensor view ISSUE ID : AICK-2858 Three commits (feature, tests, then a review fix); please use "Rebase and merge". ### Why The Python tensor view supports global, LDS and buffer views. Its C++ twin only supports global and LDS: `rocke_tensor_view_t` has nowhere to hold a `BufferResource`, so every `ROCKE_ADDR_BUFFER` branch in `cpp/helpers/tensor_view.cpp` is a "NAMED GAP" that returns NULL, and `make_buffer_view` has no C++ twin. That has three costs: - **Duplicate code.** Seven C++ instances that need buffer access roll their own private buffer-resource structs and shims (pooling, img2col, conv_implicit_gemm wgrad and build glue, deep_fused_conv_pool on gfx950 and gfx1201). Each copy has to be kept in step with Python by hand. - **New twins.** A Python instance that calls `make_buffer_view` (today `transpose_bc`, #14538) cannot use the shared helper in its C++ twin. - **The translator** (#14534) can port Python helpers into their C++ twin, but not `make_buffer_view`, because there was no struct to fill. ### What this does | Python (`helpers/tensor_view.py`) | C++ twin | |---|---| | `BufferResource` | `rocke_tv_buffer_resource_t` | | `make_buffer_resource` | `rocke_tv_make_buffer_resource` | | `make_buffer_view` | `rocke_tv_make_buffer_view` | | `TensorView.buffer` | `rocke_tensor_view_buffer` | | buffer branches of `load_scalar`, `store_scalar`, `load_vec`, `store_vec`, `load_vec_at`, `store_vec_at` | same functions, branch bodies filled in | | `load_vec_at(..., mask=)`, `store_vec_at(..., mask=)` | `rocke_tensor_view_load_vec_at_masked`, `rocke_tensor_view_store_vec_at_masked` | - `rocke_tensor_view_t` gains one field, `buffer`, at its end. All zeros means "no buffer", and it is only read when `addr_space == ROCKE_ADDR_BUFFER`. The existing view factories zero it. - Every port follows Python line for line: same builder calls, same order. Python raises become builder errors with the same type and message. - The struct grows, so `ROCKE_ABI_VERSION` (and `portable_ir` `BINARY_ABI`) goes from 4 to 5, as `abi.h` requires. - The `rocke_tv_` prefix avoids a clash with pooling.cpp's private `rocke_buffer_resource_t`, which has a different layout. ### Existing lowerings do not change The global and LDS branches are untouched, and nothing in develop builds a buffer view through the shared helper today. To prove it, the full gate set was run on develop first and then on this branch: | Check | develop | this branch | |---|---|---| | Byte-identity gate, llvm22 and llvm20 | 85 + 4 (gfx1250 @ llvm23) families, 2,010 configs GREEN | 86 + 4 families, 2,115 configs GREEN (+ `tensor_view_buffer`, 105 configs) | | Byte-identity gate, llvm23 | 88 families, 1,897 configs GREEN | 89 families, 2,002 configs GREEN | | clang `-Wall -Werror` build of `rocke_core` | pass | pass | | ctest | 33/33 | 34/34 (+ `rocke_tensor_view_buffer_errors`) | | Differential golden check | OK | OK; only the new family's hashes are added, no existing line changes | | Python suites (platform runner, both backends; library CPU lane under `ROCKE_BACKEND=both`) | 5,796 platform tests pass; library 66,803 tests, 39 fail | 5,811 pass (+15 new); library: same 66,803 tests, the same 39 fail (none related) | Numbers are for the current develop (`b0f07f86451`) and this PR head. Full test plan: #14706 (comment) Full results (develop vs this PR, including the Python platform and library suites): #14706 (comment) ### Tests - Parity family `tensor_view_buffer` (`tests/instances/parity/tensor_view_buffer_emit.{py,c}`): 35 cases on gfx950, gfx1151 and gfx1201. 17 accepted f16 cases cover every ported path, with default and explicit soffset, packed and runtime strides, with and without a mask; all byte-identical. 18 rejected cases hit each Python raise (unsupported dtype, width outside {2, 4, 8}, mask on a global view, buffer view with no resource). - `test_tensor_view_buffer_errors` (`.py` and native `.cpp`) checks that both engines raise the same error kind and message for each rejected case, since the gate only checks that both reject. ### Not in this PR - Moving the seven instances onto the shared helper (follow-ups, one instance at a time). - `load_scalar_at`, `store_scalar_at`, `async_load_lds_at`, `load_vec_tr_at`: these have no C++ twin for any address space yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rocKE: Python to C++ translator as a porting assistant (version 2.0)
ISSUE ID : AICK-2762
Draft. An experiment: an alternative design to the translator in #14447 (version 1.0). Both stay open so the two can be compared; which one goes forward, if either, is for the review to decide. 35 commits, meant to be read one by one; please use "Rebase and merge". The first three are #14447's; the rest change the design, open the tool to platform kernel instances, fix its defects and widen its rules.
What this does
Adds a translator that writes the first draft of the C++ twin of a rocKE Python kernel-builder module from the Python itself. It reads the Python module, the types recorded from its parity configs, and the C headers of the modules it calls, and writes the module's
.cppand its C header. Every function either translates by a fixed rule or becomes a hole that a person fills (holes/<module>/); the translator writes nothing while a hole is open.From then on the C++ is ordinary hand code, like every other twin: people and agents edit it,
tr.regennever writes over an existing file (--overwriteto replace,--pretendto preview), and the byte-identity gate (platform/tools/check_byte_identity.py) stays the only acceptance test.The design is in the RFC added here:
platform/dsl_docs/architecture/python_cpp_translator_rfc.md. Its Appendix A lists every change from version 1.0, with the reasons, and is meant to be removed before merge.What changed from version 1.0 (#14447)
tr.regenwrites over them--overwriteto replacetr.regen --checkin CIrocke_py.hppplatform/cpp/include/rocke/platform/cpptools/translator/tests(pytest, 113 tests, about 85 s, run by hand)library/kernels/commonThe commits
tr.regenwrites a first draft and never overwrites; RFC and HOWTO; RFC Appendix A (remove before merge)platform/python/rocke/instances/<arch>/) can be read; parent-package imports;$TR_PARITYrocke_py_adapters_attention_bwd.hpponly when used);except KeyError as e--overwritekeeps the shared support headers; reports count fills, holes and ABI refusals correctly; the platformis_valid_specABI isboolplus a reason buffer; callers of hand-filled functions translaterocke.*helpers, their out-parameter results and class methods through their C twins;b.paramkeywords; IR scalar type constants; float arithmetic; documented C getters of platform structs;rocke.*dataclasses through their C init;int | Valuestructsint(an unresolved ABI mismatch is now a counted hole that blockstr.regen); the validity gate writes its reason; never two headers that define one type differently; formattingWhat this does not do
transpose(ported by hand; the self-test records it to check the platform paths),batched_transpose,transpose_bc,batched_contractionand four gfx1250 Qwen3 instances. The two transposes were ported with the translator in a separate PR (feat(rocke): C++ twins of the batched_transpose and transpose_bc instances #14538); the others are recorded and analyzed, not ported.Results
valu,counted,auxandaddrare byte-identical to the Python on all 31 attention families, 3,083 configs per LLVM flavor, gcc and clang. Over the eight larger backward modules, 77.0% of Python lines (3,945 of 5,122) translate mechanically; hand fills count as not mechanical.rocke_corebuilds with clang-Wall -Werror. addr needs 4 hand fills (6 before: its twoRowSlabloads are now generated); valu, counted and aux need none.-Wall -Werror.tr.regen. Of the final code, 75% comes from the translator's rules and 25% from one hand fill (build_transpose_bc, whose buffer and tensor-coordinate helpers have no C twin). The first port with this tool's early version: 4% rules, 64% fills, 32% hand edits.Checks on this branch (
developat 56ad847)Size
About 45.4k added lines in 69 files; about 21.6k of them are frozen test fixtures (pinned Python modules, recorded types, cut-down code maps, goldens).
How to try it
From
platform/tools/translator/, seeHOWTO.md: setup, packages, the one command, holes and fills, the self-test, the gate, and the current limits.🤖 Generated with Claude Code