Phase 10A/10B Inc 0: build system, handle model, first module surface

Builds n3xd-ocp end to end and publishes 7.9.3.1.dev1 to the Gitea registry,
where it installs anonymously and passes its suite.

- occt/Dockerfile: OCCT 7.9.3 compiled once into a manylinux_2_28 builder
  image (base digest + tarball sha256 pinned), Draw/VTK/Tk/Xlib/OpenGL off,
  FreeType on, -O2 without fast-math or march=native. A final layer asserts
  TKService/TKV3d exist with no libGL/libX11 DT_NEEDED, which is what lets the
  app image drop libgl1/libx11-6. Mounted into, never built FROM.
- scikit-build-core + nanobind STABLE_ABI -> one cp312-abi3 extension that
  registers every OCP.* submodule via PyImport_AddModule, so `import
  OCP.TopoDS` needs no shim and cls.__module__ is right. Version <occt>.N is
  asserted against the OCCT found, keeping occt_version() truthful.
- occt_handle.h: type caster for opencascade::handle<T> over OCCT's intrusive
  refcount. Wrappers are non-owning instances holding exactly one handle in
  their keep-alive list, reusing an existing wrapper so identity survives a
  round trip. Transient constructors go through ocp_new (never nb::init<>,
  which would let OCCT delete nanobind's storage); the caster refuses a
  refcount-0 object rather than corrupt the heap. Verified under ASAN with no
  memory-safety errors, plus an RSS bound over 50k create/destroy cycles.
- Sub-shapes are returned by value everywhere, making the TShape lifetime class
  that segfaulted a process-global face memo unrepresentable.
- Standard_Failure derives RuntimeError, with ~20 concrete types dispatched on
  the dynamic OCCT type (cad_pool marshals failures home by type name).
- Inc 0 surface: gp subset, TopAbs, TopoDS (+ downcasts), TopExp, TopLoc,
  TopTools, BRep, BinTools, Poly, Standard. 34 of the app's 139 symbols.
- n3xd_ocp: additive APIs kept out of the OCP namespace so parity testing stays
  meaningful. bintools (shape <-> bytes, GIL-free, byte-identical) and _debug.

Two findings worth the record, both verified against the stock wheel rather
than assumed: upstream binds __hash__ but leaves __eq__ at identity, which is
exactly what geom_memo.py's hash-bucket + IsSame scan is built around, so we
match it instead of "fixing" it; and BinTools can release the GIL after all, by
slurping the file-like object instead of bridging a streambuf that would call
back into Python.

Gate: BREP round-trips are byte-identical to cadquery-ocp-novtk across six
fixtures (the generator asserts stock idempotency first). That matters beyond
IPC — derive.py content-addresses BREP payloads by sha256 and stores the ref.
This commit is contained in:
2026-08-10 16:10:28 +02:00
parent b757d6e8d6
commit 6139852768
58 changed files with 3446 additions and 13 deletions

99
docs/adding-symbols.md Normal file
View File

@@ -0,0 +1,99 @@
# Adding symbols
The routine task: the app needs an OCCT class this binding does not expose yet.
Read [design.md](design.md) first if you are touching the machinery instead.
## 1. Find what is missing
```bash
python tools/inventory.py --emit # re-scan the app for OCP usage
python tools/inventory.py --check # what the installed wheel lacks, by module
```
`--check` groups gaps by module, which is how increments are scoped. Note it
only sees symbols reached through an import (`TopExp.MapShapes_s`), not methods
called on instances (`shape.IsSame(...)`) — a green `--check` is necessary, not
sufficient. Running the app's own tests in the parity venv is what catches the
rest, loudly, as an `AttributeError`.
## 2. Write the module
One file per OCP module: `src/modules/mod_<Name>.cpp`.
```cpp
#include "../common/occt_module.h" // brings in the handle caster and "_a"
#include "../common/occt_policies.h" // OCP_RETURN_COPY, OCP_NOGIL
#include <Some_Class.hxx>
void register_Some(nb::module_ &root) {
nb::module_ m = ocp_submodule(root, "Some");
nb::class_<Some_Class>(m, "Some_Class")
.def(nb::init<>())
.def("Value", &Some_Class::Value, "index"_a, OCP_RETURN_COPY)
.def("Build", &Some_Class::Build, OCP_NOGIL);
}
```
Declare and call `register_Some` in `src/core.cpp`. Registration order matters
only in that a base class must precede its derived classes.
## 3. The checklist
- **Shape-returning API** → `OCP_RETURN_COPY`. Explorers, iterators, map
lookups, `Generated`/`Modified` lists — anything handing out a reference into
storage the caller does not own.
- **Static method** → `OCP_DEF_S(cls, "Name", ...)`, which appends `_s`. Every
static, without exception.
- **Transient (handle-managed) class** → derive from `Standard_Transient` in the
`nb::class_` declaration and bind constructors with `ocp_new<T, Args...>()`.
Never `nb::init<>` — see design.md.
- **Long kernel call** → `OCP_NOGIL`, but only if it cannot re-enter Python.
- **Executing constructor** → do not bind it. Bind the default constructor plus
the `SetX`/`Build` sequence.
- **Enum** → `nb::is_arithmetic()` and `.export_values()`.
- **`Message_ProgressRange` parameters** → omit them. The app never passes one
(no `OCP.Message` import anywhere), and leaving them out keeps signatures
small. Add the module if `--check` ever reports it.
- **Out-parameters** stay out-parameters. `BRep_Tool.Triangulation_s(F, L)`
writes through `L` because the app calls it that way; returning a tuple would
be tidier and wrong.
When in doubt about a signature, ask the stock wheel rather than guessing:
```bash
cd ../app && .venv/bin/python -c "from OCP.BRep import BRep_Tool; print(BRep_Tool.Triangulation_s.__doc__)"
```
## 4. New toolkits
If the linker cannot find a symbol, the class lives in a toolkit not yet listed
in `CMakeLists.txt` (`target_link_libraries(_OCP PRIVATE ...)`). Add it there;
`auditwheel` bundles whatever the linker records, so nothing else changes.
## 5. Verify and ship
```bash
make dev # compile + tests
make test-asan # if you touched ownership or added transients
make wheel # bump the .devN in pyproject.toml first
make publish
tools/parity_venv.sh && python tools/inventory.py --check
```
Then run the slice of the app's suite the increment claims — Inc 1 is gated on
`backend/tests/test_geom_memo.py`, Inc 2 on the tessellation tests plus
`pytest -m perf`, Inc 4 on `test_cad_pool.py` and `test_derive.py`, and the
cutover on the full suite plus `backend/tools/rebuild_sweep.py --diff` over the
project store.
## Adding to `n3xd_ocp` instead
Anything that is not a faithful mirror of an upstream symbol belongs in
`src/ext/` under the `n3xd_ocp` namespace: bulk array APIs, batched measurement,
anything GIL-free that upstream does not offer. `OCP.*` staying a
symbol-for-symbol drop-in is what makes parity testing meaningful, so keep
additive work out of it. Register leaf modules with
`ocp_named_module("n3xd_ocp.<name>")` and re-export them in
`python/n3xd_ocp/__init__.py`.

110
docs/building.md Normal file
View File

@@ -0,0 +1,110 @@
# Building
Everything runs through `make`; `make help` lists the targets. All compilation
happens inside the OCCT builder image, so the only host requirements are Docker
and (for publishing) `uv`.
## The builder image
`occt/Dockerfile` compiles OCCT 7.9.3 once inside
`quay.io/pypa/manylinux_2_28_x86_64` (both the base digest and the source
tarball's sha256 are pinned) and installs it to `/opt/occt`.
```bash
make image # ~40 min on 16 cores
make image-push # needs: docker login git.stroblme.de
```
It is a **compiler appliance**: wheel builds mount the repo into it rather than
`FROM` it, so iterating on the binding never re-layers the kernel. Add new
packages at the *end* of the Dockerfile — earlier layers stay cached and the
kernel is not recompiled.
The configuration turns Draw, VTK, Tk, Xlib, OpenGL and GLES off and FreeType
on, and the last layer asserts the result: TKService and TKV3d exist (text
emboss reaches `Font_BRepFont` through them), neither carries a `libGL`/`libX11`
`DT_NEEDED`, and freetype is linked. That is what lets the app image eventually
drop `libgl1` and `libx11-6`. Flags are `-O2`, no `-ffast-math`, no
`-march=native`: OCCT's version is a determinism input for assay's goldens, so
the binding must not introduce a different FP contract than the kernel it wraps.
**Production never compiles OCCT.** The host has 4 cores and `make update` is a
`git pull` plus a compose build; the kernel arrives prebuilt inside the wheel,
which is the whole reason this image exists.
## Build cache
Object files, ccache, the pip cache and the build venv live under `$(CACHE)`,
default `/mnt/cache/n3xd/ocp` — off the root filesystem, which is tight on the
dev box. Change it per invocation with `make CACHE=/somewhere wheel`, or reset
it with `make clean-cache`.
## The loop
```bash
make dev # incremental compile + pytest — the inner loop, seconds
make test-asan # handle-model memory-safety check
make wheel # full build: compile, stubs, repack, auditwheel, smoke test
```
`make wheel` compiles twice on purpose: stubs are produced by importing the
freshly built extension, so they cannot exist before the first compile, and the
wheel is packed from the source tree. The second pass is incremental.
The smoke test installs the repaired wheel into a bare venv and imports it with
`LD_LIBRARY_PATH` unset — the only honest proof that `auditwheel` made it
self-contained.
## Fixtures
`tests/data/*.brep` are the byte-identity references and are generated under the
**stock** wheel, from the app checkout:
```bash
cd ../app && .venv/bin/python ../ocp/tools/gen_fixtures.py
```
The generator asserts stock is idempotent for each fixture before recording its
digest — otherwise the gate would compare against a moving target. Regenerate
only when deliberately re-blessing (e.g. the OCCT 8.0 bump).
## Publishing
```bash
make version # confirm what you are about to publish
make publish # uv publish -> https://git.stroblme.de/api/packages/N3XD/pypi
```
Credentials come from `.secrets` (gitignored) as `UV_PUBLISH_USERNAME` /
`UV_PUBLISH_PASSWORD`; the username is a real Gitea username, not the PyPI
`__token__` convention, and the token needs `package: Read and Write`.
A version can never be republished. Iteration builds therefore carry a `.devN`
suffix and are the only ones the registry's cleanup rule collects; bump `N` in
`pyproject.toml` for each upload.
Consumers read anonymously — the N3XD org is public:
```bash
uv pip install --index-url https://git.stroblme.de/api/packages/N3XD/pypi/simple/ \
--prerelease=allow n3xd-ocp
```
Gitea serves no root `/simple/` listing (404), only the per-package path, which
is all pip and uv ask for.
## Parity
`tools/parity_venv.sh` builds a side environment where the app runs against this
wheel instead of the stock one. The app's manifests are never edited: both
distributions own the `OCP/` import path and a process can hold only one OCCT
build, so a swap is per-environment and reversible by re-syncing.
```bash
tools/parity_venv.sh # from the registry
tools/parity_venv.sh --local # from wheelhouse/
python tools/inventory.py --check
```
Coverage is expected to be partial until the increments land — `--check` prints
what is still missing, grouped by module, which is the work queue.

195
docs/design.md Normal file
View File

@@ -0,0 +1,195 @@
# Design record
Decisions that are expensive to revisit, and the evidence behind them. If you
are adding symbols rather than changing the machinery, read
[adding-symbols.md](adding-symbols.md) instead.
## What this is
A hand-written [nanobind](https://github.com/wjakob/nanobind) binding of the
OCCT surface the N3XD backend actually uses — 139 symbols across 48 `OCP.*`
modules, per `tools/inventory.py`, not all of OCCT. It installs as a top-level
`OCP`, so it is a drop-in replacement for `cadquery-ocp-novtk` and the app's 442
import sites stay untouched.
Binding *call* overhead was never the bottleneck — the CAD hotspots are inside
the C++ kernel — so the payoff is version velocity, footprint, correctness at
the ownership boundary, and the freedom to add APIs upstream cannot (GIL
release, bulk array extraction).
## The handle model
`opencascade::handle<T>` is cast by `src/common/occt_handle.h`, modelled on
nanobind's own `stl/shared_ptr.h`.
- **C++ → Python**: the wrapper is a *non-owning* nanobind instance pointing at
the C++ object, plus one handle stored in its keep-alive list. An existing
wrapper is reused (`is_new == false`), so identity holds while a wrapper is
alive, and a long-lived object crossing the boundary repeatedly does not pile
up redundant references. Transients are polymorphic, so `nb_type_put_p`
downcasts on the dynamic type.
- **Python → C++**: a plain handle copy, balanced when the caster dies after the
call. Unlike `shared_ptr`, no Python reference is taken: OCCT's intrusive
atomic refcount owns the object's memory, not the Python instance. That is
precisely what makes releasing the GIL safe — OCCT may copy handles on its own
worker threads without touching the interpreter.
**Transient constructors never use `nb::init<>`.** nanobind's normal
constructor placement-news the object into the Python instance's storage, which
OCCT would later `delete`. Use `ocp_new<T, Args...>()` (`occt_transient.h`),
which heap-allocates and returns a handle. This is not hypothetical: the app
builds a `Geom_BSplineCurve` in `sketch_builder/edges.py` and hands it to
`BRepBuilderAPI_MakeEdge`, which keeps a handle past the call.
The caster enforces the rule rather than trusting it: a transient whose
`GetRefCount()` is zero is not handle-owned, and converting it would hand OCCT
the right to free a nanobind instance. `from_python` refuses instead — a
TypeError beats heap corruption, and only a binding bug can reach it.
*Alternative considered*: `nb::intrusive_ptr` plus a side table mapping
`Standard_Transient*` to `PyObject*` (OCCT objects have no self-py slot). It
loses on the property that matters most here — nanobind's intrusive protocol
unifies the C++ count with the PyObject refcount, so `Py_INCREF` from a
GIL-free OCCT thread becomes a crash class. Kept as the documented fallback if
the caster above ever proves unworkable.
*Verification*: `tests/test_handles.py`, run both normally and under
`make test-asan`. It covers wrapper identity across a round trip, survival in
both directions when one side drops its reference, refcount balance across 1000
conversions and across 100 raising calls, null-handle ↔ `None`, and RSS growth
across 50 000 create/destroy cycles. ASAN covers memory *safety*; the RSS
assertion covers *growth* and is skipped under ASAN, whose quarantine retains
freed memory (139 MB of it, which is what a naive reading would call a leak).
## Fidelity rules
Verified against the installed stock wheel, not assumed.
**`__hash__` is bound; `__eq__` is not.** Upstream binds `__hash__` (TShape ⊕
Location) and leaves `__eq__` at Python's default identity comparison, so two
re-extracted copies of one face hash equal but compare unequal. That pairing
looks like a bug and is load-bearing: `cad/topology/geom_memo.py` buckets on
`hash(face)` and disambiguates with `IsSame` *because* `==` cannot be trusted.
Binding `__eq__` to `IsEqual` would quietly collapse entries that memo keeps
apart. Hash *values* need not match upstream — only the semantics do.
**Sub-shapes are returned by value** (`OCP_RETURN_COPY`) from explorers,
iterators, map lookups and history lists. A `TopoDS_Shape` is a small value
holding a handle to its TShape, so a copy is one incref and owns what it points
at. This makes the lifetime class that segfaulted a process-global face memo
unrepresentable.
**Executing constructors are not bound.** `BRepAlgoAPI_*` gets a default
constructor plus `SetArguments`/`SetTools`/`Build` — the two-argument forms run
the algorithm immediately, which is how a latent double-execution survived in
the app for a while.
**`_s` on every static**, via `OCP_DEF_S`. The rule is blanket rather than
clash-driven, so no static can ship without it; the app calls 176 of them.
**Enums** use `nb::is_arithmetic()` + `export_values()`, reproducing pybind11's
int comparison and module-scope members (`from OCP.TopAbs import TopAbs_FACE`).
**Exceptions**: `Standard_Failure` derives `RuntimeError`, which is what keeps
the backend's `except RuntimeError` sites working — it never names an OCCT
class. About 20 concrete types are bound under `OCP.Standard` / `OCP.StdFail`
and dispatched on the dynamic OCCT type, because cad_pool's children marshal
failures home as `f"{type(exc).__name__}: {exc}"`, making the name observable.
**`TopoDS` is a namespace in OCCT 7.9**, not a class. Upstream still presents it
as a class carrying the `_s` statics, and the app calls `TopoDS.Face_s(...)`, so
`mod_TopoDS.cpp` binds an empty carrier struct under that name.
## GIL policy
Released around calls that stay inside the kernel and cannot re-enter Python:
`Build`/`Perform`, meshing, `BRepCheck_Analyzer`, file readers and writers, and
every `n3xd_ocp` bulk API. Applied from an explicit list, never blanket.
`BinTools` is *also* GIL-free for the kernel half, which the original plan
assumed impossible. Rather than bridging a `streambuf` that calls back into
Python per chunk, `occt_stream.h` slurps the file-like object first and hands
the kernel a pure C++ stream. That is correct regardless of how BinTools seeks,
and costs one extra copy of the payload — which `n3xd_ocp.bintools` avoids
entirely for the pool paths that care.
**None as an argument** is rejected before the caster for simple overloads, so
handle parameters that legitimately accept a null handle need an explicit
`nb::arg("x").none()`. Null *returns* map to `None` unconditionally. The app
passes no null handles today; `inventory.py` plus the app suite are the guard.
## Packaging
**One extension** (`OCP/_OCP.abi3.so`) registering every `OCP.*` submodule via
`PyImport_AddModule`, so `import OCP.TopoDS` works with no shim module per name
and `cls.__module__` reads `OCP.TopoDS`. The whole surface traffics in
`TopoDS_Shape`, `gp_*` and handles, so sharing types in-process is free here and
would otherwise lean on nanobind's cross-extension registry; registration order
stays an explicit sequence in `core.cpp`; and cad_pool's forkserver warms one
dlopen. One `.cpp` per module keeps incremental compiles cheap — only the link
is shared.
**abi3 (`cp312-abi3`)** from the first build, matching forge/assay, so the
Python 3.13 bump (roadmap 10D) needs no rebuild. Escape hatch if a nanobind
STABLE_ABI limitation ever bites: drop `STABLE_ABI` and `wheel.py-api`, since
every deployed environment is 3.12. Stub generation works fine under abi3.
**Version `<occt>.N`**, asserted at configure time against the OCCT actually
found, so `occt_version()` keeps reporting the truth for assay's goldens.
Iteration builds carry `.devN`; the registry never allows republishing.
**No sdist is ever published** — it could not build without the builder image,
and offering one invites the 4-core production host to try. That is enforced by
only ever building wheels. Note `sdist.exclude` is *not* the way to do it:
scikit-build-core feeds it into the wheel's package-file mapping too, so
excluding `*` silently ships a wheel containing the compiled extension and none
of the Python package. Relatedly, file selection is git-based, so the generated
`.pyi` stubs are gitignored *and* re-included through `sdist.include`.
**Fork safety**: import starts no threads and creates no fork-hostile state, so
cad_pool's forkserver keeps costing ~30 ms per job instead of a ~1.3 s spawn.
Pinned by `tests/test_forksafety.py`.
## The `n3xd_ocp` module
`OCP.*` stays a symbol-for-symbol drop-in so parity testing means something;
anything additive lives in `n3xd_ocp`, shipped in the same wheel and backed by
the same OCCT build. Only the leaf submodules are registered from C++ — creating
the parent would put a bare module in `sys.modules` and a later `import
n3xd_ocp` would skip the package's `__init__.py`.
Shipped: `bintools` (shape ↔ `bytes`, GIL-free, byte-identical to
`OCP.BinTools` and asserted so) and `_debug` (test-only ownership
introspection).
Designed, landing with the increment that binds their types:
```python
# with Inc 1 (BRepGProp/GProp) — attacks the measured 94 % of
# face_candidate_anchors (0.99 s of 1.05 s for 690 faces) that is
# BRepGProp.SurfaceProperties_s called once per face from Python
def face_surface_props(shape, *, parallel=True) -> tuple[ndarray, ndarray]
# areas[F], centroids[F,3], ordered by TopExp.MapShapes_s(FACE) index
# with Inc 2 (BRepMesh/Poly) — replaces the per-node and per-triangle Python
# loops in cad/tessellation.py
class FaceMesh: nodes, triangles, normals, uv, face_index
def extract_meshes(shape, *, want_normals=True, want_uv=False,
apply_location=True, flip_reversed=True) -> list[FaceMesh]
# with Inc 4 — opt-in experiment: OSD::SetSignal turns some native faults into
# catchable Standard_Failure subclasses inside cad_pool children. Never called
# at import; subprocess isolation stays regardless.
def set_signal(arm_fpe: bool = False) -> None
```
The backend adopts these after cutover, one call site at a time.
## Open questions
- **S5** — whether STEP needs `CSF_*` resource files shipped in the wheel.
Modern OCCT code-initialises most `Interface_Static` defaults; settle it when
Inc 3 (I/O) lands, in a container without system OCCT resources.
- **Byte-identity beyond 7.9.3** — the gate compares against the stock wheel, so
it necessarily retires at the OCCT 8.0 bump (roadmap 10E), where the fixtures
are re-blessed deliberately alongside assay's goldens.