merge-items: Merge output directories; incremental --update; index entry metadata

CLAUDE.md

@@ -31,6 +31,8 @@ npm install --prefix preppy/node        # KTX2 embed helper deps (once)
 preppy -i config.json -o out/           # batch: variants → self-contained glbs + manifest.json + index.json
 preppy-check-tools                       # report external toolchain status
 preppy-obj2glb -i mesh.obj -o mesh.glb  # LEGACY single OBJ → Draco GLB (deprecated)
+preppy-merge-items out1/ out2/ -o merged/          # merge output dirs (assets + index.json)
+preppy-merge-items new/ --update catalog/         # incremental merge into an existing catalog
 preppy-merge-items a.json b.json -o merged.json   # LEGACY items.json merge (deprecated)
 ```
 
@@ -46,15 +48,16 @@ Console entrypoints in `preppy/apps/` are thin argparse CLIs over the library mo
 - **`assemble.py`** — `embed()` runs the bundled Node helper to swap each material's baseColorTexture for its KTX2 (`KHR_texture_basisu`), preserving `EXT_meshopt_compression` (only if the meshopt encoder is registered — F3) and `KHR_texture_transform`.
 - **`preview.py`** — `render_preview()` renders a **proxy model preview** for the default-variant thumbnail (trimesh loads the OBJ + computes normals; pyrender renders one offscreen 3/4-view frame). A `FilePathResolver` subclass swaps the OBJ's `map_Kd` names for the already-normalized PNGs, so the raw (possibly gigapixel) sources are never decoded. It renders the OBJ, **not** the delivered meshopt/KTX2 glb (no offline renderer reads those) — recognizable, not pixel-identical. Raises `PreviewUnavailable` when the toolchain/GL backend is missing so the orchestrator falls back to the texture crop. Gotcha: these OBJs mix `f v/vt` and bare `f v` faces, so trimesh's "mixed data" fallback drops UVs for the whole mesh (and pyrender then compile-fails on a shader sampling a compiled-out `uv_0`); `_recover_uv_by_index` rebuilds `uv[i]=vt[i]` from the file (validated: one `vt` per vertex, matching `v/vt` indices, `process=False` keeps file order), else the model renders untextured.
 - **`cache.py`** — content hash over **inputs + config + tool versions** (never the output glb — basis encoding is non-deterministic), `hashed_name()`, and `prune()`.
-- **`manifest.py`** (replaced `voyager.py`) — pure builders for the per-object `manifest.json` (flat `variants[] {id,label,uri,default}` + object metadata, `units:"cm"`, per-variant overrides) and the optional `index.json`.
+- **`manifest.py`** (replaced `voyager.py`) — pure builders for the per-object `manifest.json` (flat `variants[] {id,label,uri,default}` + object metadata, `units:"cm"`, per-variant overrides) and the optional `index.json` (entries carry `inventory`/`sort` when set, so a host can group + order a catalog without opening every manifest). **`index.json`'s shape has a second producer outside this repo** — `acquisition-workflow`'s `pipeline/pipeline_deploy.py` writes it from its own deploy state (it can't import preppy, which lives in a container); a change to `build_index`/`index_entry` has to be mirrored there and nothing fails loudly if it isn't.
 - **`apps/file_prep.py`** — the orchestrator. Per object, per **variant** (no grouping): resolve texture(s) → normalize + encode each → gltfpack → optional Hausdorff gate → embed all by name → one self-contained glb → manifest entry. Emits `manifest.json` per object + a top-level `index.json`. The default-variant thumbnail is a rendered model preview (`preview.render_preview`, `_render_thumbnail` helper) that falls back to a texture crop (`--thumbnail-mode texture`, or automatically when the render toolchain is unavailable).
+- **`apps/merge_items.py`** — merges separately-built catalogs. Input shape is auto-detected: output directories (or their `index.json`) are merged by copying each per-object subdirectory into `-o` and combining the `index.json` `objects[]` (merge key is object `id`; collisions are planned up front so nothing is copied before the whole merge validates). `--update CATALOG` seeds the plan from an existing catalog and rewrites it in place, copying only the named sources' objects (a base object a source redeclares is replaced without `--overwrite`); `--index-only` merges the listings and copies nothing, for a catalog whose assets aren't reachable. Plain `.json` inputs take the deprecated `items.json` document-list path.
 - **`convert.py` + `apps/obj_to_glb.py`** — the deprecated legacy OBJ→Draco-GLB path (kept until removed).
 
 ### Input config format
 
 The `preppy` input JSON is a flat array of **objects**, validated by `templates/prep-models.schema.json` (examples: `prep-models-example.json`, `mvs-example.json`):
 
-- An **object** needs `id`, `title`, and a `variants` array. Optional `prefix` (output folder/file prefix; defaults to `id`), `titles`, `inventory`, `description`, `credit`, `date`, `units` (default `cm`), `nodataFill`.
+- An **object** needs `id`, `title`, and a `variants` array. Optional `prefix` (output folder/file prefix; defaults to `id`), `titles`, `inventory`, `description`, `credit`, `date`, `units` (default `cm`), `nodataFill`, `sort` (index-only ordering key; not copied into the manifest).
 - A **variant** needs `suffix` (stable key: names the file + is the manifest variant `id`) and `obj`. Optional `label`, `default`, `nodataFill` (resolved variant ?? object ?? CLI), and per-variant `credit`/`date`/`method`/`description`. Textures are resolved transitively from the OBJ's `map_Kd`. Relative `obj` paths resolve against `--data-root` (default CWD), not the config file's location.
 
 ### Output layout

README.md

@@ -82,7 +82,12 @@ out/
 
 Asset filenames carry an inputs+config content hash by default (served
 `immutable`); `manifest.json` / `index.json` keep stable names and hold the
-current hashed URIs. Useful flags:
+current hashed URIs. Each `index.json` entry is `{id, title, manifest}` plus
+the object's `inventory` and `sort` when the config sets them and `thumb` when
+one was built — enough for a host to build a grouped, ordered listing from the
+catalog alone, without opening every manifest.
+
+Useful flags:
 
 | Flag | Effect |
 | --- | --- |
@@ -110,6 +115,47 @@ texture`) automatically.
 
 Run `preppy -h` for the complete list.
 
+### Merging catalogs
+
+Objects built in separate runs (or on separate machines) combine into one
+servable catalog with `preppy-merge-items`. Each per-object subdirectory is
+copied into the output directory and the `index.json` listings are merged:
+
+```shell
+preppy-merge-items run1/ run2/ run3/ -o catalog/
+```
+
+Object `id` is the merge key, and relative `manifest`/`thumb` URIs are preserved
+by keeping each object's subdirectory name. Duplicate ids across sources are an
+error — pass `--overwrite` to let the last source win, which is how a rebuilt
+object replaces its older assets in place. The whole merge is validated before
+anything is copied, and `--no-sort` keeps source order instead of sorting by
+title (`--sort-key` also takes `id`, `sort` or `inventory`).
+
+To add a few objects to a catalog that already exists, `--update` merges them in
+place instead of rebuilding the whole thing — only the named sources' object
+directories are replaced, and every other object is left untouched:
+
+```shell
+preppy-merge-items rebuilt/ new/ --update catalog/
+```
+
+An object already in the catalog is replaced without `--overwrite` (that is what
+an update is); duplicate ids *among the sources* are still an error. The catalog
+is rewritten in place, so `-o` is unused.
+
+`--index-only` merges the listings and copies no assets, for a catalog whose
+assets live somewhere this machine cannot reach — point it at a copy of the
+remote `index.json`:
+
+```shell
+preppy-merge-items new/ --update ./catalog-index/index.json --index-only
+```
+
+Passing `.json` files instead of directories runs the deprecated `items.json`
+document-list merge (`--merge-duplicates`); `--update` and `--index-only` do not
+apply to it.
+
 ### Hosting / caching
 
 Serve the catalog with a two-tier cache policy so cache-busting is safe:

docs/implementation-plan.md

@@ -115,6 +115,8 @@ flat listing for host pages only.
 // index.json (optional — host archive UI only)
 { "objects": [
     { "id":"PHerc1061Cr05", "title":"P.Herc. 1061, Cr. 5",
+      "inventory":"P.Herc. 1061",       // optional: group key for a host listing
+      "sort":"1061-05",                 // optional: explicit ordering key
       "manifest":"PHerc1061Cr05/manifest.json",
       "thumb":"PHerc1061Cr05/PHerc1061Cr05_thumb.jpg" } ] }
 
@@ -160,7 +162,10 @@ mogrify — cheap, no offscreen GL. (Loose end: rendered 3D thumbs later.)
 
 ### A6. CLI
 Keep console-script entry; add `--ktx2-mode {uastc,etc1s}`, `--decimate-error`,
-`--no-decimate`, keep `--uri`, `--keep-tmp`. `merge_items` → `merge_index` later.
+`--no-decimate`, keep `--uri`, `--keep-tmp`. `merge_items` keeps its name (no
+rename to `merge_index`); it gained the output-directory merge alongside the
+legacy `items.json` one, plus `--update CATALOG` (merge in place, touching only
+the named objects) and `--index-only` (listings without assets).
 
 ---
 

preppy/apps/file_prep.py

@@ -275,7 +275,8 @@ def process_object(object_cfg: Mapping, *, data_root: Path, out_dir: Path,
 
     return manifest.index_entry(
         object_id, man.get('title', object_id),
-        manifest_uri=f'{prefix}/manifest.json', thumb=thumb_rel)
+        manifest_uri=f'{prefix}/manifest.json', thumb=thumb_rel,
+        inventory=man.get('inventory'), sort=object_cfg.get('sort'))
 
 
 def _build_parser() -> argparse.ArgumentParser: