Skip to content

Model the domain as typed records with named, tested rules - #4530

Draft
Mpdreamz wants to merge 94 commits into
mainfrom
feature/erd-model
Draft

Mpdreamz wants to merge 94 commits into
mainfrom
feature/erd-model

Conversation

@Mpdreamz

@Mpdreamz Mpdreamz commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Configuration files load into typed domain records, and the rules about them are named constraints that the loaders check once and a generator tests. The build reports more problems it used to accept silently, as warnings and hints, and the tests found and fixed several real bugs on the way.

Affects: Configuration, Assembler builds, Release notes

Prompt summary: Make the domain model of docs-builder explicit. Each configuration file loads once into a typed record, its rules become named constraints instead of scattered checks, and those rules are tested against generated data. What is accepted, reported and written stays the same unless a change is called out: an error stays an error, and a new check starts as a warning.

Why

Rules about configuration live in loaders, validators and constructors, each written a little differently. They drift apart, so one command can accept a file that another rejects, and nothing tests properties such as "loading twice gives the same result" or "the order of the entries does not change the output". A typed record with named rules puts each rule in one place and lets a generator exercise it.

Writing the rules as tests found defects that example-based tests had not: an infinite loop and null references in the navigation and assembly loaders, an ordering bug in SemVersion, and mount resolution at path-segment boundaries.

What

Configuration files load into typed records

docset.yml, assembler.yml, navigation.yml, changelog.yml, codex.yml, the product catalog, search tuning and links.json (now LinkManifest) are read into a *Dto, then into a domain record with value equality. Golden snapshots over a frozen baseline check that the new loaders produce what the old ones did, and a test fails when a DTO field is not accounted for in its projection.

Rules are named constraints, checked once

Rules are Constraint<T> values collected in 32 [Constraints] classes, described in src/Elastic.Documentation/Constraints-Conventions.md. A loader checks a set once and reports each violation under the rule's id. New rules start as warnings or hints, for example an empty Markdown page in a table of contents is D108 (a warning). ConstraintSeverityLedgerTests pins the declared severity of all 287 rules, so a rule weakened from error to warning shows up as a diff.

Rules are tested on generated data

Each constraint set is tested on nine axes by Nullean.Entities.CsCheck, which generates records with hostile values (blank strings, duplicate ids, undefined enum values) and reports a failing case as the smallest one with a seed that replays it. Hand-written counter-examples remain only where a machine cannot find one. Four rules (CodexGroupKnown, GroupIdsUnique, QueryRuleIdUnique, BundleEntryFilesUnique) are declared once as relations, so the rule and the data generator cannot drift apart.

Nullean.Entities comes from NuGet

Six PackageReference entries at 0.1.0 replace project references to a checkout at a fixed path on one machine, with their versions in Directory.Packages.props. The source generator ships inside the Nullean.Entities package, and Nullean.Entities.props is gone.

Main is merged in

The branch contains main as of the SharpYaml migration. Main's readers feed the typed loaders, unknown keys in applies_to are errors as on main, and the S3 retained-assets plan, the changelog cache and the unified release-notes commands run on the domain records.

Verify

dotnet test tests/Elastic.Documentation.Tests/
dotnet test tests/Elastic.Documentation.Configuration.Tests/
dotnet test tests/Documentation.Builder.Tests/
dotnet test tests/Elastic.Documentation.Deploying.Tests/
dotnet test tests/Elastic.Documentation.OpenApiIndex.Tests/
dotnet test tests/Elastic.Documentation.Api.Tests/

The unit suites pass on the merge commit. ./build.sh unit-test restores again now that main pins MessagePack 2.5.303.

🤖 Generated with Claude Code

Mpdreamz and others added 30 commits September 29, 2026 20:36
The domain-model refactor must not change behavior. These snapshots record
what today's loaders produce, values and diagnostics, for a frozen set of
real configuration files, so every later change can be checked against them.
The provider now refreshes each temporary config file after writing it, so
its cached state matches the file on any file system.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The domain model uses C# unions and pure loaders. The SDK's compiler
already supports unions under LangVersion=preview; only UnionAttribute and
IUnion are missing on net10, so the domain project polyfills them. Result<T>
carries a value plus its diagnostics, so loaders no longer throw or thread a
collector. The purity test keeps the new domain namespaces free of file
systems, YAML and collectors. Golden failures now record only the message,
since a refactored loader may report the same failure as a diagnostic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Products, versioning systems and legacy URL mappings were separate
configuration types in Elastic.Documentation.Configuration, threaded through
every context one by one. They are now immutable records in
Elastic.Documentation.Catalog, grouped into a GlobalCatalog that
CatalogLoader loads once per process. Contexts expose it as Catalog.
Related learning becomes its own aggregate, loaded on first use rather than
re-parsed for every BuildContext. The per-file parsers are internal, so the
loaders are the only way in. The golden snapshots, the unit suite and a docs
build diffed against the base commit show no change in behavior. The dropped
[YamlSerializable] attributes generated no metadata for these init-only
records; the base assembly has none for them.

The build stamp now reads its inputs through each file's own file system, and
the onboarding test fake returns the whole catalog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three tests set FEATURE_*, MCP_SERVER_PROFILE and OTEL_EXPORTER_OTLP_ENDPOINT
on the process, which races with tests running in parallel.
ApiNavGroupingEnabled_SetEnablesIt failed whenever the override test ran
at the same time. FeatureFlags now reads its overrides through
IEnvironmentVariables, and the MCP server reads its profile from
configuration, so tests pass values per instance. Production behavior is
unchanged: both still resolve to the process environment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SearchConfiguration and its query rule records are now immutable records in
Elastic.Documentation.Search, loaded by SearchTuningLoader. The contexts
expose them as SearchTuning. The parser is internal, so the loader is the only
way in. The golden snapshot for search.yml is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records compare collections and file handles by reference, so loading the
same catalog twice gave unequal aggregates. Only the records that hold
those members now use [Equatable], with an equality attribute on each such
member; records with only value members keep compiler equality. Synonym
groups become a SynonymGroup record instead of string arrays. A guard test
flags records that need [Equatable] and ones that do not, and round-trip
tests check that each aggregate loads equal twice. That caught the
display-name lookup, a derived cache that now opts out of equality.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The toc: tree was one record family serving as both the YAML parse output
and the resolved tree, so a parser sentinel (InvalidTocItemRef) and
placeholder contexts could leak to consumers, and every switch over it
ended in `_ => null`. The parser now produces internal *Entry DTOs, and
resolution maps them to domain records under a closed TocItem union with
value equality.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ConfigurationFile validated docset.yml, read redirects.yml and joined the
catalog inside its constructor, writing straight to the build's collector
and carrying the runtime feature flags. DocumentationSetResolver now
returns the same work as a Result, with diagnostics in their original
order, and the resolved docset is an immutable domain record. Feature
flags move to BuildContext, which owns their runtime overrides; glob
matching and CTA lookup stay loader-side as extensions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BuildContext read docset.yml by threading its collector through the TOC
resolver. The loader now returns a Result, so both the build and serve's
reload go through the same load, resolve and emit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ader

AssemblyConfiguration, Repository and PublishEnvironment were mutable YAML
DTOs whose Deserialize read CI, GITHUB_ACTIONS and GITHUB_TOKEN from the
process environment and mutated repositories in place. They are now
immutable records in Elastic.Documentation.Publishing; AssemblyLoader maps
internal DTOs to them with an injectable IEnvironmentVariables, and the
content-source Match moves to a loader-side extension since it logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntry union

SiteNavigationFile was the YAML DTO, the declared site tree and its
validator in one mutable class, with entries behind an open interface.
The declared tree is now an immutable Publishing record whose entries are
a closed union; SiteNavigationLoader parses through an internal DTO and
keeps the path-prefix validation on the loading side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodexConfiguration moves to Elastic.Documentation.Codex as an immutable
record, loaded by CodexSiteLoader; the unused GetGitUrl, which read
GITHUB_TOKEN from the process environment, is removed.

YamlDotNet puts the bound type's full name in the error for an unknown key.
The assembler.yml and navigation.yml DTOs introduced earlier had new names,
which changed those messages; every DTO now keeps its original type name,
and tests pin the messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lue equality

ChangelogConfiguration and its rules, bundle and pivot records were
already immutable and YAML-free but lived in the loader project. They now
sit in Elastic.Documentation.ReleaseNotes, which the domain purity and
equality guards cover, so every record holding collections compares by
value. The renderer's entry-keyed maps take a reference comparer so equal
entries from different bundles keep their own repo and owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RepositoryLinks becomes LinkManifest, the name agreed for the entity; the
JSON keys are untouched, so links.json and link-index.json stay
byte-identical. The wire (de)serialization helpers move from the records
to Serialization.LinkJson so the Links domain holds no streams, and the
Links namespace joins the domain purity and equality guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ApplicableTo and its deployment, serverless and product applicability
records had public setters, and parsers and renderers cleared Diagnostics
on the shared instance after emitting them. They are init-only now; the
YAML and JSON converters build them with `with`, and callers hand out a
copy without diagnostics instead of mutating the parsed one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SiteNavigationDefinition.WithoutRepositories drops a repository's mounts
and phantoms and promotes the mounts' children, and
SiteNavigationLoader.ForAssembly adds the local docs-builder mount, as
ConfigurationFileProvider.CreateNavigationFile's line rewrite does. Tests
prove both produce the same navigation on the baseline and on nested
fixtures. The typed version also handles a private top-level mount, which
the text rewrite turns into invalid YAML.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loaders replaced AssemblyConfiguration.Create and SiteNavigationFile
in earlier commits; these call sites were left out of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FeatureFlags had a setter per flag, and serve, the assembler and the
OpenAPI step mutated the instance a BuildContext shared with every
reader. Flags are now built with With(...), and BuildContext.OverrideFeatures
is the single place runtime overrides are layered on. Keys, defaults and
FEATURE_* precedence are unchanged; no flag is added or enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TOC entries carried two string paths from different roots: relative to the
docset and relative to the toc.yml that declared the entry. They are now
distinct value types that can only be built with From(...), so resolution
cannot hand one where the other is expected. Both read as strings, so
rendered output is unchanged. Detection rules, which have no declaring
toc.yml, are the one place the two coincide, and that is explicit now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A TOC entry's Context was an absolute path in a plain string, next to two
relative ones. It is now its own value type, so the three kinds of path on
an entry cannot be mixed up. Diagnostics still read it as the same string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ChangelogCommands built a new ChangelogConfigurationLoader at each of its
eight call sites; it now holds a single lazily created one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every {changelog} directive re-read assembler.yml to find private
repositories, and the CLI and API roots parsed it again for DI.
AssemblyLoader.LoadOnce caches the result per configuration provider,
whose files are fixed for its lifetime, and all three go through it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ten test helpers built the same ProductCatalog from a product map and the
same empty SearchTuning and LegacyUrlMappingCatalog by hand.
ProductCatalog.FromPublicProducts, SearchTuning.None and
LegacyUrlMappingCatalog.None name those values once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ConfigurationFileProvider.CreateNavigationFile rewrote navigation.yml line
by line into navigation.filtered.yml and swapped the provider's
NavigationFile, so which navigation a reader saw depended on call order,
and AssembleSources parsed the result a fourth time with its own YAML
walker. AssembleContext now loads navigation.yml once and filters it with
SiteNavigationLoader.ForAssembly; the link mappings are derived from that
typed tree by SiteNavigationLoader.GetTocMappings, which reports a missing
path_prefix at the entry's recorded position as the walker did.

The derivation was checked equal to the walker on config/navigation.yml
before the walker was removed, and golden snapshots pin the mappings with
and without private repositories. The configuration exporter and build
stamp now always see navigation.yml. A bare toc: name with an explicit
path_prefix uses the path_prefix regardless of key order; the walker let
key order decide, which no navigation file relies on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ConfigurationFileProvider copied every config file into a fresh
config-runtime/<guid> directory, so diagnostics, the build stamp and the
configuration exporter saw throwaway paths, and UseNavigationPreview
repointed the shared NavigationFile for every later reader. Local and
remote configuration is now read where it lives; only embedded
configuration, which has no file, is written out once. The preview is an
explicit NavigationPreviewFile, chosen per environment by
AssembleContext.NavigationFile, which the build stamp now hashes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config records mirrored their files under a mix of names: *Dto, *Yaml,
*Entry, and the assembler, codex and navigation records under the same
names as their domain records. They are now all internal *Dto types, so a
type name alone says whether it is file-shaped or domain. YamlDotNet's
unknown-key errors name the Dto type now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docset.yml was a public DTO that doubled as the declared docset, and its
rules lived in the resolver and the TOC loader, so a docset built any other
way was never checked. DocumentationSetDto is now internal; loaders produce
a DocumentationSetDefinition, and DocumentationSetConstraints lists every
rule the model must satisfy, built on a small domain constraint framework.
The resolver reports the violations and keeps what the same
DocumentationSetRules accept. The loader still rejects what it cannot build
a model from. A test requires a counter-example that breaks each
constraint, and only it. Folder and listing sort are typed, so an invalid
one cannot be represented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raints

Their rules lived in the parsers: a GTM id check in a DTO setter that only
fired when 'enabled' came before 'id', a ToDictionary that threw when a
reference was named docs-content, path-prefix checks in the navigation
loader and link-mapping derivation, and related-learning checks that threw
on the first bad link. AssemblyConstraints, SiteNavigationConstraints and
RelatedLearningConstraints now hold them, and the loaders report every
violation. The GTM rule no longer depends on key order. A shared test
base applies the same laws to every constraint set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The changelog loader validated as it parsed and stopped at the first
fault. Its semantic rules now live in ChangelogConfigurationConstraints
and run on the built model, with the same messages and severities. The
loader keeps parse errors and product id resolution, because only it
still sees the authored spelling. An error in products.default now fails
the load; before, the loader reported it and returned a configuration.
The constraint test classes also gain [InheritsTests], without which
TUnit never ran the exhaustiveness laws.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ckout

Six PackageReferences replace the ProjectReferences to a checkout at a fixed path on one
machine, with their versions in Directory.Packages.props. The source generator ships inside
the Nullean.Entities package, so the separate analyzer reference is gone, and
Nullean.Entities.props with its import is removed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Brings in the SharpYaml migration, the S3 retained-assets plan and the
changelog cache, and ports them onto the domain model: main's readers now
feed the typed DTO loaders, unknown keys in applies_to are errors as on main,
and the Codex, API Explorer and style-guide previews use BuildContext.Features.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Admonition and dropdown applies_to blocks reported every reader finding since
the merge, so `{ ece:, ess: }` and similar existing docs failed strict builds
with errors main never raised there; only hints are reported now. A build with
no git checkout no longer warns that the link manifest names no repository.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Main never reported them for admonitions and dropdowns, and `{ ece:, ess: }`
reads as the key `ece:`, an error that fails strict builds of existing docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@theletterf theletterf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big one! Woof!

New rules that fire on existing docs-content and docs-builder content are
hints until that content is cleaned up, global-configuration warnings reach
the build collector as hints, admonition and applies-item blocks report
applies_to findings as hints, and the configuration export reads the config
files through the file system they came from. Tests now follow main: unknown
keys are errors, and the new Backports and Implies members are accounted for.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Goldens no longer record YAML character offsets (they depend on line endings),
test fixtures build /docs paths with forward slashes so the scoped file system
accepts them on Windows, and the redirect-target, version-readable and
value-is-text rules report hints like the other rules that fire on real docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Martijn Laarman and others added 2 commits October 21, 2026 16:28
The Windows mock file system names the docset file '/docs\_docset.yml' but its
directory differently, so the implicit scope rejected it; the fixtures now pass
the scope. The opentelemetry docs nest a table of contents past max_toc_depth,
which fails strict builds as a warning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Regenerates docs/cli-schema.json, the only conflict.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Martijn Laarman and others added 2 commits October 21, 2026 17:18
On Windows a mock file system whose working directory is '/docs' resolves the
'/docs' scope root against itself, so the scoped reads of the docset file were
refused; the other scoped fixtures use the default working directory.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Takes main's removal of the curator frontmatter override test and keeps both
using directives in the API navigation builder.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The mock file system roots '/docs' at a drive on Windows, so the snapshot's
path placeholder missed the 'C:' prefix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
docs-preview — da4b9d52 Deployed Oct 21, 2026 by github-actions[bot]
assembler-preview — da4b9d52 Deployed Oct 21, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants