Skip to content

fix: handle named value references the way APIM resolves them - #337

Open
Chris Dayne (cdayne) wants to merge 18 commits into
Azure:mainfrom
cdayne:fix/named-value-references
Open

Chris Dayne (cdayne) wants to merge 18 commits into
Azure:mainfrom
cdayne:fix/named-value-references

Conversation

@cdayne

@cdayne Chris Dayne (cdayne) commented Nov 6, 2026 •

Copy link
Copy Markdown

Summary

Makes extract and publish treat a {{reference}} to a named value the way APIM does — by display name, ignoring case, in the policy's own scope, never by resource name — and makes everything that depends on that follow: Liquid bodies and XML comments, named values in logger and backend credentials, resources a policy names through a named value, the publish order around them, and the references publish rewrites when an environment prefix or suffix is set.

Changes

Matching a reference as APIM does

  • A {{reference}} names the named value whose display name it is, ignoring case, in its own scope (service or workspace). APIM rejects a policy whose reference is only a named value's resource name; this was confirmed against a live instance. The rule is in one module, src/services/named-value-names.ts, used by extract, publish, ordering and the rewrites.
  • Extract looks references up in the named values APIM lists. If the listing fails, it tries each reference as a resource name instead, which it usually also is.
  • Publish follows references by the display name a named value is published with (overrides applied) or, where an override changes it, its display name in the source. On main, publish matched references by resource name, so a filtered publish missed a named value whose display name differs.
  • Publish stops before making any change — and dry-run fails — when a policy, fragment, or logger's or backend's credentials reference a named value only by its resource name, naming the display name to use, instead of APIM rejecting it partway through the run. An incremental publish also checks the policies it republishes with a changed API or product, as it does for redaction markers.

What counts as a reference

  • References inside XML comments are ignored (Transitive dependency scan includes backend references inside XML comments #305). Comments and policy expressions are found in one pass, so a <!-- inside an expression string hides nothing and an @( inside a comment starts no expression.
  • A backend or policy fragment named by an attribute is found wherever the attribute sits on the element and in either quote (<set-backend-service id="…" backend-id='…'>), and so are the backends <llm-content-safety backend-id> and the semantic cache lookups' embeddings-backend-id name. Extract and the prefix rewrite now recognise these attributes from one list; on main, extract only followed backend-id or fragment-id written first and double-quoted.
  • An attribute after one holding a policy expression is found too. APIM's raw XML keeps an expression's quotes as written (sf-service-instance-name="@("fabric:/app")"), so expressions are skipped as C# — comments, and string, character, verbatim and interpolated literals — rather than read as attribute text.
  • Inside a <set-body template="liquid"> body, a bare {{name}} is followed only when a named value has that display name; APIM leaves every other {{ … }} ({{body.id}}, {{ name }}, {{x | upcase}}) to Liquid. On main these were fetched as named values and failed the extract (extract reports extraction errors / exits with code 1 for APIs using set-body template="liquid" with {{ }} expressions #276). If the named values can't be listed, extract reports an error, since it can't tell them apart.

Named values as dependencies

Publish order

  • Within tier 1, a backend, logger or schema is published before the named value that names it, then policy fragments, because APIM rejects a named value change that would leave a policy using it pointing at a resource that doesn't exist. Backends and loggers that use such a named value in their credentials go after it, pools after their members, in an order where each comes after what it needs; only the named values in a genuine cycle go first.

Environment prefixes and suffixes

  • {{tokens}} in policies, fragments, and logger and backend credentials are rewritten to the display name the named value they resolve to is deployed with (affixed, or an explicit displayName override as written). On main, only tokens equal to a resource name were affixed, so the usual display-name references were left pointing at nothing, and backend credentials were never rewritten.
  • A token using a source display name that an explicit displayName override renames is rewritten to the new name, with or without a prefix, as logger credentials already were.
  • A reference to an auto-generated logger key that publish skips (no override provides it) keeps its source display name, since that named value isn't published with the affix.
  • Every policy attribute that names a resource is rewritten when its type is in appliesTo: backend-id (set-backend-service, llm-content-safety), embeddings-backend-id, fragment-id, logger-id, schema-id.
  • A named value a policy uses as a resource name has its value rewritten to that resource's deployed name.
  • References are matched against every artifact in the source, so they are rewritten even when the filter or an incremental publish leaves out what they name.
  • XML comments are left as written. APIM doesn't substitute named values inside them, and a rewritten name with -- would make the comment invalid XML. On main, tokens inside comments were rewritten.

Docs

  • The filtering guide describes how references are matched and what they pull in; the multi-environment guide's rewrite table and the dependency-graph tier 1 row match the new behaviour.

Why the added structure

The constitution asks for added complexity to be justified (§V). Each piece below answers a failure seen on main or in review:

  • One module for the reference rules (named-value-names.ts): matching was implemented separately in the extractor, the publish resolver, the logger credential rewrite and the prefix rewriter, and they disagreed (case, resource names, scope, overrides). Fixes to one kept missing the others; now every caller uses the same rule.
  • Two display names per named value on publish (as published, and as in the source): an explicit displayName override renames a named value, and references in the source use the old name. main already followed that rename for logger credentials; matching both keeps that behaviour and applies it everywhere instead of in one place.
  • Per-run shared reads (SourceReferences in publish-service.ts, the named value reader cache, and skipping artifacts that can't hold references): the expansion, the order, the early stop and the rewrites all need the source's named values and policies. Shared, a filtered publish lists the source twice instead of four times and reads each named value four times instead of six.
  • The tier 1 order graph: the incremental case in the validation table fails on main because a named value is published before the backend it names. Credential users go after their named values for the same reason in the other direction, pools after their members as on main, and a cycle (a backend whose credentials use the named value that names it) can't be ordered, so only the named values in it go first.
  • The early stop on resource-name references: without it APIM rejects the policy only after everything before it has been published (62 resources in the validation run).
  • Reading policy expressions as C# (policySpans, policyExpressionEnd, literalEnd in transitive-resolver.ts): a regex can't find an attribute that follows one whose expression holds quotes, a comment or an interpolated string, and review found each of those cases hiding a backend from the scan and from the prefix rewrite. Finding XML comments in the same pass keeps a <!-- in an expression string from hiding the rest of the policy.
  • Skipping auto-generated logger keys in the rewrite: publish already skips those named values unless an override provides them, so pointing a reference at an affixed name would point it at nothing. One shared check now decides both.

Behaviour changes users may notice

  • Extracts that exited with code 1 because of Liquid output, commented-out references or backend-id="{{…}}" now exit with code 0.
  • A reference that is only a named value's resource name is no longer followed by extract, and publish now stops on it, naming the display name to use. Artifacts extracted by v0.1.7-alpha.0 and v0.2.0-alpha.0 reference the logger key named value by its ID; the publish-time rewrite that repaired those is removed, so re-extract or change the reference.
  • Filtered extract and publish bring in named values used in credentials, and the resource a named value names.
  • Publish may create a backend, logger or schema that only a policy outside the filter uses, when a published named value names it.
  • With an environment prefix or suffix, more references are rewritten (see above), and a named value used as a resource name gets the affixed name as its value — wherever else the named value is used, too.
  • Policy tokens follow a display name an explicit override changes, even without a prefix.
  • Extract and filtered publish follow backends named by llm-content-safety and embeddings-backend-id, and backend or fragment attributes written after another attribute (including one holding a policy expression) or in single quotes.
  • With an environment prefix or suffix, references inside XML comments are no longer rewritten.

Validation

  • npm test: all pass except the 2 environment-related package-build integration tests that also fail on main. npm run lint and npm run build are clean, and every commit passes on its own.
  • Integration runs on 2026-11-06 against StandardV2 source and target instances in the public cloud (the filtering fixture), each case on main and on this branch:
Case main This branch
Extract orders and billing with dependencies exit 1: four false "not found" errors (old-backend from a comment, body.id and context.Request.Url from Liquid, a backend named {{demo-backend-id}}) exit 0, no warnings; also extracts function-test-key from function-backend's credentials
Extract orders alone exit 1, same errors exit 0; pulls in function-backend through backend-id="{{demo-backend-id}}" and function-test-key from its credentials
Filtered publish of orders publishes only demo-backend-id (its display name equals its name); misses ai-key ({{AI_Key}}), function-backend, function-test-key publishes all four, function-backend before demo-backend-id
Full publish to an empty target — exit 0; pool after function-backend, demo-backend-id after both
Incremental publish where only demo-backend-id changes, to a new backend APIM rejects the named value ("breaks policies"), then creates the backend; exit 1 new-backend, then demo-backend-id; exit 0
A source policy using {{ai-key}} (resource name) publishes 62 resources, then APIM rejects the policy; exit 1 stops before any change (and in dry-run), naming {{AI_Key}}; exit 2
Full publish with namePrefix: dev- (appliesTo: [NamedValue, Backend]) policies keep {{AI_Key}}; dev-demo-backend-id keeps the value function-backend; dev-function-backend keeps {{function-test-key}} — dev reads the other environment's resources; the spaced Liquid {{ demo-backend-id }} is rewritten {{dev-AI_Key}}, backend-id="{{dev-demo-backend-id}}", dev-demo-backend-id = dev-function-backend, dev-function-backend uses {{dev-function-test-key}}, dev-pool → dev-function-backend, logger ai uses {{dev-AI_Key}}; Liquid output and the commented reference untouched; APIM accepts each
Full publish with the default appliesTo — the workspace logger ws-ai publishes (on main it fails: its key reference isn't affixed)
  • APIM behaviour confirmed: a policy referencing a named value by resource name is rejected on save; a reference to a missing named value inside an XML comment is accepted and kept as written, while the same reference outside a comment is rejected; references match display names ignoring case, and APIM stores them in the display name's case ({{ai_key}} is saved as {{AI_Key}}), so extracted policies always use it.

Not verified

  • log-to-eventhub logger-id and schema-id with a prefix (the fixture has no Event Hub logger; GlobalSchema isn't in the default appliesTo). Both are covered by unit tests.
  • Found while testing, unchanged by this PR and also on main: with a prefix on a shared instance, tags, groups, products and APIs keep their display names, which APIM (StandardV2) rejects as duplicates with HTTP 502; the built-in azuremonitor logger can't be affixed ("Logger id and type combination is reserved"); and a second auto-generated logger key in the same workspace is rejected.

Related Issue(s)

Closes #276
Closes #315
Closes #331
Refs #305 (closed by its reporter before this was fixed)

Copilot AI balanced review requested due to automatic review settings November 6, 2026 06:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved dependency-scanning and incremental-publication gaps can omit required backends, leave incorrect references, and let policy tokens bypass validation before a publish fails after partial changes.

3 open issues
What changed in this PR

Aligns named-value reference handling across extraction and publishing with APIM’s scope-aware, case-insensitive display-name matching. Addresses Liquid and commented references, credential dependencies, and resources named indirectly through named values.

Changes:

  • Centralizes reference matching and expands dependency discovery.
  • Adds publish-time validation, dependency ordering, and environment-aware reference rewrites.
  • Adds regression coverage and updates filtering and multi-environment documentation.
File Description
tests/​unit/​services/​workspace-extractor.test.ts Updates workspace named-value fixtures.
tests/​unit/​services/​transitive-resolver.test.ts Tests scanning, credentials, and indirect dependencies.
tests/​unit/​services/​transitive-extractor.test.ts Tests display-name resolution and Liquid extraction.
tests/​unit/​services/​resource-publisher.test.ts Tests logger reference handling and overrides.
tests/​unit/​services/​resource-publisher.env-mapping.test.ts Tests scoped display-name and credential rewrites.
tests/​unit/​services/​publish-service.test.ts Covers dependency expansion, validation, rewrites, and ordering.
tests/​unit/​services/​policy-ref-rewriter.test.ts Covers additional attributes and Liquid-aware rewrites.
tests/​unit/​services/​named-value-names.test.ts Tests shared matching and target-name rules.
tests/​unit/​services/​extract-service.test.ts Updates named-value listing fixtures.
src/​services/​transitive-resolver.ts Adds shared scanning and artifact reference resolution.
src/​services/​transitive-extractor.ts Resolves display names and handles Liquid candidates.
src/​services/​secret-redaction-guard.ts Reuses the auto-generated named-value skip check.
src/​services/​resource-publisher.ts Rewrites references and indirect resource names.
src/​services/​publish-service.ts Adds reference validation, expansion, and tier-one ordering.
src/​services/​policy-ref-rewriter.ts Uses shared matching for tokens and resource attributes.
src/​services/​override-merger.ts Centralizes auto-generated named-value skipping.
src/​services/​named-value-names.ts Defines scoped display-name matching rules.
src/​models/​config.ts Expands the known-artifact lookup interface.
docs/​reference/​dependency-graph.md Documents tier-one dependency ordering.
docs/​guides/​multi-environment-shared-apim.md Updates automatic reference rewrite behavior.
docs/​guides/​filtering-resources.md Explains matching and transitive dependencies.
docs/​guides/​environment-overrides.md Documents expanded rewrites and legacy logger references.

Review effort: Balanced

Comment thread src/services/publish-service.ts Resolved
Comment thread src/services/publish-service.ts Resolved
Comment thread src/services/transitive-resolver.ts Resolved
…th quoted strings

APIM's raw XML keeps a policy expression's string literals as written, so
an attribute value can hold double quotes:
`<set-backend-service sf-service-instance-name="@("fabric:/app")"
backend-id="orders" />`. The attribute patterns ended that value at the
first inner quote and never reached backend-id, so the dependency scan
missed the backend and the prefix rewrite left it unaffixed.

The patterns now run on a copy with each policy expression's inside
blanked (blankPolicyExpressions: `@(...)` and `@{...}`, balancing brackets
and skipping C# string and character literals), the same length, so the
rewriter applies its matches to the original text by offset. A value that
is an expression keeps its leading `@`.
…publish checks

In an incremental publish, a changed API is published with its unchanged
child policies (planApiPublication), and a changed product with its
policy (planProductPolicyPublication), although those aren't publish
targets themselves. The checks that stop publish before any change, for
a reference to a named value's resource name and for the redaction
marker, scanned only the targets, so such a policy got past them and APIM
rejected it after earlier resources were already published.

Both checks now scan the targets and those policies. In a full publish
they are targets already, so nothing more is read.
Copilot AI balanced review requested due to automatic review settings November 6, 2026 07:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved expression-comment parsing and workspace-casing mismatches can hide live dependencies, bypass reference validation, and cause backend pools to publish before their members exist.

1 open issue
2 previously missed issues

In code that hasn't changed since last review

Medium severity Workspace casing prevents pool member ordering matches

src/​services/​publish-service.ts:1082

The ordering key leaves workspace casing unchanged, so a pool member ID containing /workspaces/TEAM-A/backends/caller does not match the caller descriptor in workspace team-a. If that backend is deferred because its credentials use a late named value, its pool remains in the earlier wave and can be published before its member exists. Lowercase the workspace as well, matching the case-insensitive identity used by getResourceDescriptorKey().

Medium severity Expression comments hide live references from dependency scanning

src/​services/​transitive-resolver.ts:154

A <!-- inside a policy-expression string consumes everything through a later real XML comment, so live backend and {{Api_Key}} references between them disappear from the scan. Filtered extract/publish then omits dependencies, and the reference gate can miss invalid tokens. Identify comment spans with expression awareness while preserving references inside expressions, and add a regression with a real comment after the live references.

3 resolved since last review

Review effort: Balanced

Comment thread src/services/transitive-resolver.ts Resolved
…d expressions in one pass

An apostrophe in a C# comment inside a policy expression was read as a
character literal, so the expression stayed unblanked and a quoted string
in it hid the reference attributes after it from the scan and from
prefixed publishing. A `<!--` in an expression string was taken for an
XML comment running to a later real `-->`, hiding the references between.
Expressions now skip `//` and `/* */` comments, and XML comments and
expressions are found in one pass, so neither starts inside the other.
…d values

The Tier 1 ordering key lowercased the name but not the workspace, so a
pool member ID spelling its workspace differently (`TEAM-A`) didn't match
its backend in `team-a`, and the pool could be published before a member
deferred for a named value.
Copilot AI balanced review requested due to automatic review settings November 6, 2026 07:26
@cdayne

Copy link
Copy Markdown
Author

On the two items the latest review lists as previously missed:

  • <!-- inside an expression string (transitive-resolver.ts): fixed in 7a08cae. XML comments and policy expressions are now found in one pass, so a <!-- inside an expression starts no comment, and an @( or @{ inside a comment starts no expression. Comments are blanked rather than removed, and every scan that ignored comments uses the same pass. Added a regression with your case: the <!-- string, then a live backend and {{Api_Key}}, then a real comment. Both references are found.
  • Workspace casing in the pool member ordering key (publish-service.ts): fixed in 9150d66. The key now lowercases the workspace as well as the name. Added a test where the pool's member ID spells the workspace TEAM-A, the member's backend is in team-a, and the member is deferred for a named value; the pool is published after it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Interpolated-string parsing can miss backend dependencies and leave IDs unchanged, while token replacement inside XML comments can make policies invalid.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Handle nested literals in interpolated C# strings

src/​services/​transitive-resolver.ts:175

Interpolated C# strings can contain quoted literals inside {...}, but this loop treats those quotes as ending the outer string. For example, @($"fabric:/{context.Variables.GetValueOrDefault<string>("s)v")}") stops blanking at the ) inside "s)v"; when this precedes backend-id, dependency scanning misses the backend and prefix rewriting leaves its ID unchanged. Make string scanning handle interpolation holes and nested literals, and add scan/rewrite regressions.

Medium severity Skip XML comments when rewriting tokens

src/​services/​transitive-resolver.ts:207

The blanked copy only locates Liquid bodies; replaceIn still rewrites tokens inside XML comments in the original slices. An override from Key to Prod--Key therefore turns <!-- {{Key}} --> into <!-- {{Prod--Key}} -->, making the policy invalid because XML comments cannot contain --, even without an environment mapping. Match replacements against a comment-blanked copy and preserve the original comment text.

1 resolved since last review

Review effort: Balanced

A quote inside an interpolated string's `{...}` (`$"{ctx.Get("s)v")}"`)
was taken for the end of the string, so a bracket in the nested literal
ended the expression early, and the attributes after it were hidden from
the scan and from prefixed publishing. Each `{...}` in a `$"..."` string
is now read as code, nested literals included.
Publish rewrote `{{name}}` tokens and resource-name attributes inside XML
comments, and a rewritten name with `--` (an override to `Prod--Key`)
made the comment, and the policy, invalid XML. APIM doesn't substitute
named values inside comments: on a StandardV2 instance it accepts a
commented reference to a missing named value and rejects the same
reference outside a comment. Comments are now left as written, as the
scans already ignore them.
Copilot AI balanced review requested due to automatic review settings November 6, 2026 07:50
@cdayne

Copy link
Copy Markdown
Author

On the two items the latest review lists as previously missed:

  • Nested literals in interpolated strings (transitive-resolver.ts): fixed in c4534f0. Each {...} in a $"..." string (and $@"..." or @$"...") is now read as code, including any string or character literals inside it; {{ is still a literal brace. Added regressions with your example: the scan finds the backend after it, and a prefixed publish rewrites that backend's ID.
  • Tokens inside XML comments (transitive-resolver.ts): fixed in dffdab2. Publish now leaves XML comments as written, for {{name}} tokens and for the resource-name attributes, matching the scans, which already ignore comments. I checked APIM before choosing this: on a StandardV2 instance, a policy fragment with <!-- {{no-such-nv}} --> saved, with the comment kept as written, while the same reference outside a comment failed with "Cannot find a property 'no-such-nv'". So APIM doesn't substitute named values inside comments. Added a test with a comment holding a token and a backend-id, followed by an expression string holding <!--. The multi-environment guide's note on comments is updated.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because subtle issues in policy parsing, dependency ordering, and workspace scoping can cause missed references, publish failures, or omitted credential dependencies.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants