APIOps Toolkit artifacts only partly compatible #304

Command that triggered the bug

apiops publish (also affects apiops extract)

Expected behavior

The migration guide (docs/guides/migration-from-v1.md) says existing APIOps Toolkit artifacts "should work as-is with apiops-cli" and that "the artifact format is backward compatible". We therefore expected apiops publish, run against a repository extracted by the Toolkit, to publish every resource in it, including named values, policy fragments and version sets. If some artifacts couldn't be read, we'd at least expect a warning or error, so that a partial publish can't go unnoticed.

Any of these would help:

1. Accept the Toolkit names on publish: "named values", "policy fragments", "version sets", apiVersionSetInformation.json, and a policy fragment's policy.xml when properties.value is absent.
2. Warn about artifact paths that aren't recognised, especially unknown top-level folders.
3. Update the migration guide to list these differences, and provide an in-place conversion step for existing repositories (for example an apiops migrate command; related to Migration guide from APIOps Toolkit to APIOps CLI #133 and Add interactive prompt for migrating from APIOps Toolkit to APIOps CLI #146), so teams aren't pushed towards re-extracting.
4. Store policy fragment bodies as policy.xml, as the CLI already does for API, product and operation policies, and fall back to properties.value when there is no policy.xml. This keeps fragment changes reviewable and removes one of the incompatibilities with the Toolkit.

Actual behavior

APIs, backends, products and their policies are published, but named values, policy fragments and version sets in the Toolkit layout are silently skipped, with no warning or error. The dry run that the migration guide recommends also passes, because the skipped resources simply don't appear in the plan. You'd only notice by counting them.

The differences below still exist in the latest Toolkit release (v7.0.4) and in apiops-cli main.

Differences between the Toolkit and apiops-cli layouts

1. Named values folder

   • Toolkit v7.0.4: "named values//namedValueInformation.json"
   • apiops-cli: "namedValues//namedValueInformation.json"

2. Policy fragments folder

   • Toolkit v7.0.4: "policy fragments//..."
   • apiops-cli: "policyFragments//..."

3. Policy fragment body

   • Toolkit v7.0.4: a separate policy.xml next to policyFragmentInformation.json. v7 deliberately removes properties.value and properties.format from the JSON (FormatInformationFileDto in src/common/PolicyFragment.cs).
   • apiops-cli: read only from properties.value (format: rawxml) in policyFragmentInformation.json. A separate policy.xml in the fragment folder is ignored.

4. Version sets folder

   • Toolkit v7.0.4: "version sets//..."
   • apiops-cli: "versionSets//..."

5. Version set information file (v5 only)

   • Toolkit v5.0.1: apiVersionSetInformation.json
   • Toolkit v7.0.4 and apiops-cli: versionSetInformation.json

Sources:

• Azure/apiops v7.0.4: src/common/NamedValue.cs, src/common/PolicyFragment.cs, src/common/VersionSet.cs
• Azure/apiops v5.0.1: tools/code/common/NamedValue.cs, tools/code/common/PolicyFragment.cs, tools/code/common/ApiVersionSet.cs
• apiops-cli main: RESOURCE_TYPE_METADATA in src/models/resource-types.ts

What we observed

On our repository, mostly extracted with Toolkit v7.0.4 (809 artifact files), running apiops-cli 1.0.4 with no changes:

• 217 files under "named values/", "policy fragments/" and "version sets/" were not recognised, with no warning or error.
• The CLI's artifact discovery found 507 resources, compared with 701 after converting the layout.
• After converting, all 23 policy fragment bodies were read back exactly as they appear in the Toolkit output.

Extract has the same problem in reverse

apiops extract always writes namedValues/, policyFragments/ and versionSets/. Run against an existing Toolkit repository, it creates these alongside the existing "named values/", "policy fragments/" and "version sets/" folders instead of replacing them. The old folders stay behind and still look authoritative, but publish ignores them. So a team can't start using apiops extract until the repository has been migrated, which means publish and extract have to switch over at the same time.

Policy fragment bodies become hard to review

The Toolkit stores each fragment's XML in its own policy.xml. apiops-cli stores it as a string in policyFragmentInformation.json (properties.value). A JSON string can't contain real line breaks, so the whole policy ends up on one escaped line. Every fragment change then shows up as that one line changing, which is hard to review and conflicts on any concurrent edit. This is also inconsistent with the rest of the CLI layout, where API, product and operation policies are stored as policy.xml files.

Why not just re-extract?

The obvious workaround is to re-extract from an APIM instance with apiops extract, so the artifacts come out in the CLI layout. That doesn't really work for us:

• Our git repository is the source of truth, not any single APIM instance. Artifacts are reviewed and merged in git, then published out to environments. Re-extracting would mean treating one instance as authoritative and hoping it exactly matches the repository.
• There are always several feature branches with work in progress on the artifacts. Git's rename detection deals with most of a folder rename. However, new files added under the old folder names on those branches are reported as conflicts, and any fragment policy edited on a branch conflicts with policy.xml being folded into the JSON file. Each of those has to be fixed by hand, branch by branch.
• A re-extract would also change every API's specification from specification.json to specification.yaml (see Add specification format selection to apiops extract for feature parity with Azure/apiops #257). Every spec would then show up as a new file, which hides the real changes in review.

What we need is a migration path where the CLI can publish our existing repository as-is while we move over. We can then switch the layout once, deliberately, when no branches are in flight.

Workaround

For now, before each publish we convert a copy of the Toolkit artifacts into the CLI layout. We rename the three folders, rename the v5 version set file, and inline each fragment's policy.xml into properties.value with format rawxml. The source is never modified, so the repository and the open branches stay unchanged. Happy to share the script.

apiops CLI version

1.0.4 (layout also checked against main)

Environment details

• Node.js 24.x
• Windows, self-hosted Azure DevOps agent; the pipeline steps run in Windows PowerShell 5.1
• Artifacts extracted with APIOps Toolkit (Azure/apiops) v5.0.1; layout also checked against v7.0.4

CI/CD environment

None

Is this bug blocking you?

Yes - too hard to switch to apiops-cli without backwards compatibility or a clear migration path