[netomi]

Towards #1450.

The bug

createSignatureFile streamed the package into BouncyCastle's Ed25519Signer in 1 KiB chunks. That reads as the memory-safe way to do it and is the opposite of one:

final class org.bouncycastle.crypto.signers.Ed25519Signer$Buffer extends java.io.ByteArrayOutputStream

Pure Ed25519 (RFC 8032) hashes the message twice — once for the nonce H(prefix ‖ M), once for the challenge H(R ‖ A ‖ M) — so a signer cannot discard what it is fed. BouncyCastle keeps it in a ByteArrayOutputStream, which doubles its capacity as it fills and allocates every intermediate array on the way.

The ~300 MB package in #1450 therefore ended up in a 512 MB array with the 256 MB one it had been copied from still live — three quarters of a gigabyte to sign 300 MB. That is why 1 GB of heap failed and 2 GB worked, and the reported stack trace comes straight out of that growth: ByteArrayOutputStream.write → ensureCapacity → Arrays.copyOf.

Worth noting #1458 fixed a different OOM on this path (the MAX_CONTENT_SIZE check). At 304.6 MB the reporter's file is under the 512 MB limit, so it passes that check and dies later, in signing — as they predicted in the thread.

The change

Read the file into one exactly sized array and sign that. Measured on a 9 MiB package by the test below:

	allocated	ratio
streaming into Ed25519Signer	33,560,752 bytes	3.6×
read in one go	9,445,000 bytes	1.0×

The signature is unchanged. Ed25519Signer does no more than accumulate the message and hand it to Ed25519PrivateKeyParameters.sign, which is what this now calls directly with the same algorithm and a null context. signsExactlyAsTheStreamingSignerDid asserts the two produce identical bytes, so signatures already published keep matching what a re-sign produces.

Verification had the same defect the other way round — on the mirroring path, via MirrorExtensionService — and is fixed the same way. It now also reports a PEM holding something other than an Ed25519 key, instead of letting a ClassCastException out of Ed25519Signer.init.

What this does not fix

A smaller heap, not a bounded one. It still scales with the package, so an instance with the integrity service enabled wants a heap above ovsx.publishing.max-content-size (512 MB by default). Constant memory isn't reachable through BouncyCastle: every sign overload it exposes for pure Ed25519 takes a byte[], and its only streaming signer implements Ed25519ph — a different scheme, whose signatures nothing verifying these packages today would accept. Switching to it, or shelling out to an external signer, is the discussion in #1450 and belongs in its own issue.

The silent failure is untouched. #1450's second complaint is that the version is left active = false while the CLI reports success. That is still true, and there are two reasons for it worth recording:

• publishAsync catches Exception to call markScanAsErrored, but OutOfMemoryError is an Error, so even an instance with scanning enabled records nothing.
• doPublish carries @Retryable, and the comment above it assumes retries happen — but it is a private method invoked from publishAsync in the same class, so the proxy is bypassed and the annotation never applies. (@EnableResilientMethods is on RegistryApplication, and the @Retryable on the public createExtensionVersion does work.)

Both are behavioural changes rather than a memory fix, so they are deliberately not in here.

Tests

• signsExactlyAsTheStreamingSignerDid — byte-identical signatures, the guard against a format change.
• allocatesThePackageOnceWhereStreamingAllocatedItSeveralTimesOver — measures both ways round with per-thread allocation counters, prints the numbers, and asserts the shipped path stays under 1.5× the package while the streaming one exceeds 2×. Thresholds sit well clear of the measured 1.0× and 3.6× so it fails on a regression rather than on JVM noise; it skips where the JVM does not report per-thread allocation.
• The pre-existing testGenerateSignature, which checks a real .vsix against a recorded sigzip fixture, still passes.

Full server suite green (1167 tests).

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

It introduces user-facing and robustness issues (misleading key-read error message, missing signature-length validation, and a potentially flaky allocation-measurement helper in tests) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR changes the extension signature generation and verification flow to avoid excessive heap allocation when signing/verifying large VSIX packages by bypassing BouncyCastle’s streaming Ed25519Signer buffering behavior and using direct Ed25519 byte[] APIs instead.

Changes:

• Switch signing to Ed25519PrivateKeyParameters.sign(...) over a single byte[] message buffer.
• Switch verification to Ed25519PublicKeyParameters.verify(...) over a single byte[] message buffer, with improved public-key type diagnostics.
• Add regression tests asserting signature equivalence vs the prior streaming approach and measuring allocation behavior.

File summaries

File	Description
server/src/main/java/org/eclipse/openvsx/publish/ExtensionVersionIntegrityService.java	Updates signing/verification to avoid multi-growth buffering and improves key-type error reporting.
server/src/test/java/org/eclipse/openvsx/publish/ExtensionVersionIntegrityServiceTest.java	Adds equivalence + allocation-regression tests for the updated signing approach.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.