Repository navigation
fix: limit decompressed size when serving vscode/unpkg web resources #2060
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
30324b1
fix: limit decompressed size when serving vscode/unpkg web resources
netomi 452f705
fix: bound webResourceCache by both entry count and total byte size
netomi 2127cbd
fix: disable web resource size limit by default
netomi 0c6e571
Merge branch 'main' into fix/web-resource-decompression-limit
netomi 48c4ed1
ci: raise test task heap to 6144m to fix CI OutOfMemoryError
netomi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
64 changes: 64 additions & 0 deletions
64
server/src/main/java/org/eclipse/openvsx/cache/FileSizeWeigher.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| /****************************************************************************** | ||
| * Copyright (c) 2026 Contributors to the Eclipse Foundation. | ||
| * | ||
| * See the NOTICE file(s) distributed with this work for additional | ||
| * information regarding copyright ownership. | ||
| * | ||
| * This program and the accompanying materials are made available under the | ||
| * terms of the Eclipse Public License 2.0 which is available at | ||
| * https://www.eclipse.org/legal/epl-2.0. | ||
| * | ||
| * SPDX-License-Identifier: EPL-2.0 | ||
| *****************************************************************************/ | ||
| package org.eclipse.openvsx.cache; | ||
|
|
||
| import java.io.IOException; | ||
| import java.nio.file.Files; | ||
| import java.nio.file.Path; | ||
|
|
||
| import com.github.benmanes.caffeine.cache.Weigher; | ||
| import org.jspecify.annotations.NonNull; | ||
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
|
|
||
| /** | ||
| * Weighs a cached {@link Path} by its file size in bytes, so a {@code maximumWeight} cache bounds | ||
| * its total disk footprint instead of just the number of cached files. | ||
| * <p> | ||
| * Every entry is floored to at least {@code floorWeight} of the weight budget, even if its actual | ||
| * file is smaller. Combined with a cache's {@code maximumWeight(maxTotalSize)}, setting | ||
| * {@code floorWeight = ceil(maxTotalSize / maxEntries)} makes the same cache respect both a total | ||
| * byte budget and a maximum entry count at once: no more than {@code maxEntries} entries can ever | ||
| * fit, since each one claims at least a {@code 1/maxEntries} share of the budget, while entries | ||
| * larger than that share are still weighed by their real size so the byte budget itself holds. | ||
| */ | ||
| public class FileSizeWeigher implements Weigher<Object, Object> { | ||
| private static final Logger logger = LoggerFactory.getLogger(FileSizeWeigher.class); | ||
|
|
||
| private final long floorWeight; | ||
|
|
||
| /** | ||
| * @param floorWeight the minimum weight charged to every entry, regardless of its actual file | ||
| * size; pass {@code 0} to weigh purely by actual file size. | ||
| */ | ||
| public FileSizeWeigher(long floorWeight) { | ||
| this.floorWeight = floorWeight; | ||
| } | ||
|
|
||
| @Override | ||
| public int weigh(@NonNull Object key, @NonNull Object value) { | ||
| if (!(value instanceof Path path)) { | ||
| return 1; | ||
| } | ||
|
|
||
| try { | ||
| var size = Math.max(Files.size(path), floorWeight); | ||
| return (int) Math.min(size, Integer.MAX_VALUE); | ||
| } catch (IOException e) { | ||
| // Can't determine the size, e.g. the file was already deleted; weigh it heavily so | ||
| // it doesn't linger and displace entries whose size is known. | ||
| logger.warn("Failed to determine size of cached file {}", path, e); | ||
| return Integer.MAX_VALUE; | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
138 changes: 138 additions & 0 deletions
138
server/src/test/java/org/eclipse/openvsx/adapter/WebResourceServiceTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| /******************************************************************************** | ||
| * Copyright (c) 2026 Contributors to the Eclipse Foundation | ||
| * | ||
| * See the NOTICE file(s) distributed with this work for additional | ||
| * information regarding copyright ownership. | ||
| * | ||
| * This program and the accompanying materials are made available under the | ||
| * terms of the Eclipse Public License 2.0 which is available at | ||
| * https://www.eclipse.org/legal/epl-2.0 | ||
| * | ||
| * SPDX-License-Identifier: EPL-2.0 | ||
| ********************************************************************************/ | ||
| package org.eclipse.openvsx.adapter; | ||
|
|
||
| import java.nio.file.Files; | ||
| import java.nio.file.Path; | ||
| import java.nio.file.Paths; | ||
|
|
||
| import org.junit.jupiter.api.Test; | ||
|
|
||
| import org.eclipse.openvsx.cache.CacheService; | ||
| import org.eclipse.openvsx.cache.FilesCacheKeyGenerator; | ||
| import org.eclipse.openvsx.repositories.RepositoryService; | ||
| import org.eclipse.openvsx.storage.StorageUtilService; | ||
| import org.eclipse.openvsx.util.ErrorResultException; | ||
|
|
||
| import static org.assertj.core.api.Assertions.assertThat; | ||
| import static org.assertj.core.api.Assertions.assertThatThrownBy; | ||
| import static org.mockito.Mockito.mock; | ||
| import static org.springframework.http.HttpStatus.CONTENT_TOO_LARGE; | ||
| import static org.springframework.http.HttpStatus.INTERNAL_SERVER_ERROR; | ||
|
|
||
| class WebResourceServiceTest { | ||
|
|
||
| private final FilesCacheKeyGenerator filesCacheKeyGenerator = new FilesCacheKeyGenerator(); | ||
|
|
||
| @Test | ||
| void testRejectsEntryDeclaredLargerThanLimit() throws Exception { | ||
| var extensionDownloadPath = resourcePath("wrong-size.zip"); | ||
| var service = newService(8192L); | ||
|
|
||
| assertThatThrownBy( | ||
| () -> service.getWebResource( | ||
| "ns", | ||
| "ext", | ||
| null, | ||
| "1.0.0", | ||
| "extension/README.md", | ||
| extensionDownloadPath)) | ||
| .isExactlyInstanceOf(ErrorResultException.class) | ||
| .hasMessage("The file extension/README.md exceeds the size limit of 8 KB.") | ||
| .satisfies(e -> assertThat(((ErrorResultException) e).getStatus()).isEqualTo(CONTENT_TOO_LARGE)); | ||
|
|
||
| // rejected before any extraction: nothing should have been written to the file cache | ||
| var cachedPath = filesCacheKeyGenerator | ||
| .generateCachedWebResourcePath("ns", "ext", null, "1.0.0", "extension/README.md", ".md"); | ||
| assertThat(Files.exists(cachedPath)).isFalse(); | ||
| } | ||
|
|
||
| @Test | ||
| void testStopsCopyWhenActualBytesExceedDeclaredSize() throws Exception { | ||
| // extension/package.json declares a size of 0 but its compressed payload decompresses to | ||
| // more than 0 bytes, simulating a zip entry whose header lies about the decompressed size | ||
| var extensionDownloadPath = resourcePath("wrong-size.zip"); | ||
| var service = newService(33_554_432L); | ||
|
|
||
| assertThatThrownBy( | ||
| () -> service.getWebResource( | ||
| "ns", | ||
| "ext", | ||
| null, | ||
| "1.0.0", | ||
| "extension/package.json", | ||
| extensionDownloadPath)) | ||
| .isExactlyInstanceOf(ErrorResultException.class) | ||
| .hasMessageContaining("File size exceeds limit of 0 bytes") | ||
| .satisfies(e -> assertThat(((ErrorResultException) e).getStatus()).isEqualTo(INTERNAL_SERVER_ERROR)); | ||
|
|
||
| // the partial file written before the limit tripped must not be left behind | ||
| var cachedPath = filesCacheKeyGenerator | ||
| .generateCachedWebResourcePath("ns", "ext", null, "1.0.0", "extension/package.json", ".json"); | ||
| assertThat(Files.exists(cachedPath)).isFalse(); | ||
| } | ||
|
|
||
| @Test | ||
| void testAllowsLargeFileWhenLimitIsDisabled() throws Exception { | ||
| // extension/README.md declares 31781 bytes -- larger than the 8 KB limit rejected in | ||
| // testRejectsEntryDeclaredLargerThanLimit -- but a negative maxFileSize (the default) means | ||
| // no file is too large, matching the pre-existing behavior of this endpoint. | ||
| var extensionDownloadPath = resourcePath("wrong-size.zip"); | ||
| var service = newService(-1L); | ||
|
|
||
| var cachedPath = service.getWebResource( | ||
| "ns", | ||
| "ext", | ||
| null, | ||
| "1.0.0", | ||
| "extension/README.md", | ||
| extensionDownloadPath); | ||
|
|
||
| assertThat(cachedPath).isNotNull(); | ||
| assertThat(Files.size(cachedPath)).isEqualTo(31781); | ||
| Files.deleteIfExists(cachedPath); | ||
| } | ||
|
|
||
| @Test | ||
| void testServesFileWithinLimit() throws Exception { | ||
| var extensionDownloadPath = resourcePath("todo-tree.zip"); | ||
| var service = newService(33_554_432L); | ||
|
|
||
| var cachedPath = service.getWebResource( | ||
| "ns", | ||
| "ext", | ||
| null, | ||
| "1.0.0", | ||
| "extension/package.json", | ||
| extensionDownloadPath); | ||
|
|
||
| assertThat(cachedPath).isNotNull(); | ||
| assertThat(Files.size(cachedPath)).isEqualTo(44712); | ||
| Files.deleteIfExists(cachedPath); | ||
| } | ||
|
|
||
| private WebResourceService newService(long maxFileSize) { | ||
| return new WebResourceService( | ||
| mock(StorageUtilService.class), | ||
| mock(RepositoryService.class), | ||
| mock(CacheService.class), | ||
| filesCacheKeyGenerator, | ||
| maxFileSize); | ||
| } | ||
|
|
||
| private Path resourcePath(String name) throws Exception { | ||
| var url = getClass().getResource("/org/eclipse/openvsx/util/" + name); | ||
| assertThat(url).isNotNull(); | ||
| return Paths.get(url.toURI()); | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.