Repository navigation
fix(update_stats): fetch downloads with a throttle npm accepts - #44
Open
pyramation wants to merge 1 commit into
Open
pyramation wants to merge 1 commit into
pyramation wants to merge 1 commit into
Conversation
update_stats.sh ran npm:fetch:downloads with the fetcher's defaults (2 at a time, 2500ms apart), which npm blocks with 429 after ~20 packages, so a run with ~1000 packages to fetch ran out of its 10 attempts. Fetch one package at a time, 3000ms apart (overridable with FETCH_CONCURRENT / FETCH_DELAY), for both the normal fetch and the backfill. The README's conservative 1500ms delay was also blocked after ~20 packages in Oct 2026; 3000ms fetched 931 packages with no 429s, so the README now recommends that. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012tvNcsFXLSSXxetcU8UgV4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
update_stats.shrannpm:fetch:downloadswith the fetcher's defaults (2 at a time, 2500ms apart). npm blocks that with 429 after ~20 packages, so the 2026-10-16 refresh used up all 10 retry attempts with ~950 packages still to fetch.FETCH_CONCURRENT/FETCH_DELAY.packages/stats-db/README.md: the "conservative" setting goes from--delay 1500to--delay 3000.Measured against npm:
-c 2 -d 2500)-c 1 -d 1500(old README recommendation)-c 1 -d 3000A full refresh of ~1150 packages now takes about an hour.
🤖 Generated with Claude Code
https://claude.ai/code/session_012tvNcsFXLSSXxetcU8UgV4