Skip to content

Share one concurrency budget between multiple crawlers #2133

Open
Open
Share one concurrency budget between multiple crawlers#2133
Assignees
Mantisus
Labels
enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.
Milestone
2.0

Description

@vdusek
vdusek
opened on Aug 7, 2026

Each crawler builds its own AutoscaledPool and SystemStatus, so several crawlers in one process oversubscribe the host - there is no way to cap their combined concurrency. JS v4 split the scaling logic out into a ConcurrencySystem that can be constructed once and passed to several crawlers.

JS implementation: apify/crawlee#3917

✍️ Drafted by Claude Code

Activity

  1. vdusek
    added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 7, 2026
  2. vdusek
    added this to the 2.0 milestone on Aug 7, 2026
  3. Mantisus
    self-assigned this
    on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions