Skip to content

Throttle migration item scheduling to avoid crowding out real jobs - #2096

Merged
netomi merged 4 commits into
mainfrom
feature/throttle-migration-item-scheduling
Aug 25, 2026
Merged

netomi merged 4 commits into
mainfrom
feature/throttle-migration-item-scheduling

Conversation

@netomi

@netomi netomi commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What

MigrationItemJobRequestHandler used to pull up to 25000 not-yet-migrated migration_item rows every 15 minutes and enqueue() all of them immediately (due "now"). I confirmed by inspecting JobRunr's own polling code (Paging.AmountBasedList.ascOnUpdatedAt, used by OnboardNewWorkTask) that every ENQUEUED/due job across the whole server sits in one strict FIFO queue ordered by due time — there's no per-job-type priority in open-source JobRunr. So a burst of 25000 jobs all due "now" meant any real, user-triggered job (extension published, signature generated, etc.) enqueued around the same time would sort in behind however many of those 25000 came first, delaying it until the burst drained.

How

  • MigrationItemJobRequestHandler.batchSize: configurable via ovsx.migrations.batch-size (default 200, down from the hardcoded 25000) — directly bounds how many migration jobs can ever be due ahead of a real job at once.
  • Each item in a batch is now scheduler.schedule()d at a staggered offset (200ms apart) instead of enqueue()d all for "now" — a full batch at the default size spreads over ~40s, comfortably inside the default 15-minute recurring interval.
  • MigrationScheduler's recurring cron: configurable via ovsx.migrations.cron (default */15 * * * *, identical to the previous hardcoded Cron.every15minutes()).
  • MigrationService.enqueueMigration now takes the target Instant and calls scheduler.schedule(...) instead of scheduler.enqueue(...) so the staggering above actually takes effect.
  • Logging: now reports how many items remain after each batch (new RepositoryService.countNotMigratedItems()), since draining a large backlog can now take many recurring runs instead of one, and the old per-batch count alone gave no sense of overall progress. Skipped entirely when a batch finds nothing to do, so it doesn't log a pointless "scheduled 0 items" line (and doesn't run the count query behind it) on ticks where there's nothing left — an empty slice already means hasNext() is false too, so the "migration completed" branch still fires correctly on its own.

Testing

  • MigrationItemJobRequestHandlerTest (new): read-only short-circuit, configured batch size is honored, items get strictly-increasing evenly-spaced schedule times, remaining-count query runs only when items were found, recurring-job deletion logic in both directions.
  • MigrationSchedulerTest (new): configured cron is passed through to scheduleRecurrently.
  • Full unitTests suite passes (957 tests).

Note

I also looked into detecting "the current batch has fully drained" and immediately scheduling the next one, instead of relying on the cron interval at all. That's possible but needs a schema change (a way to distinguish "still in flight" from "permanently failed and parked for investigation", which currently look identical in migration_item) — happy to pursue as a follow-up if the cron-based approach here turns out not to be responsive enough in practice.

🤖 Generated with Claude Code

JobRunr processes every ENQUEUED/due job across the whole server in one
strict FIFO queue ordered by due time (confirmed by inspecting JobRunr's
own polling code -- Paging.AmountBasedList.ascOnUpdatedAt). Enqueuing up
to 25000 migration jobs at once, all due immediately, meant any real,
user-triggered job that happened to get enqueued around the same time
would sit behind however many of those 25000 sorted ahead of it by due
time, delaying it until the burst drained.

- MigrationItemJobRequestHandler.batchSize is now configurable via
  ovsx.migrations.batch-size (default 200, down from the hardcoded
  25000), directly bounding how many migration jobs can ever be due
  ahead of a real job at once.
- Each item in a batch is now scheduled at a staggered offset (200ms
  apart) instead of all being enqueued for "now" -- a full batch spreads
  over ~40s at the default batch size, comfortably inside the default
  15-minute recurring interval.
- MigrationScheduler's recurring cron is now configurable via
  ovsx.migrations.cron (default */15 * * * *, identical to the previous
  hardcoded Cron.every15minutes()).
- MigrationService.enqueueMigration now takes the target Instant and
  calls scheduler.schedule(...) instead of scheduler.enqueue(...) so the
  staggering above actually takes effect.
- Logging: now reports how many items remain after each batch
  (RepositoryService.countNotMigratedItems()), since draining a large
  backlog can now take many recurring runs instead of one and the old
  per-batch count alone gave no sense of overall progress. Skipped
  entirely when a batch finds nothing to do, avoiding a pointless
  "scheduled 0 items" log line (and the count query behind it) on runs
  where there's nothing left.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces the impact of migration backlogs on user-triggered JobRunr jobs by throttling how many migration items are scheduled per run and by staggering their due times, while also making migration scheduling configurable and improving progress logging/visibility.

Changes:

  • Introduces configurable migration batch sizing and staggered scheduling so migration items don’t all become due “now”.
  • Makes the recurring migration scheduling cron configurable.
  • Adds backlog-counting and new unit tests covering the scheduling/throttling behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
server/src/main/java/org/eclipse/openvsx/migration/MigrationItemJobRequestHandler.java Limits batch size via config and staggers scheduled instants; adds backlog logging.
server/src/main/java/org/eclipse/openvsx/migration/MigrationScheduler.java Makes recurring cron configurable and passes it to JobRunr.
server/src/main/java/org/eclipse/openvsx/migration/MigrationService.java Changes migration job submission from immediate enqueue to scheduled-at-instant.
server/src/main/java/org/eclipse/openvsx/repositories/RepositoryService.java Adds a service method to count not-yet-migrated items for logging/progress.
server/src/main/java/org/eclipse/openvsx/repositories/MigrationItemRepository.java Adds a Spring Data count query for not-yet-migrated items.
server/src/test/java/org/eclipse/openvsx/migration/MigrationItemJobRequestHandlerTest.java New tests for batch sizing, staggering, backlog-count query behavior, and recurring-job deletion behavior.
server/src/test/java/org/eclipse/openvsx/migration/MigrationSchedulerTest.java New test verifying the configured cron is passed through to scheduleRecurrently.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

netomi and others added 3 commits August 25, 2026 10:17
- Clamp ovsx.migrations.batch-size to a minimum of 1 (with a warning)
  instead of letting a misconfigured 0/negative value make
  PageRequest.ofSize(...) throw and fail the recurring job on every
  single run.
- Rename MigrationService.enqueueMigration to scheduleMigration: it now
  schedules a job for a future Instant rather than enqueueing it for
  immediate execution, so the old name no longer matched what it does.

(Two other comments flagged the 5-field default cron ("*/15 * * * *")
as potentially invalid if JobRunr expects 6 fields elsewhere in this
codebase. That doesn't apply here: I decompiled JobRunr's own
CronExpression constructor, which splits the expression and accepts
either 5 fields (minute resolution) or 6 (second resolution) per
string, with no notion of a codebase-wide mode -- the 6-field cron
Copilot pointed at (UsageStatsService) uses Spring's unrelated
@scheduled(cron=...), not JobRunr's scheduler at all. Left as-is.)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@netomi
netomi merged commit e7ba541 into main Aug 25, 2026
5 checks passed
@netomi
netomi deleted the feature/throttle-migration-item-scheduling branch August 25, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants