Syncpoint-enabled throughput drop due to dispatcher memory-quota starvation #4605

Background

When syncpoint is enabled, each dispatcher independently sends events to EventCollector.
Under DS memory quota pressure, fast dispatchers can continuously fill shared memory,
while slower dispatchers cannot enqueue enough events to make progress.

This creates unfair progress among dispatchers and hurts overall throughput.

Problem statement

In the current model, EventCollector memory is effectively consumed by whoever sends faster.
There is no global syncpoint-stage coordination in EventBroker to force balanced progress before
moving syncpoint forward.

As a result:

• Fast dispatchers keep advancing and occupy most DS memory quota.
• Slow dispatchers are starved and lag on checkpoint/syncpoint progression.
• Syncpoint completion is delayed by stragglers, causing long tail latency.
• End-to-end throughput drops because the pipeline repeatedly waits on the slowest blocked paths.

ASCII view of the original problem

Before fix (single-dispatcher-driven pushing)

  Dispatcher A (fast)  ---->  EventCollector queue  [##########]  (quota almost full)
  Dispatcher B (slow)  --x    enqueue blocked / tiny share
  Dispatcher C (slow)  --x    enqueue blocked / tiny share

  Shared DS memory quota:
  +---------------------------------------------------------------+
  | A: 85-95%                                                     |
  | B: 3-8%                                                       |
  | C: 2-7%                                                       |
  +---------------------------------------------------------------+

  Syncpoint view:
    A: sentResolvedTs/checkpointTs keeps moving
    B,C: lagging, cannot catch up
    => global syncpoint commit waits for lagging dispatchers
    => high syncpoint latency + lower throughput

Why this is a performance issue

It is a steady-state performance issue under skewed
workloads: memory quota is monopolized by fast paths, causing starvation and prolonged syncpoint
barriers, which directly reduces throughput and increases p99 latency.