Skip to content

Live sessions: emit an instrument-scoped Gap across a provider reconnect #88

Open

Description

@Masaga

A live session has no in-band signal for the prints lost during a provider outage, and on the iceoryx2 data plane a consumer cannot recover that information from the stream at all.

Observed (v0.10.0).

  1. The Alpaca provider reports a websocket drop as a ProviderDisconnected / ProviderConnected pair and reconnects with backoff, re-applying the subscription set (
    //! # Reconnect
    //!
    //! On websocket failure the streaming task tears down the client, sleeps with
    //! exponential backoff per the configured [`ReconnectPolicy`], and reconnects
    //! — re-applying the active subscription set. A `ProviderDisconnected` /
    //! `ProviderConnected` control pair is emitted in-band so consumers see the
    //! gap window in the event stream.
    ).
  2. The session feeds those controls into stats and accounting only (
    fn note_provider_arrival(&self, ev: &MarketEvent) {
    if let MarketEvent::Control(c) = ev {
    let change = self.stats.record_connection(c);
    self.accounting.record_provider_control(&c.kind, change);
    }
    }
    ). No ControlKind::Gap is emitted for the outage window.
  3. seq is stamped at delivery, so prints the provider never received leave no seq hole. The Seq documentation says the delivered stream is contiguous "only while nothing is lost" and that honest holes come from resume-buffer eviction or a late join (
    ///
    /// - **Per-symbol, not global.** `seq` orders one symbol's stream; there is no
    /// cross-instrument order. The multiplex ordering key is `(instrument, seq)`.
    /// - **Stamped at the source.** The authoritative controller assigns `seq` once,
    /// in delivery order, before the event reaches any sink — so it is a property
    /// of the shared stream, not of a particular consumer's poll timing.
    /// - **Identical across a symbol's consumers.** Every consumer of one symbol's
    /// authoritative stream observes the same `(seq, source_ts)` for each event.
    /// - **Controls occupy slots.** In-band [`Control`] events are stamped here too,
    /// so they consume a `seq` slot like data events.
    /// - **Holes are real.** A consumer that misses events (resume-buffer eviction,
    /// late join) observes a real `seq` hole, surfaced in-band as
    /// [`ControlKind::Gap`]. The delivered stream is contiguous *only while
    /// nothing is lost*.
    ). A provider-side loss is neither, so it is invisible.
  4. On the iceoryx2 data plane, ProviderConnected, ProviderDisconnected and ProviderError are suppressed outright and ride the diagnostics plane only (
    fn control_to_pod(
    c: &Control,
    table: &mut SymbolTable,
    ) -> Result<Option<DataPayload>, InstrumentTooLong> {
    let payload = match &c.kind {
    // Connection-scoped, free-text: suppressed on the data plane.
    ControlKind::ProviderConnected { .. }
    | ControlKind::ProviderDisconnected { .. }
    ).

Why it matters. A consumer that builds path-dependent state from the trade stream (anything where the order of prices matters, not just the latest value) cannot distinguish a thirty-second outage from a quiet market. It already handles Gap by discarding and rebuilding, so the ring-eviction case is safe; the provider-outage case silently is not. Polling the diagnostics plane for a disconnect counter is possible but puts the burden of correlating two planes on every consumer, and the health view is sampled, so a short outage between two samples is lost there too.

Relation to #86. That issue makes a silent dead socket detectable (a probe turns silence into a ProviderDisconnected). This issue makes a detected outage visible on the data plane. Both are needed: without the probe this issue covers only outages the provider notices on its own; without this issue the probe's result still never reaches a data-plane consumer.

Proposal. When a ProviderConnected follows a ProviderDisconnected, emit one ControlKind::Gap { span: [disconnect_ts, reconnect_ts) } per live (instrument, kind) the session carries, through forward() so it takes a seq slot the same way the ring-eviction gaps already do (

/// One `Gap` for anything the ring evicted, then the survivors, each through
/// `forward` (which stamps **then** tees **then** delivers). Going through
/// `forward` rather than `emit` is load-bearing: it persists the
/// seam-assigned source `seq` to the tap log verbatim. Teeing here (not at
/// receipt) means an event the ring evicted is never teed — correct, since
/// it is not delivered either: it is reported as the `Gap` above, so the tap
/// log stays a faithful record of the delivered live tail.
async fn flush_backfill_pending(&mut self, ring: EventRing) {
let (dropped, _dropped_first_seq, events) = ring.into_parts();
if let Some(span) = dropped {
self.emit_gap(span.from_source_ts, span.from_source_ts, span.to_source_ts)
.await;
}
). Consumers then see both kinds of hole the same way. The disconnect timestamp is already available from the control that opened the window. No schema or wire change: Gap already crosses both transports.

Tests. A session-level test with a fake provider that emits disconnect, then connect: every live substream receives exactly one Gap whose span covers the window, stamped after the last pre-outage event and before the first post-outage one; a connect with no preceding disconnect emits nothing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions