Description
A live session has no in-band signal for the prints lost during a provider outage, and on the iceoryx2 data plane a consumer cannot recover that information from the stream at all.
Observed (v0.10.0).
- The Alpaca provider reports a websocket drop as a
ProviderDisconnected/ProviderConnectedpair and reconnects with backoff, re-applying the subscription set ().datamancer/crates/datamancer/src/providers/alpaca.rs
Lines 9 to 15 in f67e5c4
//! # Reconnect //! //! On websocket failure the streaming task tears down the client, sleeps with //! exponential backoff per the configured [`ReconnectPolicy`], and reconnects //! — re-applying the active subscription set. A `ProviderDisconnected` / //! `ProviderConnected` control pair is emitted in-band so consumers see the //! gap window in the event stream. - The session feeds those controls into stats and accounting only ( ). No
datamancer/crates/datamancer/src/session.rs
Lines 2233 to 2238 in f67e5c4
fn note_provider_arrival(&self, ev: &MarketEvent) { if let MarketEvent::Control(c) = ev { let change = self.stats.record_connection(c); self.accounting.record_provider_control(&c.kind, change); } } ControlKind::Gapis emitted for the outage window. seqis stamped at delivery, so prints the provider never received leave no seq hole. TheSeqdocumentation says the delivered stream is contiguous "only while nothing is lost" and that honest holes come from resume-buffer eviction or a late join (). A provider-side loss is neither, so it is invisible.datamancer/crates/datamancer-core/src/event.rs
Lines 16 to 29 in f67e5c4
/// /// - **Per-symbol, not global.** `seq` orders one symbol's stream; there is no /// cross-instrument order. The multiplex ordering key is `(instrument, seq)`. /// - **Stamped at the source.** The authoritative controller assigns `seq` once, /// in delivery order, before the event reaches any sink — so it is a property /// of the shared stream, not of a particular consumer's poll timing. /// - **Identical across a symbol's consumers.** Every consumer of one symbol's /// authoritative stream observes the same `(seq, source_ts)` for each event. /// - **Controls occupy slots.** In-band [`Control`] events are stamped here too, /// so they consume a `seq` slot like data events. /// - **Holes are real.** A consumer that misses events (resume-buffer eviction, /// late join) observes a real `seq` hole, surfaced in-band as /// [`ControlKind::Gap`]. The delivered stream is contiguous *only while /// nothing is lost*. - On the iceoryx2 data plane,
ProviderConnected,ProviderDisconnectedandProviderErrorare suppressed outright and ride the diagnostics plane only ().datamancer/crates/datamancer-transport-iceoryx2/src/payload.rs
Lines 213 to 220 in f67e5c4
fn control_to_pod( c: &Control, table: &mut SymbolTable, ) -> Result<Option<DataPayload>, InstrumentTooLong> { let payload = match &c.kind { // Connection-scoped, free-text: suppressed on the data plane. ControlKind::ProviderConnected { .. } | ControlKind::ProviderDisconnected { .. }
Why it matters. A consumer that builds path-dependent state from the trade stream (anything where the order of prices matters, not just the latest value) cannot distinguish a thirty-second outage from a quiet market. It already handles Gap by discarding and rebuilding, so the ring-eviction case is safe; the provider-outage case silently is not. Polling the diagnostics plane for a disconnect counter is possible but puts the burden of correlating two planes on every consumer, and the health view is sampled, so a short outage between two samples is lost there too.
Relation to #86. That issue makes a silent dead socket detectable (a probe turns silence into a ProviderDisconnected). This issue makes a detected outage visible on the data plane. Both are needed: without the probe this issue covers only outages the provider notices on its own; without this issue the probe's result still never reaches a data-plane consumer.
Proposal. When a ProviderConnected follows a ProviderDisconnected, emit one ControlKind::Gap { span: [disconnect_ts, reconnect_ts) } per live (instrument, kind) the session carries, through forward() so it takes a seq slot the same way the ring-eviction gaps already do (
datamancer/crates/datamancer/src/session.rs
Lines 1393 to 1405 in f67e5c4
| /// One `Gap` for anything the ring evicted, then the survivors, each through | |
| /// `forward` (which stamps **then** tees **then** delivers). Going through | |
| /// `forward` rather than `emit` is load-bearing: it persists the | |
| /// seam-assigned source `seq` to the tap log verbatim. Teeing here (not at | |
| /// receipt) means an event the ring evicted is never teed — correct, since | |
| /// it is not delivered either: it is reported as the `Gap` above, so the tap | |
| /// log stays a faithful record of the delivered live tail. | |
| async fn flush_backfill_pending(&mut self, ring: EventRing) { | |
| let (dropped, _dropped_first_seq, events) = ring.into_parts(); | |
| if let Some(span) = dropped { | |
| self.emit_gap(span.from_source_ts, span.from_source_ts, span.to_source_ts) | |
| .await; | |
| } |
Gap already crosses both transports.
Tests. A session-level test with a fake provider that emits disconnect, then connect: every live substream receives exactly one Gap whose span covers the window, stamped after the last pre-outage event and before the first post-outage one; a connect with no preceding disconnect emits nothing.
Activity