work-unit: logfwd-output — sink retry and HTTP output #1495

Activity

1. strawgate

   added

   bugSomething isn't workingSomething isn't working

   work-unitScheduling ticket for one agent runScheduling ticket for one agent run

   on Apr 7, 2026
2. coderabbitai

   added

   component:output/commonCross-cutting output behavior spanning multiple sinksCross-cutting output behavior spanning multiple sinks

   P1: highSignificant user-visible bug or major UX breakageSignificant user-visible bug or major UX breakage

   productionRequired for production readinessRequired for production readiness

   on Apr 7, 2026
3. github-actions commented on Apr 7, 2026

   

   github-actionsBot

   on Apr 7, 2026 – with GitHub Actions

   Contributor

   More actions

   tl;dr: Outcome A — Ready for implementation. Retry classification is already fixed in current main; remaining mechanical work is (1) wire real http output support in sink factory/runtime and (2) make Arrow IPC receiver 429 handling all-or-nothing to prevent duplicate delivery on retried requests.

   1) Recommendation

   Implement issue #1495 as two code changesets:

   1. HTTP output support in logfwd-output factory/runtime.
   2. Arrow IPC receiver atomic enqueue on backpressure.

   No additional design decision is required; behavior targets are already explicit in code/tests and issue done-criteria.

   2) Findings

   Code evidence (file:line)

   • Retry/permanent classification is already correct

     • crates/logfwd-output/src/sink.rs:30-40 — SendResult already distinguishes IoError, RetryAfter, Rejected.
     • crates/logfwd-output/src/http_classify.rs:86-89 — non-408/429 4xx returns SendResult::Rejected.
     • crates/logfwd/src/worker_pool.rs:805-808 — Rejected is treated as terminal (no retry loop).

   • HTTP output type exists in config but is not wired in sink factory

     • crates/logfwd-config/src/lib.rs:92-96, 139-141 — OutputType::Http enum + deserialization exist.
     • crates/logfwd-output/src/lib.rs:540-735 — build_sink_factory(...) match has no OutputType::Http arm and falls to _ => ... not yet supported.
     • crates/logfwd-output/src/lib.rs:1108-1122 — test explicitly asserts http is "not yet supported".

   • Arrow IPC receiver can partially accept then return 429

     • crates/logfwd-io/src/arrow_ipc_receiver.rs:199-208 — request decodes to Vec(RecordBatch).
     • crates/logfwd-io/src/arrow_ipc_receiver.rs:213-223 — batches are enqueued one-by-one via tx.try_send(...); Full stops mid-loop.
     • crates/logfwd-io/src/arrow_ipc_receiver.rs:232-239 — returns 429 after partial enqueue, enabling sender retry of full payload.

   • Docs currently conflict on HTTP support status

     • book/src/config/outputs.md:20-31 — documents HTTP output usage as available.
     • book/src/config/reference.md:226-239, 299-303 — marks HTTP output as "not yet supported".

   3) Verification

   Commands and output

   $ cargo test -p logfwd-output test_build_sink_factory_http_not_yet_supported
   test tests::test_build_sink_factory_http_not_yet_supported ... ok
   
   $ cargo test -p logfwd-output send_payload_returns_rejected_on_4xx
   test otlp_sink::tests::send_payload_returns_rejected_on_4xx ... ok
   
   $ cargo test -p logfwd-io receiver_reports_degraded_on_backpressure_and_recovers
   test arrow_ipc_receiver::tests::receiver_reports_degraded_on_backpressure_and_recovers ... ok

   4) Detailed Action Plan

   Mechanical implementation spec

   A. HTTP output wiring

   1. Add/enable an async HTTP sink factory (NDJSON POST) in crates/logfwd-output/src/.

      • Reuse existing serialization path from JsonLinesSink (crates/logfwd-output/src/json_lines.rs:15-117) for payload generation.
      • Ensure send path returns SendResult via shared HTTP classification (http_classify).

   2. Wire OutputType::Http in build_sink_factory(...):

      • File: crates/logfwd-output/src/lib.rs:530-735
      • Add explicit OutputType::Http => { ... } arm (endpoint required; auth headers pass-through; compression handling explicit).
      • Remove dependence on fallback _ => not yet supported for Http.

   3. Tests to update/add

      • Replace/flip test_build_sink_factory_http_not_yet_supported at crates/logfwd-output/src/lib.rs:1108-1122 to assert successful factory creation.
      • Add unit test for HTTP 4xx => Rejected and 429/408 => RetryAfter on the HTTP sink path (pattern from existing otlp_sink status tests).

   4. Docs sync

      • Align book/src/config/reference.md with implemented status (remove "not yet supported" once code lands).

   B. Arrow IPC 429 duplicate-prevention

   1. Make request enqueue atomic (all-or-nothing per HTTP request)

      • File: crates/logfwd-io/src/arrow_ipc_receiver.rs:210-239
      • Replace per-batch try_send loop with a single enqueue unit per request.

   2. Coalesce decoded batches before enqueue

      • Use decoded Vec(RecordBatch) from decode_ipc_stream (:363-373) and combine into one RecordBatch (same-schema concat) before try_send.
      • On backpressure (Full), return 429 with no partial request acceptance.

   3. Regression test to add

      • File: crates/logfwd-io/src/arrow_ipc_receiver.rs tests section (near receiver_reports_degraded_on_backpressure_and_recovers, :529-565).
      • New test: multi-batch payload + forced full queue; assert 429 and assert receiver queue contains no prefix from rejected request.

   4. Verification commands after implementation

      • cargo test -p logfwd-output test_build_sink_factory_http
      • cargo test -p logfwd-output http
      • cargo test -p logfwd-io arrow_ipc_receiver
      • just test

   5) Related Items

   Issues / PRs / files

   Type	Link	Relevance
   Issue	#1495	Current work-unit
   Issue	#1342	Permanent failure retry-loop scope item
   Issue	#1230	Missing HTTP output scope item
   Issue	#1046	Arrow IPC 429 duplicate-delivery scope item
   PR	#1471	Shared HTTP classification already landed
   File	crates/logfwd-output/src/lib.rs	Missing OutputType::Http sink-factory arm
   File	crates/logfwd-io/src/arrow_ipc_receiver.rs	Partial enqueue before 429
   File	crates/logfwd-output/src/http_classify.rs	Current retry/permanent mapping

   Open PRs in area: checked current open PR list; no active PR appears to target HTTP sink wiring or Arrow IPC 429 duplicate handling directly.

   Note

   🔒 Integrity filtering filtered 11 items

   Integrity filtering activated and filtered the following items during workflow execution.
   This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

   • issue:work-unit: logfwd-output — sink retry and HTTP output #1495 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:bug(sink): permanent failures (4xx, 413, encode errors) retried forever via IoError (#1156) #1342 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:work-unit: logfwd-output — sink retry and HTTP output #1499 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:bug: http output is unsupported on latest master despite being documented and implemented previously #1230 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:bug: async fanout duplicates successful outputs when a sibling sink returns RetryAfter #1232 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • resource:search_pull_requests (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • pr:fix(ci): add dedicated verification guardrail (#1486) #1501 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • pr:refactor(arrow): make conflict_schema classifier Kani-friendly #1492 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • pr:io: strengthen input lifecycle status proofs #1491 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • pr:DO NOT MERGE: Prototype source metadata injection #1370 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:bug: Arrow IPC receiver returns 429 after partial batch delivery — client retry causes duplicate records #1046 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

   ---

   What is this? | From workflow: Issue Triage

   Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
4. strawgate

   added

   component:output/httpGeneric HTTP / JSON lines outputGeneric HTTP / JSON lines output

   on Apr 7, 2026
5. strawgate commented on Apr 7, 2026

   

   strawgate

   on Apr 7, 2026

   OwnerAuthor

   More actions

   Execution Owner: Codex

   Objective
   Complete #1495 by fixing output delivery correctness across retry classification, HTTP sink wiring, partial-delivery duplicate risk, and benchmark signal validity (#1342 #1230 #1046 #1537).

   Scope lock

   • In scope: output classification/factory logic, Arrow IPC partial-delivery handling, benchmark send-result checks.
   • Out of scope: unrelated output feature additions.

   Implementation steps (in order)

   1. Read context: AGENTS.md, DEVELOPING.md, issue bodies bug(sink): permanent failures (4xx, 413, encode errors) retried forever via IoError (#1156) #1342 bug: http output is unsupported on latest master despite being documented and implemented previously #1230 bug: Arrow IPC receiver returns 429 after partial batch delivery — client retry causes duplicate records #1046 fix(bench): elasticsearch_arrow ignores IoError/RetryAfter send results #1537.
   2. Add failing tests for each issue path before code changes.
   3. Fix outcome classification so permanent vs transient failures cannot collapse incorrectly.
   4. Wire/validate HTTP output in sink factory path.
   5. Fix partial-delivery behavior to prevent duplicate-risk retry semantics.
   6. Make benchmark path fail fast on non-success send outcomes.

   Required verification

   • cargo test -p logfwd-output
   • cargo test -p logfwd-io
   • cargo test -p logfwd-bench
   • just ci

   PR requirements

   • Title: fix(output): delivery semantics and HTTP/benchmark correctness (#1495)
   • Close keywords: Fixes #1342, Fixes #1230, Fixes #1046, Fixes #1537, Refs #1495.
   • PR body must include per-issue before/after behavior table.

   Done checklist

   • permanent/transient outcomes classified correctly
   • HTTP output path works as expected
   • duplicate-risk partial-delivery behavior fixed with regression coverage
   • benchmark no longer reports false success on non-OK outcomes
   • required verification commands pass
6. strawgate commented on Apr 7, 2026

   

   strawgate

   on Apr 7, 2026

   OwnerAuthor

   More actions

   Cloud task launched (Codex queue): task_e_69d577fd28ec832e8ac919a68c76dfe9 (3 attempts).\nhttps://chatgpt.com/codex/tasks/task_e_69d577fd28ec832e8ac919a68c76dfe9
7. strawgate

   added 2 commits that reference this issue on Apr 7, 2026

   fix(output): delivery correctness and fail-fast propagation (#1495)

   8a64abf

   fix(output): delivery correctness and fail-fast propagation (#1495) (#…

   ...

   d898573
8. strawgate commented on Apr 8, 2026

   

   strawgate

   on Apr 8, 2026

   OwnerAuthor

   More actions

   Audit update (2026-04-08): this work-unit is mostly complete via PR #1568, and linked issues #1230, #1342, and #1537 are now closed. Remaining blocker for full closure is #1046 (Arrow IPC partial-delivery/429 semantics).
9. strawgate commented on Apr 8, 2026

   

   strawgate

   on Apr 8, 2026

   OwnerAuthor

   More actions

   Consolidated as of 2026-04-08. Completed scope was closed in prior pass (#1230, #1342, #1537). Remaining ingress-focused correctness work is now tracked in focused work-unit #1581 (#1046, #1551, #1557).
10. strawgate

    closed this as completedon Apr 8, 2026