Skip to content

fix(security): repair rollback pins, lifecycle cleanup and bounded processing - #142

Merged
LIghtJUNction merged 4 commits into
mainfrom
fix/security-pr-review-20260928-69befd0
Sep 28, 2026
Merged

LIghtJUNction merged 4 commits into
mainfrom
fix/security-pr-review-20260928-69befd0

Conversation

@LIghtJUNction

@LIghtJUNction LIghtJUNction commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Current revision and scope

Reworks current-architecture security proposals #121–#128, with a follow-up for rollback serialization and upload pin cancellation. Refs #80, #81, #83, #85; does not close umbrella acceptance issues.

  • Current head: 27917aa1080f55953871a7fa42188562450fdeda.
  • Current complete tree: 54b39853ab2ac0f25164ed90ec0fd6b7e300a065.
  • Integrated main: e94202899ab48399a586b0c613a1b817f9783d58 (includes integrate: deterministic clearing, bounty and task C acceptance increments #119). The follow-up is a normal child of cbc5741; existing history and account-scope/recovery protections are preserved.
  • Entire PR versus that main: 20 files, 1038 additions / 25 deletions. Patch SHA-256: 27c18f99c408769fe1f24964c64ee29d9ce7d4c40f3a96208151b6cdb0c4e261.
  • This follow-up alone: 7 files, 376 additions / 15 deletions, including two new test files. Patch SHA-256: 931f889e681e443d3d25bd6edb432fb3d8c25a7f2ad232c01b3d4a9d481e5e95.
  • No product CI, published dictionary/schema, token budget, skip, or production configuration changes.

Follow-up repairs in 27917aa

  1. Keep writers excluded until rollback compensation completes. PostgreSQL retains the existing advisory-lock key at connection scope; SQLite adds a cancellation-aware, bounded-wait OS file lock. Both fences survive SQL rollback, including automatic SQLite transaction abort and PostgreSQL constraint failure. Releasing SQL locks before restoring external pins allowed another writer to observe partially compensated state.
  2. Do not abandon compensation on repeated cancellation. Shield the compensation task, drain it before releasing the writer fence, preserve the original failure/cancellation, retain reverse-order/savepoint ownership, and continue independent cleanup after an individual callback failure.
  3. Compensate upload and revision pins. Register compensation before chunk/sealed-output pinning. Failed authenticated requests, including response projection failure after publication and a pin operation failing after its side effect, leave no new durable pin or committed resource/revision. Shared revision publication compensates newly acquired pins without removing a pre-existing explicit revision pin. Successful retry, cached replay, and fresh idempotent replay retain their content.
  4. Finish an in-flight Git pin mutation before propagating cancellation. Cancelling to_thread does not stop Git; rollback could previously remove a pin before the still-running thread wrote it back. Pin/unpin mutations now settle before the cancellation reaches metadata rollback. Tests exercise a real delayed Git ref update, not just a schema check or mocked success.

Operational note: PostgreSQL connections must be direct or session-pooled, not transaction-pooled. SQLite's reference backend uses a persistent .writer.lock sidecar; do not mix old/new SQLite writers during rollout. No deployment was performed.

Earlier repairs retained

Actual verification

Local verification used Python 3.13.5, PostgreSQL 15.19, actual SQLite, Git, Ed25519 and age. It is not the target Python 3.15 / PostgreSQL 16 / Valkey 9 environment.

Source / stage Observed result
Unchanged cbc5741, final 21 follow-up regressions 18 failed / 3 passed, no errors/skips
Current complete tree 54b3985, 17-file related suite 129 passed, no failures/errors/skips, 232.65 seconds
Current tree static/build checks compileall, Ruff for both new test files, git diff --check, and wheel/sdist build passed

The 129-case suite includes all 21 follow-up regressions, previous security regression files, both metadata backends, transfers and their endpoint, post/query-ref lifecycle, hosting, market contract/account-scope and store-foundation checks, and published dictionary/route checks. It is a focused related suite, not a claim that all repository tests ran locally.

Earlier evidence remains historical, not relabeled as current-head results: original-main initial regressions 10 failed / 2 passed; original #125 predicate 2 failed; unindexed topic query 1 failed; 41bb241 related suite 110 passed; 8eb8b1d expanded suite 284 passed / 1 failed on immutable preview schema; final schema correction and signed boundary checks 2 passed; cbc5741 focused suite 88 passed. The schema failure was fixed, not waived. The previous 285-case combination was not rerun locally after that correction.

The prior cbc5741 Rewrite contracts and Recovery safety runs both completed successfully. They are superseded for merge purposes by fresh exact-head runs:

These new runs were queued/running when this record was updated. Merge still requires exact-head four-shard completeness/disjointness, conformance/token-budget, applicable recovery/build gates, and review. No timeout, allowlist, skip, or assertion weakening was used.

Boundaries and publication

  • Bound Git pushes over SSH by size and runtime #129–Batch inbox alias resolution to prevent database lock exhaustion #141 target legacy src/msgd: not ported or claimed fixed. No legacy architecture was restored; other PR branches were not changed.
  • Compensation covers live-process transaction failures, not universal crash-orphan recovery or guaranteed cleanup despite permanent storage failure. GC is throttled, not fully incremental.
  • Source preparation used separate never-merge workbench refs. Transport typos were corrected behind mandatory patch SHA-256 and complete-tree checks before publication. The prepared commit's tree was verified against the locally tested tree, then this PR branch was fast-forwarded with force=false.
  • Helper workflows, encoded transport data, downloaded dependencies and artifacts are absent from the product diff. Helper successes are not product CI evidence.
  • No merge to main, deployment, production Root/keys/PIN/funding/recovery/deletion, or real external delivery.

…anup

Reworks #121 through #128 against current main. Includes 24 new authenticated PostgreSQL/SQLite regressions. Exact source tree locally passed 110 related tests and wheel/sdist build; target-runtime CI is still required. No deployment, merge, production access, or workflow changes in this product tree.
Copilot AI lite review requested due to automatic review settings September 28, 2026 12:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-28T12:56:31.741107Z 27917aa Manual request
🔒 Security Review ✅ Completed 2026-09-28T12:25:42.345281Z 41bb241 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…pe protections

Preserve both parents and all main changes. Complete tree d3f0250 independently reconstructed locally from the main CI source artifact and unchanged document blob identities. Only the security repair delta remains relative to e942028. Product target-runtime CI and review remain required; no production access or main merge.
The expanded integration regression caught short-code contract drift. Preserve the immutable v1 schema and enforce 128 entries at runtime after authorization, before source traversal/materialization. Replace schema-only assertions with actual signed 129-entry rejection (zero storage IO/writes) and successful 128-entry positive control. Existing immutable-shortcode assertion stays unchanged. Target-runtime CI remains required.

Copy link
Copy Markdown
Contributor Author

Follow-up pushed as 27917aa, verified complete tree 54b39853ab2ac0f25164ed90ec0fd6b7e300a065. Please review this head rather than the previous 41bb241/cbc5741 snapshots.

Reproduced against unchanged cbc5741: the final 21 new cases give 18 failed / 3 passed. The same cases plus related storage, upload, lifecycle, hosting, market/account-scope and dictionary checks give 129 passed / 0 failed / 0 errors / 0 skipped on the published tree. Local Python 3.13.5 / PostgreSQL 15.19; fresh target-environment CI is separate evidence.

Review focus:

  • SQL rollback (including SQLite implicit abort) must not release writer exclusion before external compensation finishes.
  • Repeated cancellation must not abandon compensation or let rollback overtake a still-running Git pin update.
  • Failed chunk/sealed/revision publication must release only its newly acquired pins; successful retries/replays and existing references remain intact.
  • PostgreSQL now uses connection-scoped advisory locking: direct/session-pooled connections only. SQLite adds a writer-lock sidecar. No production rollout was performed.

Fresh CI: Rewrite contracts, Recovery safety. PR body contains hashes, exact verification scope and remaining boundaries. No merge, deployment, test waiver or legacy-code restoration.

@codex security review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27917aa108

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/msg/storage/sqlite.py
Comment on lines +634 to +635
fd=os.open(self.path.with_name(self.path.name+'.writer.lock'),
os.O_CREAT|os.O_RDWR|os.O_CLOEXEC|os.O_NOFOLLOW,0o600)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Canonicalize the database path before choosing the fence

When two processes open the same SQLite database through different path aliases—such as its real path and a symlink—this derives different .writer.lock files, so the new fence does not serialize them. After an INSERT OR ROLLBACK implicitly releases SQLite's own lock, the alias-backed writer can enter while the first transaction is still running external compensation, exposing the half-compensated state this fence is intended to prevent. Resolve or otherwise canonicalize the database path before deriving the sidecar path.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants