Skip to content

BE-930: Add memory, thread and disk limits to the workspace - #10003

Open
thehabbos007 wants to merge 1 commit into
asa/hash-integrations/04a-engine-snapshotsfrom
asa/hash-integrations/04b-engine-limits
Open

thehabbos007 wants to merge 1 commit into
asa/hash-integrations/04a-engine-snapshotsfrom
asa/hash-integrations/04b-engine-limits

Conversation

@thehabbos007

Copy link
Copy Markdown
Contributor

🌟 Purpose

This PR lets a caller limit a workspace's memory, threads and disk use through WorkspaceConfig.

🔍 Changes

  • memory_limit, temp_directory_limit and threads set DuckDB's memory_limit, max_temp_directory_size and threads settings.
  • disk_limit limits the combined size of the database file, its log and DuckDB's temporary directory. The workspace checks it on open and after each query and create_table. When the files are over the limit, it checkpoints the log into the database file and measures again before it fails with DiskLimit.

⚠️ Known issues

  • The check runs after a statement, so one create_table can write past the limit, and the table it created stays in the database.

@thehabbos007 thehabbos007 self-assigned this Oct 12, 2026
@github-actions github-actions Bot added area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team area/tests New or updated tests area/apps > hash-integrations labels Oct 12, 2026
@thehabbos007
thehabbos007 added this pull request to stack #9985 October 12, 2026 14:49
@vercel

vercel Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Oct 13, 2026 9:22pm UTC
petrinaut Ready Ready Preview Oct 13, 2026 9:22pm UTC
petrinaut-docs Ready Ready Preview Oct 13, 2026 9:22pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Oct 13, 2026 9:22pm UTC

Request Review

@codecov

codecov Bot commented Oct 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.68085% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.78%. Comparing base (4341978) to head (b241522).

Files with missing lines Patch % Lines
...al/integrations/engine/src/workspace/connection.rs 93.97% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@                              Coverage Diff                               @@
##           asa/hash-integrations/04a-engine-snapshots   #10003      +/-   ##
==============================================================================
+ Coverage                                       66.77%   66.78%   +0.01%     
==============================================================================
  Files                                            1982     1983       +1     
  Lines                                          217586   217659      +73     
  Branches                                         8513     8520       +7     
==============================================================================
+ Hits                                           145297   145368      +71     
- Misses                                          70662    70663       +1     
- Partials                                         1627     1628       +1     
Flag Coverage Δ
hash-integrations-engine 89.95% <94.68%> (+0.96%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from b241522 to 8f8ddc7 Compare October 13, 2026 07:45
@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from 8f8ddc7 to fe46e4a Compare October 13, 2026 09:05
@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from fe46e4a to 29f3584 Compare October 13, 2026 10:15
@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from 29f3584 to 9812312 Compare October 13, 2026 19:53
@cursor

cursor Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes workspace open API and enforces disk limits after queries; misconfigured limits or checkpoint failures could block opens or statements in production integrations.

Overview
Introduces WorkspaceConfig and a ByteSize helper so callers can cap DuckDB memory, temp directory size, thread count, and overall workspace disk use when opening a workspace. Workspace::open now takes (path, config) and applies the DuckDB settings on connect.

When disk_limit is set, the engine totals the database file, WAL, and .tmp directory (with recursive sizing), runs CHECKPOINT if over limit, then rejects open or post-query / create_table with DiskLimit / DiskUsage errors. A single statement can still grow files past the cap before the check; data written in that request is not rolled back.

Integration tests cover configured DuckDB settings, open refusal, post-statement limits, temp-dir accounting, and checkpoint-before-fail behavior.

Reviewed by Cursor Bugbot for commit e28208b. Bugbot is set up for automated code reviews on this repo. Configure here.

This branch was successfully deployed

3 active deployments
Preview – petrinaut-docs — e28208bb Deployed Oct 13, 2026 by vercel[bot]
Preview – hash — e28208bb Deployed Oct 13, 2026 by vercel[bot]
Preview – petrinaut — e28208bb Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash-integrations area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

1 participant