ci: ratchet type errors in apps/sim test files by waleedlatif1 · Pull Request #8959 · simstudioai/sim

waleedlatif1 · 2026-10-13T03:39:52Z

Summary

apps/sim's tsconfig excludes *.test.ts(x), so about 3,363 test files are never type-checked. This adds a shrink-only ratchet so new type errors can't enter test files, without forcing anyone to fix the existing ones now.

apps/sim/tsconfig.tests.json extends the app config and includes the test files. Only the ratchet uses it.
check:test-types (scripts/check-test-types.ts)
- Runs the native TS7 tsc (the repo's @typescript/native) on that config and counts errors per file.
- Fails if any file's count rises or a new file has errors. A drop also fails until --update shrinks the baseline, the same as check:explicit-any.
- --update never raises a count.
- Unparseable compiler output fails outright: a config error, a crash, an error total that doesn't match the parsed count, or an exit code that disagrees with it. It never reads as "fewer errors".
Its own test-types job (4 vCPU) runs alongside lint, test and build, so it stays off the critical path, at about 1.5 min. It's gated on the test flag and included in ci-gate.sh. On failure it uploads diagnostics.txt and counts.json, so a shrink never needs a local compile: gh run download <run> -n test-types -D <dir>, then bun run scripts/check-test-types.ts --counts <dir>/counts.json --update. It isn't in check:audits, because that also runs on laptops.
Baseline comes from this PR's first CI run (run 38342772080): 3,209 errors across 874 files. The one-time bootstrap path was removed before merge, so the shipped script cannot create or raise a baseline.

Known gap: an @ts-expect-error or @ts-ignore added to a test still hides an error. Counting those could be a follow-up.

Test plan

test-types job passes against the committed baseline
scripts/check-test-types.test.ts (parsing and comparison; each guard verified by revert), ci-gate.test.ts

apps/sim's tsconfig excludes *.test.ts(x), so ~3.4k test files are never type-checked and any new test can add errors. check:test-types runs the native tsc on apps/sim/tsconfig.tests.json (the app config with tests left in), counts errors per file, and compares with a shrink-only baseline, matching check:explicit-any's semantics: a rise or a new file fails, a drop fails until --update shrinks it, --update never raises, --update --init only creates a missing baseline. Output it cannot place (global diagnostics, a crash) fails outright instead of reading as fewer errors. It type-checks the whole app, so it runs in its own test-types job gated like test (wired into ci-gate.sh), not in check:audits. On failure the job uploads counts.json and the compiler output; --counts <file> lets the baseline be bootstrapped or shrunk from that artifact without a local compile.

vercel · 2026-10-13T03:39:55Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 13, 2026 4:41am UTC

… --init path Baseline from run 38342772080's test-types counts: 3209 errors in 874 files. --update stays shrink-only; a missing baseline is restored from git.

waleedlatif1 · 2026-10-13T03:42:39Z

@greptile

waleedlatif1 · 2026-10-13T03:42:41Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T03:42:45Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 11 files

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-13T03:46:32Z

Confidence Score: 5/5

[High impact] The PR appears safe to merge, with no outstanding findings.

Summary

Adds a shrink-only check for type errors in apps/sim tests.

The test type check tracks errors by file and error code.
CI runs the test type check as its own gated job.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[CI test flag] --> B[test-types job]
  B --> C[Compile app with tests]
  C --> D[Check compiler output and exit status]
  D --> E[Count errors by file and code]
  E --> F[Compare with committed baseline]
  F --> G[Unchanged counts pass]
  F --> H[Higher or lower counts fail]
  H --> I[Upload compiler reports]
  G --> J[CI gate]
  H --> J

_{Reviews (4) · Last reviewed commit: "ci: read the saved compiler status throu..." · Reviewed by Greptile}

… tests A per-file count let one new error replace a fixed one with the total flat. The baseline is now { file: { TSnnnn: count } }: any (file, code) count that rises fails, including a new code in a baselined file, and --update stays shrink-only per pair. The key stops at the code: line and column move with edits, and messages print types that change with unrelated source or compiler versions. --diagnostics <file> replaces --counts: it reads a saved compiler output (the CI artifact's diagnostics.txt) through the same parser and undercount checks, so a fix can shrink the baseline without a local compile. The baseline was migrated from run 38342772080's diagnostics.txt with the script's parser: 3209 errors in 874 files across 52 codes, every per-file total equal to the previous baseline's. Command-level tests run the real script on fixture diagnostics: unchanged, a rising count, a new code replacing a fixed error, a new file, a dropped count, --update refusing a raise, --update shrinking, and unparseable or inconsistent output. The checks.yml comment now describes the artifact as input for shrinking the existing baseline.

waleedlatif1 · 2026-10-13T04:09:59Z

@greptile

waleedlatif1 · 2026-10-13T04:10:01Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T04:10:09Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 11 files

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

…ratchet

--diagnostics read an empty or truncated diagnostics.txt as fewer errors, so a failed compile's artifact plus --update could erase the baseline. --report now writes status.json (exit code and printed error count) once the compile's output and exit code agree, and --diagnostics requires it: exit 0 must mean no errors, the parse must match the recorded count, and a missing or malformed status fails. Command-level tests cover each case and that --update never writes the baseline from such input. Also passes a route context to the socket-token route test's POST call, the one new test type error (TS2554) the ratchet caught after #8958.

waleedlatif1 · 2026-10-13T04:23:10Z

@greptile

waleedlatif1 · 2026-10-13T04:23:11Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T04:23:17Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 12 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-13T04:36:02Z

@greptile

waleedlatif1 · 2026-10-13T04:36:04Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T04:36:10Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 12 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

ci: commit the test type-error baseline from CI and drop the one-time…

8fe3658

… --init path Baseline from run 38342772080's test-types counts: 3209 errors in 874 files. --update stays shrink-only; a missing baseline is restored from git.

waleedlatif1 marked this pull request as ready for review October 13, 2026 03:42

waleedlatif1 requested a review from a team as a code owner October 13, 2026 03:42

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread scripts/check-test-types.ts Outdated

Comment thread .github/workflows/checks.yml Outdated

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread scripts/check-test-types.test.ts

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread scripts/check-test-types.ts Outdated

vercel Bot deployed to Preview October 13, 2026 04:14 View deployment

waleedlatif1 added 2 commits October 12, 2026 21:21

Merge remote-tracking branch 'origin/staging' into ci/test-typecheck-…

4feabc8

…ratchet

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

vercel Bot deployed to Preview October 13, 2026 04:27 View deployment

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread scripts/check-test-types.ts Outdated

ci: read the saved compiler status through toRecord

c1d25fc

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

vercel Bot deployed to Preview October 13, 2026 04:41 View deployment

waleedlatif1 merged commit 7d090d1 into staging Oct 13, 2026
59 checks passed

TheodoreSpeaks mentioned this pull request Oct 13, 2026

v0.9.20: project membership enforcement, db and perf improvements, realtime and auth hardening, CI and test fixes #8963

Merged

waleedlatif1 deleted the ci/test-typecheck-ratchet branch October 13, 2026 05:51

Conversation

waleedlatif1 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

vercel Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Oct 13, 2026 •

edited

Loading

vercel Bot commented Oct 13, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 13, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading