Skip to content

fix: update code for nightly-2026-01-08 toolchain - #6982

Closed
mikedep333 wants to merge 1 commit into
Eventual-Inc:mainfrom
mikedep333:info_remainder-new-rust
Closed

mikedep333 wants to merge 1 commit into
Eventual-Inc:mainfrom
mikedep333:info_remainder-new-rust

Conversation

@mikedep333

@mikedep333 mikedep333 commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

What

Bump rust nightly toolchain from nightly-2025-09-03 to nightly-2026-01-08.
This picks up the ArrayChunks::into_remainder() return type change
(rust-lang/rust#149127) while staying before the cargo --timings=html
removal (rust-lang/cargo#16420), which CI still relies on.

Changes to accommodate the new nightly:

  • Fix into_remainder() call in daft-minhash to match new return type
    (no longer wrapped in Option) (This enables distributions to build
    with Rust 1.93 stable and newer when RUSTC_BOOTSTRAP is set.)
  • Upgrade cargo-llvm-cov from 0.7.1 to 0.8.7 to fix corrupt profraw
    files due to LLVM profdata format incompatibility

Clippy lint fixes for newer nightly:

  • Fix use_self by replacing self-referential type names with Self
    across multiple enum/struct definitions
  • Allow clippy::use_self on snafu-derived Error variants that use
    Arc (snafu macro generates code with concrete type names,
    incompatible with Self): daft-io CachedError, daft-parquet
    RemoteFetchFailed
  • Allow clippy::result_large_err in daft-io/src/tos.rs (IO error
    types are inherently large)
  • Rename clippy::only_used_in_recursion allow attributes to
    clippy::self_only_used_in_recursion (lint renamed in newer clippy)
  • Fix ref_as_ptr by using std::ptr::from_ref
  • Fix unnecessary_unwrap in daft-sql by using if-let
  • Fix unchecked_time_subtraction by using checked_sub().unwrap()
  • Fix useless_vec in daft-io test
  • Fix derivable_impls on PreviewFormat by using #[derive(Default)]

Why

Iterator::array_chunks().into_remainder() returns IntoIter<T, N> directly in stable Rust 1.93 and later, not Option<IntoIter<T, N>> as in daft's current target nightly (2025-09-03). The current code does not compile on stable Rust 1.95.0 because of this API difference.

We are successfully building daft with stable Rust 1.95.0 (using RUSTC_BOOTSTRAP=1 to accept nightly feature gates) in our internal build system, and this is the one call site that needs updating.

Testing

  • Verified this is the only call site of into_remainder() in the codebase
  • Confirmed the replacement compiles and works correctly on stable Rust 1.95.0

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

@github-actions github-actions Bot added the fix label May 21, 2026
@greptile-apps

greptile-apps Bot commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adapts the SIMD minhash implementation to the ArrayChunks::into_remainder() API change introduced in stable Rust 1.95.0, where the method returns IntoIter<T, N> directly rather than Option<IntoIter<T, N>>. The nightly toolchain pin is updated from nightly-2025-09-03 to nightly-2026-02-27 (the branch point for 1.95.0) so both development and production builds stay consistent.

  • src/daft-minhash/src/lib.rs: Removes the if let Some(remainder) wrapper and iterates over chunks.into_remainder() directly, matching the stabilized API signature.
  • rust-toolchain.toml: Bumps the pinned nightly from 2025-09-03 to 2026-02-27 so the dev toolchain exposes the same into_remainder() return type as stable 1.95.0.

Confidence Score: 5/5

Safe to merge — two-line targeted fix with the necessary toolchain update included.

The code change correctly removes the Option-unwrapping layer that no longer applies in the new API, and the toolchain pin is updated in the same commit to keep the dev environment consistent with stable 1.95.0. The logic is unchanged; only the wrapping construct is removed.

No files require special attention.

Important Files Changed

Filename Overview
rust-toolchain.toml Bumps pinned nightly toolchain from 2025-09-03 to 2026-02-27 to match the API surface of stable Rust 1.95.0
src/daft-minhash/src/lib.rs Removes the if let Some(remainder) wrapper around chunks.into_remainder() to match the stabilized 1.95.0 API

Reviews (2): Last reviewed commit: "fix(daft-minhash): fix into_remainder() ..." | Re-trigger Greptile

Comment thread src/daft-minhash/src/lib.rs Resolved
@mikedep333 mikedep333 changed the title fix(daft-minhash): adapt into_remainder() API for stable Rust 1.95.0 Draft: fix(daft-minhash): adapt into_remainder() API for stable Rust 1.95.0 May 21, 2026
@universalmind303
universalmind303 marked this pull request as draft May 21, 2026 23:13
@universalmind303

universalmind303 commented May 21, 2026 •

Copy link
Copy Markdown
Member

hey @mikedep333, We have no qualms with bumping the toolchain version, as long as all breaking changes are addressed, and CI is updated + passing I dont see any issue.

@mikedep333

mikedep333 commented May 26, 2026 •

Copy link
Copy Markdown
Contributor Author

hey @mikedep333, We have no qualms with bumping the toolchain version, as long as all breaking changes are addressed, and CI is updated + passing I dont see any issue.

Thanks! What would you prefer? Stable Rust release with RUSTC_BOOTSTRAP=1, or a newer nightly? We're testing the former.

EDIT: Nevermind, I just bumped the nightly instead because I read that the Rust project doesn't want people using RUSTC_BOOTSTRAP=1

@mikedep333
mikedep333 force-pushed the info_remainder-new-rust branch from febc9d7 to ee70619 Compare May 26, 2026 20:49
@mikedep333 mikedep333 changed the title Draft: fix(daft-minhash): adapt into_remainder() API for stable Rust 1.95.0 fix(daft-minhash): adapt into_remainder() API for stable Rust 1.95.0 May 26, 2026
@mikedep333
mikedep333 marked this pull request as ready for review May 26, 2026 20:55
@mikedep333
mikedep333 requested a review from a team as a code owner May 26, 2026 20:55
@madvart
madvart requested a review from colin-ho May 26, 2026 20:56
@mikedep333
mikedep333 force-pushed the info_remainder-new-rust branch 7 times, most recently from 7eaeb0e to d86f3e6 Compare May 27, 2026 13:59
@mikedep333 mikedep333 changed the title fix(daft-minhash): adapt into_remainder() API for stable Rust 1.95.0 fix: update code for nightly-2026-01-08 toolchain May 27, 2026
@mikedep333
mikedep333 force-pushed the info_remainder-new-rust branch 10 times, most recently from 01865fb to 47ab584 Compare May 27, 2026 19:00
@mikedep333
mikedep333 force-pushed the info_remainder-new-rust branch 2 times, most recently from cafc11b to 848e8d3 Compare May 27, 2026 20:03
@mikedep333

Copy link
Copy Markdown
Contributor Author

@universalmind303 I managed to resolve a ton of new clippy errors with the help of Claude Opus. CI is now green.

@srilman srilman left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM overall @mikedep333, but before merging, we should run some integration and performance tests just to make sure that there isn't a regression, just to be safe

@mikedep333
mikedep333 force-pushed the info_remainder-new-rust branch from 848e8d3 to a264598 Compare May 28, 2026 15:15
@mikedep333

Copy link
Copy Markdown
Contributor Author

I just did an automatice rebase. Apologies, I didn't realize that would invalidate the review.

Bump rust nightly toolchain from nightly-2025-09-03 to nightly-2026-01-08.
This picks up the ArrayChunks::into_remainder() return type change
(rust-lang/rust#149127) while staying before the cargo --timings=html
removal (rust-lang/cargo#16420), which CI still relies on.

Changes to accommodate the new nightly:
- Fix into_remainder() call in daft-minhash to match new return type
  (no longer wrapped in Option) (This enables distributions to build
  with Rust 1.93 stable and newer when RUSTC_BOOTSTRAP is set.)
- Upgrade cargo-llvm-cov from 0.7.1 to 0.8.7 to fix corrupt profraw
  files due to LLVM profdata format incompatibility

Clippy lint fixes for newer nightly:
- Fix use_self by replacing self-referential type names with Self
  across multiple enum/struct definitions
- Allow clippy::use_self on snafu-derived Error variants that use
  Arc<Error> (snafu macro generates code with concrete type names,
  incompatible with Self): daft-io CachedError, daft-parquet
  RemoteFetchFailed
- Allow clippy::result_large_err in daft-io/src/tos.rs (IO error
  types are inherently large)
- Rename clippy::only_used_in_recursion allow attributes to
  clippy::self_only_used_in_recursion (lint renamed in newer clippy)
- Fix ref_as_ptr by using std::ptr::from_ref
- Fix unnecessary_unwrap in daft-sql by using if-let
- Fix unchecked_time_subtraction by using checked_sub().unwrap()
- Fix useless_vec in daft-io test
- Fix derivable_impls on PreviewFormat by using #[derive(Default)]

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mikedep333
mikedep333 force-pushed the info_remainder-new-rust branch from a264598 to 8a7dc31 Compare May 28, 2026 17:28
@srilman

srilman commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

@mikedep333 sorry for the late reply. I got a chance to run the branch on a distributed TPCH benchmark and here are the results I got:

Question Main Branch Speedup
q1 26.0676581 28.0766075 0.9284x
q2 32.07873285 44.10331833 0.7274x
q3 49.10234449 66.14377418 0.7424x
q4 28.07571647 30.08533663 0.9332x
q5 88.18369736 115.2236393 0.7653x
q6 22.06034057 18.06100293 1.2214x
q7 56.11132756 74.1367581 0.7569x
q8 100.1790575 141.2632608 0.7092x
q9 145.2682338 175.3278191 0.8286x
q10 48.10861888 58.13599004 0.8275x
q11 34.16380277 34.08179067 1.0024x
q12 38.09343539 36.09127376 1.0555x
q13 38.08180197 38.08577202 0.9999x
q14 31.07141488 39.09201161 0.7948x
q15 78.15956215 76.15276887 1.0264x
q16 36.09026515 38.09308122 0.9474x
q17 108.2086526 116.2235113 0.9310x
q18 72.14562614 88.18933489 0.8181x
q19 32.08821383 34.08709192 0.9414x
q20 64.1199002 72.15588798 0.8886x
q21 119.1922928 148.3015068 0.8037x
q22 32.07084237 38.09896915 0.8418x
Total 1278.72 1509.21 0.8473x

As you can see, its an overall slowdown of 15%. That seems kind of concerning. Any ideas on how to reduce it?

@mikedep333

Copy link
Copy Markdown
Contributor Author

Hey @srilman ,

I asked my team, and we don't think there is any way we can try to address this performance regression that isn't very time consuming for us.

When you have a need to update Rust for some other reason, then both this MR and we may be of help.

Thank you
-Mike

@mikedep333 mikedep333 closed this Jul 2, 2026
srilman added a commit that referenced this pull request Sep 26, 2026
## Changes Made

Upgrade Rust nightly from 2025-09-03 to 2026-05-27. Based on Rustc
performance tracking numbers
(https://perf.rust-lang.org/index.html?start=2025-09-03&end=2026-05-27&kind=raw&stat=instructions%3Au),
this version should get around the regression in nightly performance
that we saw in #6982.

Similar to the last attempt:
* `daft-minhash`: Change into_remainder() call since it now returns an
iterator directly
* Addressing a bunch of Clippy fixes
* Switch back to unrestricted maturin by changing `--timings=html` to
just `--timings`
* Tested by running some rough performance numbers on MinHash and
ClickBench on AWS to ensure no regression. Will follow up this PR with a
deeper breakdown for ClickBench and TPCH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants