Repository navigation
Add Spark accelerators #517
Description
Motivation
There are several Spark plugins and extensions that claim to offer significantly better performance than vanilla Spark for analytical workloads -for example, Apache Comet (based on DataFusion), Blaze (DataFusion), and Apache Gluten (mainly Velox, ClickHouse). Since ClickBench already allows GPU-based engines (#410), NVIDIA RAPIDS could also be considered.
I believe these are worth evaluating for a few reasons:
- Independent benchmark: These accelerators are typically benchmarked only on their own TPC-H/TPC-DS variants. Including them in ClickBench would offer a more independent comparison. This is valuable to both users and developers.
- Distributed vs. single-node overhead: Some of these tools have single-node counterparts (e.g., DataFusion). Measuring how much overhead comes from different factors (such as Spark’s architecture, quality of implementation) is useful in itself.
- Spark user insights: For Spark users, this shows how much they can gain (or lose) by adopting such plugins (which are mostly easy to install having already deployed Spark) to compare with other distributed engines and Spark itself.
Request
Would you be open to including these in ClickBench? If I checked correctly, no similar issue has been created yet.
If yes, I’d be glad to take on this task and submit the necessary PRs incrementally (likely one per engine).
Activity
rschu1ze commentedon Jul 6, 2025
Yes, that would be nice. Please send PR(s).
Iskander14yo commentedon Jul 7, 2025
Can I ask some questions?
Do we pursue a goal to put these engines in the most equal conditions, or is that not necessary? I mean mainly Spark versions here — since hardware is fixed, and I haven’t identified other major factors that could impact results yet.
ClickBench isn’t Spark-specific, so it seems fine to use whatever works best per engine. But if we want comparisons to be more fair (and helpful for Spark users), maybe it makes sense to align Spark versions across vanilla Spark, Comet, Gluten, etc.
So first option is to keep Spark version consistent within the Spark “family,” and allow exceptions with clear notes.
Another option is to use "latest stable Spark version per engine" approach. My thoughts were:
Let me know which direction seems reasonable for you. Currently I'm leaning towards second option because it's easier to maintain and still "fair" (we use same VM anyway and let each engine use latest Spark version it supports).
SparkApplicationMaster commentedon Jul 8, 2025
@Iskander14yo +1 for the second "full power" approach, outdated frameworks will become only more outdated...
If the engines support pyspark, you can start with some of this code:
https://github.com/ClickHouse/ClickBench/blob/main/spark/query.py
https://github.com/ClickHouse/ClickBench/blob/main/sail/query.py
For example, sail is a spark replacement framework written in pure rust that uses spark connect and datafusion, so no code changes were needed except the SparkSession configuration, at least on ClickBench
It would be nice to see if the engines you mentioned can do ClickBench without rewriting the code or queries
Maybe some common module can be created with common functions for all the engines?
Iskander14yo commentedon Jul 8, 2025
Thanks for the feedback!
Yep, I definitely want to reuse current Spark code as much as possible. At least because part of these backends claims to work "without code changes". In practice, that's not always true, so we might face some difficulties here.
I'm not sure about introducing a shared module though. It could become harder to maintain over time, especially as different engines diverge and require different setup or tweaks. And ClickBench doesn’t seem to be structured for cross-engine code reuse at the moment.
That said, I’ll definitely document any reusable patterns and relations in the README. Moreover, I’ll start sending PRs soon — and from there we can see whether some form of reuse makes sense in practice.
rschu1ze commentedon Jul 9, 2025
True, though this was not an intentional decision, and it is fine to share code (e.g. setup code) to cover the Spark situation.
I don't have a strong opinion on the first vs. the second approach. Once GPUs enter the game, a "fair" comparison with CPU-only databases is no longer feasible anyways. But if at last all Spark submissions want to standardize on the same hardware platform (with a GPU), that would be helpful to compare performance across different Spark flavors.
Iskander14yo commentedon Jul 14, 2025
I have another Q:
In general, these engines don’t fully support the same set of operators, functions, or expressions as Spark. In such cases, they (often) gracefully fall back to Spark execution or (rarely) fail due to semantic mismatches between engines.
Currently I observe falling back while failing apparently is not the case here but we may take it into account as well. Should we rewrite queries in those cases, or keep them as-is?
Reasons to rewrite:
Reasons not to rewrite:
In fact, if correctness of this decision is concern we can avoid dichotomy and set 2 launches - with both default and rewritten queries (same folder, different
queries.sqland results). That's the option I prefer now. WDYT?rschu1ze commentedon Nov 14, 2025
There are now submissions for Spark Auron, Spark Comet and Spark Gluten. I think we can close this ticket. Thanks @Iskander14yo