vortex-java

Pure-Java reader/writer for the Vortex columnar file format. 100% Java, no JNI, no sun.misc.Unsafe. Uses the FFM API (MemorySegment/Arena, Java 25+) for zero-copy memory-mapped reads — good performance out of the box, without native dependencies.

Checked against the Rust reference in both directions (Rust writes, Java reads and vice versa), and against 172 real-world Raincloud datasets, each matching its Parquet copy value for value.

Who is this for

• JVM analytics engines and OLAP systems
• Anyone who wants mmap-backed, zero-copy columnar reads without native-library management
• Small footprint. The Rust reference's JNI bindings (vortex-jni) are a 75 MB jar (native binaries for four platforms) that pulls in Arrow, Netty, Guava and RoaringBitmap. vortex-java's reader is ~800 KB of plain Java (vortex-reader + vortex-core + vortex-fsst) with no other required dependencies, and runs on any platform with a JDK. CI builds the full reactor on Linux + macOS + Windows × JDK 25 + 26.

Quickstart

<dependency>
  <groupId>io.github.dfa1.vortex</groupId>
  <artifactId>vortex-reader</artifactId>
  <version>0.16.0</version>
</dependency>

Minimal read example

try (VortexReader vf = VortexReader.open(Path.of("data/example.vortex"));
     var iter = vf.scan(ScanOptions.all())) {
    while (iter.hasNext()) {
        try (Chunk chunk = iter.next()) {
            LongArray ts = chunk.column(ColumnName.of("timestamp"));
            for (long i = 0; i < ts.length(); i++) {
                System.out.println(ts.getLong(i));
            }
        }
    }
}

Lifecycle. ScanIterator implements Iterator<Chunk> and Chunk implements AutoCloseable. Each chunk owns a confined Arena; closing it releases the decoded buffers. Calling iter.next() while a prior chunk is still open throws IllegalStateException. Use try-with-resources, or iter.forEachRemaining(c -> ...) which closes each chunk for you. See docs/explanation.md#memory-model.

Minimal write example

<dependency>
  <groupId>io.github.dfa1.vortex</groupId>
  <artifactId>vortex-writer</artifactId>
  <version>0.16.0</version>
</dependency>

DType.Struct schema = DType.structBuilder()
        .field(ColumnName.of("timestamp"), DType.I64)
        .field(ColumnName.of("symbol"),    DType.UTF8)
        .field(ColumnName.of("price"),     DType.F64)
        .field(ColumnName.of("volume"),    DType.I64.asNullable())     // boxed Long[] → nullable
        .build();

try (var ch = FileChannel.open(Path.of("data/example.vortex"),
                               StandardOpenOption.CREATE, StandardOpenOption.WRITE);
     var writer = VortexWriter.create(ch, schema, WriteOptions.cascading(3))) {
    writer.writeChunk(c -> c
            .put(ColumnName.of("timestamp"), new long[]   {1_700_000_000_000L, 1_700_000_001_000L})
            .put(ColumnName.of("symbol"),    new String[] {"AAPL", "AAPL"})
            .put(ColumnName.of("price"),     new double[] {189.95, 190.10})
            .put(ColumnName.of("volume"),    new Long[]   {100L, null}));  // null in nullable col
}

Each .put is validated at the call site: unknown column names, mismatched array types, and boxed arrays for non-nullable columns throw IllegalArgumentException immediately. Missing columns surface as IllegalStateException when the lambda returns.

For more examples — projection, filtering, custom encodings, and the CLI — see the tutorial.

Documentation

Docs follow the Diátaxis framework.

Document	Mode	Contents
docs/tutorial.md	Tutorial	Step-by-step: write and read your first Vortex file
docs/how-to.md	How-to	Recipes: count rows, convert Parquet, filter, project, custom encodings
docs/reference.md	Reference	API surface, CLI subcommands, operator tables, file-format trailer
docs/compatibility.md	Reference	Encoding support table, S3 fixture status
docs/explanation.md	Explanation	Design rationale, memory model, benchmarks
docs/testing.md	Explanation	Test strategy: layers, counts per module, what each layer verifies

Contributing

Requirements: Java 25+. Build: ./mvnw verify.

Forks welcome. See CONTRIBUTING.md for full build reference, coding conventions, and how to add a new encoding.

This project uses Claude Code for implementation work. Architecture, API design, and all decisions are human-driven.

References

• vortex.dev — the Vortex project's website
• Vortex: A Columnar File Format for GPU Streaming — InfoQ presentation
• Vortex: One Format for Any Shape — SpiralDB blog post
• BtrBlocks: The Compressor Behind Vortex — the cascading, cost-based encoding selection this project's own cascading compressor is modeled on
• A Tour of the Vortex Columnar Format with vortex-java — walkthrough of the format on a real dataset

Other Vortex implementations

Project	Language	Notes
vortex-data/vortex	Rust	Reference implementation + JNI bindings
LaurieRhodes/vortex-go	Go	Pure-language port
Evariops/Vorticity	C#	Pure .NET, dependency-free

Acknowledgments

YourKit supports open source projects with innovative and intelligent tools for monitoring and profiling Java and .NET applications. YourKit is the creator of YourKit Java Profiler, YourKit .NET Profiler, and YourKit YouMonitor.