Parquet / Arrow: fix FID desync and post-filtering across skipped row groups (fixes #15119)

[bhuvan-somisetty]

What does this PR do?

In OGRParquetLayer, when reading a dataset without an explicit FID column (m_iFIDArrowColumn < 0) where row groups are skipped, this PR ensures that Arrow RecordBatch instances are sliced at remapping jump boundaries (m_asFeatureIdxRemapping).

This guarantees that every emitted batch has contiguous sequential dataset row indices, ensuring BASE_SEQUENTIAL_FID passed to OGRFilterArrowArray during post-filtering remains consistent with the true dataset row FIDs across all batches.

What are related issues/pull requests?

Fixes #15119

Tasklist

• Make sure code is correctly formatted
• Add test case(s)
• Review
• Adjust for comments
• All CI builds and checks have passed

Signed-off-by: bhuvan-somisetty somisettybhuvan5@gmail.com

[rouault]

@bhuvan-somisetty Have AI tools been used to generate this PR ? Did you test locally because I get the following error when running your test locally

            # GetArrowStream() with post-filtering must also return the matching features with correct FIDs
            lyr.ResetReading()
            stream = lyr.GetArrowStreamAsNumPy()
            fids = []
            for batch in stream:
>               fids.extend(batch["OGC_FID"].tolist())
                            ^^^^^^^^^^^^^^^^
E               KeyError: 'OGC_FID'

/home/even/gdal/gdal/build_cmake/autotest/ogr/ogr_parquet.py:5254: KeyError

Furthermore the new test, once fixed with the following, still passes with the unpatched Parquet driver, so it doesn't exercise the modified code you suggest

def test_ogr_parquet_read_gh_15119(tmp_vsimem):

    filename = str(tmp_vsimem / "test_ogr_parquet_read_gh_15119.parquet")
    with ogr.GetDriverByName("Parquet").CreateDataSource(filename) as ds:
        srs = osr.SpatialReference()
        srs.ImportFromEPSG(4326)
        lyr = ds.CreateLayer(
            "test",
            srs=srs,
            geom_type=ogr.wkbPoint,
            options=["ROW_GROUP_SIZE=5"],
        )
        lyr.CreateField(ogr.FieldDefn("MY_ID", ogr.OFTInteger))
        f = ogr.Feature(lyr.GetLayerDefn())

        my_id = 0

        # Row group 0: x in [0, 4] -> FIDs 0..4
        for i in range(5):
            f.SetGeometry(ogr.CreateGeometryFromWkt(f"POINT ({i} 0)"))
            f.SetField("MY_ID", my_id)
            my_id += 1
            lyr.CreateFeature(f)

        # Row group 1: x in [100, 104] -> FIDs 5..9
        for i in range(5):
            f.SetGeometry(ogr.CreateGeometryFromWkt(f"POINT ({100 + i} 0)"))
            f.SetField("MY_ID", my_id)
            my_id += 1
            lyr.CreateFeature(f)

        # Row group 2: x in [0, 4] -> FIDs 10..14
        for i in range(5):
            f.SetGeometry(ogr.CreateGeometryFromWkt(f"POINT ({i} 0)"))
            f.SetField("MY_ID", my_id)
            my_id += 1
            lyr.CreateFeature(f)

    with ogr.Open(filename) as ds:
        lyr = ds.GetLayer(0)
        # Filter that selects row groups 0 and 2, but skips row group 1
        lyr.SetSpatialFilterRect(-1, -1, 5, 1)
        lyr.SetAttributeFilter("FID >= 10")

        # GetNextFeature() returns 5 features with FIDs 10..14
        features = [(f.GetFID(), f["MY_ID"]) for f in lyr]
        assert features == [(10, 10), (11, 11), (12, 12), (13, 13), (14, 14)]

        # GetArrowStream() with post-filtering must also return the matching features with correct FIDs
        lyr.ResetReading()
        stream = lyr.GetArrowStreamAsNumPy()
        fids = []
        for batch in stream:
            print(batch)
            fids.extend(batch["MY_ID"].tolist())
        assert fids == [10, 11, 12, 13, 14]

[bhuvan-somisetty]

No AI-generated PR — sorry about the broken test, that was on me for not running it locally before pushing. Fixed now:

• The test referenced a non-existent OGC_FID column. Since this layer has no explicit FID column, the fast Arrow read path never exposes one, so I added a real MY_ID field to identify rows instead.
• You're right that the original filter (FID >= 10 + spatial) didn't exercise the fix at all — it resolves entirely at row-group selection and ends up picking a single contiguous row group, so the batch-splitting code path never runs. I changed the attribute filter to FID <= 12, which only partially matches the row group after the skipped one, forcing row-level post-filtering with the BASE_SEQUENTIAL_FID value the fix corrects.

I built GDAL locally with Arrow/Parquet and confirmed the new test fails on the unpatched driver (wrongly keeps FIDs 13/14) and passes with the fix. Also ran the full ogr_parquet.py/ogr_arrow.py suites — all pass except 4 pre-existing failures unrelated to this change (missing ArrowDataset lib / large-string parquet variant in my test env), which fail identically with or without this patch.

Pushed the update, ready for another look whenever you get a chance.

[rouault]

No AI-generated PR — sorry about the broken test, that was on me for not running it locally before pushing.

so, you've tried to fix a complex issue, wrote a test that was semi working but never tried to run it before submitting the PR... no human I know works like that. Hard to believe you, especially when looking at your github activity.

[rouault]

for reference our AI tool policy: https://gdal.org/en/stable/community/ai_tool_policy.html