Skip to content

Possible out-of-bounds cluster label in _clustering.cluster #82

Open

Description

@akaszynski

While benchmarking on the stock bunny, _clustering.cluster raised out of a std::vector::at bounds check and then reported heap corruption:

File ".../clustering.py", line 212, in cluster
  self.clusters, invalid, self.nclus = _clustering.cluster(
IndexError: vector::_M_range_check: __n (which is 18446744073709551615) >= this->size() (which is 8000)
corrupted size vs. prev_size

18446744073709551615 is (size_t)-1. The natural reading is a null or unassigned cluster label of -1 reaching an unsigned index, and corrupted size vs. prev_size on the same run says something wrote out of bounds before the check caught it.

I could not reproduce it. Seven runs on download_bunny().triangulate() with subdivide(3) at 4000, 8000 (twice), 20000 and 200000 clusters all completed normally. So this is filed on one observation, from a session that also hit an OOM kill caused by #81 — subdivide() mutating the caller's mesh, so a loop reusing one mesh compounds the subdivision. A mesh far larger than intended is a plausible precondition, which would make this reachable rather than routine.

Worth looking at regardless, because the failure mode is memory corruption rather than a clean error, and -1 labels are real: cluster() returns an invalid count, and _cluster_centroid and _projection_limit both carry explicit clusters == -1 remaps, so the null label is expected to exist somewhere in this path.

Places to start:

  • Wherever a cluster label indexes a container in Cluster / FastCluster (src/clustering.cpp), check whether a -1 can reach it before the null-cluster remap, and whether the index is signed at that point.
  • int n_faces = faces.shape(0) and similar truncations in the same file. These were flagged as unreachable in practice during Use pyvista's public API and int32 face indices #80, but a compounded mesh makes "in practice" a weaker claim.
  • Running the suite, or a bunny reproduction, under ASAN would settle it quickly. We already have a working recipe from Use pyvista's public API and int32 face indices #80: build with -fsanitize=address -fno-omit-frame-pointer -g -O1 and run under LD_PRELOAD of libasan.so with detect_leaks=0.

Filing so it is not lost. Anyone hitting a spurious IndexError or a glibc corruption message out of cluster() should add what they have here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions