Migrate from cisd:jhdf5 to janelia-jhdf5 #314

Follow-up to #181. We are now on cisd:jhdf5 19.04.1, which is unmaintained. It bundles HDF5 1.10 native libraries for only a few platforms. janelia-jhdf5 (org.scijava:janelia-jhdf5) is a maintained fork with the same ch.systemsx.cisd.hdf5.* API. It runs on HDF5 2.x with natives from org.bytedeco:hdf5 for Linux, macOS and Windows on x86-64 and arm64. It vendors cisd:base under the same package names, so it must not be on the class path together with cisd:jhdf5 or cisd:base.

Status

janelia-jhdf5 is not released yet (22.03.01-SNAPSHOT, depending on org.bytedeco hdf5 and javacpp snapshots). Until it is, the PRs below are drafts that carry temporary plumbing (snapshot repository, javacpp.version / hdf5.version overrides, a CI step that builds and installs the janelia-jhdf5 snapshot, -Denforcer.skip=true), each marked "remove once janelia-jhdf5 is released".

Plan:

1. Release janelia-jhdf5. This is blocked on a stable javacpp-presets 1.5.15 release that includes HDF5 2.2.0 (Upgrade presets for HDF5 2.2.0 bytedeco/javacpp-presets#1808, merged) and the Windows fixes (#1814, merged). Both are in the 1.5.15-SNAPSHOT builds only; the latest release is 1.5.14 with HDF5 1.14.3. Multi-platform natives (Depend on hdf5-platform instead of only the macosx-arm64 natives JaneliaSciComp/janelia-jhdf5#11) are merged.
2. Manage org.scijava:janelia-jhdf5 in pom-scijava.
3. Take the draft PRs out of draft, with the temporary plumbing removed.
4. Drop the cisd:jhdf5 exclusions that some PRs add while the migration is in progress (see below).

API differences from cisd:jhdf5 19.04.1

Compared by signature: all public classes of 19.04.1 exist in janelia-jhdf5. The only missing public members:

• HDF5ObjectInformation.getAddress(): use getToken() / getFileNumber() (HDF5 2.x identifies objects by token).
• The old JNI helper natives in HDFHelper (getPointerSize, compoundCpyVLStr, freeCompoundVLStr, createVLStrFromCompound); replaced by pure Java.

Note that IHDF5Reader.getDoubleArrayAttribute(...) and the other IHDF5LegacyReader methods were already removed in 19.04.0. Code that only still compiled because cisd:jhdf5 14.12.x was on the class path through some dependency needs reader.float64().getArrayAttr(...) etc. (see saalfeldlab/cremi-deform#3).

Tier 1: direct dependency, source uses the API

Needs an actual migration PR. [x] = merged and released.

• https://github.com/ome/bioformats: Switch from cisd:jhdf5 to janelia-jhdf5 ome/bioformats#4490 (formats-bsd; most important: bigwarp, multiview-reconstruction and others get jhdf5 through it)
• https://github.com/saalfeldlab/n5-hdf5: toward HDF5 2.2.0 saalfeldlab/n5-hdf5#50 (existing draft; suggestions posted)
• https://github.com/bigdataviewer/bigdataviewer-core: Switch from cisd:jhdf5 to janelia-jhdf5 bigdataviewer/bigdataviewer-core#228
• https://github.com/fiji/H5J_Loader_Plugin: Switch from cisd:jhdf5 to janelia-jhdf5 fiji/H5J_Loader_Plugin#7
• https://github.com/saalfeldlab/transform-helpers: Switch from cisd:jhdf5 to janelia-jhdf5 saalfeldlab/transform-helpers#8
• https://github.com/saalfeldlab/cremi-deform: Switch from cisd:jhdf5 to janelia-jhdf5 saalfeldlab/cremi-deform#3
• https://github.com/saalfeldlab/bigcat: Switch from cisd:jhdf5 to janelia-jhdf5 saalfeldlab/bigcat#118 (could not be built here: pom-scijava 19.2.0, Java 8 only; unverified)
• https://github.com/mastodon-sc/mastodon: Switch from cisd:jhdf5 to janelia-jhdf5 mastodon-sc/mastodon#382 (CI will fail until org.litt:geff:1.1.1 is released; unrelated to this change)
• https://github.com/ilastik/ilastik4ij: Switch from cisd:jhdf5 to janelia-jhdf5 ilastik/ilastik4ij#144
• https://github.com/JaneliaSciComp/workstation: Switch from cisd:jhdf5 to janelia-jhdf5 JaneliaSciComp/workstation#62 (modules/Viewer3D, moves from ch.systems.cisd:jhdf5:14.12.0; compiles, but Viewer3D pins org.bytedeco:javacpp:0.10, which will likely break HDF5 loading at runtime until the ffmpeg/javacpp stack is upgraded)

Tier 2: direct dependency, no (or test-only) source use

Verify whether the dependency is needed at all; if it is only a transitive workaround, it can simply be removed.

• https://github.com/saalfeldlab/n5: Switch from cisd:jhdf5 to janelia-jhdf5 saalfeldlab/n5#232 (test scope, N5Benchmark)
• https://github.com/saalfeldlab/n5-utils: Switch from cisd:jhdf5 to janelia-jhdf5 saalfeldlab/n5-utils#43 (test use only, CopyTest)
• https://github.com/JaneliaSciComp/STIM: Switch from cisd:jhdf5 to janelia-jhdf5 JaneliaSciComp/STIM#14 (no source use at all; HDF5 I/O goes through n5-hdf5)
• https://github.com/scifio/scifio-hdf5: Switch from cisd:jhdf5 to janelia-jhdf5 scifio/scifio-hdf5#4 (Imaris writer uses the low-level hdf.hdf5lib API directly; the module has no tests)
• https://github.com/fiji/register_virtual_stack_slices: Switch from cisd:jhdf5 to janelia-jhdf5 fiji/register_virtual_stack_slices#17 (runtime scope; unused by its sources, only there so Bio-Formats can read HDF5, so it could be removed instead)
• https://github.com/embl-cba/cats: Switch from cisd:jhdf5 to janelia-jhdf5 embl-cba/cats#33 (dependency looks unused; the project does not resolve its dependencies even without this change, so the PR is unverified)

Tier 3: transitive through bioformats (or n5-hdf5 / bigdataviewer-core)

These only exclude ch.systems.cisd:jhdf5 today and get cisd:jhdf5 transitively. They need no change of their own once bioformats migrates, other than dropping the exclusion for the old groupId and checking that cisd:jhdf5 / cisd:base no longer appear in mvn dependency:tree.

• saalfeldlab/bigwarp, saalfeldlab/template-building
• fiji/SPIM_Registration
• JaneliaSciComp/multiview-reconstruction, JaneliaSciComp/multiview-simulation, JaneliaSciComp/BigStitcher-Spark
• PreibischLab/mvrecon, PreibischLab/Blob_Tracker4D (inactive since 2017/2020)
• juglab/labkit-ui
• UU-cellbiology/BigTrace, UU-cellbiology/bigvolumebrowser
• mianalysis/mia, BIOP/bigdataviewer-image-loaders, BIOP/quick-start-czi-reader

Temporary exclusions to remove

Until n5-hdf5 and bigdataviewer-core are released with janelia-jhdf5, projects that depend on them get cisd:jhdf5 19.04.1 transitively, so both jars would be on the class path. The PRs for n5-utils, STIM, transform-helpers and cremi-deform therefore add cisd:jhdf5 exclusions on n5-hdf5 / bigdataviewer-core / bigcat. Remove them once those are released.

Finding more consumers

gh search code jhdf5 --owner <org> --filename pom.xml per organization, then check each pom for a direct dependency vs. an exclusion and for HDF5Factory in the Java sources. The script used for this list:

#!/usr/bin/env bash
# List GitHub repos whose pom.xml mentions jhdf5, classified as direct dependency / exclusion-only,
# plus whether Java sources use HDF5Factory. Code search is limited to 10 req/min, hence the sleeps.
# Usage: find-jhdf5-consumers.sh org1 org2 ...
set -u
for o in "$@"; do
  gh search code jhdf5 --owner "$o" --filename pom.xml --limit 100 --json repository,path \
    --jq '.[]|.repository.nameWithOwner+" "+.path'
  sleep 7
done | sort -u | while read -r repo path; do
  pom=$(gh api -H "Accept: application/vnd.github.raw" "repos/$repo/contents/$path" 2>/dev/null) || continue
  kind=$(python3 -I -c '
import re,sys
t=sys.stdin.read(); k=set()
for m in re.finditer(r"<dependency>(.*?)</dependency>",t,re.S):
    b=m.group(1)
    if re.search(r"<artifactId>\s*jhdf5\s*</artifactId>",b):
        # outside any <exclusion> block => direct
        k.add("direct" if re.search(r"<artifactId>\s*jhdf5\s*</artifactId>",re.sub(r"<exclusions>.*?</exclusions>","",b,flags=re.S)) else "exclusion")
print(",".join(sorted(k)) or "mention-only")' <<<"$pom")
  src=$(gh search code HDF5Factory --repo "$repo" --json path --jq 'length'); sleep 7
  echo "$repo $path | pom: $kind | HDF5Factory files: $src"
done

CC @ctrueden