[5.5.4 SNAPSHOT]: NPEs reading some Zarr variables

[rschmunk]

Versions impacted by the bug

v5.x

What went wrong?

A user who is interested in plotting Zarr data has sent me 4 example directory stores that are lon-lat gridded.

I am able to plot the 2D data in 3 of them, albeit without lon-lat coordinates since the NJ Zarr code does not yet speak NcZarr. (Maybe that could an optional enhancement?) The fourth I haven't a clue about yet. Maybe the data actually are all NaN!

But ignore that for now. More troublesome is that in trying to make line plots of the longitude and latitude variable in these Zarr directory stores, I get an NPE when reading the data for 7 out of 8 cases. The stacktrace in those 7 cases leads to RandomAccessFile.java:737.

The compressor for the lon and lat vars in two of these files is null. Another is blosc and the other is zlib.

I might have to ping the user about it, but if you need one or samples to work with, please ask.

Relevant stack trace

java.lang.NullPointerException
	at ucar.unidata.io.RandomAccessFile.read_(RandomAccessFile.java:737)
	at ucar.unidata.io.RandomAccessFile.readBuffer(RandomAccessFile.java:503)
	at ucar.unidata.io.RandomAccessFile.seek(RandomAccessFile.java:491)
	at ucar.unidata.io.RandomAccessFile.readBytes(RandomAccessFile.java:676)
	at ucar.unidata.io.RandomAccessFile.read(RandomAccessFile.java:764)
	at ucar.unidata.io.zarr.RandomAccessDirectory.read_(RandomAccessDirectory.java:205)
	at ucar.unidata.io.RandomAccessFile.readBuffer(RandomAccessFile.java:503)
	at ucar.unidata.io.RandomAccessFile.seek(RandomAccessFile.java:491)
	at ucar.nc2.iosp.zarr.ZarrLayoutBB$DataChunk.getByteBuffer(ZarrLayoutBB.java:174)
	at ucar.nc2.iosp.LayoutBBTiled.hasNext(LayoutBBTiled.java:101)
	at ucar.nc2.iosp.zarr.ZarrLayoutBB.hasNext(ZarrLayoutBB.java:98)
	at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:360)
	at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:288)
	at ucar.nc2.iosp.zarr.ZarrIosp.readData(ZarrIosp.java:121)
	at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122)
	at ucar.nc2.Variable.reallyRead(Variable.java:797)
	at ucar.nc2.Variable._read(Variable.java:736)
	at ucar.nc2.Variable.read(Variable.java:614)
	at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:459)
	at ucar.nc2.dataset.VariableDS._read(VariableDS.java:432)
	at ucar.nc2.dataset.VariableDS._read(VariableDS.java:442)
	at ucar.nc2.Variable.read(Variable.java:600)
	at ucar.nc2.Variable.read(Variable.java:546)
	at gov.nasa.giss.data.nc.array.NcArray1D.doSlice(NcArray1D.java:533)

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

N/A

Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

[mnlerman]

Hello @rschmunk, I'm going to take a look at this. Could you provide me with a couple of the sample files to use for debugging?

[rschmunk]

@haileyajohnson, See example here.

Trying to read the longitude and latitude variables throws the NPE.

When the rainfall variable is read, it comes up entirely NaN. That may or may not be correct, but I have seen it happen with a couple other example Zarr stores which I believe to have good data in them.

[rschmunk]

Another example here.

In this one the reading the latitude variable throws an NPE, but reading the longitude variable throws an exception that "last source index 15076 out of bounds".

java.lang.ArrayIndexOutOfBoundsException: arraycopy: last source index 15076 out of bounds for byte[8092] at java.base/java.lang.System.arraycopy(Native Method) at ucar.unidata.io.RandomAccessFile.readBytes(RandomAccessFile.java:662)

The "variable" array can be read and the data look good.

[mnlerman]

@rschmunk I was able to resolve the issue that was causing the null pointer exceptions for both of the files that you provided. I also looked at the issue that you mentioned about the NaNs. I opened your file using Python and saw that there is in fact data in the lat, lon and rainf-tavg variables but the file is compressed using blosc. Netcdf-java doesn't support reading zarr files compressed using blosc. So it appears that the NaN filling is the expected behavior in this case. Are you seeing the same issue on files that are compressed in other ways?

[rschmunk]

@mnlerman, Poking bout, it looks like all the sample stores that I have are (which is not a lot) either blosc or uncompressed.

[haileyajohnson]

Right now we're only able to read zarr data that's compressed with one of the filters implemented in netcdf-java: Deflate (zlib), Shuffle, 32-bit Checksum (CRC, Fletcher, and Adler), or ScaleOffset

[mnlerman]

I'm going to close this issue because the fix for the null pointer exceptions is merged into the baseline (/pull/1121). Please open a new issue if you encounter additional related problems