How to read the structure member variables from a section of the Structure array

[rschmunk]

I've been coding up the ability to plot data stored in structure member variables, and have it working if I read in the member variable for the entire Structure array, e.g.,

Variable var = njfile.findVariable(sname);
Structure struct = (Structure)var;
Variable memberVar = struct.findVariable(mmame)
Array a = memberVar.read()

However, in one of test files I am working with, there are 226 million structures in the array. I'm not able to do much with a member Array extracted as above before memory trouble arises. So I'd like to be able to read the member values in sections. I blithefully tried doing so by using the Variable.read(int[], int[]) method, but that in turn calls Variable.read(Section), which just throws an exception. Okay, the API for Variable.read(Section) does say…

"If the Variable is a member of an array of Structures, this returns only the variable's data in the first Structure, so that the Array shape is the same as the Variable. To read the data in all structures, use ncfile.readSectionSpec()."

That apparently should read ncfile.readSection(String), and I have no idea how to work with that method. The API for that method has a "See Also" notation for SectionSpecification, but the webpage that it links to says "page not found, try here", which in turn returns a 404. and I have been unable to find it it in the netCDF-Java documentation.

ETA: Just noticed the ParsedSectionSpec class in the API, which looks useful. Same broken link to Section Specification.

[lesserwhirls]

Interesting...I cannot find any pages focusing on the section specification in our docs (back to at least 4.6 days). However, I did find this page on the wayback machine: https://web.archive.org/web/20080706185524/https://www.unidata.ucar.edu/software/netcdf-java/reference/SectionSpecification.html

That seems like pretty important information! I will add a new page in our docs to reflect this content, and get any existing links updated to point to the right location.

[lesserwhirls]

A concrete example of reading data from a structure memeber can be found here, but specifically something like:

ncfile.readSection("/Chromosomes/Summary.StartIndex")

for all data in the member, or, for a subset of data from the member:

ncfile.readSection("/Chromosomes/Summary(12:20).StartIndex")

where the relevant ncdump of that dataset looks like:

netcdf cdmUnitTest/formats/hdf5/20130212_CN021_P3_222k_B02_WD7195FBPAT10231Nat_Nat_Std_CHTNWD_OP3_14.mip222k.oschp {
  ...
  group: Chromosomes {
    variables:

      Structure {
        ushort Chromosome;
        int Display_&size;
        char Display(4);
        ulong StartIndex;
        ulong MarkerCount;
        float MinSignal;
        float MaxSignal;
        float MedianCnState;
        float HomFrequency;
        float HetFrequency;
        float Mosaicism;
        float LOH;
        float MedianSignal;
      } Summary(24);
  ...
  }
  ...
}

[rschmunk]

@lesserwhirls The day after I originally posted, I was able to accomplish what I wanted doing something like the following:

Variable var = njfile.findVariable(sname);
Structure struct = (Structure)var;
Variable memberVar = struct.findVariable(mmame);

Section sect = new Section(origin, shape);        
ParsedSectionSpec spec = new ParsedSectionSpec(memberVar, sect);
String specStr = spec.makeSectionSpecString();

Array a =njfile.readSection(specStr);

The only negative was that in the example case of an array of 226 million Structures, it took roughly twice as long to retrieve all the member values in chunks of about 40 million than it did to grab them all at once. OTOH, no memory errors.