Ref #221

[EdwardSafford-NOAA]

Modify the IodaObsSpace object and time_series mechanism to make these changes:

1. Allow missing files in a time_series, including the first file.
2. Allow input yaml files using the IodaObsSpace object to specify multiple data files within a dataset specification. This specification is now supported by time_series:

datasets:
  - name: experiment
    type: IodaObsSpace
    filenames:
      - /scratch1/NCEPDEV/da/Edward.Safford/noscrub/test_data/sample_gsi2ioda/abi_g16_obs_2025030600.nc4
      - /scratch1/NCEPDEV/da/Edward.Safford/noscrub/test_data/sample_gsi2ioda/abi_g16_obs_2025030606.nc4
      - /scratch1/NCEPDEV/da/Edward.Safford/noscrub/test_data/sample_gsi2ioda/abi_g16_obs_2025030612.nc4
      - /scratch1/NCEPDEV/da/Edward.Safford/noscrub/test_data/sample_gsi2ioda/abi_g16_obs_2025030618.nc4
    channels: &channels 7,8
    groups:
      - name: GsiEffectiveQC
        variables: &variables [brightnessTemperature]

3. Modify time_series utilities to treat all file names as lists. This allows time_series to iterate over the specified filenames.

I've verified that yaml files using IodaObsSpace and JediVariationalBiasCorrection objects work with these changes. Note that the JediVeriationalBiasCorrection object does not support specifications of multiple data files (i.e. multiple instances of either or both the bias_file or lapse_file) within a specified data_collection. Changing that limitation will be done in a new issue.

Dependencies

None

Impact

None

Closes #221

[kevindougherty-noaa]

Looks good to me up front. I did have a question for everyone if it were possible. Could we use some sort of logic where we can feed the yaml the filename as a template i.e. abi_g16_obs_{$datetime}.nc4 and have an option for start date and stop date? I can see if you have to make a month long time series, adding 120 cycle filenames would be a pain.

[EdwardSafford-NOAA]

Yes we have most of that working now in obs-monitor. The plot templates use this construction to set up the 120 (or more) filenames:

# Data read
# ---------
datasets:
  - name: time
    satellite: {{SAT}}
    sensor: {{SENSOR}}
    type: MonDataSpace
    control_file:
      - {{DATA}}/time.{{SENSOR}}_{{SAT}}.ctl
    filenames:
      {% for i in range(0, 720, 6) %}
      {% set offset_hr = i | string %}
      {% set offset_str = "-" + offset_hr + "H" %}
      {% set offset_td = offset_str | to_timedelta %}
      {% set PDATEnew = PDATE | add_to_datetime(offset_td) %}
      - {{ DATA }}/time.{{ SENSOR }}_{{ SAT }}.{{ PDATEnew | to_YMDH }}.ieee_d
      {% endfor %}

The 120 filenames are constructed using wxflow functions (mostly jinja), and the resulting file is fed into eva. Similar substitutions could be made for the time-series start and stop dates.

[CoryMartin-NOAA]

I think @kevindougherty-noaa is suggesting adding that capability to EVA so that the list of filenames gets constructed in the script and the YAML only provides a template and start/end dateTimes. Something to consider, but probably outside the scope of this PR?

[kevindougherty-noaa]

@CoryMartin-NOAA yes not needed for this PR just a thought to be added to eva capabilities.

[Dooruk]

Thanks again, it works great, here is an example with two different time windows (6-hr & 24-hr):

I'm not sure if it's possible to arrange max,min, or xticks for datetime in EVA?

[EdwardSafford-NOAA]

@Dooruk you can specify the tick location in eva by adding lines like this to the plots entry:

graphics:
  - figure
    plots:
      set_xticks: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22]
      set_yticks: [100, 150, 200, 250, 300, 350, 400]

[EdwardSafford-NOAA]

For setting min and max values either axis I don't have an example of that but it has to be something like adding
set_xlim: [x_min, x_max] to the plots: dictionary.