Managing package contents

[martinjanssens]

In #26 (comment), we're starting to think about where to draw the boundary for the to-be published code. Just creating an issue for discussing what we'd eventually want to include. I'd propose:

• Getting rid of the entire Download/ and Preprocess/ folders and all their contents. These only provide functions for downloading MODIS data and preparing that data so it can be used by the data handling framework I used for my paper. These functions should all work, and so they can be used with v. 0.1.0 if anyone wants to. Getting rid of these routines should also remove the need for pyhdf.
• Removing the main.py file, since it describes the procedure for reproducing the paper results, which should stay with v 0.1.0.
• Rewriting the outcome anlaysis routines in Postprocess/ such that they read data that is compatible with the new metric functions. Remove the sparse PCA functions that rely on Ristretto, as those aren't part of any core analysis that I'd think we want to support. Happy to discuss what we want to support, though.
• Making a set of wrapper examples, which, given a dataset of a particular structure, loops over a bunch of cloud fields, computes a set of metrics and stores these. One of these examples could be a slight rewrite of the old data handling for reading MODIS downloads, another would be reading in LES fields from a netCDF4 file.

[leifdenby]

hey @martinjanssens I definitely agree with stripping out this "book-keeping" functionality, but I've actually been working on a separate python module for doing a lot of this work (https://github.com/cloudsci/cloudmetrics-pipeline#cloudmetrics-pipeline). I'm quite excited with how it is working out 😄 The concept is create an analysis "pipeline" by daisy-chaining functions. For every step you get intermediate netCDF files that can be opened, plotted, etc. As you can see from the readme I've tried to expose the functionality that I would need as a user, but behind the scenes everything can be run in parallel using luigi. At the moment I'm working on the MODIS download part and turning those HDF files into netCDF. Anyway, I'm interested to hear what you think of it :)

I forgot to add: I've tried to design the code so that it can be used on any netCDF files as input. I've already used it on files from LES simulations 😄

[martinjanssens]

Wow, this is really nice. This is the more generalised, cleaner, more pythonic and more user-friendly version of what I had in mind - fantastic work! I'd be really excited to add an analysis bit to this, based on the old Postprocess functions. Adding support for various data sources is a great idea, I think that will lower the threshold for using the metrics quite a bit.

[martinjanssens]

@leifdenby I've been looking at this a little bit more, and have a few thoughts that we should probably discuss before I continue refactoring metrics:

The pipeline is currently constructed to operate on cloud_mask fields only, I think. It's probably not hard to add operations on scalar fields, but many of the metrics that still require refactoring are cloud_scalars (e.g. cloud_top_height, cloud_water_path and their derivatives, the fourier metrics and the woi (wavelet) metrics). We should probably decide on how we'd like to integrate these. I rewrote the fourier metric functions to operate not on the cloud_scalar fields directly, but on the spectra, requiring the user to explicitly first compute the spectra and then pass those to the metric functions. This is probably the clearest exposition of what's going on, but I'm not sure it's compatible with the pipeline construction we currently have. Can it easily work with constructions like these?

k1d, psd_1d_rad, psd_1d_azi = compute_spectra(cloud_scalar,...)
anisotropy = spectral_anisotropy(psd_1d_azi)
...

If not, we may have to rewrite that so it works similar to how the object metrics are currently implemented (i.e. there is one under-the-hood function that computes the spectra, stores this in a quasi-cache and gets called by each of the fourier metrics, which themselves operate on the cloud_scalar). What do you think is the best course of action here? A similar question arises for the wavelet metrics, which I've started on, but didn't wan't to finish until we've agreed on how to best integrate it.

[leifdenby]

The pipeline is currently constructed to operate on cloud_mask fields only, I think

Yes, you're right, sort of. What I was thinking of was that each stage in the pipeline would output an xr.Dataset. The very first stage would be able to open not just files containing single 2D fields (for example cloud-mask), but also other scalar fields.

Some steps in the pipeline could add extra variables to this dataset (for example produce a cloud-mask from a some cloud scalar). The metric functions which require additional scalars would then check if the dataset contains the variables required. If not, then we raise an exception suggesting which pipeline steps to call to add the extra variables (to derive them from what is already there) or simply say that the user needs to load files to make them available to.

Maybe what we need is a list of what additional fields (besides ones that can be derived from a cloud mask) are needed by some metrics? I'm struggling a bit to keep this in my head at the moment. I'll create an issue for discussions around an API now and start a list.

[martinjanssens]

Nice, that's really smooth and will work very well I think. We could then consider to automatically handle the computation of derived data that is necessary to compute a metric in the pipeline, i.e. if I want to compute the fourier script's spectral_length, I'd add spectral_length to the pipeline's metric list, and it goes and checks whether the required input (k1d, psd_1d_rad) already has been computed or needs to be added to the dataset before computing it?

[leifdenby]

I'd add spectral_length to the pipeline's metric list, and it goes and checks whether the required input (k1d, psd_1d_rad) already has been computed or needs to be added to the dataset before computing it?

Exactly!

[leifdenby]

I'm going to complete the clean-up @martinjanssens by assuming that all the functionality which isn't directly computing any metrics will done by https://github.com/cloudsci/cloudmetrics-pipeline rather than this package. If you think this is the wrong approach feel free to re-open this issue and we can discuss further :)