Skip to content

Backend

The zonda-request repository provides a backend system for processing ICON grid and external parameter requests. It automates the workflow of validating user input, generating grid configurations, running EXTPAR calculations, and archiving results. This backend integrates with GitHub issues and Jenkins pipelines to ensure a seamless and automated process.

Structure of the Repository

The repository is structured as follows (as of v2.0):

zonda-request/
|
|__ .github/                                 # Default directory for GitHub specific configurations
|   |
|   |__ ISSUE_TEMPLATE/                      # Contains GitHub templates for issues
|   |   |
|   |   |__ bug_report.yml                   # Template for bug report issues
|   |   |__ data_request.md                  # Template for issues containing Zonda requests
|   |   |__ feature_request.yml              # Template for feature request issues
|   |
|   |__ workflows/                           # Contains recipes for GitHub Actions workflows
|       |
|       |__ cleanup_branches_and_issues.yml  # Deletes branches and closes Zonda requests older than 7 days
|       |__ create_branch_from_issue.yml     # Creates new branch when a new Zonda request is submitted
|       |__ ping_devs_on_issue.yml           # Pings devs on bug reports and feature requests
|       |__ rerun_request.yml                # Re-runs an existing Zonda request via a trigger phrase
|
|__ img/                                     # Contains general images
|   |
|   |__ zonda_logo.png                       # The Zonda logo on a transparent background
|
|__ jenkins/                                 # Contains Jenkins-specific files
|   |
|   |__ common/                              # Contains code common to multiple Jenkins pipelines
|   |   |
|   |   |__ steps.groovy                     # Jenkins pipeline steps common to multiple pipelines
|   |   |__ variables.groovy                 # Collection of global variables used in multiple pipelines
|   |
|   |__ Cleanup                              # Pipeline cleaning the output data server and unused container images
|   |__ Request                              # Pipeline processing Zonda requests
|   |__ Testsuite                            # Pipeline processing requests labeled as testsuite on PR branches
|
|__ scripts/                                 # Contains scripts used by the different pipelines and workflows
|   |
|   |__ archive_output.py                    # Moves Zonda output and logs to the public data server
|   |__ cleanup_data_server.py               # Deletes files on the output data server older than a threshold (e.g., 7 days)
|   |__ create_config_file.py                # Extracts and validates the JSON config given by users and creates config.json
|   |__ delete_old_branches.sh               # Deletes branches older than a threshold (e.g., 7 days)
|   |__ hash.py                              # Creates a hash for each Zonda request based on the current time
|   |__ report.py                            # Reports status of pipelines to GitHub via the REST API
|
|__ src/                                     # Contains the source code to process Zonda requests
|   |
|   |__ processing/                          # Contains the main files used to process requests
|   |   |
|   |   |__ __init__.py
|   |   |__ extpar_manager.py                # Manages everything needed to run EXTPAR based on config.json
|   |   |__ grid_manager.py                  # Manages everything needed to generate/retrieve grids
|   |   |__ output_manager.py                # Manages all the output data and its location
|   |   |__ process_request.py               # Uses all managers and utilities, and defines the workflow to process a request
|   |
|   |__ utilities/                           # Contains files defining general utilities
|   |   |
|   |   |__ __init__.py
|   |   |__ utilities.py                     # Provides general utilities used by the different managers
|   |
|   |__ visualization/                       # Contains files/scripts related to the visualization of the output
|   |   |
|   |   |__ __init__.py
|   |   |__ visualization_manager.py         # Manages the visualization of different output data
|   |
|   |__ __init__.py
|
|__ LICENSE
|__ README.md
|__ environment.lock.yml                     # Defines the Conda environment for Zonda (with pinned package versions)
|__ environment.yml                          # Defines the direct dependencies of Zonda

Workflow Summary

  1. A user submits a request via a GitHub issue using the provided template (labeled as "data request").
  2. A new branch request-[ISSUE_NUMBER] is created on zonda-request via the GitHub workflow create_branch_from_issue.yml.
  3. The multi-branch pipeline Request is triggered after the new branch is found on the repository (Jenkins scans the repository every 2 minutes).
  4. The pipeline extracts the JSON config, generates the ICON grids, runs EXTPAR, visualizes output data, if requested, and archives the results on a public data server.
  5. The pipeline reports the status of the request on the GitHub issue via report.py, including links to the output data and error log-files.
  6. The output data is available for download for up to 7 days.

Output Files

The provided output folder contains the following files:

zonda_output_my_request/
|
|__ data/                                            # Contains the output data generated by the Zonda request
|   |
|   |__ DOMXX/                                       # Contains the files specific to DOMXX
|       |
|       |__ visualizations/                          # Contains plots and visualizations
|       |   |
|       |   |__ my_request_DOMXX.html                # Visualization of the grid boundary cells provided by ICON Tools
|       |   |__ my_request_DOMXX_[EXTPAR_FIELD].png  # Visualization of EXTPAR_FIELD from my_request_DOMXX_external_parameter.nc
|       |
|       |__ my_request_DOMXX.nc                      # Grid file for DOMXX
|       |__ my_request_DOMXX.parent.nc               # Grid file for the parent of DOMXX (only available for the first domain, i.e. DOM01)
|       |__ my_request_DOMXX_external_parameter.nc   # External parameters file for DOMXX generated by EXTPAR
|       |__ my_request_DOMXX_latlon[_rotated].nc     # Grid file for DOMXX remapped to a (rotated) lat-lon grid
|
|__ logs/                                            # Contains log-files from ICON Tools, EXTPAR and Zonda
|   |
|   |__ DOMXX/                                       # Contains the files specific to DOMXX
|       |__ [ALL EXTPAR LOGS]                        # Placeholder for all the EXTPAR logs (these may change depending on the configuration)
|
|__ namelists/                                       # Contains namelist files used to run ICON Tools and EXTPAR
|   |
|   |__ DOMXX/                                       # Contains the files specific to DOMXX
|       | 
|       |__ [ALL EXTPAR NAMELISTS]                   # Placeholder for all the EXTPAR namelists (these may change depending on the configuration)
|       |__ extpar_config.json                       # EXTPAR configuration file extracted from config.json
|       |__ nml_gridgen                              # ICON Tools namelist file generated based on config.json
|    
|__ config.json                                      # The JSON configuration file extracted from user input

Note that multiple DOMXX may exists, where XX is a placeholder for the domain ID, e.g. DOM02. The structure of these directories is generally the same, with some files missing depending on the configuration options of each domain.

Remote Server

The remote server is located at ETH Zurich and owned by the Institute for Atmospheric and Climate Science (IAC). It has the following specifications:

  • 128 cores
  • 3.0 TB RAM
  • 40 Gbit/s LAN
  • 64-bit Linux system

With the current setting, a maximum of 6 concurrent builds can be run on this machine. In the case of more concurrent requests, a queuing system is used.

Container Tools

The workflow described above makes use of Podman to manage the container images of ICON Tools and EXTPAR. However, for users that do not follow the usual workflow (e.g., DWD), the Zonda backend also supports Apptainer .

To use Apptainer instead of Podman, users can call the script process_request.py with an additional flag: --apptainer. Note that before running process_request.py like this, users have to make sure that the Singularity Image Files (SIF) for ICON Tools and EXTPAR exist in the workspace directory passed to process_request.py via the --workspace argument. Additionally, the SIF files must be named icon_tools.sif and extpar.sif, respectively.