This repository contains YANG Catalog's REST API and the bulk of its internal infrastructure for processing YANG modules and extracting their properties. It also serves information to YANG Catalog's frontend and implements the functionality of the Admin UI.

The scripts in this repository serve as a backend to add, update, remove and manage YANG module files in yangcatalog. It is composed of:

• scripts that run every day as a cron job
• an API which lets users add, remove or find the modules they expect
• scripts that parse new/updated modules from different sources and populate them into yangcatalog database

This repository works directly with the yangModels/yang repository. That repository contains all the modules structured by vendors (Cisco, Huawei and others) and SDOs (IETF, IEEE, MEF, BBF and others).

The most important directory in this repository is parseAndPopulate. This directory contains parsing scripts to parse all the modules of given directories. This gives us all the metadata of the modules according to draft-clacla-netmod-model-catalog-03. Parsed metadata is subsequently populated to Redis and ConfD databases. These databases are used for the yang-search part of yangcatalog.org.

We can parse modules either with the --sdo option, which will go through a given directory and parse all of its yang modules one by one, or without this option, which will try to find a platform-metadata.json file in the directory which contains paths to capability.xml files and parse all the modules according to those files with vendor metadata added.

To find all the modules with missing or wrong revisions, namespaces, imports, includes or modules that according to a capability.xml file should be in the folder but are missing, we can use the integrity script.

The API module runs as a UWSGI emperor vassal (using the yang-catalog.ini file) and contains several endpoints. Most of the endpoints serve to find modules in different ways. This is described deeper in the API documentation. If the user is registered, she/he can add, modify or delete modules based on a pre-approved path. Once a user has filled in the registration form, one of yangcatalog's admin users needs to approve user using Admin UI and give the user specific rights, so he is able to add, remove or update only certain modules.

Some requests may take a longer period of time to process. Because of this some endpoints will give the user a job-id. The user can check this job at any time to see if the job has been completed or is still processing or failed during the execution by using the job-id.

The Yangcatalog API is also used by some automated external jobs. Every time new modules are merged into the yangModels/yang repository a job is triggered to populate all the new modules to the yangcatalog database.

The backend API also receives IETF Yang models every day and if there are any new drafts it will automatically populate the yangcatalog database and update the repository with all the new IETF modules if GitHub Actions pass successfully.

There are several cron jobs that run every day.

• statistics job which goes through all the modules that are in yangcatalog and generates an HTML file which has information about what vendors' and SDOs' modules we have and the number of modules that we have.
• resolve_expiration job that checks all the IETF draft modules and their expiration dates and updates its metadata accordingly.
• remove_unused job that removes data on the server that is not used anymore.
• user_reminder script that will be triggered once a month to show us which users we have in our database.
• In the ietfYangDraftPull directory there is one job.
  1. pull_local clones the https://github.com/YangModels/yang repo, updates RFCs data with the latest data from the YANG-RFC.tgz, which is created during running of this script: https://github.com/YangCatalog/module-compilation/blob/develop/ietf_modules_extraction/run_ietf_module_extraction.sh. Then goes through RFC, experimental, and IANA modules data and populates yangcatalog.
• In the automatic_push directory there are two jobs.
  1. ietf_push retrieves and adds new IETF RFC and draft modules to the https://github.com/yang-catalog/yang repository if there are any new/updated modules.
  2. iana_push rsyncs and pushes new IANA modules to the https://github.com/yang-catalog/yang repo.
• recovery script which pulls all the data from Redis and creates a json file which is saved on the server as a backup. If we lose all the data for some reason we can use this script to upload it back with no loss of data.
• revise_tree_type reevaluates the tree type for modules that were previously of type nmda-compatible.
• reviseSemver reevaluates the derived semantic versions of modules.

Yang admin users are informed about every new module added to the yangcatalog database in a Cisco Webex teams room and by email. Also, there are other emails and Webex messages sent for different events, they can be found in the message_factory module.

ConfD Premium has to be accessible

Since this is just a small part of the whole functional environment, you need to build using the docker-compose file from the deployment folder. Then the catalog_backend_api:latest image can be used to run a docker container where everything will start as it is supposed to.

See the README.md file in the documentation/ directory.

Using backend/recovery/recovery.py --load --file /var/yang/cache/redis-json/<specific>.json for loading some specific backup, or using backend/recovery/recovery.py --load to load the latest backup.

To be localized to your configuration.

        location /doc {
            alias /usr/share/nginx/html/slate/build;
        }

        location /api {
            rewrite /api(/.*)$ $1 break;
            include uwsgi_params;
            uwsgi_pass 127.0.0.1:8443;
            uwsgi_read_timeout 900;
        }