This tool is a collection of scripts that helps to automatically generate FIWARE smart data model and agent configuration. It takes a list of variables from a process control system (PCS) datasource as input and generates resources based on name system 'Identification System for Power Plants' (German: Kraftwerk-Kennzeichen-System - KKS).

It has been developed because manual creation of smart data model (schema.yaml) and agent configuration (config.js) in big data models (1,000 attributes +) is a lot of effort and prone to errors.

• Create FIWARE ressources from KKS variable names
  • Table of Contents
  • Overview
  • Background
    • IoT-Agent Configuration
    • Smart Data Model
  • Usage
    • Preparation
    • Common concepts
    • 1) GroupTagsAndAssignMeaning
      • configuration
      • manual steps
      • automated steps
    • 2) CreateFiwareSDM
      • configuration
      • manual steps
      • automated steps
    • 3) CreateFiwareAgentConfiguration
      • configuration
      • manual steps
      • automated steps

The tool consists of 3 submodules that can be executed after each other or as single steps depending on configuration:

1. GroupTagsAndAssignMeaning: Split relevant variable names with systematic of Identification System for Power Stations (Kraftwerkkennzeichensystem - KKS) and assign meanings to the name parts, attributes and entities
2. CreateFiwareSmartDataModel: Use splitted tags and meanings to generate a FIWARE smart data model
3. CreateFiwareIotagentConfiguration: Use splitted tags and meanings to create an IoT-Agent configuration

The (OPC UA) IoT-Agent uses a configuration file to extract desired variables from an OPC UA datasource if you prefer manual configuration. An example for a FIWARE IoT-Agent configuration can be found here.

The configuration exists basically out of three sections:

1. types: This section summarizes all FIWARE entities as types and summarizes their active, lazy and command attributes in general
2. contexts: This section maps (OPC DA) tags to specific FIWARE entities and attribute defined in types using an id
3. contextSubscriptions: This sections creates context subscriptions for (opc da) tags and optionally specifies their data types

Contexts and contextSubscriptions section is the same, so there will be two sections created that can be copied into actual Agent-configuration sections.

Configuration will always be mapped to FIWARE context and data types, we will also need a Smart Data Model to introduce new entities and attributes to the FIWARE. Therefore a schema.yaml file will be generated by the tool, that can be tested using https://editor.swagger.io/

Examples for Smart Data Models and desired schema.yaml can be found here.

Python 3+ environment needed (tested with: 3.10.4). Make sure to install all dependencies from requirements.txt.

This tools automizes the steps described above and might needs some human input to create ressources correctly (adopt tool configuration according to your needs). Execute the scripts in the defined order / alltogether and generate neccessary ressources stepwise / at once.

The settings in the beginning let you configure the tool. Set "true" for all steps that you want to execute or "false" for all steps that you don't want to execute.

Moreover there is a language setting that you can use to choose a language for entity names and attribute descriptions. Make sure to also adopt text templates in configuration according to desired language.

Each script has an own input / output folder which name is the number of the script. You can find it in ressource folder:

• Script "GroupTagsAndAssignMeaning.py" has input / output folder ressources/1/*
• Script "CreateFiwareSDM.py" has input folder ressources/2/*
• Script "CreateFiwareConfiguration.py" has input folder ressources/3/*

Executing the script you have the option to seperate input / output files from each script by only using the dedicated folder of the script (useful if you execute the steps as single steps). Alternatively you can choose to use the output file of the script before as an input file for the next step (mandatory if you want to execute all steps after eachother).

Script 1 parses given attributes with the use of 'Identification System for Power Stations' (Kraftwerkkennzeichensystem - KKS) and assigns an entity as well as meanings to the tagnames.

1. Create an excel export with fields at least for propterty name and data type
2. Adopt tool confiugration by setting variables in config section for script 1.
   1. filepath_input: Path to input excel-file to parse. The file should at least contain the columns for attribute name and data type. Further columns can be added as well if needed, but aren't neccessary.
   2. filepath_output: Path to output-excel for parsed names
   3. input_column_attribute_name: Name of the column that holds the Property Name
   4. input_column_attribute_type: Name of the column that holds the Property Type Name
   5. output_column_attribute_cleaned_kks_name: Column name of mandatory output column with cleaned attribute name property (according to NGSI-LD standard)
   6. output_columnn_attribute_kks_0: Column name of mandatory output column with property for kks layer 0 (plant overall)
   7. output_columnn_attribute_kks_1: Column name of mandatory output column with property for kks layer 1 (function)
   8. output_columnn_attribute_kks_1_nr: Column name of mandatory output column with number of property at kks layer 1
   9. output_columnn_attribute_kks_2: Column name of mandatory output column with property for kks layer 2
   10. output_columnn_attribute_kks_2_nr: Column name of mandatory output column with number of property at kks layer 2
   11. output_columnn_attribute_kks_3: Column name of mandatory output column with property for kks layer 3
   12. output_columnn_attribute_entity: Column name of mandatory output column with entity (logical group of tags)
   13. output_column_attribute_entity_other_lang: Column name of mandatory output column with entity in an alternative language
   14. output_columnn_attribute_meaning: Column name of mandatory output column with meaning of property
   15. output_columnn_attribute_ending: Column name of mandatory output column with property ending
   16. output_columnn_attribute_type_1: Column name of mandatory output column with property for opc ua data type mapping eg. map an OpenAPI3.0 data type to opc type to create a FIWARE data model (step 3)
   17. output_columnn_attribute_type_2: Column name of mandatory output column with property for opc ua data type mapping eg. map an NGSI-LD data type to opc type to create FIWARE IoT-Agent configuration (step 4)
   18. filepath_kks_meaning: Path to meaning-tables of kks name system (maps abbreviation to meaning)
   19. mapping_column_type_name: Column name of mandatory column in mapping table with original variable type (default: 'OPC_Name')
   20. mapping_column_map_type_name: Column name of mandatory column in mapping table with Smart Data Model type (default: 'OpenAPI_3.0_Type')
   21. mapping_column_abbreviation: Column name of mandatory column in mapping table with abbreviation for function (default: 'abbreviation')
   22. mapping_column_aggregate: Column name of mandatory column in mapping table with abbreviation for aggregate (default: 'aggregate')
3. Create or check existing mapping file configured in filepath_kks_meaning to assign correct entitiy / attribute meaning
4. Configure execution of this step and set execute_script_1 from general config section to true. Afterwards execute tool: py Main.py
5. Afterwards you should find an output file with splitted names and assigned meanings in configured output file

Make sure that:

• The KKS-rules used in that tool meet the nature of your extracted variables (check name parts of output file)
• Make sure you assign correct meanings to file at filepath_kks_meaning
• Adopt tool parametrization and copy input / output files to the folder of the script
• Check that KKS-names have been parsed correctly

The input file will be parsed and an output file will be created with the help of the file at filepath_kks_meaning. Therefore:

• Create new columns in Excel which will be filled with the KKS name parts (see configuration for output columns)
• Moreover a meaning to KKS name will be assigned by filling column entity (meaning of function at kks level 1) and meaning (composed of [<Function> <Function-Nr> <Aggregate> <Aggregate-Nr> <OperatingRessource (optional)> (<Meaning Suffix / Suffix>)])
• Tags which doesn't fullfill requirements of KKS name system will be added at Name column and other columns remain empty
• Entity and attribute names will be cleaned, so that their names are according to NGSI-LD rules

Script 2 parses attributes with assigned KKS names and creates a FIWARE smart data model.

1. Create an excel export with tool 1 to group tags and assign meanings (see step 1)
2. Adopt tool confiugration by setting configuration of this script section
   1. filepath_input: Path to excel-file to parse
   2. filepath_output: Path to output-yaml with data model
   3. clear_existing_file_from_line: If you want to have some content in the schema.yaml output file and want to append new content set that parameter to a line number. All content after the line number will be deleted and new data will be added after the line.
   4. entitiy_description: Descriptive text of entity. You can use variables ENTITY_NAME (Name of the entity derived by cleaned function meaning at kks level 1), ENTITY_NAME_OL (Name of entity in an alternative language) and FUNCTION_CODE (KKS function code of level 1) in your description if you like.
   5. subsystem_description: Descriptive text for default property of each generated entity for relation to PLS (property 'isSubSystemOf')
   6. mandatory_properties: Smart Data Model properties that should be marked as mandatory
3. Configure execution of this step and set execute_script_2 from general config section to true. Afterwards execute tool: py Main.py
4. Afterwards you should find an output file with yaml specification for FIWARE smart data model

Make sure that:

• You either correct non KKS attributes and entities before executing this step by adding an entity and a meaning to these items or do something about them after script execution (mentioned at the end of the script execution as CMD output)
• You adopt tool parametrization and copy input / output files to the folder of the script
• The created data model is valid and meets your project requirements. Use swagger editor to check validity (see https://github.com/FIWARE/tutorials.Understanding-At-Context/tree/41f06aca6fe9fd66b5f1693bfb115da475c72334 for further help)
• You correct tags that appear twice in the list and are mentioned at the end of the script

In the moment the input file will be parsed and an output file will be created. Therefore:

• KKS output level 1, called function, will be used as entity name. This function name will be cleaned (remove whitespaces and special characters) and used to group common attributes.
• As description of the entity we will use [Process management function '<KKS Name> (KKS function code: <KKS function code>)']
• Each entity has some default attributes:
  • isSubsystemOf (type relation) reference to PCS entity
  • id (type property) NGSI-LD ID (unique urn)
  • type (type property) NGSI-LD entity type
  • dateObserved timestamp for measured values
• All other attributes are the attributes which have the same function at KKS level 1. The official KKS name will be used as attribute name
• The description of the attributes is composed of [<Funktion> <Funktion-Nr> <Aggregate> <Aggregate-Nr> <Operational Resouces (optional)> (<Meaning Suffix>)] The type of the attributes is either boolean (KKS-type 'Bitwert') or number (KKS-type '32-Bit IEEE 754') in our case, but adjusting the type mappings in mapping file from step 1 should allow you to use further types and custom mappings.

Script 3 parses attribute list with assigned KKS names and creates relevant sections of FIWARE OPC UA IoT-Agent configuration.

1. Create an excel export and data model with tools from step 1 and 2
2. Adopt tool confiugration by adjusting the config section of step 3
   1. filepath_input: Path to excel-file to parse
   2. filepath_output_types: Path to output with configuration types section (mapping of FIWARE entity and attribute names and types)
   3. filepath_output_context: Path to output with configuration context / contextSubscriptions section (mapping of FIWARE entity and attribute names and with OPC UA attribute names and types)
   4. opc_ua_namespace: OPC UA namespace of the variable. This text will be used in agent mapping section as opcua_id / object_id eg. ns=<NR>;s=0GCN10AP010//WAB_DosPu_NH4OH.SPOP_HL
3. Configure execution of this step and set execute_script_3 from general config section to true. Afterwards execute tool: py Main.py
4. Afterwards you should find two output files with json arrays that can be used for IoT-Agent tool configuration. Copy them to the specific sections of you IoT Agent configuration.

Make sure that:

• You have a complete data model from step 2
• You have a complete list of attributes and meanings from step 1
• You copy output json-objects to corresponding fields in config.js file to use them with IoT-Agent
• You adopt tool parametrization and copy input / output files to the folder of the script

In the moment the input file will be parsed and an output file will be created. Therefore:

• The first json object called 'types' will be created from column entity, name and datatype. It contains all relevant FIWARE entities and their attribute names and types.
• The second json object called 'context' or 'contextSubscriptions' will be created from column entity and name. It contains relevant FIWARE entities and maps (OPC) Tag names to attribute names of an entity.