• Clone the project

git clone git@github.com:dica-solution/exam_elt.git
cd exam_etl

• Setting up the environment

poetry install

Set up the Environment variables in the .env.dev depend on the installed environment. If the installed environment is on dica-server, you can skip this step.

DATABASE_URL

• Create the tracking_logs table if not existed

python create_id_mapping_table.py

• Create the sync_logs table if not existed, the lastest runtime is set to 7 days ago (can choose how many days ago by set the d, for example --d 3)

python create_sync_logs_table.py -d 7

• Depending on specific requirements, you will choose one from the following commands:

# Import by list of IDs or just one ID
python data_integration.py --file [file_path]
python data_integration.py --id [ID]

# Sync
python data_integration.py --sync

• Runtime logs in logs/runtime_log

python data_integration.py --file example_ids.txt # import with list of IDs
python data_integration.py --id 99 # import with a single ID