Microframework to improve the experience of running and managing archival records of computational experiments, such as machine learning and data science experiments, in Python.

• Automatically create (nested) folder structure for results of each run of an experiment
• Simply attach metadata such as performance metrics to experiment object and they will be automatically stored as JSON file
• Easily attach file artifacts such as matplotlib figures to experiment records
• Log messages to stdout as well as permanently store into log file
• Ready-to-use automatically generated boilerplate code for the analysis and post-processing of experiment data after experiments have terminated.
• Experiment inheritance: Experiment modules can inherit from other modules and extend their functionality via parameter overwrites and hooks!
• Configuration files: Create YAML config files to run parameter variations without duplicating code
• Built-in hyperparameter optimization with Optuna integration (available with full installation)

Install the stable version with pip

pip3 install pycomex

Or the most recent development version by cloning the source:

git clone https://github.com/the16thpythonist/pycomex.git

and then installing with either pip

cd pycomex
pip3 install -e .

or poetry

cd pycomex
poetry install

Each computational experiment has to be bundled as a standalone python module. Important experiment parameters are placed at the top of this module. All variable names written in upper case will automatically be detected as parameters of the experiment.

The actual implementation of the experiment execution is placed into a single file which will have to be decorated with the Experiment decorator.

Upon execution the experiment, a new archive folder is automatically created. This archive folder can be used to store all the file artifacts that are created during the experiment. Some artifacts are stored automatically by default, such as a JSON file containing all data stored in the main experiment storage, a snapshot of the experiment module and more...

Archiving of metadata, file artifacts and error handling is automatically managed on context exit.

# my_experiment.py
"""
A minimal example demonstrating PyComex experiment structure.
This docstring is saved as experiment metadata.
"""
from pycomex.functional.experiment import Experiment
from pycomex.utils import file_namespace, folder_path

# Experiment parameters (uppercase variables are auto-detected)
MESSAGE: str = "Hello PyComex!"
ITERATIONS: int = 5

# Debug mode: reuses same archive folder for development
__DEBUG__ = True

@Experiment(
    base_path=folder_path(__file__),     # Results stored relative to this file
    namespace=file_namespace(__file__),  # Creates folder based on filename
    glob=globals(),                      # Provides access to parameters
)
def experiment(e: Experiment) -> None:
    e.log("Starting experiment...")

    # Store structured data (creates nested JSON structure)
    e["config/message"] = MESSAGE
    e["config/iterations"] = ITERATIONS

    # Run experiment loop
    for i in range(ITERATIONS):
        metric = i * 0.1
        e.track("metrics/value", metric)  # Track time-series data
        e.log(f"Iteration {i}: {MESSAGE} (metric: {metric})")

    # Save final results and artifacts
    e["results/final_metric"] = metric
    e.commit_raw("results.txt", f"Final result: {metric}")

# Run experiment when executed directly
experiment.run_if_main()

Running the Experiment:

# print help
python my_experiment.py --help

# Basic execution
python my_experiment.py

# Override parameters via command line
python my_experiment.py --MESSAGE "Custom message!" --ITERATIONS 10

This example would create the following folder structure:

my_experiment/
└── debug/
    ├── experiment_out.log      # Complete execution log
    ├── experiment_meta.json    # Experiment metadata and parameters
    ├── experiment_data.json    # All tracked data and stored values
    ├── experiment_code.py      # Snapshot of the original experiment code
    ├── results.txt            # Custom artifact saved via commit_raw()
    └── .track/                # Time-series visualizations
        └── metrics_value_001.png  # Auto-generated plot of tracked metrics

Key Features:

• Automatic Archiving: Each experiment run creates a timestamped folder with complete execution records
• Parameter Management: Uppercase variables are automatically detected as configurable parameters
• Command-line Overrides: Parameters can be modified without editing code
• Structured Data Storage: Nested data organization using path-like keys (e.g., "config/learning_rate")
• Time-series Tracking: Built-in support for tracking metrics over time with automatic visualization
• Artifact Management: Easy saving of files, figures, and custom data formats

PyComex provides a powerful CLI accessible via the pycomex command:

Creating New Experiments:

# Create a new experiment module from template
pycomex template experiment my_new_experiment.py

# Create a configuration file from an existing experiment
pycomex template config -e experiment.py -n config_name

Running Experiments:

# Run an experiment directly
pycomex run experiment.py

# Run a configuration file
pycomex run config.yml

Managing Experiment Archives:

# List recent experiments
pycomex archive list

# Show detailed information about an experiment
pycomex archive overview

# Compress and archive old experiments
pycomex archive compress results/

For more command line options use pycomex --help.

NOTE. For an introduction to more advanced features take a look at the examples in pycomex/examples ( https://github.com/the16thpythonist/pycomex/tree/master/pycomex/examples )

Complete documentation is available at: https://the16thpythonist.github.io/pycomex/

Additional details on specific topics can be found in the DOCUMENTATION.rst file.

The pycomex/examples ( https://github.com/the16thpythonist/pycomex/tree/master/pycomex/examples ) folder contains practical example modules that illustrate key features of the framework.

PyComex is built on top of these excellent open source libraries:

• Click - Command line interface toolkit
• Rich - Rich text and beautiful formatting in the terminal
• Jinja2 - Modern and designer-friendly templating language
• NumPy - The fundamental package for scientific computing
• Matplotlib - Comprehensive 2D plotting library
• pytest - Testing framework