Dynamic Toolkit & Experiment Loading – Documentation, Testing & Verification

[ilayfalach]

Task Summary

This issue covers all required work to ensure that dynamic toolkits and experiments in Hera load correctly, are properly documented, and include complete and reliable unit tests.
Sphinx documentation setup will be handled in a separate issue.

✅ Task List

1. Verify Dynamic Toolkit Loading

Ensure that loading a dynamic toolkit through the repository mechanism works correctly.

Test loading:

A simple dynamic toolkit

Multiple dynamic toolkits within a project

Confirm correct behavior of:

dataToolkit

Toolkit repository resolution

Dynamic class loading

Identify missing docstrings that are required for clarity.

2. Understand and Validate Experiment Loading

Document what an experiment is within Hera (structure, purpose, and workflow).

Verify the complete loading flow:

experimentHome

Getting the experiment datasource document

Loading /code/.py

Initializing experimentSetupWithData

Validate object behavior:

trialSet

entityType

Methods such as

getExperimentData()

getDataFromDateRange()

TrialWithData.getData()

Identify missing or unclear docstrings that need to be added.

3. Create a Full Notebook Tutorial for Experiments

Produce a clear Jupyter Notebook that explains:

What an experiment is

How to load experimentHome

How to access an experiment

How to inspect datasets, trial sets, and entities

How to perform basic data operations

Include runnable examples using a real project.

Add explanations + comments in Markdown cells.

4. Create Complete Unit Tests (New Test Suite)
   4.1. Unit Tests for Dynamic Toolkit Loading

Create a new JSON test definition (e.g., test_dynamic_toolkits.json) that verifies:

Listing toolkits

Loading a dynamic toolkit

Running at least one real method on it

Saving expected outputs (PREPARE mode)

Validating future changes (RUN mode)

4.2. Unit Tests for Experiment Loading

Create a second JSON definition (e.g., test_experiment_loading.json) that verifies:

Listing experiments (keys / experimentsTable)

Loading a specific experiment

Running basic data queries on it

Ensuring the experiment object initializes correctly

4.3. Integrate tests into the general test runner

Add both JSON files to run_all_json_tests.sh

Ensure that run_all_definitions.py fully supports the experiment use-cases

[yehudarav]

@ilayfalach : Did you finish it?

[ilayfalach]

I’m close to finishing, but I’m not done yet. @yehudarav