NBPipeliner is a Python library designed to automate the execution of Jupyter notebooks, convert them to HTML reports, and serve them via a Flask web application. It also includes scheduling capabilities to run notebooks at regular intervals.

• Features
• Prerequisites
• Installation
• Configuration
• Usage
• Logging
• Contributing
• License

• Automated Notebook Execution: Execute Jupyter notebooks using Papermill.
• HTML Conversion: Convert executed notebooks to HTML reports using nbconvert.
• Web Serving: Serve HTML reports via a Flask web application.
• Scheduling: Schedule notebook executions at configurable intervals.
• Logging: Comprehensive logging for monitoring and debugging.

• Python 3.8 or higher
• Poetry for dependency management

git clone https://github.com/compartia/nbpipeline.git
cd nbpipeliner

Ensure you have Poetry installed. If not, install it using:

curl -sSL https://install.python-poetry.org | python3 -

Then, install the project's dependencies:

poetry install

Activate Poetry's virtual environment:

poetry shell

NBPipeliner can be configured using environment variables. Below are the available configurations:

You can set environment variables in your shell or use a .env file with tools like python-dotenv.

Example:

export LOG_LEVEL=DEBUG
export NPB_SERVER_PORT=5000

Ensure that your Jupyter notebooks are placed in the notebooks_dir directory. The names of these notebooks should match the names defined in your pipeline stages. This ensures that each stage can correctly locate and execute the corresponding notebook.

For example, if you have a stage defined as ('sample_stage_1', 'stage1'), there should be a notebook named sample_stage_1.ipynb in your notebooks_dir directory.

Define your pipeline stages. Each stage is a tuple containing the notebook name and the URL path.

    from nbpipeline.run import NBPipeliner

    PIPELINE_STAGES = [  
        ('sample_stage_1', 'stage1'),
        ('sample_stage_2', 'stage2'),    
    ]
    pipeline = NBPipeliner(__PIPELINE_STAGES, notebooks_dir)
    pipeline.start()

Upon running, the application will:

• Execute the defined Jupyter notebooks.
• Convert them to HTML reports.
• Serve the reports via a Flask web application accessible at http://<host>:<port>/.

Open your web browser and navigate to http://localhost:8088/ (replace 8088 with your configured port). You'll see a navigation page listing all the reports. Click on any link to view the corresponding HTML report.

The pipeline is scheduled to execute notebooks at intervals defined by NBP_DEFAULT_SCHEDULE_INTERVAL_MINUTES. By default, it's set to run every 10 minutes. You can adjust this interval by setting the environment variable accordingly.

Example:

To schedule the pipeline to run every 30 minutes:

export NBP_DEFAULT_SCHEDULE_INTERVAL_MINUTES=30

• data/reports/: Generated HTML reports will be saved here.

Contributions are welcome! Please follow these steps:

1. Fork the Repository

2. Create a Feature Branch

   git checkout -b feature/YourFeature

3. Commit Your Changes

   git commit -m "Add your feature"

4. Push to the Branch

   git push origin feature/YourFeature

5. Open a Pull Request

This project is licensed under the MIT License.