Overview

Relevant source files

• go.mod
• go.sum
• pkg/backend/backend.go
• test/mocks/backend/backend.go

Purpose and Scope

modctl is a command-line tool for managing machine learning model artifacts as OCI-compliant packages. It enables packaging, distributing, and retrieving models with their associated code, documentation, and configuration using container registry infrastructure. This page provides a high-level introduction to the tool's architecture, key components, and design principles.

For detailed information on specific subsystems, see:

• Architecture details: Architecture Overview
• External integrations: Dependencies and External Systems
• Installation procedures: Installation
• Command reference: CLI Reference

Sources: go.mod1-30 pkg/backend/backend.go1-87

What is modctl?

modctl is a model artifact management tool that treats ML models as first-class distributable artifacts. It packages model weights, training code, documentation, and metadata into standardized OCI-compliant images that can be stored in container registries and distributed using P2P networks. The tool follows the ModelPack specification to ensure interoperability across the ML ecosystem.

The tool provides a Docker-like experience for ML practitioners:

Operation	Description	Backend Method
modctl build	Package model files into an artifact	Build()
modctl push	Upload artifact to a registry	Push()
modctl pull	Download artifact from a registry	Pull()
modctl extract	Unpack artifact contents locally	Extract()
modctl inspect	View artifact metadata and layers	Inspect()
modctl ls	List locally stored artifacts	List()

Sources: pkg/backend/backend.go27-69 test/mocks/backend/backend.go30-761

Key Features

OCI-Compliant Artifact Format

modctl uses the OCI image specification for model artifacts, enabling:

• Storage in any OCI-compliant registry (Docker Hub, GitHub Container Registry, Harbor)
• Content-addressed storage using SHA256/SHA512 digests
• Layer-based composition with deduplication
• Standard manifest and descriptor formats

Sources: go.mod18-19

Modelfile-Driven Builds

Artifacts are defined using a Modelfile that specifies:

• NAME, ARCH, FAMILY: Model metadata
• CONFIG: Configuration files
• MODEL: Model weights and checkpoints
• CODE: Training/inference code
• DOC: Documentation and notebooks

The modctl modelfile generate command can automatically create a Modelfile by scanning a directory.

Sources: Referenced in diagrams and table of contents

High-Performance Distribution

• Parallel Transfer: Concurrent layer downloads with configurable limits using errgroup
• Dragonfly P2P: Optional peer-to-peer distribution for large models via gRPC
• Retry Logic: Exponential backoff with retry-go/v4
• Progress Tracking: Real-time feedback using mpb progress bars
• Hardware Acceleration: SIMD-optimized hashing with sha256-simd

Sources: go.mod8 go.mod16 go.mod24

Flexible Build Targets

modctl build [--output-local] [--output-remote ghcr.io/org/model:v1]

• Build directly to local storage
• Build and push to remote registry in one operation
• Attach additional files to existing artifacts

Sources: pkg/backend/backend.go40-41

System Architecture

High-Level Component Model

Figure 1: Core Architecture Mapping to Code Entities

The architecture follows a layered design with clear separation of concerns:

1. CLI Layer (cmd/): Cobra-based command structure with Viper configuration management
2. Backend Layer (pkg/backend/): Unified interface abstracting all 14 operations
3. Storage Layer (pkg/storage/): OCI distribution library wrapper for manifest and blob operations
4. Supporting Systems: Progress tracking, archiving, Dragonfly integration, Modelfile parsing

Sources: pkg/backend/backend.go27-69 pkg/backend/backend.go72-86

Backend Interface Operations

Figure 2: Backend Interface Method Categories

The Backend interface defines 14 operations grouped by functionality:

Category	Methods	Purpose
Authentication	Login(), Logout()	Registry credential management
Creation	Build(), Attach(), Upload()	Artifact construction and augmentation
Distribution	Pull(), Push(), Fetch()	Registry transfer operations
Introspection	List(), Inspect()	Query local artifacts
Maintenance	Remove(), Prune(), Tag(), Extract()	Storage management

Sources: pkg/backend/backend.go27-69 test/mocks/backend/backend.go43-747

Component Responsibilities

Backend (pkg/backend/)

The backend struct pkg/backend/backend.go72-74 implements the Backend interface and serves as the orchestration layer. It holds a storage.Storage instance and delegates storage operations while coordinating higher-level workflows like building artifacts, managing transfers, and handling progress reporting.

The New() function pkg/backend/backend.go77-86 instantiates the backend with a storage directory path.

Sources: pkg/backend/backend.go72-86

Storage Abstraction

The storage layer abstracts OCI registry operations through the storage.Storage interface, which wraps the distribution/distribution/v3 library. This abstraction enables:

• Uniform API for local filesystem and remote registry access
• Manifest CRUD operations
• Blob CRUD operations with digest validation
• Repository enumeration
• Garbage collection

Sources: go.mod9 pkg/backend/backend.go23

CLI Integration

The CLI layer uses the Cobra framework go.mod21 for command parsing and Viper go.mod22 for configuration management. Commands create a Backend instance and invoke the appropriate interface method with operation-specific configuration structs from pkg/config/.

Sources: go.mod21-22

Technology Stack

Core Dependencies

Library	Purpose	Version
oras.land/oras-go/v2	OCI artifact transfer	v2.6.0
distribution/distribution/v3	OCI registry storage	v3.0.0
opencontainers/image-spec	OCI specification types	v1.1.1
modelpack/model-spec	ModelPack specification	v0.0.7
d7y.io/api/v2	Dragonfly P2P API	v2.1.94

Sources: go.mod5-30

Supporting Libraries

Library	Purpose	Version
spf13/cobra	CLI framework	v1.10.2
spf13/viper	Configuration management	v1.21.0
vbauerster/mpb/v8	Progress bars	v8.11.2
avast/retry-go/v4	Retry logic with backoff	v4.7.0
minio/sha256-simd	Hardware-accelerated hashing	v1.0.1
klauspost/compress	Fast compression	v1.18.0

Sources: go.mod5-30

Git Integration

Two Git libraries provide repository metadata extraction:

• go-git/go-git/v5 (v5.16.4): Pure Go implementation
• libgit2/git2go/v34 (v34.0.0): CGo bindings to libgit2

Sources: go.mod14-15

Standards Compliance

OCI Specifications

modctl adheres to the OCI image specification for:

• Descriptors: Content-addressed references with media types and digests
• Manifests: Lists of layers with platform metadata
• Image Layout: Filesystem structure for local storage
• Distribution Spec: Registry API for push/pull operations

Sources: go.mod18-19

ModelPack Specification

Model artifacts follow the ModelPack spec (github.com/modelpack/model-spec v0.0.7), which extends OCI with:

• Model-specific layer types (weights, code, config, documentation)
• Metadata schema for model family, architecture, and framework
• Conventions for layer ordering and annotation

Sources: go.mod17

Operational Model

Typical Workflow

Figure 3: End-to-End Artifact Lifecycle

1. Generate: Scan directory and create Modelfile
2. Build: Package files into OCI artifact
3. Push: Upload to registry (can be combined with build via --output-remote)
4. Pull: Download artifact (optionally via Dragonfly P2P)
5. Extract: Unpack to filesystem (supports glob patterns for selective extraction)

Sources: pkg/backend/backend.go40-50

Storage Locations

• Local Storage: ~/.modctl/ (configurable via --storage-dir)
• Remote Registries: Any OCI-compliant registry
• Dragonfly Cache: P2P distributed cache layer

The New() function pkg/backend/backend.go77-86 accepts a storageDir parameter to override the default location.

Sources: pkg/backend/backend.go77-86

Design Principles

Interface-Driven Architecture

All major subsystems are defined as interfaces (Backend, Storage, Builder, OutputStrategy) with concrete implementations. This enables:

• Testability via mocks (generated by mockery)
• Multiple implementations (local/remote output strategies)
• Dependency inversion

The mock in test/mocks/backend/backend.go30-761 demonstrates the full interface contract used for testing.

Sources: pkg/backend/backend.go26-69 test/mocks/backend/backend.go30-761

Content-Addressed Storage

All artifacts are identified by cryptographic digests (SHA256). This ensures:

• Immutability: Content cannot change without changing the digest
• Deduplication: Identical layers are stored once
• Integrity: Corruption is detected via digest validation

Sources: go.mod18

Concurrent Operations with Bounded Parallelism

Transfer operations use errgroup.WithContext with configurable concurrency limits to parallelize layer operations while preventing resource exhaustion. Individual layer operations include retry logic via avast/retry-go/v4.

Sources: go.mod8 go.mod26

Summary

modctl provides a comprehensive toolkit for ML model lifecycle management by:

• Packaging models as OCI artifacts using the ModelPack specification
• Leveraging existing container registry infrastructure
• Supporting high-performance distribution via P2P networks
• Offering a familiar CLI experience for ML practitioners
• Maintaining standards compliance for ecosystem interoperability

The tool's architecture emphasizes modularity, testability, and extensibility through interface-driven design and clear separation of concerns across its layered subsystem structure.

Sources: go.mod1-30 pkg/backend/backend.go1-87 test/mocks/backend/backend.go1-761