Research and systems substrate for artifact-first training and evaluation of LLM-powered code agents on GPU programming tasks.

The repository centers on one idea:

GPU-agent work gets much easier to reason about when traces, benchmarks, build artifacts, profile summaries, candidate transitions, and training-readiness decisions are all first-class objects instead of side effects hidden in ad hoc scripts.

This repo is the resulting cockpit:

• a local execution and evaluation surface for run, bench, eval, build, inspect, compare, replay, and bundle
• a task and benchmark substrate spanning internal Triton/CUDA-style workloads and curated public benchmark imports
• a transition-aware collection layer for trajectories, patch-bearing repair traces, reformulation episodes, and SFT packaging

• Artifact-first: every serious step emits inspectable bundle state rather than ad hoc console noise
• Transition-aware: episodes capture candidate lineage, patch hashes, patch kinds, and repair/reformulate transitions
• Governed packaging: run-level readiness and episode-level readiness are separated on purpose
• Task-rich: internal Triton/CUDA-style tasks coexist with curated public benchmark adapters
• Training-targeted: the current data and config surface is oriented toward bounded tool-use on a strong sub-40B model, not open-ended frontier RL

gpu_rl/
├── README.md                                # top-level project overview, usage guide, and architecture map
├── pyproject.toml                           # package metadata and the `gpc` console entrypoint
├── .gitignore                               # runtime, dataset, and planning-artifact exclusions
├── configs/
│   └── training/
│       ├── first_target_splits_v1.json      # frozen train/dev split definition for the first training target
│       ├── rollout_debug_repair_heldout_v1.json
│       ├── rollout_debug_repair_v1.json     # scripted rollout configs for local and held-out evaluation
│       └── sft_qwen32b_debug_repair_lora.json
│                                             # first checked-in SFT smoke config for the initial training target
├── docs/
├── gpu_cockpit/
│   ├── cli/
│   │   └── main.py                          # public `gpc` CLI surface
│   ├── contracts/
│   │   ├── compare.py
│   │   ├── environment.py
│   │   ├── evidence.py
│   │   ├── patch.py
│   │   ├── replay.py
│   │   ├── summary.py
│   │   ├── training.py
│   │   └── trajectory.py                    # schema-first contract layer for bundles, episodes, and training configs
│   ├── engine/
│   │   ├── benchmark.py
│   │   ├── environment.py
│   │   ├── evaluator.py
│   │   ├── evidence.py
│   │   ├── inspector.py
│   │   ├── knowledge.py
│   │   ├── patching.py
│   │   ├── replay.py
│   │   ├── rollout.py
│   │   ├── runner.py
│   │   ├── sft.py
│   │   ├── training.py
│   │   └── trajectory.py                    # execution, eval, inspection, retrieval, and data-packaging engines
│   ├── backends/
│   │   ├── amd/                             # ROCm trace/profile normalization and AMD mirrored-path logic
│   │   └── nvidia/                          # Nsight, sanitizer, disassembly, and profile normalization
│   ├── executors/                           # local host and docker execution backends
│   ├── workloads/                           # adapter registration and workload-facing package helpers
│   └── artifacts/
│       └── schemas/
│                                             # exported JSON schemas for the contract layer
├── knowledge/
│   ├── README.md
│   ├── benchmark_notes/
│   ├── hardware_notes/
│   ├── operator_families/
│   ├── profiler_playbooks/
│   └── transformation_cards/                # human-written retrieval corpus for operators, bottlenecks, and transforms
├── scripts/
│   ├── build_heldout_baseline_report.py
│   ├── build_first_target_training_assets.py
│   ├── export_schemas.py
│   ├── generate_transition_goldens.py
│   ├── smoke_rollout_eval.py
│   └── smoke_sft_train.py                   # reproducible builders and smoke paths for schemas, datasets, and training assets
├── tests/
│   ├── golden_datasets/
│   ├── golden_episodes/
│   ├── golden_retrieval/
│   ├── golden_runs/
│   ├── test_environment.py
│   ├── test_evaluator.py
│   ├── test_inspector.py
│   ├── test_knowledge.py
│   ├── test_sft.py
│   ├── test_training.py
│   └── test_trajectory.py                  # regression suite plus checked-in golden bundles and training-facing fixtures
└── workloads/
    ├── baselines/
    ├── benchmarks/
    ├── fixtures/
    ├── public_benchmarks/
    ├── reference/
    ├── tasks/
    └── tests/                              # task specs, baselines, curated imports, reference kernels, and hook scripts

These documents freeze the semantics and boundaries that matter for the first training wave:

For schema work, bundle inspection, knowledge indexing, and non-GPU smoke paths:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .

gpc doctor

gpc task list
gpc adapter list
gpc adapter summary kernelbench
gpc adapter summary computeeval

gpc eval \
  --task task/reduction_debug/eval/v1 \
  --determinism-runs 2 \
  -- python3 workloads/reference/triton_row_sum_patchable_candidate.py --benchmark-repeats 5

gpc inspect runs/<run_id> --section quality
gpc compare runs/<run_a> runs/<run_b>
gpc replay runs/<run_id>
gpc bundle runs/<run_id> --full

python3 scripts/build_first_target_training_assets.py
python3 scripts/build_heldout_baseline_report.py
python3 scripts/smoke_sft_train.py
python3 scripts/smoke_rollout_eval.py

gpc build \
  --task task/attention_score/eval/v1 \
  --triton-build-spec workloads/reference/triton_attention_score_kernel.py:get_build_spec \
  --source-file workloads/reference/triton_attention_score_kernel.py

Useful when the immediate goal is inspecting:

• generated PTX
• SASS or disassembly output
• Triton IR stages
• source-to-PTX/SASS mapping summaries

gpc run \
  --task task/attention_score/eval/v1 \
  --trace-system \
  --profile-kernel \
  --profile-pack quick \
  --sanitize \
  --sanitize-tool memcheck \
  --emit-disassembly \
  --triton-build-spec workloads/reference/triton_attention_score_kernel.py:get_build_spec \
  -- python3 workloads/reference/triton_attention_score_optimize_candidate.py --benchmark-repeats 5

gpc env scripted \
  --task task/reduction_debug/eval/v1 \
  --out /tmp/reduction_debug_episode.json \
  --step-budget 12 \
  --workflow debug \
  --with-build \
  --triton-build-spec workloads/reference/triton_row_sum_kernel.py:get_build_spec \
  -- python3 workloads/reference/triton_row_sum_patchable_candidate.py --benchmark-repeats 5

gpc sft package \
  datasets/first_target_transition_train_v1 \
  --out-dir datasets/first_target_sft_train_v1 \
  --split train \
  --patch-bearing-only \
  --governance usable_positive_sft \
  --governance usable_negative_debug \
  --governance usable_negative_transition \
  --transition-kind repaired \
  --transition-kind reformulated \
  --transition-kind patch_applied \
  --verb debug \
  --verb reformulate

gpc knowledge build-index

gpc knowledge query \
  --query "mask bug failed repair register pressure" \
  --verb debug \
  --prefer-mixed \
  --limit 8

gpc inspect runs/<run_id> --section build
gpc inspect runs/<run_id> --section profile
gpc inspect runs/<run_id> --section replay
gpc inspect runs/<run_id> --section quality

gpc bench --adapter kernelbench --case case/kernelbench/level1/40_layernorm/v0_1
gpc bench --adapter computeeval --case case/computeeval/2025_1/cuda_16/v1

gpc train validate-config configs/training/sft_qwen32b_debug_repair_lora.json
gpc rollout scripted configs/training/rollout_debug_repair_heldout_v1.json --out-dir /tmp/heldout_rollout

• Internal tasks carry richer semantics for repair, reformulation, hidden failures, patch transitions, and training governance.
• Public adapters provide external provenance, broader operator coverage, and a reality check against overfitting to bespoke tasks.
• The packaging defaults deliberately prefer transition-rich internal traces over thin public benchmark wrappers unless explicitly configured otherwise.

Every serious action in the cockpit writes a run directory that can be inspected, replayed, compared, or packaged.

runs/<run_id>/
├── manifest.json
├── events.jsonl
├── summary.json
├── summary.md
├── prompt/
│   └── task_spec.json
├── meta/
│   ├── doctor_report.json
│   ├── hardware_fingerprint.json
│   └── task_spec_full.json
├── command/
│   ├── stdout.txt
│   ├── stderr.txt
│   └── summary.json
├── correctness/
│   ├── correctness.json
│   ├── determinism.json
│   └── *_summary.json
├── eval/
│   ├── anti_hack_report.json
│   ├── eval_envelope.json
│   └── gate_summary.json
├── perf/
│   ├── raw_timings.json
│   └── benchmark.json
├── build/
│   ├── build_record.json
│   ├── source_map_summary.json
│   ├── tri_view.json
│   └── source_ptx_sass_map.json
├── patches/
│   ├── request.json
│   ├── unified_diff.patch
│   └── applied_patch.json
├── candidate/
│   ├── state.json
│   └── transition.json
└── replay/
    ├── command.json
    ├── environment.json
    └── replay_pack.json

The local knowledge base is intentionally small and structured rather than sprawling.

Retrieval is designed to return a mix of:

• docs
• prior run examples
• patch-bearing repair traces
• reformulation examples
• similar tasks

That bias is deliberate: useful training and debugging context usually mixes prose with concrete examples.

A large fraction of GPU-agent signal is not “one prompt, one answer.” The important learning unit is often:

1. inspect a candidate
2. identify a failure mode or weak strategy
3. patch or transform the candidate
4. rebuild, benchmark, compare, and re-evaluate
5. decide whether the resulting trace is usable for training

The repository therefore treats the following as first-class:

• patch_candidate transitions
• candidate lineage and parent-child state
• episode-level governance separate from run-level readiness
• usable negative traces for failed repair and failed reformulation

pip install -e .
python3 scripts/export_schemas.py
python3 -m unittest discover -s tests -v

# Rebuild checked-in transition fixtures
python3 scripts/generate_transition_goldens.py

# Rebuild train/dev assets for the first training target
python3 scripts/build_first_target_training_assets.py

# Validate training config and rollout config
gpc train validate-config configs/training/sft_qwen32b_debug_repair_lora.json
gpc rollout scripted configs/training/rollout_debug_repair_heldout_v1.json --out-dir /tmp/heldout_rollout

1. Start with gpu_cockpit/cli/main.py to see the public command surface.
2. Read gpu_cockpit/engine/runner.py, gpu_cockpit/engine/evaluator.py, and gpu_cockpit/engine/inspector.py for the core run lifecycle.
3. Read gpu_cockpit/engine/environment.py, gpu_cockpit/engine/trajectory.py, and gpu_cockpit/engine/sft.py for the training-facing data model.
4. Read the training configs in configs/training/ before changing training assumptions.

• source code
• tests
• schemas
• benchmark metadata
• golden runs / episodes / datasets used for regression coverage
• training configs

• runs/
• datasets/
• docs_tmp/
• artifacts/
• generated knowledge/index/

This split is deliberate:

• checked-in goldens define the stable verification surface
• generated runtime outputs remain local and disposable

If you are here for the shortest orientation path:

• inspect the checked-in training configs
• run gpc doctor
• inspect gpc --help
• execute one gpc eval task
• inspect a run bundle
• build the training assets

If you are here to extend the project:

• prefer richer internal repair and reformulate traces over shallow benchmark breadth
• preserve candidate lineage and governance semantics
• treat patch-bearing episodes as the highest-signal assets in the current system

If you are here to start training:

• finish validation on the checked-in smoke path
• use the checked-in training configs and smoke scripts
• validate on the intended training environment before launching larger jobs
• treat the smoke sequence as the gate to more expensive runs