• 🚀 Full RL training pipeline runnable on your laptop (via Tinker API)
• 📈 AIME 2024: 43.3% → 56.7% (+13.3%)
• 🛡️ Novel redundancy penalty to prevent reward hacking
• 💰 Total cost: < $150 (including failed experiments)

TL;DR: A minimal implementation of JustRL-style reasoning model training using the Tinker platform (run it on your Macbook!).

JustRL: Simplicity is all you need. No KL penalty, no entropy regularization, no length penalty—just RL.

This repository demonstrates a two-stage training pipeline to transform a standard instruction-tuned model into a reasoning model with explicit thinking capabilities:

Based on the paper "JustRL: Simplicity at Scale", we adopt a minimalist approach:

• No KL penalty (kl_coef=0)
• No length penalty (found harmful in experiments)
• Clip-higher (clip_ratio=1.28)

Our implementation follows the evolution from standard GRPO to the simplified JustRL approach, with our own targeted improvements:

Our Additional Contributions (JustTinker):

The original JustRL paper uses DeepSeek-R1-Distill-Qwen-1.5B, which was distilled from DeepSeek-R1 (671B) and natively outputs <think>...</think> format. This model already "thinks out loud" by default.

In contrast, Qwen3-4B-Instruct-2507 is an internalized reasoning model:

Qwen3-4B-Instruct-2507 was trained to produce concise, direct answers without exposing the step-by-step reasoning process. The reasoning capability is "compressed" into the model weights through distillation, optimizing for fewer output tokens.

Why use it anyway? It's the most suitable small model available on Tinker platform. To align with the JustRL setup, we perform a cold-start SFT to teach the model the <think>...</think> format before RL training.

Since Qwen3-4B-Instruct-2507 doesn't output thinking tokens by default, we need to "awaken" this capability:

Before SFT: "The answer is 42."
After SFT:  "<think>Let me analyze this step by step...</think>\n\nThe answer is \boxed{42}."

This is a very short training phase (~800 steps, <$30) that teaches format compliance, not reasoning ability. The model already has strong reasoning capabilities—we're just teaching it to show its work.

The trained model is made publicly available on Tinker:

tinker://b0af3bd0-9638-583f-8c2c-2bb348453023:train:0/weights/coldstart_sft_final

You can load this checkpoint directly for Stage 2 (JustRL) training:

python scripts/tinker/justrl_math_reasoning.py \
    --checkpoint {coldstart_sft_final} \
    --reasoning \
    --scale medium

The cold-start SFT successfully taught the model to produce structured thinking:

Input: How many terms will there be if we expand (4x³ + x⁻³ + 2)²⁰¹⁶?

Output:
<think>
Okay, let's see. I need to figure out how many terms there will be when we
expand (4x³ + x⁻³ + 2)²⁰¹⁶ and combine like terms. Hmm, expanding such a
high exponent might be complicated, but maybe there's a pattern or formula
I can use instead of multiplying everything out.

First, let me recall...
</think>

The number of distinct terms is \boxed{12097}.

We conducted three experiments to explore JustRL-style training:

Experiment ID: failed_exp_001_training_collapse_20260111 Status: Failed — Documented for learning purposes

During our first JustRL training run, we observed a classic reward hacking phenomenon where the model exploited the reward function in unintended ways.

# This configuration led to training collapse
algorithm:
  clip_ratio_low: 0.8
  clip_ratio_high: 1.28
  kl_coef: 0.0              # ❌ No KL penalty - allowed unconstrained drift

training:
  learning_rate: 1e-6
  batch_size: 32
  rollout_n: 8
  max_response_length: 8192

eval:
  eval_interval: 20         # ❌ Too infrequent to catch collapse early
  eval_samples: 200

# - No format reward weight

The plot shows clear signs of collapse after step ~120: accuracy drops sharply while response length explodes.

The model discovered that generating longer responses occasionally led to correct answers by chance. Without constraints, this behavior was reinforced:

Feedback Loop:
┌─────────────────────────────────────────────────────────────┐
│  Occasionally long response → correct answer → reward       │
│         ↓                                                   │
│  Policy reinforces "generate longer"                        │
│         ↓                                                   │
│  Quality drops → fewer correct samples                      │
│         ↓                                                   │
│  Remaining correct samples are mostly long → more bias      │
│         ↓                                                   │
│  Collapse: 35,000+ char responses, no reasoning, ~10% acc   │
└─────────────────────────────────────────────────────────────┘

We extracted typical reward hacking samples from Step 140 evaluation:

Common patterns:

• Responses hit max_length (8192 tokens) and get truncated
• No <think>...</think> structure
• Repetitive text loops (e.g., "Therefore, the three sides..." repeated 100+ times)
• Self-aware "Wait" statements showing the model recognizes issues but can't stop

Full analysis: docs/research/reward_hacking_mechanism.md

Key insight: JustRL's "simplicity" works for models already trained to reason (DeepSeek-R1-Distill), but may need guardrails for models learning to reason from scratch.

Rather than abandoning JustRL's simplicity, we developed targeted interventions:

1. Format reward: format_reward_weight=0.1 to incentivize <think> usage
2. Redundancy penalty ⭐: Penalize repetitive content (our original contribution)
   • Uses compression ratio + N-gram analysis
   • Only activates when redundancy > 30%
   • Max penalty: 0.3 on correct answers
3. Early stopping: Stop if eval accuracy drops 5%+ for 3 consecutive evals
4. Health monitoring: Warn if response length > 5000, thinking rate < 60%, or redundancy > 40%

What we deliberately avoided (following JustRL):

• ❌ KL penalty
• ❌ Length penalty (found harmful in JustRL experiments)

Experiment ID: exp_002_with_redundancy_penalty Status: Not fully trained — Step 120/800

After implementing the mitigations above, we resumed training from Step 81 with the new configuration.

Following JustRL's minimalist philosophy, we avoid KL penalty and length penalty. Instead, we introduce an original redundancy penalty to combat reward hacking.

algorithm:
  name: grpo
  rollout_n: 8
  clip_ratio_low: 0.8
  clip_ratio_high: 1.28    # clip-higher (JustRL style)
  kl_coef: 0.0             # ❌ No KL penalty (JustRL style)
  entropy_coef: 0.0        # ❌ No entropy regularization

reward:
  type: binary
  correct_reward: 1.0
  incorrect_reward: 0.0
  length_penalty: false           # ❌ No length penalty (harmful per JustRL)
  format_reward_weight: 0.1       # ✅ Encourage <think> token usage
  redundancy_weight: 0.3          # ✅ Original: penalize repetitive content
  redundancy_threshold: 0.3       # ✅ Only penalize when redundancy > 30%

training:
  learning_rate: 1e-6
  max_response_length: 8192
  eval_interval: 10

Key Observations:

• No collapse at Step 120 (unlike Exp 001 which collapsed at Step 120-140)
• Redundancy penalty keeping repetitive content in check (~38%, threshold 30%)
• Eval accuracy improved from 83% to 85.5% peak
• Response length stable, not exploding

Experiment ID: exp_003_justrl_aligned Status: In Progress — Step 60/800

This experiment aligns more closely with the JustRL paper's training setup, using the DAPO-Math-17k dataset and adding AIME 2024 as an additional benchmark.

Key Observations:

• MATH accuracy remains stable around 88-91% (no degradation)
• AIME accuracy improved significantly: 43.33% → 56.67% peak (+13.3%)
• Thinking rate stable on both benchmarks
• No reward hacking observed (redundancy score ~39%, within limits)

Same reward/algorithm setup as Experiment 002, with key changes:

tinker://fbadbbce-0cfc-53dd-ad26-9117748c5070:train:0/weights/checkpoint_step_50

RLVR/
├── README.md                    # This file
├── scripts/
│   ├── launchers/               # Shell scripts (run_coldstart_sft.sh, run_justrl_reasoning.sh)
│   ├── tinker/                  # Tinker API scripts (coldstart_sft.py, justrl_math_reasoning.py)
│   └── utils/                   # Utilities (plot_rlvr_training.py, plot_sft_training.py)
├── src/
│   ├── configs/                 # Training configurations (SFTConfig, RLConfig)
│   ├── data/                    # Dataset loading (MATH, GSM8K, DAPO-Math-17k, AIME)
│   ├── evaluation/              # Math verification (MathVerifier)
│   └── prompts/                 # Prompt formatting utilities
└── resources/                   # Training curves and artifacts
    ├── coldstartSFT/            # Stage 1 results
    ├── justRL_exp001/           # Exp 001 (collapsed)
    ├── justRL_exp002/           # Exp 002 (with redundancy penalty)
    └── justRL_exp003/           # Exp 003 (DAPO + AIME)

# Install dependencies
pip install -r requirements.txt

# Set API key
export TINKER_API_KEY=your_api_key

./scripts/launchers/run_coldstart_sft.sh small

# Exp 003:
./scripts/launchers/run_justrl_reasoning.sh medium --reasoning \
      --checkpoint tinker://b0af3bd0-9638-583f-8c2c-2bb348453023:train:0/weights/coldstart_sft_final

Credits: All experiments were conducted using $150 free credits gifted by Tinker.

JustRL Cost Breakdown:

• Exp 001 (collapsed): ~120 steps before reward hacking
• Exp 002 (with redundancy penalty): 40 steps (Step 81-120)
• Average cost: ~$0.45/step

Note: Costs are based on Tinker pricing for Qwen3-4B-Instruct-2507 ($0.22/M tokens). Actual costs are often lower than estimates due to early stopping and response length variance.

• JustRL: Simplicity at Scale — Core methodology
• DeepSeek-R1 Technical Report — GRPO algorithm
• DAPO: Decoupled Clip and Dynamic Sampling — Advanced techniques
• RL's Razor: On-Policy Implicit Regularization — Why KL penalty may be unnecessary

• Tinker Platform — Training infrastructure

MIT License

If you find this repository helpful, please cite:

@misc{ning2026justtinker,
  author       = {Ning, Guanghan},
  title        = {JustTinker: Minimal Reinforcement Learning with Verifiable Rewards},
  year         = {2026},
  publisher    = {GitHub},
  url          = {https://github.com/Guanghan/JustTinker},
  note         = {A minimal implementation of JustRL-style reasoning model training with redundancy penalty}
}