feat: Terminal-Bench eval harness (MVP Phase 1) #178
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.0% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.1% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
jeremyeder
added a commit
that referenced
this pull request
Dec 7, 2025
Fixed 2 root causes affecting 14 total tests: 1. PatternExtractor attribute access (10 tests fixed): - Changed finding.attribute.attribute_id → finding.attribute.id - Fixed extract_specific_patterns() method - Added create_dummy_finding() helper for Assessment validation - Fixed 8 pattern extractor tests + 4 downstream test failures 2. Anthropic API error mocks (2 tests fixed): - Updated RateLimitError mock with response and body kwargs - Updated APIError mock with request and body kwargs - Adapted to evolved Anthropic SDK error class signatures Test status: 34 failed → 20 failed (14 tests fixed) Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.1% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
jeremyeder
added a commit
that referenced
this pull request
Dec 7, 2025
Changed assertion from "90%" to "90.0%" to match actual output format. The SkillGenerator formats confidence as "90.0%" not "90%". Test status: 20 failed → 19 failed Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.1% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
jeremyeder
added a commit
that referenced
this pull request
Dec 7, 2025
Updated temp_repo fixture to use attribute IDs that PatternExtractor recognizes (claude_md_file, type_annotations) instead of generic test attributes. This allows PatternExtractor to extract skills from the test assessment. Identified root cause for remaining 6 CLI failures: output directory is created relative to cwd, not repo_path. Fix planned for next session. Test status: Still 19 failed (investigation in progress) Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 33.1% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 10 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
jeremyeder
added a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 8, 2025
Fixed 2 root causes affecting 14 total tests: 1. PatternExtractor attribute access (10 tests fixed): - Changed finding.attribute.attribute_id → finding.attribute.id - Fixed extract_specific_patterns() method - Added create_dummy_finding() helper for Assessment validation - Fixed 8 pattern extractor tests + 4 downstream test failures 2. Anthropic API error mocks (2 tests fixed): - Updated RateLimitError mock with response and body kwargs - Updated APIError mock with request and body kwargs - Adapted to evolved Anthropic SDK error class signatures Test status: 34 failed → 20 failed (14 tests fixed) Related: ambient-code#178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jeremyeder
added a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 8, 2025
Changed assertion from "90%" to "90.0%" to match actual output format. The SkillGenerator formats confidence as "90.0%" not "90%". Test status: 20 failed → 19 failed Related: ambient-code#178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jeremyeder
added a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 8, 2025
* fix: resolve quick win test failures (CSV, config, research formatter)
Fixed 5 test failures across 3 categories:
**CSV Reporter Tests (4 errors → 0):**
- Added create_dummy_findings() helper to generate Finding objects
- Updated mock assessments to include required findings matching attributes_total
- Fixed test_csv_empty_batch to expect ValueError during BatchAssessment construction
**Config Model Test (1 failure → 0):**
- Updated test_config_invalid_weights_negative to test for negative weights (current validation)
- Removed outdated test_config_invalid_weights_sum (sum-to-1.0 validation was intentionally removed)
**Research Formatter Tests (2 failures → 0):**
- Fixed format_report() to ensure exactly one trailing newline
- Updated extract_attribute_ids() regex to capture malformed IDs for validation
Test status: 48→43 failures, 737→746 passed
* fix: resolve learning service test failures with proper mocks and validation
Fixed all 9 learning service test failures by addressing three issues:
1. Mock method mismatches (7 tests):
- Tests were mocking `extract_from_findings()` but code calls
`extract_all_patterns()` or `extract_specific_patterns()`
- Updated all mocks to use correct method names based on whether
`attribute_ids` parameter is passed
2. LLMEnricher import path (1 test):
- Test tried to patch `learning_service.LLMEnricher` but it's imported
inside `_enrich_with_llm()` method from `learners.llm_enricher`
- Changed patch path to actual import location
3. Repository validation (4 tests):
- Repository model requires `.git` directory
- Updated `temp_dir` fixture to run `git init`
- Updated tests to create assessment files in `.agentready/` subdirectory
(code expects assessments at `.agentready/assessment-*.json`)
4. Assessment validation (3 tests):
- Assessment requires `len(findings) == attributes_total`
- Added `create_dummy_finding()` helper
- Updated tests to include proper number of findings
All 17 learning service tests now pass.
Test progress: 48 failed → 34 failed (14 tests fixed)
* fix: resolve pattern extractor and LLM enricher test failures (14 tests)
Fixed 2 root causes affecting 14 total tests:
1. PatternExtractor attribute access (10 tests fixed):
- Changed finding.attribute.attribute_id → finding.attribute.id
- Fixed extract_specific_patterns() method
- Added create_dummy_finding() helper for Assessment validation
- Fixed 8 pattern extractor tests + 4 downstream test failures
2. Anthropic API error mocks (2 tests fixed):
- Updated RateLimitError mock with response and body kwargs
- Updated APIError mock with request and body kwargs
- Adapted to evolved Anthropic SDK error class signatures
Test status: 34 failed → 20 failed (14 tests fixed)
Related: ambient-code#178
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: correct confidence format assertion in skill generator test
Changed assertion from "90%" to "90.0%" to match actual output format.
The SkillGenerator formats confidence as "90.0%" not "90%".
Test status: 20 failed → 19 failed
Related: ambient-code#178
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: resolve CLI command test failures with path resolution and validation (12 tests)
Fixes 12 failing tests in CLI commands (extract-skills and learn):
CLI Command Fixes (Both Commands):
- Resolve output_dir relative to repo_path instead of cwd
- Fixes isolated_filesystem() test context issues
- Ensures output created in repository, not temp directory
- Add IntRange(min=1) validation for llm_budget parameter
- Prevents negative budget values
- Provides clear Click validation error
Test Assertion Fixes:
- Fix skill_md format tests: glob("*/SKILL.md") not glob("*.md")
- SKILL.md files are created in subdirectories (skill-id/SKILL.md)
- Fix github_issues format tests: glob("skill-*.md") not glob("issue-*.md")
- Issue files are named skill-{id}.md, not issue-*.md
- Add known skill IDs to test fixtures (claude_md_file, type_annotations)
- PatternExtractor requires recognizable attribute IDs to extract skills
Test Progress: 19 failed → 7 failed (12 tests fixed, 63% complete)
Files Modified:
- src/agentready/cli/extract_skills.py (path resolution, validation)
- src/agentready/cli/learn.py (path resolution, validation)
- tests/unit/test_cli_extract_skills.py (glob patterns)
- tests/unit/test_cli_learn.py (glob patterns, fixture data)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: resolve isolated test failures in code_sampler and fixer_service (2 tests)
Fixes 2 isolated test failures:
Code Sampler Fix (code_sampler.py):
- Add 'path' key check before accessing dict in _format_code_samples()
- Empty dicts in files list were causing KeyError
- Changed: if isinstance(file_item, dict) and "path" in file_item
Fixer Service Test Fix (test_fixer_service.py):
- Add passing finding to test_generate_fix_plan_no_failing_findings
- Assessment validation requires len(findings) == attributes_total
- Test was creating assessment with 0 findings but attributes_total=1
- Now creates a passing finding to satisfy validation
Test Progress: 19 failed → 5 failed (14 tests fixed, 74% complete)
Remaining: 5 GitHub scanner tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: resolve GitHub scanner test failures with proper pagination mocking (5 tests)
Fixes 5 GitHub scanner test failures by correctly mocking API pagination:
Root Cause:
- Scanner's pagination loop breaks when response.json() returns empty list
- Original mocks used return_value which returns same repos on every call
- Loop continued until hitting max_repos limit (100), returning duplicates
Fix Applied (All 5 Tests):
- Changed from `mock_get.return_value = mock_response` to:
```python
mock_response_page1 = Mock() # Returns repos
mock_response_page1.json.return_value = [repo1, repo2]
mock_response_page2 = Mock() # Empty - signals end of pagination
mock_response_page2.json.return_value = []
mock_get.side_effect = [mock_response_page1, mock_response_page2]
```
Tests Fixed:
1. test_successful_org_scan - Basic org scanning
2. test_filters_private_repos - Private repo filtering
3. test_includes_private_repos_when_requested - Include private when flagged
4. test_filters_archived_repos - Archived repo filtering
5. test_rate_limit_warning - Rate limit warning logging
Test Progress: 19 failed → 0 failed (19 tests fixed, 100% complete ✅)
Final Status: 789 passed, 2 skipped, 0 failed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 8, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-08) ### Bug Fixes * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * resolve 45 test failures across CLI, services, and assessors ([#4](#4)) ([3405142](3405142)), closes [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
jeremyeder
force-pushed
the
feature/eval-harness-mvp
branch
from
79a7b5a to
1dd1539
Compare
Contributor
|
|
Contributor
🤖 AgentReady Assessment ReportRepository: agentready 📊 Summary
Languages Detected
Repository Stats
🎖️ Certification Ladder
📋 Detailed FindingsAPI Documentation
Build & Development
Code Organization
Code Quality
❌ Type AnnotationsMeasured: 32.8% (Threshold: ≥80%) Evidence:
📝 Remediation StepsAdd type annotations to function signatures
Commands: # Python
pip install mypy
mypy --strict src/
# TypeScript
npm install --save-dev typescript
echo '{"compilerOptions": {"strict": true}}' > tsconfig.jsonExamples: ❌ Structured LoggingMeasured: not configured (Threshold: structured logging library) Evidence:
📝 Remediation StepsAdd structured logging library for machine-parseable logs
Commands: # Install structlog
pip install structlog
# Configure structlog
# See examples for configurationExamples: Context Window Optimization
❌ File Size LimitsMeasured: 2 huge, 11 large out of 153 (Threshold: <5% files >500 lines, 0 files >1000 lines) Evidence:
📝 Remediation StepsRefactor large files into smaller, focused modules
Examples: Dependency Management
Documentation
❌ Concise DocumentationMeasured: 305 lines, 47 headings, 33 bullets (Threshold: <500 lines, structured format) Evidence:
📝 Remediation StepsMake documentation more concise and structured
Commands: # Check README length
wc -l README.md
# Count headings
grep -c '^#' README.mdExamples: Features
DocumentationSee docs/ for detailed guides. Bad: Verbose proseThis project is a tool that helps you assess your repository [Many more paragraphs of prose...] Examples: Performance
Repository Structure
Security
Testing & CI/CD
🎯 Next StepsPriority Improvements (highest impact first):
📝 Assessment Metadata
🤖 Generated with Claude Code |
github-actions bot
pushed a commit
that referenced
this pull request
Dec 9, 2025
# [2.15.0](v2.14.1...v2.15.0) (2025-12-09) ### Bug Fixes * resolve all test suite failures - achieve zero failures ([#180](#180)) ([990fa2d](990fa2d)), closes [#148](#148) [#147](#147) [#145](#145) * resolve YAML syntax error in update-docs workflow and add actionlint ([#173](#173)) ([97b06af](97b06af)) ### Features * replace markdown-link-check with lychee for link validation ([#177](#177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([#178](#178)) ([d06bab4](d06bab4)), closes [#171](#171)
Contributor
|
🎉 This PR is included in version 2.15.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
jeremyeder
added a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 9, 2025
* fix: resolve quick win test failures (CSV, config, research formatter)
Fixed 5 test failures across 3 categories:
**CSV Reporter Tests (4 errors → 0):**
- Added create_dummy_findings() helper to generate Finding objects
- Updated mock assessments to include required findings matching attributes_total
- Fixed test_csv_empty_batch to expect ValueError during BatchAssessment construction
**Config Model Test (1 failure → 0):**
- Updated test_config_invalid_weights_negative to test for negative weights (current validation)
- Removed outdated test_config_invalid_weights_sum (sum-to-1.0 validation was intentionally removed)
**Research Formatter Tests (2 failures → 0):**
- Fixed format_report() to ensure exactly one trailing newline
- Updated extract_attribute_ids() regex to capture malformed IDs for validation
Test status: 48→43 failures, 737→746 passed
* fix: resolve learning service test failures with proper mocks and validation
Fixed all 9 learning service test failures by addressing three issues:
1. Mock method mismatches (7 tests):
- Tests were mocking `extract_from_findings()` but code calls
`extract_all_patterns()` or `extract_specific_patterns()`
- Updated all mocks to use correct method names based on whether
`attribute_ids` parameter is passed
2. LLMEnricher import path (1 test):
- Test tried to patch `learning_service.LLMEnricher` but it's imported
inside `_enrich_with_llm()` method from `learners.llm_enricher`
- Changed patch path to actual import location
3. Repository validation (4 tests):
- Repository model requires `.git` directory
- Updated `temp_dir` fixture to run `git init`
- Updated tests to create assessment files in `.agentready/` subdirectory
(code expects assessments at `.agentready/assessment-*.json`)
4. Assessment validation (3 tests):
- Assessment requires `len(findings) == attributes_total`
- Added `create_dummy_finding()` helper
- Updated tests to include proper number of findings
All 17 learning service tests now pass.
Test progress: 48 failed → 34 failed (14 tests fixed)
* fix: resolve pattern extractor and LLM enricher test failures (14 tests)
Fixed 2 root causes affecting 14 total tests:
1. PatternExtractor attribute access (10 tests fixed):
- Changed finding.attribute.attribute_id → finding.attribute.id
- Fixed extract_specific_patterns() method
- Added create_dummy_finding() helper for Assessment validation
- Fixed 8 pattern extractor tests + 4 downstream test failures
2. Anthropic API error mocks (2 tests fixed):
- Updated RateLimitError mock with response and body kwargs
- Updated APIError mock with request and body kwargs
- Adapted to evolved Anthropic SDK error class signatures
Test status: 34 failed → 20 failed (14 tests fixed)
Related: ambient-code#178
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: correct confidence format assertion in skill generator test
Changed assertion from "90%" to "90.0%" to match actual output format.
The SkillGenerator formats confidence as "90.0%" not "90%".
Test status: 20 failed → 19 failed
Related: ambient-code#178
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: resolve CLI command test failures with path resolution and validation (12 tests)
Fixes 12 failing tests in CLI commands (extract-skills and learn):
CLI Command Fixes (Both Commands):
- Resolve output_dir relative to repo_path instead of cwd
- Fixes isolated_filesystem() test context issues
- Ensures output created in repository, not temp directory
- Add IntRange(min=1) validation for llm_budget parameter
- Prevents negative budget values
- Provides clear Click validation error
Test Assertion Fixes:
- Fix skill_md format tests: glob("*/SKILL.md") not glob("*.md")
- SKILL.md files are created in subdirectories (skill-id/SKILL.md)
- Fix github_issues format tests: glob("skill-*.md") not glob("issue-*.md")
- Issue files are named skill-{id}.md, not issue-*.md
- Add known skill IDs to test fixtures (claude_md_file, type_annotations)
- PatternExtractor requires recognizable attribute IDs to extract skills
Test Progress: 19 failed → 7 failed (12 tests fixed, 63% complete)
Files Modified:
- src/agentready/cli/extract_skills.py (path resolution, validation)
- src/agentready/cli/learn.py (path resolution, validation)
- tests/unit/test_cli_extract_skills.py (glob patterns)
- tests/unit/test_cli_learn.py (glob patterns, fixture data)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: resolve isolated test failures in code_sampler and fixer_service (2 tests)
Fixes 2 isolated test failures:
Code Sampler Fix (code_sampler.py):
- Add 'path' key check before accessing dict in _format_code_samples()
- Empty dicts in files list were causing KeyError
- Changed: if isinstance(file_item, dict) and "path" in file_item
Fixer Service Test Fix (test_fixer_service.py):
- Add passing finding to test_generate_fix_plan_no_failing_findings
- Assessment validation requires len(findings) == attributes_total
- Test was creating assessment with 0 findings but attributes_total=1
- Now creates a passing finding to satisfy validation
Test Progress: 19 failed → 5 failed (14 tests fixed, 74% complete)
Remaining: 5 GitHub scanner tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: resolve GitHub scanner test failures with proper pagination mocking (5 tests)
Fixes 5 GitHub scanner test failures by correctly mocking API pagination:
Root Cause:
- Scanner's pagination loop breaks when response.json() returns empty list
- Original mocks used return_value which returns same repos on every call
- Loop continued until hitting max_repos limit (100), returning duplicates
Fix Applied (All 5 Tests):
- Changed from `mock_get.return_value = mock_response` to:
```python
mock_response_page1 = Mock() # Returns repos
mock_response_page1.json.return_value = [repo1, repo2]
mock_response_page2 = Mock() # Empty - signals end of pagination
mock_response_page2.json.return_value = []
mock_get.side_effect = [mock_response_page1, mock_response_page2]
```
Tests Fixed:
1. test_successful_org_scan - Basic org scanning
2. test_filters_private_repos - Private repo filtering
3. test_includes_private_repos_when_requested - Include private when flagged
4. test_filters_archived_repos - Archived repo filtering
5. test_rate_limit_warning - Rate limit warning logging
Test Progress: 19 failed → 0 failed (19 tests fixed, 100% complete ✅)
Final Status: 789 passed, 2 skipped, 0 failed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
jeremyeder
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 9, 2025
* disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * resolve 45 test failures across CLI, services, and assessors ([#4](#4)) ([3405142](3405142)), closes [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 10, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-10) ### Bug Fixes * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * resolve 45 test failures across CLI, services, and assessors ([#4](#4)) ([3405142](3405142)), closes [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * convert AgentReady assessment to comment-triggered workflow ([#8](#8)) ([350f21b](350f21b)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 10, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-10) ### Bug Fixes * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * resolve 45 test failures across CLI, services, and assessors ([#4](#4)) ([3405142](3405142)), closes [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * convert AgentReady assessment to comment-triggered workflow ([#8](#8)) ([350f21b](350f21b)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * redesign assessment reports with badge-style compact format ([#10](#10)) ([35836d4](35836d4)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
This was referenced Dec 10, 2025
jeremyeder
added a commit
that referenced
this pull request
Dec 10, 2025
* chore: update leaderboard data [skip ci] Generated from submissions/ directory at 2025-12-05 17:38:42 UTC * fix: resolve 45 test failures across CLI, services, and assessors (#4) * fix: resolve quick win test failures (CSV, config, research formatter) Fixed 5 test failures across 3 categories: **CSV Reporter Tests (4 errors → 0):** - Added create_dummy_findings() helper to generate Finding objects - Updated mock assessments to include required findings matching attributes_total - Fixed test_csv_empty_batch to expect ValueError during BatchAssessment construction **Config Model Test (1 failure → 0):** - Updated test_config_invalid_weights_negative to test for negative weights (current validation) - Removed outdated test_config_invalid_weights_sum (sum-to-1.0 validation was intentionally removed) **Research Formatter Tests (2 failures → 0):** - Fixed format_report() to ensure exactly one trailing newline - Updated extract_attribute_ids() regex to capture malformed IDs for validation Test status: 48→43 failures, 737→746 passed * fix: resolve learning service test failures with proper mocks and validation Fixed all 9 learning service test failures by addressing three issues: 1. Mock method mismatches (7 tests): - Tests were mocking `extract_from_findings()` but code calls `extract_all_patterns()` or `extract_specific_patterns()` - Updated all mocks to use correct method names based on whether `attribute_ids` parameter is passed 2. LLMEnricher import path (1 test): - Test tried to patch `learning_service.LLMEnricher` but it's imported inside `_enrich_with_llm()` method from `learners.llm_enricher` - Changed patch path to actual import location 3. Repository validation (4 tests): - Repository model requires `.git` directory - Updated `temp_dir` fixture to run `git init` - Updated tests to create assessment files in `.agentready/` subdirectory (code expects assessments at `.agentready/assessment-*.json`) 4. Assessment validation (3 tests): - Assessment requires `len(findings) == attributes_total` - Added `create_dummy_finding()` helper - Updated tests to include proper number of findings All 17 learning service tests now pass. Test progress: 48 failed → 34 failed (14 tests fixed) * fix: resolve pattern extractor and LLM enricher test failures (14 tests) Fixed 2 root causes affecting 14 total tests: 1. PatternExtractor attribute access (10 tests fixed): - Changed finding.attribute.attribute_id → finding.attribute.id - Fixed extract_specific_patterns() method - Added create_dummy_finding() helper for Assessment validation - Fixed 8 pattern extractor tests + 4 downstream test failures 2. Anthropic API error mocks (2 tests fixed): - Updated RateLimitError mock with response and body kwargs - Updated APIError mock with request and body kwargs - Adapted to evolved Anthropic SDK error class signatures Test status: 34 failed → 20 failed (14 tests fixed) Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: correct confidence format assertion in skill generator test Changed assertion from "90%" to "90.0%" to match actual output format. The SkillGenerator formats confidence as "90.0%" not "90%". Test status: 20 failed → 19 failed Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: resolve CLI command test failures with path resolution and validation (12 tests) Fixes 12 failing tests in CLI commands (extract-skills and learn): CLI Command Fixes (Both Commands): - Resolve output_dir relative to repo_path instead of cwd - Fixes isolated_filesystem() test context issues - Ensures output created in repository, not temp directory - Add IntRange(min=1) validation for llm_budget parameter - Prevents negative budget values - Provides clear Click validation error Test Assertion Fixes: - Fix skill_md format tests: glob("*/SKILL.md") not glob("*.md") - SKILL.md files are created in subdirectories (skill-id/SKILL.md) - Fix github_issues format tests: glob("skill-*.md") not glob("issue-*.md") - Issue files are named skill-{id}.md, not issue-*.md - Add known skill IDs to test fixtures (claude_md_file, type_annotations) - PatternExtractor requires recognizable attribute IDs to extract skills Test Progress: 19 failed → 7 failed (12 tests fixed, 63% complete) Files Modified: - src/agentready/cli/extract_skills.py (path resolution, validation) - src/agentready/cli/learn.py (path resolution, validation) - tests/unit/test_cli_extract_skills.py (glob patterns) - tests/unit/test_cli_learn.py (glob patterns, fixture data) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: resolve isolated test failures in code_sampler and fixer_service (2 tests) Fixes 2 isolated test failures: Code Sampler Fix (code_sampler.py): - Add 'path' key check before accessing dict in _format_code_samples() - Empty dicts in files list were causing KeyError - Changed: if isinstance(file_item, dict) and "path" in file_item Fixer Service Test Fix (test_fixer_service.py): - Add passing finding to test_generate_fix_plan_no_failing_findings - Assessment validation requires len(findings) == attributes_total - Test was creating assessment with 0 findings but attributes_total=1 - Now creates a passing finding to satisfy validation Test Progress: 19 failed → 5 failed (14 tests fixed, 74% complete) Remaining: 5 GitHub scanner tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: resolve GitHub scanner test failures with proper pagination mocking (5 tests) Fixes 5 GitHub scanner test failures by correctly mocking API pagination: Root Cause: - Scanner's pagination loop breaks when response.json() returns empty list - Original mocks used return_value which returns same repos on every call - Loop continued until hitting max_repos limit (100), returning duplicates Fix Applied (All 5 Tests): - Changed from `mock_get.return_value = mock_response` to: ```python mock_response_page1 = Mock() # Returns repos mock_response_page1.json.return_value = [repo1, repo2] mock_response_page2 = Mock() # Empty - signals end of pagination mock_response_page2.json.return_value = [] mock_get.side_effect = [mock_response_page1, mock_response_page2] ``` Tests Fixed: 1. test_successful_org_scan - Basic org scanning 2. test_filters_private_repos - Private repo filtering 3. test_includes_private_repos_when_requested - Include private when flagged 4. test_filters_archived_repos - Archived repo filtering 5. test_rate_limit_warning - Rate limit warning logging Test Progress: 19 failed → 0 failed (19 tests fixed, 100% complete ✅) Final Status: 789 passed, 2 skipped, 0 failed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> * chore(release): 2.10.0 [skip ci] # [2.10.0](jeremyeder/agentready@v2.9.0...v2.10.0) (2025-12-08) ### Bug Fixes * disable attestations for Test PyPI to avoid conflict ([#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](jeremyeder@a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](jeremyeder@de28cd0)) * resolve 45 test failures across CLI, services, and assessors ([#4](jeremyeder#4)) ([3405142](jeremyeder@3405142)), closes [#178](https://github.com/jeremyeder/agentready/issues/178) [#178](https://github.com/jeremyeder/agentready/issues/178) * resolve broken links and workflow failures ([#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](jeremyeder@fbf5cf7)) * skip PR comments for external forks to prevent permission errors ([#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](jeremyeder@2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](jeremyeder@621152e)) * add quay/quay to leaderboard ([#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](jeremyeder@d6e8df0)) * Add weekly research update skill and automation ([#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](jeremyeder@7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](jeremyeder@71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) ### Performance Improvements * implement lazy loading for heavy CLI commands ([#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](jeremyeder@6a7cd4e)) * fix: resolve 45 test failures across CLI, services, and assessors (#4) * fix: resolve quick win test failures (CSV, config, research formatter) Fixed 5 test failures across 3 categories: **CSV Reporter Tests (4 errors → 0):** - Added create_dummy_findings() helper to generate Finding objects - Updated mock assessments to include required findings matching attributes_total - Fixed test_csv_empty_batch to expect ValueError during BatchAssessment construction **Config Model Test (1 failure → 0):** - Updated test_config_invalid_weights_negative to test for negative weights (current validation) - Removed outdated test_config_invalid_weights_sum (sum-to-1.0 validation was intentionally removed) **Research Formatter Tests (2 failures → 0):** - Fixed format_report() to ensure exactly one trailing newline - Updated extract_attribute_ids() regex to capture malformed IDs for validation Test status: 48→43 failures, 737→746 passed * fix: resolve learning service test failures with proper mocks and validation Fixed all 9 learning service test failures by addressing three issues: 1. Mock method mismatches (7 tests): - Tests were mocking `extract_from_findings()` but code calls `extract_all_patterns()` or `extract_specific_patterns()` - Updated all mocks to use correct method names based on whether `attribute_ids` parameter is passed 2. LLMEnricher import path (1 test): - Test tried to patch `learning_service.LLMEnricher` but it's imported inside `_enrich_with_llm()` method from `learners.llm_enricher` - Changed patch path to actual import location 3. Repository validation (4 tests): - Repository model requires `.git` directory - Updated `temp_dir` fixture to run `git init` - Updated tests to create assessment files in `.agentready/` subdirectory (code expects assessments at `.agentready/assessment-*.json`) 4. Assessment validation (3 tests): - Assessment requires `len(findings) == attributes_total` - Added `create_dummy_finding()` helper - Updated tests to include proper number of findings All 17 learning service tests now pass. Test progress: 48 failed → 34 failed (14 tests fixed) * fix: resolve pattern extractor and LLM enricher test failures (14 tests) Fixed 2 root causes affecting 14 total tests: 1. PatternExtractor attribute access (10 tests fixed): - Changed finding.attribute.attribute_id → finding.attribute.id - Fixed extract_specific_patterns() method - Added create_dummy_finding() helper for Assessment validation - Fixed 8 pattern extractor tests + 4 downstream test failures 2. Anthropic API error mocks (2 tests fixed): - Updated RateLimitError mock with response and body kwargs - Updated APIError mock with request and body kwargs - Adapted to evolved Anthropic SDK error class signatures Test status: 34 failed → 20 failed (14 tests fixed) Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: correct confidence format assertion in skill generator test Changed assertion from "90%" to "90.0%" to match actual output format. The SkillGenerator formats confidence as "90.0%" not "90%". Test status: 20 failed → 19 failed Related: #178 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: resolve CLI command test failures with path resolution and validation (12 tests) Fixes 12 failing tests in CLI commands (extract-skills and learn): CLI Command Fixes (Both Commands): - Resolve output_dir relative to repo_path instead of cwd - Fixes isolated_filesystem() test context issues - Ensures output created in repository, not temp directory - Add IntRange(min=1) validation for llm_budget parameter - Prevents negative budget values - Provides clear Click validation error Test Assertion Fixes: - Fix skill_md format tests: glob("*/SKILL.md") not glob("*.md") - SKILL.md files are created in subdirectories (skill-id/SKILL.md) - Fix github_issues format tests: glob("skill-*.md") not glob("issue-*.md") - Issue files are named skill-{id}.md, not issue-*.md - Add known skill IDs to test fixtures (claude_md_file, type_annotations) - PatternExtractor requires recognizable attribute IDs to extract skills Test Progress: 19 failed → 7 failed (12 tests fixed, 63% complete) Files Modified: - src/agentready/cli/extract_skills.py (path resolution, validation) - src/agentready/cli/learn.py (path resolution, validation) - tests/unit/test_cli_extract_skills.py (glob patterns) - tests/unit/test_cli_learn.py (glob patterns, fixture data) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: resolve isolated test failures in code_sampler and fixer_service (2 tests) Fixes 2 isolated test failures: Code Sampler Fix (code_sampler.py): - Add 'path' key check before accessing dict in _format_code_samples() - Empty dicts in files list were causing KeyError - Changed: if isinstance(file_item, dict) and "path" in file_item Fixer Service Test Fix (test_fixer_service.py): - Add passing finding to test_generate_fix_plan_no_failing_findings - Assessment validation requires len(findings) == attributes_total - Test was creating assessment with 0 findings but attributes_total=1 - Now creates a passing finding to satisfy validation Test Progress: 19 failed → 5 failed (14 tests fixed, 74% complete) Remaining: 5 GitHub scanner tests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: resolve GitHub scanner test failures with proper pagination mocking (5 tests) Fixes 5 GitHub scanner test failures by correctly mocking API pagination: Root Cause: - Scanner's pagination loop breaks when response.json() returns empty list - Original mocks used return_value which returns same repos on every call - Loop continued until hitting max_repos limit (100), returning duplicates Fix Applied (All 5 Tests): - Changed from `mock_get.return_value = mock_response` to: ```python mock_response_page1 = Mock() # Returns repos mock_response_page1.json.return_value = [repo1, repo2] mock_response_page2 = Mock() # Empty - signals end of pagination mock_response_page2.json.return_value = [] mock_get.side_effect = [mock_response_page1, mock_response_page2] ``` Tests Fixed: 1. test_successful_org_scan - Basic org scanning 2. test_filters_private_repos - Private repo filtering 3. test_includes_private_repos_when_requested - Include private when flagged 4. test_filters_archived_repos - Archived repo filtering 5. test_rate_limit_warning - Rate limit warning logging Test Progress: 19 failed → 0 failed (19 tests fixed, 100% complete ✅) Final Status: 789 passed, 2 skipped, 0 failed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> * chore(release): 2.10.0 [skip ci] * disable attestations for Test PyPI to avoid conflict ([#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](jeremyeder@a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](jeremyeder@de28cd0)) * resolve 45 test failures across CLI, services, and assessors ([#4](jeremyeder#4)) ([3405142](jeremyeder@3405142)), closes [#178](https://github.com/jeremyeder/agentready/issues/178) [#178](https://github.com/jeremyeder/agentready/issues/178) * resolve broken links and workflow failures ([#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](jeremyeder@fbf5cf7)) * skip PR comments for external forks to prevent permission errors ([#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](jeremyeder@2a29fb8)) * add ambient-code/agentready to leaderboard ([#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](jeremyeder@621152e)) * add quay/quay to leaderboard ([#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](jeremyeder@d6e8df0)) * Add weekly research update skill and automation ([#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](jeremyeder@7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](jeremyeder@71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * implement lazy loading for heavy CLI commands ([#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](jeremyeder@6a7cd4e)) * feat: add Harbor framework integration for real Terminal-Bench evaluations Implements complete Harbor integration to enable real-world Terminal-Bench assessor validation, replacing mocked results with actual Claude Code agent benchmarks. This enables empirical measurement of assessor effectiveness across real repositories. Key Components: - HarborConfig: Validated configuration with model/agent allowlists - Real benchmark execution: Secure subprocess integration with Harbor CLI - Parallel execution: ProcessPoolExecutor with resource limits (4 workers) - Aggregation: Pandas-based statistical analysis of assessor effectiveness - Security: Environment sanitization, path traversal prevention Implementation follows strict TDD (red-green-refactor): - 41 unit tests (100% coverage for aggregator, batch_runner, harbor_config) - 89% coverage for tbench_runner - All security validations tested Files Created: - src/agentready/services/eval_harness/{aggregator,batch_runner,harbor_config,tbench_runner}.py - tests/unit/test_{harbor_config,eval_harness_{services,cli}}.py - specs/002-harbor-real-integration/ (complete feature documentation) Tested with: black, isort, ruff (all passing) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: implement blocking test strategy with tiered CI jobs Fixed all 41 CLI tests and implemented a comprehensive blocking test strategy to improve CI reliability and development velocity. Test Fixes (41/41 CLI tests passing): - Fixed Pydantic validation error handling in config loading - Added extra="forbid" to Config model for strict validation - Fixed macOS path resolution for sensitive directories - Added /private/etc and refined /var handling - Fixed large repo warning exception handling E2E Critical Tests (11 tests - <1 min runtime): - Self-assessment end-to-end test - JSON/HTML/Markdown report generation validation - CLI command tests (help, version, research-version) - Error handling tests (nonexistent dir, invalid config) - Config application tests CI Workflow Changes: - Tier 1: critical-tests job (BLOCKS merge) - E2E tests, CLI tests, model tests - Runs on Python 3.12 and 3.13 - Fast (<5 min total) - Tier 2: linting job (BLOCKS merge) - black, isort, ruff checks - Tier 3: full-test-suite (WARNING only) - All tests with coverage reporting - Uploads coverage artifacts - continue-on-error: true - Tier 4: platform-tests (macOS - informational) - Platform-specific validation - continue-on-error: true Coverage Settings: - Removed global 90% fail-under threshold from pyproject.toml - Critical tests run without coverage (speed priority) - Full suite generates coverage reports without blocking Documentation: - Added plans/blocking-tests-strategy.md with complete implementation guide - 4-phase migration plan for future enhancements Impact: - Critical tests provide fast feedback (<5 min vs 15+ min) - Trivial PRs no longer blocked by flaky tests - Platform-specific tests don't cause false failures - All CLI tests reliable on macOS 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix(security): implement critical security fixes from code review Addressed 3 critical security vulnerabilities and 1 important reliability issue identified by feature-dev:code-reviewer agent (ID: 027604dd). Security Fixes: 1. TOCTOU path traversal vulnerability (Issue #1 - Confidence 85%) - Fixed double resolve() call that created race condition - Now use already-resolved path to avoid TOCTOU 2. Incomplete macOS path boundary checking (Issue #2 - Confidence 95%) - Replaced startswith() with proper is_relative_to() checking - Created _is_path_in_directory() helper for correct boundary checking - Prevents bypass via directories like /var/log-backup 3. Inconsistent sensitive directory lists (Issue #3 - Confidence 90%) - Centralized SENSITIVE_DIRS and VAR_SENSITIVE_SUBDIRS in security.py - CLI now imports from security module instead of duplicating - Ensures consistent protection across all entry points Reliability Fix: 4. Missing job-level timeouts in CI (Issue #4 - Confidence 82%) - Added timeout-minutes to all 4 GitHub Actions jobs - Prevents hung jobs from consuming CI resources - Critical tests: 15min, Linting: 10min, Full suite: 30min, macOS: 20min Changes: - src/agentready/utils/security.py: Added constants and boundary check helper - src/agentready/cli/main.py: Import centralized constants, use proper checking - .github/workflows/tests.yml: Add job-level timeouts to all jobs - plans/blocking-test-followups.md: Document remaining improvements Follow-Up: - Created issue #192 for remaining important improvements: 1. Make E2E test timeouts configurable 2. Add E2E test for sensitive directory blocking - Code simplification opportunities documented but deferred (low priority) Test Results: - All 41 CLI tests pass - All 11 E2E tests pass - Sensitive directory tests validate new boundary checking logic 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: correct Harbor results parsing to match actual Harbor 2.0 JSON structure Harbor framework writes results to timestamped subdirectories with singular "result.json" filename and different JSON schema than initially expected. This commit fixes three critical issues: 1. Find timestamped results directory (Harbor creates YYYY-MM-DD__HH-MM-SS/) 2. Use singular "result.json" instead of plural "results.json" 3. Parse actual Harbor JSON structure: - stats.evals.<eval_name>.{n_trials, n_errors, metrics, reward_stats} - n_solved calculated from reward_stats (tasks with reward > 0) - mean_score from metrics[0].mean Tested with real Harbor 2.0 output from Terminal-Bench evaluation. Resolves FileNotFoundError and KeyError exceptions when parsing Harbor results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * chore: save Harbor integration WIP before rebase onto v2.15.0 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * chore: restore version to 2.15.0 after rebase * fix: remove duplicate assessor registration for architecture_decisions and issue_pr_templates These two assessors have real implementations in documentation.py and structure.py but were also being added as stubs, creating duplicate findings in assessment reports. Fixes: - Removed StubAssessor('architecture_decisions', ...) from create_stub_assessors() - Removed StubAssessor('issue_pr_templates', ...) from create_stub_assessors() - Added warning comment to prevent future duplicates Result: 28 unique assessors instead of 30 with 2 duplicates * feat: redesign assess command output with detailed results table Changes: - Reordered summary statistics: Score, Assessed, Skipped, Total (new), Duration - Added assessment results table showing all test results inline - Table columns: Test Name, Test Result (with emojis), Notes - Notes column shows: - PASS: score (e.g., '100/100') - FAIL: failure reason from measured_value/threshold or evidence - NOT_APPLICABLE/SKIPPED: reason for skip from evidence - ERROR: error message - Auto-truncate long notes to 50 chars for readability - Improves user experience by showing all results without needing to open reports 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: validate API key before HarborConfig initialization Move API key validation before creating HarborConfig object to provide clean error message instead of ValueError traceback when ANTHROPIC_API_KEY is not set. This prevents the error from being raised in HarborConfig.__post_init__ before the validation check can run. * feat: add automatic Harbor CLI preflight checks with dataset management Implements interactive Harbor CLI installation and Terminal-Bench dataset management for benchmark command, resolving hardcoded path dependencies. ## Changes **Preflight System (NEW)** - src/agentready/utils/preflight.py: - check_harbor_cli(): Interactive Harbor installation with uv/pip fallback - ensure_terminal_bench_dataset(): Dynamic task discovery with auto-download - PreflightError exception for installation failures - tests/unit/utils/test_preflight.py: 9 comprehensive unit tests (100% coverage) **Benchmark Integration** - src/agentready/cli/benchmark.py: - Added --skip-preflight flag for advanced users - Integrated preflight checks before Harbor execution - Pass dynamic task_path to HarborConfig for smoketest mode - src/agentready/services/eval_harness/harbor_config.py: - Added task_path: Optional[Path] field - Updated docstring with task_path documentation - src/agentready/services/eval_harness/tbench_runner.py: - Replaced hardcoded task path with config.task_path - Added stdout/stderr capture for better error reporting - Enhanced error messages with stderr details - Added validation for smoketest mode task_path requirement **Documentation** - README.md: Added Harbor CLI installation section - CLAUDE.md: Added Preflight Checks architecture documentation - .gitignore: Added jobs/ directory (Harbor benchmark output) ## Security - Uses safe_subprocess_run() with 5-minute timeout for installations - User consent required before any Harbor installation - 10-minute timeout for dataset downloads with clear error messages - Sanitized environment variables for Harbor subprocess execution ## Testing - All preflight unit tests pass (9/9) - All linters pass (black, isort, ruff) - Test coverage: preflight.py at 60% (check_harbor_cli fully covered) ## Breaking Changes None - additive feature with backwards compatibility via --skip-preflight flag 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: pass full environment to Harbor subprocess The previous implementation only passed 3 environment variables (ANTHROPIC_API_KEY, PATH, HOME) which was too restrictive and broke Harbor's ability to run Claude Code agents. Harbor and Claude Code need additional environment variables like: - SHELL, TERM (shell configuration) - PYTHONPATH (Python environment) - LANG, LC_ALL (locale settings) - Other variables Harbor expects Now we pass through the full environment and explicitly set the API key to ensure it's correct. Fixes: 'Invalid API key · Please run /login' error in trajectory.json 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: set ANTHROPIC_AUTH_TOKEN for Harbor's Claude Code agent Harbor's claude-code agent looks for ANTHROPIC_AUTH_TOKEN in the environment, not ANTHROPIC_API_KEY. The agent code shows: env = { "ANTHROPIC_AUTH_TOKEN": os.environ.get( "MINIMAX_API_KEY", os.environ.get("ANTHROPIC_AUTH_TOKEN", "") ), ... } This was causing the 'Invalid API key · Please run /login' error in trajectory.json even when ANTHROPIC_API_KEY was correctly set in the user's environment. Fix: Set both ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN to ensure compatibility with Claude Code's authentication requirements. Resolves: Invalid API key error when running benchmarks Source: .venv/lib/python3.13/site-packages/harbor/agents/installed/claude_code.py 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: display trajectory file path in benchmark summary Added trajectory_path field to TbenchResult and logic to find and display the agent's trajectory.json file at the end of benchmark runs. The trajectory file contains the complete interaction history between the agent and Claude Code, which is valuable for debugging and understanding agent behavior. Changes: - Added trajectory_path: Path | None to TbenchResult dataclass - Updated _real_tbench_result() to search for trajectory.json in Harbor's output directory structure - Updated parse_harbor_results() to accept and set trajectory_path - Updated benchmark.py to display trajectory path in summary output Example output: Score: 0.00 Task Solved: False Resolved Trials: 0 Unresolved Trials: 1 Pass@1: 0.00 Trajectory: /private/var/folders/.../trajectory.json 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: override Harbor's hardcoded MiniMax API configuration Harbor's claude-code agent hardcodes ANTHROPIC_BASE_URL to MiniMax API: "ANTHROPIC_BASE_URL": "https://api.minimax.io/anthropic" This causes authentication errors when trying to use real Anthropic API keys. Fix: Set ANTHROPIC_API_BASE and ANTHROPIC_BASE_URL to point to the real Anthropic API endpoint, and remove MINIMAX_API_KEY from environment. Changes: - Set ANTHROPIC_BASE_URL=https://api.anthropic.com - Set ANTHROPIC_API_BASE=https://api.anthropic.com (alternative var) - Remove MINIMAX_API_KEY from environment if present This should override Harbor's MiniMax configuration and allow proper authentication with Anthropic's API. If this doesn't work (if Claude Code only uses ANTHROPIC_BASE_URL which is hardcoded by Harbor), we may need to patch Harbor or use a different agent implementation. Source: .venv/lib/python3.13/site-packages/harbor/agents/installed/claude_code.py:117-131 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: display Harbor command with copy/paste ready format Added comprehensive command display before Harbor execution to help with debugging and manual testing. Features: - Displays full Harbor command with proper shell escaping - Shows copy/paste ready version with environment variables - Truncates API key in display for security (first 20 chars) - Uses $ANTHROPIC_API_KEY variable in copyable version - Includes command breakdown showing all flags and options - Logs command execution to logger for debugging Example output: ====================================================================== Harbor Command (Copy/Paste Ready) ====================================================================== ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN=$ANTHROPIC_API_KEY ANTHROPIC_BASE_URL=https://api.anthropic.com ANTHROPIC_API_BASE=https://api.anthropic.com harbor run --path /path/to/task --agent claude-code --model anthropic/claude-sonnet-4-5 --jobs-dir /tmp/... --n-concurrent 1 --quiet ====================================================================== Command Breakdown: ====================================================================== Command: harbor run --path /path/to/task --agent claude-code ... Environment Variables: ANTHROPIC_API_KEY=sk-ant-oat01-MU6FQE... ANTHROPIC_AUTH_TOKEN=sk-ant-oat01-MU6FQE... ANTHROPIC_BASE_URL=https://api.anthropic.com ANTHROPIC_API_BASE=https://api.anthropic.com ====================================================================== This makes it easy to: - Copy/paste command for manual testing - Debug environment variable issues - Verify command construction - Share command with others for troubleshooting 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net>
github-actions bot
pushed a commit
that referenced
this pull request
Dec 10, 2025
# [2.17.0](v2.16.0...v2.17.0) (2025-12-10) ### Features * enhance assessors with multi-language support and security ([#200](#200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([#202](#202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [#178](#178) [#178](#178)
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 11, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-11) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
jeremyeder
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 12, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-11) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
jeremyeder
pushed a commit
that referenced
this pull request
Dec 12, 2025
# [2.10.0](jeremyeder/agentready@v2.9.0...v2.10.0) (2025-12-11) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](jeremyeder@6ecb786)), closes [#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](jeremyeder@a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](jeremyeder@de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](jeremyeder@27e87e5)), closes [#104](https://github.com/jeremyeder/agentready/issues/104) [#192](https://github.com/jeremyeder/agentready/issues/192) * resolve all test suite failures - achieve zero failures ([#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](jeremyeder@990fa2d)), closes [#148](https://github.com/jeremyeder/agentready/issues/148) [#147](https://github.com/jeremyeder/agentready/issues/147) [#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](jeremyeder@fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](jeremyeder@3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](jeremyeder@97b06af)) * skip PR comments for external forks to prevent permission errors ([#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](jeremyeder@2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](jeremyeder@621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](jeremyeder@a56e318)) * add Memory MCP server allow list to repository settings ([#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](jeremyeder@41d87bb)) * add quay/quay to leaderboard ([#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](jeremyeder@d6e8df0)) * Add weekly research update skill and automation ([#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](jeremyeder@7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](jeremyeder@71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * container support ([#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](jeremyeder@c6874ea)) * convert AgentReady assessment to on-demand workflow ([#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](jeremyeder@b5a1ce0)), closes [#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](jeremyeder@85712f2)), closes [#10](jeremyeder#10) * Harbor framework integration for Terminal-Bench evaluations ([#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](jeremyeder@d73a8c8)), closes [#4](jeremyeder#4) [#178](https://github.com/jeremyeder/agentready/issues/178) [#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](jeremyeder@570087d)), closes [#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](jeremyeder@f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](jeremyeder@d06bab4)), closes [#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](jeremyeder@6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 12, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-12) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 12, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-12) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
jeremyeder
added a commit
that referenced
this pull request
Dec 15, 2025
* chore: improve lychee link checker retry handling - Increase max_retries from 3 to 5 for better transient error handling - Reduce retry_wait_time from 30s to 2s for faster retries - Remove implementation-status exclusions (file was removed) - Total retry time: 10s (5 × 2s) vs previous 90s (3 × 30s) More attempts, faster response, better handling of transient failures. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: simplify PR review workflow and remove duplicate trigger - Remove pull_request trigger (doesn't have secret access) - Keep only pull_request_target (has secret access) - Remove broken output parsing (claude-code-action doesn't support custom outputs) - Simplify to just run /review-agentready command - Fixes duplicate workflow runs and ANTHROPIC_API_KEY missing errors The workflow was running twice - once without secrets (failing) and once with secrets (succeeding). Now runs once with secrets. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: consolidate GitHub Actions workflows by purpose PHASE 1 (completed earlier): - Delete 5 redundant workflows (tests, publish-pypi, docs-lint, 2× leaderboard) - Rename 3 workflows (ci, agentready-dev, update-docs-manual) - Add 3 new workflows (stale-issues, leaderboard, update-docs) - Optimize triggers for cost savings PHASE 2 (this commit): - Merge coverage-report.yml into ci.yml as new job - Create docs.yml combining link-check + future docs jobs - Fix actionlint issues (proper quoting, combined redirects) - Rename agentready-dev workflow - Add explicit @agentready-dev agent invocation in prompt - Update all GitHub Actions to latest versions (v6) - Delete update-docs-manual.yml (redundant with automated update-docs.yml) - Add GitHub Actions guidelines to CLAUDE.md - Reorganize README.md with TOC, research citations, expanded CLI reference NET RESULT: - 16 workflows → 13 workflows (-3 total) - Clear purpose-driven organization - 100% actionlint compliance for modified workflows - Improved maintainability All modified workflows validated with actionlint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: correct default weights to sum to 1.0 The default weights were summing to 0.99 instead of 1.0, causing validation errors: "Default weights must sum to 1.0 (got 0.9900, difference: -0.0100)" Root cause: Tier distribution was: - Tier 1: 54% (0.54) - Tier 2: 27% (0.27) - Tier 3: 15% (0.15) - Tier 4: 3% (0.03) Total: 99% (0.99) ❌ Fix: Increased dependency_security from 0.04 to 0.05 - Tier 1: 55% (0.55) - Tier 2: 27% (0.27) - Tier 3: 15% (0.15) - Tier 4: 3% (0.03) Total: 100% (1.00) ✅ This permanently fixes the floating-point validation error that appeared frequently in CI. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * chore(release): 2.10.0 [skip ci] # [2.10.0](jeremyeder/agentready@v2.9.0...v2.10.0) (2025-12-11) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](jeremyeder@6ecb786)), closes [#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](jeremyeder@a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * leaderboard workflow and SSH URL support ([#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](jeremyeder@de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](jeremyeder@27e87e5)), closes [#104](https://github.com/jeremyeder/agentready/issues/104) [#192](https://github.com/jeremyeder/agentready/issues/192) * resolve all test suite failures - achieve zero failures ([#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](jeremyeder@990fa2d)), closes [#148](https://github.com/jeremyeder/agentready/issues/148) [#147](https://github.com/jeremyeder/agentready/issues/147) [#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](jeremyeder@fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](jeremyeder@3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](jeremyeder@97b06af)) * skip PR comments for external forks to prevent permission errors ([#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](jeremyeder@2a29fb8)) ### Features * add ambient-code/agentready to leaderboard ([#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](jeremyeder@621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](jeremyeder@a56e318)) * add Memory MCP server allow list to repository settings ([#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](jeremyeder@41d87bb)) * add quay/quay to leaderboard ([#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](jeremyeder@d6e8df0)) * Add weekly research update skill and automation ([#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](jeremyeder@7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](jeremyeder@71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * container support ([#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](jeremyeder@c6874ea)) * convert AgentReady assessment to on-demand workflow ([#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](jeremyeder@b5a1ce0)), closes [#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](jeremyeder@85712f2)), closes [#10](jeremyeder#10) * Harbor framework integration for Terminal-Bench evaluations ([#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](jeremyeder@d73a8c8)), closes [#4](jeremyeder#4) [#178](https://github.com/jeremyeder/agentready/issues/178) [#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](jeremyeder@570087d)), closes [#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](jeremyeder@f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](jeremyeder@d06bab4)), closes [#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](jeremyeder@6a7cd4e)) * fix: update CHANGELOG.md links to use ambient-code/agentready * fix: update version flag test to match new format The --version output format was updated in PR #221 to show: 'AgentReady v2.20.2\nResearch Report: 2025-12-08' Updated test assertions to match the new format. * fix: make link checker simple and stable Changes: - Accept 403 status codes (sites that block bots) - Exclude academic publishers (ACM, IEEE, Springer) - Exclude research sites that commonly block scrapers (Anthropic, Microsoft Research) - Exclude placeholder/future research URLs (arxiv.org/abs/25xx) - Exclude Claude AI URLs (blocks automated scrapers) - Reduce timeout from 20s to 10s for faster failures - Reduce retries from 5 to 2 (fail fast) - Make link-check non-blocking with continue-on-error - Consolidate into single step (remove duplicate checks) - Remove verbose flag to reduce noise This makes CI more reliable by accepting that some research/academic sites will always block automated checks, and focusing only on critical infrastructure links that we can actually verify. * fix: remove continue-on-error from link checker The comprehensive exclusions should make it pass reliably. If it fails, that's a real issue we should fix. * fix: add missing v2.14.1 and v2.14.0 changelog entries Restored missing changelog entries between v2.15.0 and v2.13.0: - v2.14.1 (2025-12-05): YAML syntax error fix - v2.14.0 (2025-12-05): Container support feature Retrieved from git history (commits f67072e and 8bb403f). --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net>
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** ensure post-comment step runs after Claude Code Action ([b087e5c](b087e5c)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** ensure post-comment step runs after Claude Code Action ([b087e5c](b087e5c)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) * **workflows:** simplify post-comment step condition ([1bbf40a](1bbf40a)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** ensure post-comment step runs after Claude Code Action ([b087e5c](b087e5c)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) * **workflows:** remove if:always() to test step execution ([ff0bb12](ff0bb12)) * **workflows:** simplify post-comment step condition ([1bbf40a](1bbf40a)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** ensure post-comment step runs after Claude Code Action ([b087e5c](b087e5c)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) * **workflows:** remove if:always() to test step execution ([ff0bb12](ff0bb12)) * **workflows:** simplify post-comment step condition ([1bbf40a](1bbf40a)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** ensure post-comment step runs after Claude Code Action ([b087e5c](b087e5c)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) * **workflows:** remove if:always() to test step execution ([ff0bb12](ff0bb12)) * **workflows:** simplify post-comment step condition ([1bbf40a](1bbf40a)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
github-actions bot
pushed a commit
to jeremyeder/agentready
that referenced
this pull request
Dec 16, 2025
# [2.10.0](v2.9.0...v2.10.0) (2025-12-16) ### Bug Fixes * add bounded retry logic for LLM rate limit handling ([ambient-code#205](https://github.com/jeremyeder/agentready/issues/205)) ([6ecb786](6ecb786)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) * disable attestations for Test PyPI to avoid conflict ([ambient-code#155](https://github.com/jeremyeder/agentready/issues/155)) ([a33e3cd](a33e3cd)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * downgrade docker/metadata-action to v5 and fix shellcheck warnings ([12f5509](12f5509)) * enable Harbor task filtering for smoketest support ([ambient-code#222](https://github.com/jeremyeder/agentready/issues/222)) ([f780188](f780188)) * leaderboard workflow and SSH URL support ([ambient-code#147](https://github.com/jeremyeder/agentready/issues/147)) ([de28cd0](de28cd0)) * make E2E test timeouts configurable and add sensitive directory test ([ambient-code#206](https://github.com/jeremyeder/agentready/issues/206)) ([27e87e5](27e87e5)), closes [ambient-code#104](https://github.com/jeremyeder/agentready/issues/104) [ambient-code#192](https://github.com/jeremyeder/agentready/issues/192) * rename research report in data directory ([b8ddfdc](b8ddfdc)) * resolve all test suite failures - achieve zero failures ([ambient-code#180](https://github.com/jeremyeder/agentready/issues/180)) ([990fa2d](990fa2d)), closes [ambient-code#148](https://github.com/jeremyeder/agentready/issues/148) [ambient-code#147](https://github.com/jeremyeder/agentready/issues/147) [ambient-code#145](https://github.com/jeremyeder/agentready/issues/145) * resolve broken links and workflow failures ([ambient-code#160](https://github.com/jeremyeder/agentready/issues/160)) ([fbf5cf7](fbf5cf7)) * resolve YAML syntax error in continuous-learning workflow ([ambient-code#172](https://github.com/jeremyeder/agentready/issues/172)) ([3d40fcc](3d40fcc)) * resolve YAML syntax error in update-docs workflow and add actionlint ([ambient-code#173](https://github.com/jeremyeder/agentready/issues/173)) ([97b06af](97b06af)) * skip PR comments for external forks to prevent permission errors ([ambient-code#163](https://github.com/jeremyeder/agentready/issues/163)) ([2a29fb8](2a29fb8)) * update --version flag to show correct version and research report date ([ambient-code#221](https://github.com/jeremyeder/agentready/issues/221)) ([5a85abb](5a85abb)) * **workflows:** ensure post-comment step runs after Claude Code Action ([b087e5c](b087e5c)) * **workflows:** handle all event types in agentready-dev workflow ([9b942bf](9b942bf)) * **workflows:** improve error handling and logging for comment posting ([9ea1e6b](9ea1e6b)) * **workflows:** improve issue number extraction and add debug step ([ecd896b](ecd896b)) * **workflows:** remove if:always() to test step execution ([ff0bb12](ff0bb12)) * **workflows:** simplify post-comment step condition ([1bbf40a](1bbf40a)) ### Features * add ambient-code/agentready to leaderboard ([ambient-code#148](https://github.com/jeremyeder/agentready/issues/148)) ([621152e](621152e)) * add Harbor Terminal-Bench comparison for agent effectiveness ([ambient-code#199](https://github.com/jeremyeder/agentready/issues/199)) ([a56e318](a56e318)) * add Memory MCP server allow list to repository settings ([ambient-code#203](https://github.com/jeremyeder/agentready/issues/203)) ([41d87bb](41d87bb)) * add quay/quay to leaderboard ([ambient-code#162](https://github.com/jeremyeder/agentready/issues/162)) ([d6e8df0](d6e8df0)) * Add weekly research update skill and automation ([ambient-code#145](https://github.com/jeremyeder/agentready/issues/145)) ([7ba17a6](7ba17a6)) * automate PyPI publishing with trusted publishing (OIDC) ([ambient-code#154](https://github.com/jeremyeder/agentready/issues/154)) ([71f4632](71f4632)), closes [pypa/#action-pypi-publish](https://github.com/jeremyeder/agentready/issues/action-pypi-publish) * consolidate GitHub Actions workflows by purpose ([ambient-code#217](https://github.com/jeremyeder/agentready/issues/217)) ([717ca6b](717ca6b)), closes [ambient-code#221](https://github.com/jeremyeder/agentready/issues/221) * container support ([ambient-code#171](https://github.com/jeremyeder/agentready/issues/171)) ([c6874ea](c6874ea)) * convert AgentReady assessment to on-demand workflow ([ambient-code#213](https://github.com/jeremyeder/agentready/issues/213)) ([b5a1ce0](b5a1ce0)), closes [ambient-code#191](https://github.com/jeremyeder/agentready/issues/191) * enhance assessors with multi-language support and security ([ambient-code#200](https://github.com/jeremyeder/agentready/issues/200)) ([85712f2](85712f2)), closes [#10](#10) * Harbor framework integration for Terminal-Bench evaluations ([ambient-code#202](https://github.com/jeremyeder/agentready/issues/202)) ([d73a8c8](d73a8c8)), closes [#4](#4) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) [ambient-code#178](https://github.com/jeremyeder/agentready/issues/178) * Redesign homepage features with two-column layout and research links ([ambient-code#189](https://github.com/jeremyeder/agentready/issues/189)) ([570087d](570087d)), closes [ambient-code#187](https://github.com/jeremyeder/agentready/issues/187) * replace markdown-link-check with lychee for link validation ([ambient-code#177](https://github.com/jeremyeder/agentready/issues/177)) ([f1a4545](f1a4545)) * Terminal-Bench eval harness (MVP Phase 1) ([ambient-code#178](https://github.com/jeremyeder/agentready/issues/178)) ([d06bab4](d06bab4)), closes [ambient-code#171](https://github.com/jeremyeder/agentready/issues/171) * **workflows:** add comment posting for [@agentready-dev](https://github.com/agentready-dev) agent ([5dff614](5dff614)) ### Performance Improvements * implement lazy loading for heavy CLI commands ([ambient-code#151](https://github.com/jeremyeder/agentready/issues/151)) ([6a7cd4e](6a7cd4e))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implement Terminal-Bench evaluation harness to empirically measure the impact of AgentReady assessors on agentic development performance.
Overview
This PR implements Phase 1 (MVP) of the Terminal-Bench eval harness - a systematic A/B testing framework that measures how each AgentReady assessor improves benchmark scores.
Components Implemented
Phase 1A-1D: Core Services
Phase 1E: GitHub Pages Dashboard
/agentready/tbenchPhase 1F: Documentation & Tests
docs/eval-harness-guide.md- Step-by-step tutorialsdocs/tbench/methodology.md- Statistical methods explainedCLI Commands
Statistical Methods
Significance Criteria (both required):
Effect Size Interpretation:
Demo Results
Ran eval harness on AgentReady repository itself:
This validates the system works correctly - it identifies repos that already follow best practices.
File Structure
Phase 2 (Future)
Testing
✅ 6 CLI unit tests passing
✅ 5 integration tests passing
✅ 32 service tests passing
✅ End-to-end workflow tested
✅ Dashboard generated and verified
✅ All demos working (slides, walkthrough, terminal demo)
Files Changed
New Services:
src/agentready/services/eval_harness/*.py(5 services)src/agentready/models/eval_harness.py(data models)New CLI:
src/agentready/cli/eval_harness.py(5 commands)Tests:
tests/unit/test_eval_harness*.py(6 files)tests/integration/test_eval_harness_e2e.pyDocumentation:
docs/eval-harness-guide.mddocs/tbench/methodology.mddocs/tbench.md(dashboard)Demos:
docs/demos/slides.html(15 slides, reveal.js)docs/demos/walkthrough.md(complete guide)scripts/generate_slides.pyscripts/build_demos.py🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com