Add CoreThink Agent v1.0 SWE-bench Lite submission

[JayVaghasiya-ai]

• 62.33% success rate (187/300 resolved instances)
• Neuro-symbolic approach with Claude Sonnet-4 + DeepSeek-R1
• Complete reasoning traces and evaluation artifacts included
• Technical report: https://arxiv.org/pdf/2509.00971
• Organization: CoreThink.ai

==================================================
Resolved 187 instances (62.33%)
==================================================
Resolved by Repository
- astropy/astropy: 3/6 (50.0%)
- django/django: 79/114 (69.3%)
- matplotlib/matplotlib: 14/23 (60.87%)
- mwaskom/seaborn: 4/4 (100.0%)
- pallets/flask: 0/3 (0.0%)
- psf/requests: 4/6 (66.67%)
- pydata/xarray: 2/5 (40.0%)
- pylint-dev/pylint: 3/6 (50.0%)
- pytest-dev/pytest: 8/17 (47.06%)
- scikit-learn/scikit-learn: 16/23 (69.57%)
- sphinx-doc/sphinx: 9/16 (56.25%)
- sympy/sympy: 45/77 (58.44%)
==================================================
Resolved by Time
- 2012: 1/1 (100.0%)
- 2014: 1/3 (33.33%)
- 2015: 1/1 (100.0%)
- 2016: 1/4 (25.0%)
- 2017: 10/16 (62.5%)
- 2018: 11/21 (52.38%)
- 2019: 39/59 (66.1%)
- 2020: 40/66 (60.61%)
- 2021: 30/42 (71.43%)
- 2022: 35/57 (61.4%)
- 2023: 18/30 (60.0%)
=================================================

• Is a pass@1 submission (does not attempt the same task instance more than once)
• Does not use SWE-bench test knowledge (PASS_TO_PASS, FAIL_TO_PASS)
• Does not use the hints field in SWE-bench
• Does not have web-browsing OR has taken steps to prevent lookup of SWE-bench solutions via web-browsing

[john-b-yang]

Please reduce the number of files represented, I'm assuming you have submitted a lot of artifacts for trajectories? Please try to condense into one file per task instance.

[JayVaghasiya-ai]

Thank you for the feedback! I've already addressed this concern and consolidated the trajectory files. Here's what we've updated:

What We Changed

Previously, we had two files per task instance in the trajs/ directory:
<instance_id>.traj (main trajectory file)
<instance_id>.info.log (execution log file)
We have now merged these into a single .traj file per task instance, with the execution log content appended to the trajectory file. All .info.log files have been removed.
Current Submission Structure
Our submission now follows this structure:

evaluation/lite/20250914_CorethinkAgent/
├── README.md                    # Documentation
├── metadata.yaml                # Submission metadata
├── all_preds.jsonl             # All predictions (JSONL format)
├── preds.json                  # All predictions (JSON format)
├── results/                    # Evaluation results
│   ├── results.json
│   ├── resolved_by_repo.json
│   └── resolved_by_time.json
├── logs/                       # Evaluation artifacts (299 instances)
│   └── <instance_id>/
│       ├── patch.diff          # Generated patch
│       ├── report.json         # Test report
│       └── test_output.txt     # Test execution output
└── trajs/                      # Trajectory files (299 instances)
    └── <instance_id>/
        └── <instance_id>.traj  # Single consolidated trajectory file

Alignment with SWE-bench Lite Guidelines
✅ One file per task instance in trajs/: Each of the 299 task instances now has exactly one .traj file containing the complete reasoning trace and execution logs.
✅ Required files at root level: We include all_preds.jsonl, metadata.yaml, and README.md as specified.
✅ Results directory: Contains evaluation metrics and analysis.
✅ Logs directory: Contains the evaluation artifacts (patches, test outputs, reports) organized by instance.

The logs/ directory contains 3 files per instance (patch.diff, report.json, test_output.txt), which are the standard evaluation artifacts generated by the SWE-bench harness. These are kept separate as they represent the evaluation results rather than the agent's internal reasoning process.

Is this structure now aligned with your requirements, or would you like us to further consolidate any other files?

Best regards,
Jay
CorethinkAI