20250930 - Add report from Tingly (Code Reasoning Agent) with Qwen-Plus.

[0x0079]

Introducing Tingly Omnicoder: Supercharging Qwen-Plus for Precision Code Reasoning

This submission presents Tingly Omnicoder, a next-generation coding agent that dramatically elevates the capabilities of the mid-sized Qwen-Plus model—built upon the powerful Qwen3-30B-A3B-Instruct foundation—through a synergistic blend of intelligent tooling, structured reasoning, and error-tolerant code understanding.

While vanilla integrations like OpenHands + Qwen-Plus deliver baseline performance, Tingly Omnicoder achieves significantly higher success rates by embedding three core innovations:

Intelligent Code Search & Summarization: Our custom-built code retrieval system doesn’t just find relevant snippets—it tolerates syntactic and semantic noise, accurately locates context even in imperfect codebases, and distills complex logic into concise, actionable summaries.
Chain-of-Thought (CoT) Prompt Engineering for Debugging: We’ve meticulously designed CoT prompts that guide the agent to think like a seasoned developer—hypothesizing root causes, localizing bugs with surgical precision, and generating targeted, minimal fixes in the right places.
Tool-Agent Architecture for Clean Context: To prevent context pollution and maintain focus, we decouple tool usage (e.g., file reading, test execution) from the core reasoning agent. This ensures Qwen-Plus’s attention remains laser-focused on high-level problem-solving—not buried in raw I/O logs.
Together, these techniques unlock reasoning performance that far exceeds what Qwen-Plus achieves out-of-the-box, enabling Tingly Omnicoder to compete with—and often surpass—agents built on much larger models.

In short: we didn’t just plug in a model. We built an omniscient coding partner that sees through noise, reasons step-by-step, and acts with surgical precision—proving that with the right architecture, even a “middle-sized” model can punch far above its weight.

Performance Details

Submission summary for 20250930_tingly_qwen_plus on SWE-bench verified split
==================================================
Resolved 304 instances (60.8%)
==================================================
Resolved by Repository
- astropy/astropy: 12/22 (54.55%)
- django/django: 146/231 (63.2%)
- matplotlib/matplotlib: 19/34 (55.88%)
- mwaskom/seaborn: 0/2 (0.0%)
- pallets/flask: 1/1 (100.0%)
- psf/requests: 3/8 (37.5%)
- pydata/xarray: 17/22 (77.27%)
- pylint-dev/pylint: 2/10 (20.0%)
- pytest-dev/pytest: 13/19 (68.42%)
- scikit-learn/scikit-learn: 27/32 (84.38%)
- sphinx-doc/sphinx: 22/44 (50.0%)
- sympy/sympy: 42/75 (56.0%)
==================================================
Resolved by Time
- 2013: 2/3 (66.67%)
- 2014: 1/2 (50.0%)
- 2015: 0/1 (0.0%)
- 2016: 2/2 (100.0%)
- 2017: 9/16 (56.25%)
- 2018: 14/24 (58.33%)
- 2019: 67/98 (68.37%)
- 2020: 66/108 (61.11%)
- 2021: 44/86 (51.16%)
- 2022: 64/102 (62.75%)
- 2023: 35/58 (60.34%)

Evaluation Info

sb-cli evaluation info.
Email: yzffeng@tingly.dev
RunID: tingly-20250929

Checklist

• Is a pass@1 submission (does not attempt the same task instance more than once)
• Does not use SWE-bench test knowledge (PASS_TO_PASS, FAIL_TO_PASS)
• Does not use the hints field in SWE-bench
• Does not have web-browsing OR has taken steps to prevent lookup of SWE-bench solutions via web-browsing

[john-b-yang]

There's a lot of files, any way you could make the representation for logs more condensed?

[0x0079]

There's a lot of files, any way you could make the representation for logs more condensed?

Since we implement agent as tool, each tool will generate its own trajectory.
If it causes any inconvenience, I could merge them into a single file for each instance.
I will try to condense them too.

[DONE]

[0x0079]

@john-b-yang
@ofirpress
Will we move on?

[ofirpress]

we are working towards a deadline in a week or two and will not be reviewing submissions until probably mid november at the earliest. i'm super sorry about that- we're a very small group of people doing new research and managing swe-bench, swe-agent and swe-smith all at the same time and it's tough to juggle everything

[john-b-yang]

Please give me write access to your repository.

[0x0079]

Please give me write access to your repository.

@john-b-yang , thanks for your response.
I found the invitation is expired, I have send another one, please check again.

https://github.com/project-anders/experiments/invitations

[FFengIll]

Has there been any progress?
@john-b-yang