Skip to content

Add Tosa to LLM extension #15556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Sebastian-Larsson wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add Tosa to LLM extension #15556

Sebastian-Larsson wants to merge 2 commits into from

Conversation

@Sebastian-Larsson
Copy link
Collaborator

@Sebastian-Larsson Sebastian-Larsson commented Nov 4, 2025 •
edited by pytorch-bot bot
Loading

@Sebastian-Larsson Sebastian-Larsson added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Nov 4, 2025
@Sebastian-Larsson Sebastian-Larsson added ciflow/trunk release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava module: llm Issues related to LLM examples and apps, and to the extensions/llm/ code labels Nov 4, 2025
@pytorch-bot
Copy link

pytorch-bot bot commented Nov 4, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15556

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures

As of commit c974788 with merge base 1a086f0 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 4, 2025
@zingo
Copy link
Collaborator

zingo commented Nov 11, 2025

Hi @SS-JIA this one is outside of Arm stuff so we need help with someone to review it, thanks for your time and effort with this.

@SS-JIA
Copy link
Contributor

SS-JIA commented Nov 11, 2025

Requested review from @larryliu0820

larryliu0820
larryliu0820 reviewed Nov 11, 2025
View reviewed changes
examples/models/llama/export_llama_lib.py
llm_config.quantization.pt2e_quantize.value
)
quantizers.append(coreml_quantizer)
if llm_config.backend.tosa.enabled and llm_config.quantization.pt2e_quantize:
Copy link
Contributor

@larryliu0820 larryliu0820 Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add a unit test to cover this logic branch?

Copy link
Collaborator Author

@Sebastian-Larsson Sebastian-Larsson Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know what you think

@Sebastian-Larsson
Copy link
Collaborator Author

Sebastian-Larsson commented Dec 2, 2025

@larryliu0820 woul you mind taking another look with the changes after your comments?

@zingo
Copy link
Collaborator

zingo commented Dec 4, 2025

@metascroy Is this OK to merge?

@zingo
Copy link
Collaborator

zingo commented Dec 15, 2025

Hi @larryliu0820 (and/or @digantdesai ) is this OK after the added test? We have some follow up PR that would be good to get into PR state :)

@Sebastian-Larsson Sebastian-Larsson mentioned this pull request Dec 17, 2025
Open
@zingo zingo added this to the 1.1.0 milestone Dec 19, 2025
@SS-JIA
Copy link
Contributor

SS-JIA commented Jan 7, 2026

@larryliu0820 new test case was added - mind taking a look? Thanks in advance!

@hth000 hth000 mentioned this pull request Jan 13, 2026
Open
SS-JIA
SS-JIA approved these changes Jan 14, 2026
View reviewed changes
Copy link
Contributor

@SS-JIA SS-JIA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as the requested test was added!

SS-JIA
SS-JIA requested changes Jan 14, 2026
View reviewed changes
Copy link
Contributor

@SS-JIA SS-JIA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary RC while I double check internal importing the diff

@meta-codesync
Copy link
Contributor

meta-codesync bot commented Jan 14, 2026

@SS-JIA has imported this pull request. If you are a Meta employee, you can view this in D90692071.

@SS-JIA
Copy link
Contributor

SS-JIA commented Jan 14, 2026

Indeed, the new test is failing due to the Arm targets not being visible to the test target.

To fix, update: https://github.com/pytorch/executorch/blob/8293fa64d221c1d694723b9c209fe0eeb34a8748/examples/models/llama/tests/TARGETS

Add the "//executorch/backends/arm/quantizer:lib" target as a dependency

@larryliu0820
Copy link
Contributor

larryliu0820 commented Jan 20, 2026

Looks good as long as CI jobs are fixed.

@Sebastian-Larsson
Add Tosa to LLM extension
9f93123 
Change-Id: I2cded64d7ffdf3de441bbd7ee73357c8ba0a3749
Signed-off-by: Sebastian Larsson <sebastian.larsson@arm.com>
@zingo zingo removed this from the 1.1.0 milestone Jan 28, 2026
@zingo zingo moved this from To triage to In progress in ExecuTorch Arm Backend Jan 28, 2026
@zingo zingo requested a review from SS-JIA January 29, 2026 10:27
@zingo
Copy link
Collaborator

zingo commented Jan 29, 2026

Rebasing as we just merged a tosa spec change just to make sure all is still working :)

@zingo
Copy link
Collaborator

zingo commented Jan 30, 2026

@SS-JIA we fixed the buck2 things and now it needs to re-imported as we cant merge with the version diff vs meta internal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@larryliu0820 larryliu0820 larryliu0820 left review comments

@zingo zingo zingo approved these changes

@jackzhxng jackzhxng Awaiting requested review from jackzhxng

@mergennachin mergennachin Awaiting requested review from mergennachin mergennachin is a code owner

@lucylq lucylq Awaiting requested review from lucylq lucylq is a code owner

@SS-JIA SS-JIA Awaiting requested review from SS-JIA

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: llm Issues related to LLM examples and apps, and to the extensions/llm/ code partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava

Projects

ExecuTorch Arm Backend
Status: In progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Sebastian-Larsson @zingo @SS-JIA @larryliu0820