Arm backend: Int16 linear support #14258

per · 2025-09-12T14:22:33Z

Summary

Adds support for a16w8 for linear when targeting a backend with +int16 extension.

Fixes #13729

Test plan

Tested through unit tests.

cc @digantdesai @freddan80 @zingo @oscarandersson8218

pytorch-bot · 2025-09-12T14:22:36Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14258

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 0bdb35c with merge base 0329a8a ():

NEW FAILURES - The following jobs have failed:

pull / test-samsung-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t 1d895b71082c5899a3c15af6d32a1b51bc44b561836e769092c5e5c50fc4f143 /exec failed with exit code 1
trunk / test-arm-ootb-linux / linux-job (gh)
RuntimeError: Command docker exec -t 0ad4790d2177643b1dbd1a8878ecbf22db2250382ba29bdeaccd2a334c199022 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

per · 2025-09-15T14:45:20Z

Unrelated failures.

backends/arm/_passes/add_bias_pass.py

digantdesai · 2025-09-15T17:00:20Z

backends/arm/_passes/arm_pass_manager.py

        self.add_pass(FuseConstantArgsPass(exported_program))
+        self.add_pass(InsertTableOpsPass(exported_program))
+        # If we have a conv2d with int16 activation split up into a convolution
+        # and an addition, to work-around the lack of support for int48 in torch


and an addition, to work-around the lack of support for int48 in torch

Or can it be done by using torch.dtype.int64 instead and then detecting and lowering it as int48 downstream?

I was starting of in that direction, but it interfere a bit with the int64->int32 handling, so rather keep it separate.

yeah given int64 is treated as radioactive :P

digantdesai

This is awesome, No conv tests for 16a8w, just curious.

digantdesai · 2025-09-16T15:27:00Z

backends/arm/test/ops/test_linear.py

 )
+
+
+@common.parametrize("test_data", test_data_all_16a8w)


@per - Does it work with U55 and U85?
@Ninja91 - IIRC you already have tests seems like haven't merged?

No, not yet. Test will be added for Ethos-U with a Vela update when it's in place.

I have added tests for U55 and U85

facebook-github-bot · 2025-09-16T16:53:38Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D82552402.

zingo · 2025-09-17T08:06:23Z

I see this testfail :(
[test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job https://github.com/pytorch/executorch/actions/runs/17768877335/job/50517971935?pr=14258#logs

FAILED backends/arm/test/ops/test_linear.py::test_linear_16a8w_tosa_INT[model_linear_rank1_large_randn,per_channel_quant=True] - AssertionError: Output 0 does not match reference output.
Given atol: 0.004528127912431955, rtol: 0.001.
Output tensor shape: torch.Size([20]), dtype: torch.float32
Difference: max: 0.1481781005859375, abs: 0.1481781005859375, mean abs error: 0.008467483520507812.
-- Model vs. Reference --
Numel: 20, 20
Median: 14.398289680480957, 14.398289680480957
Mean: 2.563008487224579, 2.555246722698212
Max: 69.8498764038086, 69.84635162353516
Min: -115.46151733398438, -115.60969543457031

zingo · 2025-09-18T08:04:14Z

I removed milestone 1.0 for now Vela does not support this anyway so lets take this a bit calmer.

zingo · 2025-09-19T11:00:35Z

@digantdesai , this has come out of sync with Meta internal version and I'm not allowed merge it, would you like to assist?

digantdesai · 2025-09-19T15:44:28Z

yeah let me try to merge this.

facebook-github-bot · 2025-09-19T15:45:17Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D82552402.

digantdesai · 2025-09-19T15:45:20Z

if it doesn't work I can give you a patch..

Signed-off-by: Per Åstrand <[email protected]> Change-Id: I2b189b559f699c7eda6921ed515c0e8a849226ca

Support quantization to 16a8w. Since the resulting TOSA operator needs to have the bias in int48 which isn't avaiable as a type in torch, the conv2d needs to be decomposed into a conv + add, where the conv result is scaled down to 32 bit before the addition of the bias is done. Signed-off-by: Per Åstrand <[email protected]> Change-Id: Ib8cae694035796374a55a9909e501596e983abf5

For the case when the activation is 16 bit the bias in TOSA must be a int48_t tensor. Since that can't be represented using torch.dtypes the corresponding node.meta is set with a key 'tosa_dtype_48bit' to pass through the note to the creation of the TOSA Tensor. Also make sure to distinguish between int32 and int48 tensors in fuse constant ops pass. Signed-off-by: Per Åstrand <[email protected]> Change-Id: Iefe64f2b02f388c905c9c818ee7d2a6af40bc9e3

Signed-off-by: Per Åstrand <[email protected]> Change-Id: Ibe158d8d35a632547290f1b9a055d061ae267d77

Enable tests of int16 activations and int8 weight quantization. Test for large_rand is disabled to sort out why the test is flaky. Signed-off-by: Per Åstrand <[email protected]> Change-Id: I9de5d472f8862edebcf82c140399985db930c069

Add a enum class to handle special dtypes that can't be represented in torch (i.e. int48_t) to avoid leaking serializer types into the pass handling of the backend. Signed-off-by: Per Åstrand <[email protected]> Change-Id: I3388cec3c8a26f28790eedc3f124c336b6724cb4

zingo · 2025-09-22T18:01:16Z

Fails are unrelated
Note: Fail in trunk / test-arm-ootb-linux / linux-job is unrelated and handled by another PR

zingo · 2025-09-22T18:02:19Z

@digantdesai we had to rebase as a file was changed and as we changed it again we got back to the same problem and need your help again. Sorry about that.

facebook-github-bot · 2025-09-23T15:08:03Z

@digantdesai has imported this pull request. If you are a Meta employee, you can view this in D82552402.

per requested a review from digantdesai as a code owner September 12, 2025 14:22

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 12, 2025

per requested a review from zingo September 12, 2025 14:22

per added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm ciflow/trunk release notes: none Do not include this in the release notes labels Sep 12, 2025

zingo added this to the 1.0.0 milestone Sep 12, 2025

zingo changed the title ~~Int16 linear support~~ Arm backend: Int16 linear support Sep 12, 2025

per force-pushed the int16_linear_support branch from e170183 to 0ab797f Compare September 15, 2025 07:08

digantdesai reviewed Sep 15, 2025

View reviewed changes

backends/arm/_passes/add_bias_pass.py Outdated Show resolved Hide resolved

digantdesai reviewed Sep 15, 2025

View reviewed changes

digantdesai approved these changes Sep 15, 2025

View reviewed changes

digantdesai requested a review from Ninja91 September 15, 2025 17:10

per force-pushed the int16_linear_support branch from 5b9c54e to ac4203c Compare September 16, 2025 14:18

digantdesai reviewed Sep 16, 2025

View reviewed changes

per force-pushed the int16_linear_support branch from ac4203c to 54ad8e2 Compare September 17, 2025 15:20

zingo removed this from the 1.0.0 milestone Sep 18, 2025

per added 4 commits September 22, 2025 11:49

Arm backend: Handle 16 bit activation for conv2d

bc0f134

Signed-off-by: Per Åstrand <[email protected]> Change-Id: I2b189b559f699c7eda6921ed515c0e8a849226ca

Arm backend: Fix mult and scale calculation for int48_t

2bd09f9

Signed-off-by: Per Åstrand <[email protected]> Change-Id: Ibe158d8d35a632547290f1b9a055d061ae267d77

per added 2 commits September 22, 2025 11:51

Arm backend: Enable linear 16a8w tests

aafcede

Enable tests of int16 activations and int8 weight quantization. Test for large_rand is disabled to sort out why the test is flaky. Signed-off-by: Per Åstrand <[email protected]> Change-Id: I9de5d472f8862edebcf82c140399985db930c069

per force-pushed the int16_linear_support branch from dc7a0a7 to 18c9985 Compare September 22, 2025 10:06

Merge branch 'main' into int16_linear_support

0bdb35c

zingo merged commit bb81136 into pytorch:main Sep 24, 2025
359 of 365 checks passed

		)


		@common.parametrize("test_data", test_data_all_16a8w)

Arm backend: Int16 linear support #14258

Arm backend: Int16 linear support #14258

Uh oh!

Conversation

per commented Sep 12, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pytorch-bot bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14258

❌ 2 New Failures

Uh oh!

per commented Sep 15, 2025

Uh oh!

Uh oh!

digantdesai Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

and an addition, to work-around the lack of support for int48 in torch

Uh oh!

per Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

digantdesai Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

digantdesai Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

per Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Ninja91 Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

zingo commented Sep 17, 2025

Uh oh!

zingo commented Sep 18, 2025

Uh oh!

zingo commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

digantdesai commented Sep 19, 2025

Uh oh!

facebook-github-bot commented Sep 19, 2025

Uh oh!

digantdesai commented Sep 19, 2025

Uh oh!

zingo commented Sep 22, 2025

Uh oh!

zingo commented Sep 22, 2025

Uh oh!

facebook-github-bot commented Sep 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

per commented Sep 12, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 12, 2025 •

edited

Loading

digantdesai Sep 15, 2025 •

edited

Loading

zingo commented Sep 19, 2025 •

edited

Loading