Skip to content

Add matmul int4 for CUDA #17526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yufenglee wants to merge 28 commits into from
Closed

Add matmul int4 for CUDA #17526

yufenglee wants to merge 28 commits into from

Conversation

@yufenglee
Copy link
Member

@yufenglee yufenglee commented Sep 12, 2023

Description

Motivation and Context

github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 12, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 12, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 12, 2023
View reviewed changes
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
int32_t k = k_block_idx * block_size;
const BlockwiseQuantBlock<T, block_size, bits>* blob_ptr = src_blob + task_idx;
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);
} else {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
int32_t k = k_block_idx * block_size;
const BlockwiseQuantBlock<T, block_size, bits>* blob_ptr = src_blob + task_idx;
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);
} else {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
int32_t k = k_block_idx * block_size;
const BlockwiseQuantBlock<T, block_size, bits>* blob_ptr = src_blob + task_idx;
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);
} else {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
int32_t k = k_block_idx * block_size;
const BlockwiseQuantBlock<T, block_size, bits>* blob_ptr = src_blob + task_idx;
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);
} else {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
int32_t k = k_block_idx * block_size;
const BlockwiseQuantBlock<T, block_size, bits>* blob_ptr = src_blob + task_idx;
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
onnxruntime/contrib_ops/cpu/quantization/dequantize_blockwise.h Outdated
if (nullptr != zero_points) {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], zero_points[task_idx], k, K);
} else {
blob_ptr->dequant(dst + n * K + k, scale[task_idx], k, K);

Check warning

Code scanning / PREfast

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).

Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow (io.2).
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 18, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 18, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 18, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 18, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 26, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 26, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 27, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 28, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 28, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 9, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 9, 2023
View reviewed changes
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 9, 2023
View reviewed changes
@yufenglee yufenglee closed this Oct 11, 2023
@yufenglee
Copy link
Member Author

yufenglee commented Oct 11, 2023

A clean one: #17890

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yufenglee