[X86] AMD Zen 6 Initial enablement#179150
Merged
Merged
Conversation
llvmbot
added
clang-tools-extra
compiler-rt
backend:X86
clangd
clang:driver
Member
|
@llvm/pr-subscribers-llvm-transforms @llvm/pr-subscribers-clang-codegen Author: Ganesh (ganeshgit) ChangesAdded new ISA AVX512 BMM. Patch is 101.35 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/179150.diff 59 Files Affected:
diff --git a/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp b/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp
index d43a3ea327a39..8a5e3290ed9ef 100644
--- a/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp
+++ b/clang-tools-extra/clang-include-fixer/find-all-symbols/STLPostfixHeaderMap.cpp
@@ -39,6 +39,8 @@ const HeaderMapCollector::RegexHeaderMap *getSTLPostfixHeaderMap() {
{"include/ammintrin.h$", "<ammintrin.h>"},
{"include/avx2intrin.h$", "<immintrin.h>"},
{"include/avx512bwintrin.h$", "<immintrin.h>"},
+ {"include/avx512bmmintrin.h$", "<immintrin.h>"},
+ {"include/avx512bmmvlintrin.h$", "<immintrin.h>"},
{"include/avx512cdintrin.h$", "<immintrin.h>"},
{"include/avx512dqintrin.h$", "<immintrin.h>"},
{"include/avx512erintrin.h$", "<immintrin.h>"},
diff --git a/clang-tools-extra/clangd/index/CanonicalIncludes.cpp b/clang-tools-extra/clangd/index/CanonicalIncludes.cpp
index cbef64f351341..3a646e47a9b16 100644
--- a/clang-tools-extra/clangd/index/CanonicalIncludes.cpp
+++ b/clang-tools-extra/clangd/index/CanonicalIncludes.cpp
@@ -42,6 +42,8 @@ const std::pair<llvm::StringRef, llvm::StringRef> IncludeMappings[] = {
{"include/ammintrin.h", "<ammintrin.h>"},
{"include/avx2intrin.h", "<immintrin.h>"},
{"include/avx512bwintrin.h", "<immintrin.h>"},
+ {"include/avx512bmmintrin.h", "<immintrin.h>"},
+ {"include/avx512bmmvlintrin.h", "<immintrin.h>"},
{"include/avx512cdintrin.h", "<immintrin.h>"},
{"include/avx512dqintrin.h", "<immintrin.h>"},
{"include/avx512erintrin.h", "<immintrin.h>"},
diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index 3a3d76112a02b..a6ac25634a567 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -265,6 +265,21 @@ NVPTX Support
X86 Support
^^^^^^^^^^^
+- `march=znver6` is now supported.
+- Support ISA of ``AVX512BMM``.
+ * Support intrinsic of ``_mm512_vbmacor16x16x16_epi16``.
+ * Support intrinsic of ``_mm512_vbmacxor16x16x16_epi16``.
+ * Support intrinsic of ``_mm512_mask_vbitrev_epi8``.
+ * Support intrinsic of ``_mm512_maskz_vbitrev_epi8``.
+ * Support intrinsic of ``_mm512_vbitrev_epi8``.
+ * Support intrinsic of ``_mm256_vbmacor16x16x16_epi16``.
+ * Support intrinsic of ``_mm256_vbmacxor16x16x16_epi16``.
+ * Support intrinsic of ``_mm128_mask_vbitrev_epu8``.
+ * Support intrinsic of ``_mm256_mask_vbitrev_epu8``.
+ * Support intrinsic of ``_mm128_maskz_vbitrev_epu8``.
+ * Support intrinsic of ``_mm256_maskz_vbitrev_epu8``.
+ * Support intrinsic of ``_mm128_vbitrev_epu8``.
+ * Support intrinsic of ``_mm256_vbitrev_epu8``.
Arm and AArch64 Support
^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/clang/include/clang/Basic/BuiltinsX86.td b/clang/include/clang/Basic/BuiltinsX86.td
index 23eac47eb5e4c..7d1b513532f34 100644
--- a/clang/include/clang/Basic/BuiltinsX86.td
+++ b/clang/include/clang/Basic/BuiltinsX86.td
@@ -5055,3 +5055,31 @@ let Features = "avx10.2", Attributes = [NoThrow, Const, RequiredVectorWidth<256>
let Features = "avx10.2", Attributes = [NoThrow, Const, RequiredVectorWidth<512>] in {
def vgetmantbf16512_mask : X86Builtin<"_Vector<32, __bf16>(_Vector<32, __bf16>, _Constant int, _Vector<32, __bf16>, unsigned int)">;
}
+
+let Features = "avx10.2", Attributes = [NoThrow, Const, RequiredVectorWidth<128>] in {
+ def vsqrtbf16 : X86Builtin<"_Vector<8, __bf16>(_Vector<8, __bf16>)">;
+}
+
+let Features = "avx10.2", Attributes = [NoThrow, Const, RequiredVectorWidth<256>] in {
+ def vsqrtbf16256 : X86Builtin<"_Vector<16, __bf16>(_Vector<16, __bf16>)">;
+}
+
+let Features = "avx10.2", Attributes = [NoThrow, Const, RequiredVectorWidth<512>] in {
+ def vsqrtbf16512 : X86Builtin<"_Vector<32, __bf16>(_Vector<32, __bf16>)">;
+}
+
+let Features = "avx512bmm", Attributes = [NoThrow, Const, Constexpr, RequiredVectorWidth<512>] in {
+ def bmacor16x16x16_v32hi : X86Builtin<"_Vector<32, short>(_Vector<32, short>, _Vector<32, short>, _Vector<32, short>)">;
+ def bmacxor16x16x16_v32hi : X86Builtin<"_Vector<32, short>(_Vector<32, short>, _Vector<32, short>, _Vector<32, short>)">;
+ def bitrev512 : X86Builtin<"_Vector<64, char>(_Vector<64, char>)">;
+}
+
+let Features = "avx512bmm,avx512vl", Attributes = [NoThrow, Const, Constexpr, RequiredVectorWidth<256>] in {
+ def bmacor16x16x16_v16hi : X86Builtin<"_Vector<16, short>(_Vector<16, short>, _Vector<16, short>, _Vector<16, short>)">;
+ def bmacxor16x16x16_v16hi : X86Builtin<"_Vector<16, short>(_Vector<16, short>, _Vector<16, short>, _Vector<16, short>)">;
+ def bitrev256 : X86Builtin<"_Vector<32, char>(_Vector<32, char>)">;
+}
+
+let Features = "avx512bmm,avx512vl", Attributes = [NoThrow, Const, Constexpr, RequiredVectorWidth<128>] in {
+ def bitrev128 : X86Builtin<"_Vector<16, char>(_Vector<16, char>)">;
+}
diff --git a/clang/include/clang/Options/Options.td b/clang/include/clang/Options/Options.td
index 421208a812bbc..658ee6f7ebe60 100644
--- a/clang/include/clang/Options/Options.td
+++ b/clang/include/clang/Options/Options.td
@@ -6889,6 +6889,8 @@ def mavx512bf16 : Flag<["-"], "mavx512bf16">, Group<m_x86_Features_Group>;
def mno_avx512bf16 : Flag<["-"], "mno-avx512bf16">, Group<m_x86_Features_Group>;
def mavx512bitalg : Flag<["-"], "mavx512bitalg">, Group<m_x86_Features_Group>;
def mno_avx512bitalg : Flag<["-"], "mno-avx512bitalg">, Group<m_x86_Features_Group>;
+def mavx512bmm : Flag<["-"], "mavx512bmm">, Group<m_x86_Features_Group>;
+def mno_avx512bmm : Flag<["-"], "mno-avx512bmm">, Group<m_x86_Features_Group>;
def mavx512bw : Flag<["-"], "mavx512bw">, Group<m_x86_Features_Group>;
def mno_avx512bw : Flag<["-"], "mno-avx512bw">, Group<m_x86_Features_Group>;
def mavx512cd : Flag<["-"], "mavx512cd">, Group<m_x86_Features_Group>;
diff --git a/clang/lib/Basic/Targets/X86.cpp b/clang/lib/Basic/Targets/X86.cpp
index f00d435937b92..2d65e5f6a9ba7 100644
--- a/clang/lib/Basic/Targets/X86.cpp
+++ b/clang/lib/Basic/Targets/X86.cpp
@@ -296,6 +296,8 @@ bool X86TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
HasAVX512DQ = true;
} else if (Feature == "+avx512bitalg") {
HasAVX512BITALG = true;
+ } else if (Feature == "+avx512bmm") {
+ HasAVX512BMM = true;
} else if (Feature == "+avx512bw") {
HasAVX512BW = true;
} else if (Feature == "+avx512vl") {
@@ -308,6 +310,8 @@ bool X86TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
HasAVX512IFMA = true;
} else if (Feature == "+avx512vp2intersect") {
HasAVX512VP2INTERSECT = true;
+ } else if (Feature == "avx512bmm") {
+ HasAVX512BMM = true;
} else if (Feature == "+sha") {
HasSHA = true;
} else if (Feature == "+sha512") {
@@ -716,6 +720,9 @@ void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
case CK_ZNVER5:
defineCPUMacros(Builder, "znver5");
break;
+ case CK_ZNVER6:
+ defineCPUMacros(Builder, "znver6");
+ break;
case CK_Geode:
defineCPUMacros(Builder, "geode");
break;
@@ -833,6 +840,8 @@ void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
Builder.defineMacro("__AVX512DQ__");
if (HasAVX512BITALG)
Builder.defineMacro("__AVX512BITALG__");
+ if (HasAVX512BMM)
+ Builder.defineMacro("__AVX512BMM__");
if (HasAVX512BW)
Builder.defineMacro("__AVX512BW__");
if (HasAVX512VL) {
@@ -846,6 +855,8 @@ void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
Builder.defineMacro("__AVX512IFMA__");
if (HasAVX512VP2INTERSECT)
Builder.defineMacro("__AVX512VP2INTERSECT__");
+ if (HasAVX512BMM)
+ Builder.defineMacro("__AVX512BMM__");
if (HasSHA)
Builder.defineMacro("__SHA__");
if (HasSHA512)
@@ -1076,6 +1087,7 @@ bool X86TargetInfo::isValidFeatureName(StringRef Name) const {
.Case("avx512fp16", true)
.Case("avx512dq", true)
.Case("avx512bitalg", true)
+ .Case("avx512bmm", true)
.Case("avx512bw", true)
.Case("avx512vl", true)
.Case("avx512vbmi", true)
@@ -1196,6 +1208,8 @@ bool X86TargetInfo::hasFeature(StringRef Feature) const {
.Case("avx512fp16", HasAVX512FP16)
.Case("avx512dq", HasAVX512DQ)
.Case("avx512bitalg", HasAVX512BITALG)
+ .Case("avx512bmm", HasAVX512BMM)
+ .Case("avx512bmm", HasAVX512BMM)
.Case("avx512bw", HasAVX512BW)
.Case("avx512vl", HasAVX512VL)
.Case("avx512vbmi", HasAVX512VBMI)
@@ -1641,6 +1655,7 @@ std::optional<unsigned> X86TargetInfo::getCPUCacheLineSize() const {
case CK_ZNVER3:
case CK_ZNVER4:
case CK_ZNVER5:
+ case CK_ZNVER6:
// Deprecated
case CK_x86_64:
case CK_x86_64_v2:
diff --git a/clang/lib/Basic/Targets/X86.h b/clang/lib/Basic/Targets/X86.h
index 922e32906cd04..6bd55f9fbf4bb 100644
--- a/clang/lib/Basic/Targets/X86.h
+++ b/clang/lib/Basic/Targets/X86.h
@@ -104,6 +104,7 @@ class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo {
bool HasAVX512BF16 = false;
bool HasAVX512DQ = false;
bool HasAVX512BITALG = false;
+ bool HasAVX512BMM = false;
bool HasAVX512BW = false;
bool HasAVX512VL = false;
bool HasAVX512VBMI = false;
diff --git a/clang/lib/CodeGen/TargetBuiltins/X86.cpp b/clang/lib/CodeGen/TargetBuiltins/X86.cpp
index 9645ed87b8ef3..2c4e1f0cc8b17 100644
--- a/clang/lib/CodeGen/TargetBuiltins/X86.cpp
+++ b/clang/lib/CodeGen/TargetBuiltins/X86.cpp
@@ -2678,6 +2678,49 @@ Value *CodeGenFunction::EmitX86BuiltinExpr(unsigned BuiltinID,
return EmitX86MaskedCompareResult(*this, Shufbit, NumElts, MaskIn);
}
+ case X86::BI__builtin_ia32_bitrev128:
+ case X86::BI__builtin_ia32_bitrev256:
+ case X86::BI__builtin_ia32_bitrev512: {
+ Intrinsic::ID ID;
+ switch (BuiltinID) {
+ default: llvm_unreachable("Unsupported intrinsic!");
+ case X86::BI__builtin_ia32_bitrev128:
+ ID = Intrinsic::x86_avx512_vbitrevb_128;
+ break;
+ case X86::BI__builtin_ia32_bitrev256:
+ ID = Intrinsic::x86_avx512_vbitrevb_256;
+ break;
+ case X86::BI__builtin_ia32_bitrev512:
+ ID = Intrinsic::x86_avx512_vbitrevb_512;
+ break;
+ }
+
+ return Builder.CreateCall(CGM.getIntrinsic(ID), Ops);
+ }
+
+ case X86::BI__builtin_ia32_bmacor16x16x16_v16hi:
+ case X86::BI__builtin_ia32_bmacor16x16x16_v32hi:
+ case X86::BI__builtin_ia32_bmacxor16x16x16_v16hi:
+ case X86::BI__builtin_ia32_bmacxor16x16x16_v32hi: {
+ Intrinsic::ID ID;
+ switch (BuiltinID) {
+ default: llvm_unreachable("Unsupported intrinsic!");
+ case X86::BI__builtin_ia32_bmacor16x16x16_v16hi:
+ ID = Intrinsic::x86_avx512_vbmacor_v16hi;
+ break;
+ case X86::BI__builtin_ia32_bmacor16x16x16_v32hi:
+ ID = Intrinsic::x86_avx512_vbmacor_v32hi;
+ break;
+ case X86::BI__builtin_ia32_bmacxor16x16x16_v16hi:
+ ID = Intrinsic::x86_avx512_vbmacxor_v16hi;
+ break;
+ case X86::BI__builtin_ia32_bmacxor16x16x16_v32hi:
+ ID = Intrinsic::x86_avx512_vbmacxor_v32hi;
+ break;
+ }
+
+ return Builder.CreateCall(CGM.getIntrinsic(ID), Ops);
+ }
// packed comparison intrinsics
case X86::BI__builtin_ia32_cmpeqps:
case X86::BI__builtin_ia32_cmpeqpd:
diff --git a/clang/lib/Headers/CMakeLists.txt b/clang/lib/Headers/CMakeLists.txt
index c92b370b88d2d..3686080c1d6cf 100644
--- a/clang/lib/Headers/CMakeLists.txt
+++ b/clang/lib/Headers/CMakeLists.txt
@@ -186,6 +186,8 @@ set(x86_files
avx2intrin.h
avx512bf16intrin.h
avx512bitalgintrin.h
+ avx512bmmintrin.h
+ avx512bmmvlintrin.h
avx512bwintrin.h
avx512cdintrin.h
avx512dqintrin.h
diff --git a/clang/lib/Headers/avx512bmmintrin.h b/clang/lib/Headers/avx512bmmintrin.h
new file mode 100644
index 0000000000000..d39106f9c2276
--- /dev/null
+++ b/clang/lib/Headers/avx512bmmintrin.h
@@ -0,0 +1,63 @@
+/*===-------- avx512bmmintrin.h - AVX512BMM intrinsics *------------------===
+ *
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===---------------------------------------------------------------------===
+ */
+
+#ifndef __IMMINTRIN_H
+#error "Never use <avx512bmmintrin.h> directly; include <immintrin.h> instead."
+#endif
+
+#ifndef _AVX512BMMINTRIN_H
+#define _AVX512BMMINTRIN_H
+
+/* Define the default attributes for the functions in this file. */
+#define __DEFAULT_FN_ATTRS \
+ __attribute__((__always_inline__, __nodebug__, __target__("avx512bmm"), \
+ __min_vector_width__(512)))
+
+#if defined(__cplusplus) && (__cplusplus >= 201103L)
+#define __DEFAULT_FN_ATTRS_CONSTEXPR __DEFAULT_FN_ATTRS constexpr
+#else
+#define __DEFAULT_FN_ATTRS_CONSTEXPR __DEFAULT_FN_ATTRS
+#endif
+
+static __inline __m512i __DEFAULT_FN_ATTRS _mm512_bmacor16x16x16(__m512i __A,
+ __m512i __B,
+ __m512i __C) {
+ return (__m512i)__builtin_ia32_bmacor16x16x16_v32hi(
+ (__v32hi)__A, (__v32hi)__B, (__v32hi)__C);
+}
+
+static __inline __m512i __DEFAULT_FN_ATTRS _mm512_bmacxor16x16x16(__m512i __A,
+ __m512i __B,
+ __m512i __C) {
+ return (__m512i)__builtin_ia32_bmacxor16x16x16_v32hi(
+ (__v32hi)__A, (__v32hi)__B, (__v32hi)__C);
+}
+
+static __inline __m512i __DEFAULT_FN_ATTRS _mm512_bitrev_epi8(__m512i __A) {
+ return (__m512i)__builtin_ia32_bitrev512((__v64qi)__A);
+}
+
+static __inline __m512i __DEFAULT_FN_ATTRS
+_mm512_mask_bitrev_epi8(__mmask64 __U, __m512i __A, __m512i __B) {
+ return (__m512i)__builtin_ia32_selectb_512(
+ (__mmask64)__U, (__v64qi)_mm512_bitrev_epi8(__A), (__v64qi)__B);
+}
+
+static __inline __m512i __DEFAULT_FN_ATTRS
+_mm512_maskz_bitrev_epi8(__mmask64 __U, __m512i __A) {
+ return (__m512i)__builtin_ia32_selectb_512((__mmask64)__U,
+ (__v64qi)_mm512_bitrev_epi8(__A),
+ (__v64qi)_mm512_setzero_si512());
+}
+
+#undef __DEFAULT_FN_ATTRS
+#undef __DEFAULT_FN_ATTRS_CONSTEXPR
+
+#endif
diff --git a/clang/lib/Headers/avx512bmmvlintrin.h b/clang/lib/Headers/avx512bmmvlintrin.h
new file mode 100644
index 0000000000000..86a1f2ab410d0
--- /dev/null
+++ b/clang/lib/Headers/avx512bmmvlintrin.h
@@ -0,0 +1,85 @@
+/*===------------- avx512bmvlintrin.h - BMM intrinsics ------------------===
+ *
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===-----------------------------------------------------------------------===
+ */
+#ifndef __IMMINTRIN_H
+#error \
+ "Never use <avx512bmmvlintrin.h> directly; include <immintrin.h> instead."
+#endif
+
+#ifndef __BMMVLINTRIN_H
+#define __BMMVLINTRIN_H
+
+/* Define the default attributes for the functions in this file. */
+#define __DEFAULT_FN_ATTRS128 \
+ __attribute__((__always_inline__, __nodebug__, \
+ __target__("avx512bmm,avx512vl"), __min_vector_width__(128)))
+#define __DEFAULT_FN_ATTRS256 \
+ __attribute__((__always_inline__, __nodebug__, \
+ __target__("avx512bmm,avx512vl"), __min_vector_width__(256)))
+
+#if defined(__cplusplus) && (__cplusplus >= 201103L)
+#define __DEFAULT_FN_ATTRS128_CONSTEXPR __DEFAULT_FN_ATTRS128 constexpr
+#define __DEFAULT_FN_ATTRS256_CONSTEXPR __DEFAULT_FN_ATTRS256 constexpr
+#else
+#define __DEFAULT_FN_ATTRS128_CONSTEXPR __DEFAULT_FN_ATTRS128
+#define __DEFAULT_FN_ATTRS256_CONSTEXPR __DEFAULT_FN_ATTRS256
+#endif
+
+static __inline __m256i __DEFAULT_FN_ATTRS256
+_mm256_bmacor16x16x16(__m256i __A, __m256i __B, __m256i __C) {
+ return (__m256i)__builtin_ia32_bmacor16x16x16_v16hi(
+ (__v16hi)__A, (__v16hi)__B, (__v16hi)__C);
+}
+
+static __inline __m256i __DEFAULT_FN_ATTRS256
+_mm256_bmacxor16x16x16(__m256i __A, __m256i __B, __m256i __C) {
+ return (__m256i)__builtin_ia32_bmacxor16x16x16_v16hi(
+ (__v16hi)__A, (__v16hi)__B, (__v16hi)__C);
+}
+
+static __inline __m128i __DEFAULT_FN_ATTRS128 _mm128_bitrev_epi8(__m128i __A) {
+ return (__m128i)__builtin_ia32_bitrev128((__v16qi)__A);
+}
+
+static __inline __m256i __DEFAULT_FN_ATTRS256 _mm256_bitrev_epi8(__m256i __A) {
+ return (__m256i)__builtin_ia32_bitrev256((__v32qi)__A);
+}
+
+static __inline __m128i __DEFAULT_FN_ATTRS128
+_mm128_mask_bitrev_epi8(__mmask16 __U, __m128i __A, __m128i __B) {
+ return (__m128i)__builtin_ia32_selectb_128(
+ (__mmask16)__U, (__v16qi)_mm128_bitrev_epi8(__A), (__v16qi)__B);
+}
+
+static __inline __m256i __DEFAULT_FN_ATTRS256
+_mm256_mask_bitrev_epi8(__mmask32 __U, __m256i __A, __m256i __B) {
+ return (__m256i)__builtin_ia32_selectb_256(
+ (__mmask32)__U, (__v32qi)_mm256_bitrev_epi8(__A), (__v32qi)__B);
+}
+
+static __inline __m128i __DEFAULT_FN_ATTRS128
+_mm128_maskz_bitrev_epi8(__mmask16 __U, __m128i __A) {
+ return (__m128i)__builtin_ia32_selectb_128((__mmask16)__U,
+ (__v16qi)_mm128_bitrev_epi8(__A),
+ (__v16qi)_mm_setzero_si128());
+}
+
+static __inline __m256i __DEFAULT_FN_ATTRS256
+_mm256_maskz_bitrev_epi8(__mmask32 __U, __m256i __A) {
+ return (__m256i)__builtin_ia32_selectb_256((__mmask32)__U,
+ (__v32qi)_mm256_bitrev_epi8(__A),
+ (__v32qi)_mm256_setzero_si256());
+}
+
+#undef __DEFAULT_FN_ATTRS128_CONSTEXPR
+#undef __DEFAULT_FN_ATTRS256_CONSTEXPR
+#undef __DEFAULT_FN_ATTRS128
+#undef __DEFAULT_FN_ATTRS256
+
+#endif
diff --git a/clang/lib/Headers/immintrin.h b/clang/lib/Headers/immintrin.h
index 19064a4ff5cea..00107c44c3a55 100644
--- a/clang/lib/Headers/immintrin.h
+++ b/clang/lib/Headers/immintrin.h
@@ -58,6 +58,10 @@
#include <avx512bitalgintrin.h>
+#include <avx512bmmintrin.h>
+
+#include <avx512bmmvlintrin.h>
+
#include <avx512cdintrin.h>
#include <avx512vpopcntdqintrin.h>
diff --git a/clang/test/CodeGen/attr-target-x86.c b/clang/test/CodeGen/attr-target-x86.c
index 474fa93629d89..6a110ce38605b 100644
--- a/clang/test/CodeGen/attr-target-x86.c
+++ b/clang/test/CodeGen/attr-target-x86.c
@@ -33,7 +33,7 @@ __attribute__((target("fpmath=387")))
void f_fpmath_387(void) {}
// CHECK-NOT: tune-cpu
-// CHECK: [[f_no_sse2]] = {{.*}}"target-cpu"="i686" "target-features"="+cmov,+cx8,+x87,-aes,-amx-avx512,-avx,-avx10.1,-avx10.2,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512f,-avx512fp16,-avx512ifma,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxifma,-avxneconvert,-avxvnni,-avxvnniint16,-avxvnniint8,-f16c,-fma,-fma4,-gfni,-kl,-pclmul,-sha,-sha512,-sm3,-sm4,-sse2,-sse3,-sse4.1,-sse4.2,-sse4a,-ssse3,-vaes,-vpclmulqdq,-widekl,-xop" "tune-cpu"="i686"
+// CHECK: [[f_no_sse2]] = {{.*}}"target-cpu"="i686" "target-features"="+cmov,+cx8,+x87,-aes,-amx-avx512,-avx,-avx10.1,-avx10.2,-avx2,-avx512bf16,-avx512bitalg,-avx512bmm,-avx512bw,-avx512cd,-avx512dq,-avx512f,-avx512fp16,-avx512ifma,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxifma,-avxneconvert,-avxvnni,-avxvnniint16,-avxvnniint8,-f16c,-fma,-fma4,-gfni,-kl,-pclmul,-sha,-sha512,-sm3,-sm4,-sse2,-sse3,-sse4.1,-sse4.2,-sse4a,-ssse3,-vaes,-vpclmulqdq,-widekl,-xop" "tune-cpu"="i686"
__attribute__((target("no-sse2")))
void f_no_sse2(void) {}
@@ -41,7 +41,7 @@ void f_no_sse2(void) {}
__attribute__((target("sse4")))
void f_sse4(void) {}
-// CHECK: [[f_no_sse4]] = {{.*}}"target-cpu"="i686" "target-features"="+cmov,+cx8,+x87,-amx-avx512,-avx,-avx10.1,-avx10.2,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512f,-avx512fp16,-avx512ifma,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxifma,-avxneconvert,-avxvnni,-avxvnniint16,-avxvnniint8,-f16c,-fma,-fma4,-sha512,-sm3,-sm4,-sse4.1,-sse4.2,-vaes,-vpclmulqdq,-...
[truncated]
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
efriedma-quic
left a comment
efriedma-quic
left a comment
Contributor
There was a problem hiding this comment.
Please split AVX512BMM into a separate patch from the Zen6-specific bits.
RKSimon
requested changes
Feb 2, 2026
RKSimon
left a comment
RKSimon
left a comment
Contributor
There was a problem hiding this comment.
Mainly minor tidyup comments - it'd be much better if the BMM work was in a separate patch to the znver6 work - some BMM ref docs would be good as all I have to go off is the gcc patches. We have enough ISD::BITREVERSE functionality now that I don't see any need for the BITREV builtins/intrinsics.
| } | ||
|
|
||
| static __inline __m512i __DEFAULT_FN_ATTRS _mm512_bitrev_epi8(__m512i __A) { | ||
| return (__m512i)__builtin_ia32_bitrev512((__v64qi)__A); |
Contributor
There was a problem hiding this comment.
can you just use __builtin_elementwise_bitreverse ? This would also get you constexpr support for free (inc the mask variants) :)
| #define __DEFAULT_FN_ATTRS_CONSTEXPR __DEFAULT_FN_ATTRS | ||
| #endif | ||
|
|
||
| static __inline __m512i __DEFAULT_FN_ATTRS _mm512_bmacor16x16x16(__m512i __A, |
Contributor
There was a problem hiding this comment.
ideally we need doxygen comments for new intrinsics - but this isn't well enforced
| } | ||
|
|
||
| static __inline __m128i __DEFAULT_FN_ATTRS128 _mm128_bitrev_epi8(__m128i __A) { | ||
| return (__m128i)__builtin_ia32_bitrev128((__v16qi)__A); |
Contributor
There was a problem hiding this comment.
__builtin_elementwise_bitreverse
| } | ||
|
|
||
| static __inline __m256i __DEFAULT_FN_ATTRS256 _mm256_bitrev_epi8(__m256i __A) { | ||
| return (__m256i)__builtin_ia32_bitrev256((__v32qi)__A); |
Contributor
There was a problem hiding this comment.
__builtin_elementwise_bitreverse
| @@ -27,6 +27,7 @@ | |||
|
|
|||
| #if (defined(__GNUC__) || defined(__clang__)) && !defined(_MSC_VER) | |||
| #include <cpuid.h> | |||
| #include <stdio.h> | |||
|
|
||
| // BMM matrix multiplication operations | ||
| def x86vbmacor : SDNode<"X86ISD::VBMACOR", SDTVnni>; | ||
| def x86vbmacxor : SDNode<"X86ISD::VBMACXOR", SDTVnni>; |
| @@ -1518,3 +1518,9 @@ def X86vpmaddwd_su : PatFrag<(ops node:$lhs, node:$rhs), | |||
| return N->hasOneUse(); | |||
| }]>; | |||
|
|
|||
| // Bit reverse bytes operation | |||
| def x86vpbitrevb : SDNode<"X86ISD::VPBITREVB", SDTShuff1Op>; | |||
Contributor
There was a problem hiding this comment.
I really hope we can avoid this and use ISD::BIREVERSE legal ops directly
| @@ -1349,7 +1355,7 @@ static const char *getAMDProcessorTypeAndSubtype(unsigned Family, | |||
| return CPU; | |||
| } | |||
|
|
|||
| #undef testFeature | |||
| #undef testFeaturemake | |||
|
|
||
| let Predicates = [HasBMM] in | ||
| defm VBMACXORZ : VNNI_rmb<0x80, "vbmacxor16x16x16", x86vbmacxor, SchedWriteVecIMul.ZMM, v32i16_info, 0>, | ||
| EVEX_V512, T_MAP6, REX_W; |
| // If we have BMM, we can use VPBITREVB to reverse the bits in each byte. | ||
| // BMM implies VLX support, so this works for 128/256/512-bit vectors. | ||
| if (Subtarget.hasBMM() && VT.getSizeInBits() >= 128) { | ||
| return DAG.getNode(X86ISD::VPBITREVB, DL, VT, In); |
Contributor
There was a problem hiding this comment.
Hopefully we can remove X86ISD::VPBITREVB and just make ISD::BITREVERSE legal for v64i8 BMM + v16i8/v32i8 on BMM+VLX
Contributor
Author
There was a problem hiding this comment.
Yes will do it. Will split the patch into two.
- For znver6 skeleton enablement
- BMM removing BITREVB and using the generics.
Contributor
|
@ganeshgit Are you hoping to backport this to 22.x? |
Contributor
Author
Agreed. Will split the patch into two. The BMM doc is available as https://docs.amd.com/v/u/en-US/69192-PUB. BITREVERSE includes all integer types whereas BITREVB is byte granular. So, thought of having it separately. |
Contributor
Author
I will leave it knowledgeable approvers to take a call on it. Have included @tru here who approved znver5 backport. If it is approved, I would definitely like to have it part of 22.x as well. |
Contributor
I don't think thats necessary - SSE/AVX already have plenty of intrinsics that wrap generics but are only available for some integer types (multiply and add/sub saturation are examples that come to mind but there's plenty of others) - the more we avoid target specifics the better imo. |
Contributor
Soft -1 on backporting this. It's a feature not a bug fix and contains a relatively large diff at that which makes merge conflicts more likely. |
boomanaiden154
left a comment
boomanaiden154
left a comment
Contributor
There was a problem hiding this comment.
Premerge failures are relevant, but it looks like they're specific to the AVX512BMM changes.
boomanaiden154
left a comment
boomanaiden154
left a comment
Contributor
There was a problem hiding this comment.
It looks like a lot of the llvm tests that are modified in this patch just have redundant check-lines added.
| // Models 60h-6Fh (Krackan1). | ||
| // Models 70h-77h (Sarlak). | ||
| if (Model <= 0x4f || (Model >= 0x60 && Model <= 0x77) || | ||
| (Model >= 0xd0 && Model <= 0xd7)) { |
Contributor
There was a problem hiding this comment.
Why are we removing the model comments here?
Contributor
Author
There was a problem hiding this comment.
Weisshorn is excluded and we have few models included as well. The model numbers are broader now.
Contributor
There was a problem hiding this comment.
So update the comments to match Host.cpp?
| @@ -9,7 +9,7 @@ | |||
| // This file defines the machine model for Znver4 to support instruction | |||
| // scheduling and other instruction cost heuristics. | |||
| // Based on: | |||
| // * AMD Software Optimization Guide for the AMD Family 19h (Zen4) | |||
| // * AMD Software Optimization Guide for the AMD Family 19h (Zen4) | |||
Contributor
There was a problem hiding this comment.
All the changes in this file look like drive-by formatting changes that should probably be reverted?
Contributor
Author
There was a problem hiding this comment.
The model is related to the patch as we are using it for the initial enablement. I think we can retain this.
Contributor
There was a problem hiding this comment.
these changes are purely trailing whitespace - best not to include the change in this patch.
| @@ -1,6 +1,7 @@ | |||
| ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 3 | |||
| ; RUN: opt -passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mcpu=znver4 -S < %s | FileCheck %s | |||
| ; RUN: opt -passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mcpu=znver5 -S < %s | FileCheck %s | |||
| ; RUN: opt -passes=slp-vectorizer -mtriple=x86_64-unknown-linux-gnu -mcpu=znver6 -S < %s | FileCheck %s | |||
Contributor
There was a problem hiding this comment.
Do we need to test znver6 here? This looks like a regression test for behavior that only occurred with znver4 (probably cost model related). I don't think we need to cover all new zen architectures here.
| @@ -2,6 +2,7 @@ | |||
| ; RUN: opt -passes=debugify,loop-unroll -mcpu=znver3 -pass-remarks=TTI -pass-remarks-analysis=TTI < %s -S 2>&1 | FileCheck --check-prefixes=ALL,TTI %s | |||
| ; RUN: opt -passes=debugify,loop-unroll -mcpu=znver4 -pass-remarks=loop-unroll -pass-remarks-analysis=loop-unroll < %s -S 2>&1 | FileCheck --check-prefixes=ALL,UNROLL %s | |||
| ; RUN: opt -passes=debugify,loop-unroll -mcpu=znver5 -pass-remarks=loop-unroll -pass-remarks-analysis=loop-unroll < %s -S 2>&1 | FileCheck --check-prefixes=ALL,UNROLL %s | |||
| ; RUN: opt -passes=debugify,loop-unroll -mcpu=znver6 -pass-remarks=loop-unroll -pass-remarks-analysis=loop-unroll < %s -S 2>&1 | FileCheck --check-prefixes=ALL,UNROLL %s | |||
Contributor
There was a problem hiding this comment.
Do we need to test znver6 here?
ganeshgit
force-pushed
the
znver6
branch
2 times, most recently
from
February 3, 2026 12:27
1b1dde2 to
58a5853
Compare
boomanaiden154
left a comment
boomanaiden154
left a comment
Contributor
There was a problem hiding this comment.
A couple of nits, but I think this broadly LGTM at this point. Please update the description to remove information about AVX512BMM now that has been split to a separate patch.
| // Models 50h-57h (Weisshorn). | ||
| // Models 90h-AFh (Weisshorn). | ||
| // Models C0h-CFh (Weisshorn). | ||
| // |
Contributor
There was a problem hiding this comment.
Empty line should probably be removed?
| // Models 50h-57h (Weisshorn). | ||
| // Models 90h-AFh (Weisshorn). | ||
| // Models C0h-CFh (Weisshorn). | ||
| // |
Contributor
There was a problem hiding this comment.
Same here regarding the empty line.
Contributor
Author
There was a problem hiding this comment.
This was intentional to have a split between same codename and others. I have removed the line now.
boomanaiden154
approved these changes
Feb 4, 2026
boomanaiden154
left a comment
boomanaiden154
left a comment
Contributor
There was a problem hiding this comment.
LGTM now. Please get a review from @RKSimon before landing though.
Contributor
Author
This patch adds initial support for AMD Zen 6 architecture (znver6): - Added znver6 CPU target recognition in Clang and LLVM - Added znver6 to target parser and host CPU detection - Updated scheduling models to inherit from znver4 - Added znver6 to various optimizer tests - Updated compiler-rt CPU model detection for znver6 znver6 features: FP16, AVXVNNIINT8, AVXNECONVERT, AVXIFMA (without BMM).
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/66/builds/26096 Here is the relevant piece of the build log for the reference |
| @@ -105,6 +105,7 @@ enum ProcessorSubtypes { | |||
| INTEL_COREI7_ARROWLAKE_S, | |||
| INTEL_COREI7_PANTHERLAKE, | |||
| AMDFAM1AH_ZNVER5, | |||
| AMDFAM1AH_ZNVER6, | |||
Member
There was a problem hiding this comment.
Values in this enum have to be synced with analogous enum in libgcc. In GCC (gcc/common/config/i386/i386-cpuinfo.h) in enum processor_subtypes AMDFAM1AH_ZNVER6 has different value than here (it's after novalake). Entry in X86TargetParser.def would need to be adjusted to match the one here
The table is out-of-sync between gcc and llvm anyway, I have a PR #171172 to sync it and make it more robust, I hope to get around to having it merged soon
rishabhmadan19
pushed a commit
to rishabhmadan19/llvm-project
that referenced
this pull request
Feb 9, 2026
This patch adds initial support for AMD Zen 6 architecture (znver6): - Added znver6 CPU target recognition in Clang and LLVM - Updated compiler-rt CPU model detection for znver6 - Added znver6 to target parser and host CPU detection - Added znver6 to various optimizer tests znver6 features: FP16, AVXVNNIINT8, AVXNECONVERT, AVXIFMA (without BMM).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch adds initial support for AMD Zen 6 architecture (znver6):
znver6 features: FP16, AVXVNNIINT8, AVXNECONVERT, AVXIFMA (without BMM).