-
Notifications
You must be signed in to change notification settings - Fork 15.9k
[CIR] Add math and builtin intrinsics support #175233
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@llvm/pr-subscribers-clang @llvm/pr-subscribers-clangir Author: None (adams381) ChangesThis PR adds support for various math and builtin intrinsics to CIR: Changes
All changes include CIR, LLVM lowering, and OGCG test checks to verify correctness. Patch is 134.99 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/175233.diff 12 Files Affected:
diff --git a/clang/include/clang/CIR/Dialect/IR/CIROps.td b/clang/include/clang/CIR/Dialect/IR/CIROps.td
index 8358b076ee7b6..5ac64585f000f 100644
--- a/clang/include/clang/CIR/Dialect/IR/CIROps.td
+++ b/clang/include/clang/CIR/Dialect/IR/CIROps.td
@@ -5096,6 +5096,106 @@ def CIR_Exp2Op : CIR_UnaryFPToFPBuiltinOp<"exp2", "Exp2Op"> {
}];
}
+def CIR_LogOp : CIR_UnaryFPToFPBuiltinOp<"log", "LogOp"> {
+ let summary = "Computes the floating-point natural logarithm";
+ let description = [{
+ `cir.log` computes the natural logarithm of a floating-point operand and
+ returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_Log10Op : CIR_UnaryFPToFPBuiltinOp<"log10", "Log10Op"> {
+ let summary = "Computes the floating-point base-10 logarithm";
+ let description = [{
+ `cir.log10` computes the base-10 logarithm of a floating-point operand and
+ returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_Log2Op : CIR_UnaryFPToFPBuiltinOp<"log2", "Log2Op"> {
+ let summary = "Computes the floating-point base-2 logarithm";
+ let description = [{
+ `cir.log2` computes the base-2 logarithm of a floating-point operand and
+ returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_NearbyintOp : CIR_UnaryFPToFPBuiltinOp<"nearbyint", "NearbyintOp"> {
+ let summary = "Rounds floating-point value to nearest integer";
+ let description = [{
+ `cir.nearbyint` rounds a floating-point operand to the nearest integer value
+ and returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_RintOp : CIR_UnaryFPToFPBuiltinOp<"rint", "RintOp"> {
+ let summary = "Rounds floating-point value to nearest integer";
+ let description = [{
+ `cir.rint` rounds a floating-point operand to the nearest integer value
+ and returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_RoundOp : CIR_UnaryFPToFPBuiltinOp<"round", "RoundOp"> {
+ let summary = "Rounds floating-point value to nearest integer";
+ let description = [{
+ `cir.round` rounds a floating-point operand to the nearest integer value
+ and returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_RoundEvenOp : CIR_UnaryFPToFPBuiltinOp<"roundeven", "RoundEvenOp"> {
+ let summary = "Rounds floating-point value to nearest integer, ties to even";
+ let description = [{
+ `cir.roundeven` rounds a floating-point operand to the nearest integer
+ value, with ties rounding to even (banker's rounding).
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_SinOp : CIR_UnaryFPToFPBuiltinOp<"sin", "SinOp"> {
+ let summary = "Computes the floating-point sine";
+ let description = [{
+ `cir.sin` computes the sine of a floating-point operand and returns
+ a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_TanOp : CIR_UnaryFPToFPBuiltinOp<"tan", "TanOp"> {
+ let summary = "Computes the floating-point tangent";
+ let description = [{
+ `cir.tan` computes the tangent of a floating-point operand and returns
+ a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
+def CIR_TruncOp : CIR_UnaryFPToFPBuiltinOp<"trunc", "TruncOp"> {
+ let summary = "Truncates floating-point value to integer";
+ let description = [{
+ `cir.trunc` truncates a floating-point operand to an integer value
+ and returns a result of the same type.
+
+ Floating-point exceptions are ignored, and it does not set `errno`.
+ }];
+}
+
def CIR_FAbsOp : CIR_UnaryFPToFPBuiltinOp<"fabs", "FAbsOp"> {
let summary = "Computes the floating-point absolute value";
let description = [{
@@ -5106,6 +5206,34 @@ def CIR_FAbsOp : CIR_UnaryFPToFPBuiltinOp<"fabs", "FAbsOp"> {
}];
}
+def CIR_AbsOp : CIR_Op<"abs", [Pure, SameOperandsAndResultType]> {
+ let summary = "Computes the absolute value of a signed integer";
+ let description = [{
+ `cir.abs` computes the absolute value of a signed integer or vector
+ of signed integers.
+
+ The `poison` attribute indicates whether the result value is a poison
+ value if the argument is statically or dynamically INT_MIN.
+
+ Example:
+
+ ```mlir
+ %0 = cir.const #cir.int<-42> : s32i
+ %1 = cir.abs %0 poison : s32i
+ %2 = cir.abs %3 : !cir.vector<!s32i x 4>
+ ```
+ }];
+
+ let arguments = (ins
+ CIR_AnySIntOrVecOfSIntType:$src,
+ UnitAttr:$poison
+ );
+
+ let results = (outs CIR_AnySIntOrVecOfSIntType:$result);
+
+ let assemblyFormat = "$src ( `poison` $poison^ )? `:` type($src) attr-dict";
+}
+
def CIR_FloorOp : CIR_UnaryFPToFPBuiltinOp<"floor", "FloorOp"> {
let summary = "Computes the floating-point floor value";
let description = [{
@@ -5123,6 +5251,58 @@ def CIR_FloorOp : CIR_UnaryFPToFPBuiltinOp<"floor", "FloorOp"> {
}];
}
+class CIR_UnaryFPToIntBuiltinOp<string mnemonic, string llvmOpName>
+ : CIR_Op<mnemonic, [Pure]>
+{
+ let arguments = (ins CIR_AnyFloatType:$src);
+ let results = (outs CIR_IntType:$result);
+
+ let summary = [{
+ Builtin function that takes a floating-point value as input and produces an
+ integral value as output.
+ }];
+
+ let assemblyFormat = [{
+ $src `:` type($src) `->` type($result) attr-dict
+ }];
+
+ let llvmOp = llvmOpName;
+}
+
+def CIR_LroundOp : CIR_UnaryFPToIntBuiltinOp<"lround", "LroundOp">;
+def CIR_LLroundOp : CIR_UnaryFPToIntBuiltinOp<"llround", "LlroundOp">;
+def CIR_LrintOp : CIR_UnaryFPToIntBuiltinOp<"lrint", "LrintOp">;
+def CIR_LLrintOp : CIR_UnaryFPToIntBuiltinOp<"llrint", "LlrintOp">;
+
+class CIR_BinaryFPToFPBuiltinOp<string mnemonic, string llvmOpName>
+ : CIR_Op<mnemonic, [Pure, SameOperandsAndResultType]> {
+ let summary = [{
+ libc builtin equivalent ignoring floating-point exceptions and errno.
+ }];
+
+ let arguments = (ins
+ CIR_AnyFloatOrVecOfFloatType:$lhs,
+ CIR_AnyFloatOrVecOfFloatType:$rhs
+ );
+
+ let results = (outs CIR_AnyFloatOrVecOfFloatType:$result);
+
+ let assemblyFormat = [{
+ $lhs `,` $rhs `:` qualified(type($lhs)) attr-dict
+ }];
+
+ let llvmOp = llvmOpName;
+}
+
+def CIR_CopysignOp : CIR_BinaryFPToFPBuiltinOp<"copysign", "CopySignOp">;
+def CIR_FMaxNumOp : CIR_BinaryFPToFPBuiltinOp<"fmaxnum", "MaxNumOp">;
+def CIR_FMaximumOp : CIR_BinaryFPToFPBuiltinOp<"fmaximum", "MaximumOp">;
+def CIR_FMinNumOp : CIR_BinaryFPToFPBuiltinOp<"fminnum", "MinNumOp">;
+def CIR_FMinimumOp : CIR_BinaryFPToFPBuiltinOp<"fminimum", "MinimumOp">;
+def CIR_FModOp : CIR_BinaryFPToFPBuiltinOp<"fmod", "FRemOp">;
+def CIR_PowOp : CIR_BinaryFPToFPBuiltinOp<"pow", "PowOp">;
+def CIR_ATan2Op : CIR_BinaryFPToFPBuiltinOp<"atan2", "ATan2Op">;
+
//===----------------------------------------------------------------------===//
// Variadic Operations
//===----------------------------------------------------------------------===//
diff --git a/clang/lib/CIR/CodeGen/CIRGenBuiltin.cpp b/clang/lib/CIR/CodeGen/CIRGenBuiltin.cpp
index 85406e9f6488a..0f30de30bf405 100644
--- a/clang/lib/CIR/CodeGen/CIRGenBuiltin.cpp
+++ b/clang/lib/CIR/CodeGen/CIRGenBuiltin.cpp
@@ -150,6 +150,45 @@ static RValue emitUnaryFPBuiltin(CIRGenFunction &cgf, const CallExpr &e) {
return RValue::get(call->getResult(0));
}
+template <typename Op>
+static RValue emitUnaryMaybeConstrainedFPToIntBuiltin(CIRGenFunction &cgf,
+ const CallExpr &e) {
+ mlir::Type resultType = cgf.convertType(e.getType());
+ mlir::Value src = cgf.emitScalarExpr(e.getArg(0));
+
+ assert(!cir::MissingFeatures::fpConstraints());
+
+ auto call = Op::create(cgf.getBuilder(), src.getLoc(), resultType, src);
+ return RValue::get(call->getResult(0));
+}
+
+template <typename Op>
+static RValue emitBinaryFPBuiltin(CIRGenFunction &cgf, const CallExpr &e) {
+ mlir::Value arg0 = cgf.emitScalarExpr(e.getArg(0));
+ mlir::Value arg1 = cgf.emitScalarExpr(e.getArg(1));
+
+ mlir::Location loc = cgf.getLoc(e.getExprLoc());
+ mlir::Type ty = cgf.convertType(e.getType());
+ auto call = Op::create(cgf.getBuilder(), loc, ty, arg0, arg1);
+
+ return RValue::get(call->getResult(0));
+}
+
+template <typename Op>
+static mlir::Value emitBinaryMaybeConstrainedFPBuiltin(CIRGenFunction &cgf,
+ const CallExpr &e) {
+ mlir::Value arg0 = cgf.emitScalarExpr(e.getArg(0));
+ mlir::Value arg1 = cgf.emitScalarExpr(e.getArg(1));
+
+ mlir::Location loc = cgf.getLoc(e.getExprLoc());
+ mlir::Type ty = cgf.convertType(e.getType());
+
+ assert(!cir::MissingFeatures::fpConstraints());
+
+ auto call = Op::create(cgf.getBuilder(), loc, ty, arg0, arg1);
+ return call->getResult(0);
+}
+
static RValue errorBuiltinNYI(CIRGenFunction &cgf, const CallExpr *e,
unsigned builtinID) {
@@ -263,6 +302,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_acosl:
case Builtin::BI__builtin_acosf128:
case Builtin::BI__builtin_elementwise_acos:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::ACosOp>(cgf, *e);
case Builtin::BIasin:
case Builtin::BIasinf:
case Builtin::BIasinl:
@@ -272,6 +312,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_asinl:
case Builtin::BI__builtin_asinf128:
case Builtin::BI__builtin_elementwise_asin:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::ASinOp>(cgf, *e);
case Builtin::BIatan:
case Builtin::BIatanf:
case Builtin::BIatanl:
@@ -281,6 +322,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_atanl:
case Builtin::BI__builtin_atanf128:
case Builtin::BI__builtin_elementwise_atan:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::ATanOp>(cgf, *e);
case Builtin::BIatan2:
case Builtin::BIatan2f:
case Builtin::BIatan2l:
@@ -290,7 +332,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_atan2l:
case Builtin::BI__builtin_atan2f128:
case Builtin::BI__builtin_elementwise_atan2:
- return RValue::getIgnored();
+ return emitBinaryFPBuiltin<cir::ATan2Op>(cgf, *e);
case Builtin::BIceil:
case Builtin::BIceilf:
case Builtin::BIceill:
@@ -301,6 +343,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_ceilf128:
return emitUnaryMaybeConstrainedFPBuiltin<cir::CeilOp>(cgf, *e);
case Builtin::BI__builtin_elementwise_ceil:
+ return RValue::getIgnored();
case Builtin::BIcopysign:
case Builtin::BIcopysignf:
case Builtin::BIcopysignl:
@@ -309,7 +352,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_copysignf16:
case Builtin::BI__builtin_copysignl:
case Builtin::BI__builtin_copysignf128:
- return RValue::getIgnored();
+ return emitBinaryFPBuiltin<cir::CopysignOp>(cgf, *e);
case Builtin::BIcos:
case Builtin::BIcosf:
case Builtin::BIcosl:
@@ -386,6 +429,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_fmal:
case Builtin::BI__builtin_fmaf128:
case Builtin::BI__builtin_elementwise_fma:
+ return RValue::getIgnored();
case Builtin::BIfmax:
case Builtin::BIfmaxf:
case Builtin::BIfmaxl:
@@ -394,6 +438,8 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_fmaxf16:
case Builtin::BI__builtin_fmaxl:
case Builtin::BI__builtin_fmaxf128:
+ return RValue::get(
+ emitBinaryMaybeConstrainedFPBuiltin<cir::FMaxNumOp>(cgf, *e));
case Builtin::BIfmin:
case Builtin::BIfminf:
case Builtin::BIfminl:
@@ -402,6 +448,8 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_fminf16:
case Builtin::BI__builtin_fminl:
case Builtin::BI__builtin_fminf128:
+ return RValue::get(
+ emitBinaryMaybeConstrainedFPBuiltin<cir::FMinNumOp>(cgf, *e));
case Builtin::BIfmaximum_num:
case Builtin::BIfmaximum_numf:
case Builtin::BIfmaximum_numl:
@@ -418,6 +466,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_fminimum_numf16:
case Builtin::BI__builtin_fminimum_numl:
case Builtin::BI__builtin_fminimum_numf128:
+ return RValue::getIgnored();
case Builtin::BIfmod:
case Builtin::BIfmodf:
case Builtin::BIfmodl:
@@ -426,7 +475,9 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_fmodf16:
case Builtin::BI__builtin_fmodl:
case Builtin::BI__builtin_fmodf128:
+ return emitBinaryFPBuiltin<cir::FModOp>(cgf, *e);
case Builtin::BI__builtin_elementwise_fmod:
+ return RValue::getIgnored();
case Builtin::BIlog:
case Builtin::BIlogf:
case Builtin::BIlogl:
@@ -436,6 +487,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_logl:
case Builtin::BI__builtin_logf128:
case Builtin::BI__builtin_elementwise_log:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::LogOp>(cgf, *e);
case Builtin::BIlog10:
case Builtin::BIlog10f:
case Builtin::BIlog10l:
@@ -445,6 +497,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_log10l:
case Builtin::BI__builtin_log10f128:
case Builtin::BI__builtin_elementwise_log10:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::Log10Op>(cgf, *e);
case Builtin::BIlog2:
case Builtin::BIlog2f:
case Builtin::BIlog2l:
@@ -454,6 +507,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_log2l:
case Builtin::BI__builtin_log2f128:
case Builtin::BI__builtin_elementwise_log2:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::Log2Op>(cgf, *e);
case Builtin::BInearbyint:
case Builtin::BInearbyintf:
case Builtin::BInearbyintl:
@@ -462,6 +516,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_nearbyintl:
case Builtin::BI__builtin_nearbyintf128:
case Builtin::BI__builtin_elementwise_nearbyint:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::NearbyintOp>(cgf, *e);
case Builtin::BIpow:
case Builtin::BIpowf:
case Builtin::BIpowl:
@@ -470,7 +525,10 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_powf16:
case Builtin::BI__builtin_powl:
case Builtin::BI__builtin_powf128:
+ return RValue::get(
+ emitBinaryMaybeConstrainedFPBuiltin<cir::PowOp>(cgf, *e));
case Builtin::BI__builtin_elementwise_pow:
+ return RValue::getIgnored();
case Builtin::BIrint:
case Builtin::BIrintf:
case Builtin::BIrintl:
@@ -480,6 +538,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_rintl:
case Builtin::BI__builtin_rintf128:
case Builtin::BI__builtin_elementwise_rint:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::RintOp>(cgf, *e);
case Builtin::BIround:
case Builtin::BIroundf:
case Builtin::BIroundl:
@@ -489,6 +548,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_roundl:
case Builtin::BI__builtin_roundf128:
case Builtin::BI__builtin_elementwise_round:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::RoundOp>(cgf, *e);
case Builtin::BIroundeven:
case Builtin::BIroundevenf:
case Builtin::BIroundevenl:
@@ -498,6 +558,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_roundevenl:
case Builtin::BI__builtin_roundevenf128:
case Builtin::BI__builtin_elementwise_roundeven:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::RoundEvenOp>(cgf, *e);
case Builtin::BIsin:
case Builtin::BIsinf:
case Builtin::BIsinl:
@@ -507,6 +568,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_sinl:
case Builtin::BI__builtin_sinf128:
case Builtin::BI__builtin_elementwise_sin:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::SinOp>(cgf, *e);
case Builtin::BIsinh:
case Builtin::BIsinhf:
case Builtin::BIsinhl:
@@ -527,6 +589,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_sincosf16:
case Builtin::BI__builtin_sincosl:
case Builtin::BI__builtin_sincosf128:
+ return RValue::getIgnored();
case Builtin::BIsqrt:
case Builtin::BIsqrtf:
case Builtin::BIsqrtl:
@@ -536,6 +599,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_sqrtl:
case Builtin::BI__builtin_sqrtf128:
case Builtin::BI__builtin_elementwise_sqrt:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::SqrtOp>(cgf, *e);
case Builtin::BItan:
case Builtin::BItanf:
case Builtin::BItanl:
@@ -545,6 +609,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_tanl:
case Builtin::BI__builtin_tanf128:
case Builtin::BI__builtin_elementwise_tan:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::TanOp>(cgf, *e);
case Builtin::BItanh:
case Builtin::BItanhf:
case Builtin::BItanhl:
@@ -554,6 +619,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_tanhl:
case Builtin::BI__builtin_tanhf128:
case Builtin::BI__builtin_elementwise_tanh:
+ return RValue::getIgnored();
case Builtin::BItrunc:
case Builtin::BItruncf:
case Builtin::BItruncl:
@@ -563,6 +629,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_truncl:
case Builtin::BI__builtin_truncf128:
case Builtin::BI__builtin_elementwise_trunc:
+ return emitUnaryMaybeConstrainedFPBuiltin<cir::TruncOp>(cgf, *e);
case Builtin::BIlround:
case Builtin::BIlroundf:
case Builtin::BIlroundl:
@@ -570,6 +637,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_lroundf:
case Builtin::BI__builtin_lroundl:
case Builtin::BI__builtin_lroundf128:
+ return emitUnaryMaybeConstrainedFPToIntBuiltin<cir::LroundOp>(cgf, *e);
case Builtin::BIllround:
case Builtin::BIllroundf:
case Builtin::BIllroundl:
@@ -577,6 +645,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_llroundf:
case Builtin::BI__builtin_llroundl:
case Builtin::BI__builtin_llroundf128:
+ return emitUnaryMaybeConstrainedFPToIntBuiltin<cir::LLroundOp>(cgf, *e);
case Builtin::BIlrint:
case Builtin::BIlrintf:
case Builtin::BIlrintl:
@@ -584,6 +653,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_lrintf:
case Builtin::BI__builtin_lrintl:
case Builtin::BI__builtin_lrintf128:
+ return emitUnaryMaybeConstrainedFPToIntBuiltin<cir::LrintOp>(cgf, *e);
case Builtin::BIllrint:
case Builtin::BIllrintf:
case Builtin::BIllrintl:
@@ -591,6 +661,7 @@ static RValue tryEmitFPMathIntrinsic(CIRGenFunction &cgf, const CallExpr *e,
case Builtin::BI__builtin_llrintf:
case Builtin::BI__builtin_llrintl:
case Builtin::BI__builtin_llrintf128:
+ return emitUnaryMaybeConstrainedFPToIntBuiltin<cir::LLrintOp>(cgf, *e);
case Builtin::BI__builtin_ldexp:
case Builtin::BI__builtin_ldexpf:
case Builtin::BI__builtin_ldexpl:
@@ -677,6 +748,36 @@ RValue CIRGenFunction::emitBuiltinExpr(const GlobalDecl &gd, unsigned builtinID,
cir::VACopyOp::create(builder, dstPtr.getLoc(), dstPtr, srcPtr);
return {};
}
+
+ case Builtin::BIabs:
+ case Builtin::BIlabs:
+ case Builtin::BIllabs:
+ case Builtin::BI__builtin_abs:
+ case Builtin::BI__builtin_labs:
+ case Builtin::BI__builtin_ll...
[truncated]
|
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
5c082ff to
5b11a61
Compare
|
|
||
| let arguments = (ins | ||
| CIR_AnySIntOrVecOfSIntType:$src, | ||
| UnitAttr:$poison |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| UnitAttr:$poison | |
| UnitAttr:$min_is_poison |
|
|
||
| let results = (outs CIR_AnySIntOrVecOfSIntType:$result); | ||
|
|
||
| let assemblyFormat = "$src ( `poison` $poison^ )? `:` type($src) attr-dict"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| let assemblyFormat = "$src ( `poison` $poison^ )? `:` type($src) attr-dict"; | |
| let assemblyFormat = "$src ( `min_is_poison` $min_is_poison^ )? `:` type($src) attr-dict"; |
Seeing poison by itself gives me the impression that the operation will definitely return posion.
The LLVM dialect uses is_int_min_poison and always prints its value, but that feels overly verbose to me.
| of signed integers. | ||
|
|
||
| The `poison` attribute indicates whether the result value is a poison | ||
| value if the argument is statically or dynamically INT_MIN. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| value if the argument is statically or dynamically INT_MIN. | |
| value if the argument is statically or dynamically the minimum value for the | |
| type. |
INT_MIN is specifcally tied to the int type in C/C++, so this was misleading. (The LLVM Language Reference makes the same mistake.) Presumably, we intend for this to return poison if I pass SHORT_MIN with an !s16i type, and so on.
| } | ||
|
|
||
| def CIR_LroundOp : CIR_UnaryFPToIntBuiltinOp<"lround", "LroundOp">; | ||
| def CIR_LLroundOp : CIR_UnaryFPToIntBuiltinOp<"llround", "LlroundOp">; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| def CIR_LLroundOp : CIR_UnaryFPToIntBuiltinOp<"llround", "LlroundOp">; | |
| def CIR_LlroundOp : CIR_UnaryFPToIntBuiltinOp<"llround", "LlroundOp">; |
These are going to look really weird in CamelCase regardless, but I think we should at least be consistent with what the LLVM dialect does. I hate every variation of this I can think of.
| def CIR_LroundOp : CIR_UnaryFPToIntBuiltinOp<"lround", "LroundOp">; | ||
| def CIR_LLroundOp : CIR_UnaryFPToIntBuiltinOp<"llround", "LlroundOp">; | ||
| def CIR_LrintOp : CIR_UnaryFPToIntBuiltinOp<"lrint", "LrintOp">; | ||
| def CIR_LLrintOp : CIR_UnaryFPToIntBuiltinOp<"llrint", "LlrintOp">; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| def CIR_LLrintOp : CIR_UnaryFPToIntBuiltinOp<"llrint", "LlrintOp">; | |
| def CIR_LlrintOp : CIR_UnaryFPToIntBuiltinOp<"llrint", "LlrintOp">; |
| return emitBuiltinBitOp<cir::BitPopcountOp>(*this, e); | ||
|
|
||
| case Builtin::BI__builtin_unpredictable: { | ||
| if (cgm.getCodeGenOpts().OptimizationLevel != 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no need to put the assert inside a condition
|
|
||
| // If this is a predefined lib function (e.g. malloc), emit the call | ||
| // using exactly the normal call path. | ||
| if (getContext().BuiltinInfo.isPredefinedLibFunction(builtinID)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm curious as to why you needed to add this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This handles target-specific builtins that can have aggregate return values (e.g., __builtin_arm_mve_vld2q_u32). When the result is an aggregate and returnValue is null, we need to report this as NYI rather than silently producing incorrect code. This matches the pattern in classic codegen.
| case Builtin::BI__abnormal_termination: | ||
| case Builtin::BI_abnormal_termination: | ||
| case Builtin::BI_setjmpex: | ||
| case Builtin::BI_setjmp: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to insert a call to errorNYI here to keep these from falling through.
| case Builtin::BIforward_like: | ||
| case Builtin::BIas_const: | ||
| return RValue::get(emitLValue(e->getArg(0)).getPointer()); | ||
| case Builtin::BIforward_like: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you move this one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
std::forward_like has different semantics from std::forward - it adjusts value category based on a template parameter, so it needs different handling. Moved it to the NYI group since it requires separate implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Classic codegen has it in the same group as the others.
| case Builtin::BIforward: | ||
| case Builtin::BIforward_like: | ||
| case Builtin::BIas_const: | ||
| return RValue::get(emitLValue(e->getArg(0)).getPointer()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was this needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This provides the actual implementation for std::move, std::move_if_noexcept, std::forward, and std::as_const. These builtins simply return a pointer to their argument - emitLValue(e->getArg(0)).getPointer() is the correct codegen matching OGCG behavior. Previously these were falling through to NYI.
This patch adds support for floating-point math intrinsics in CIR, enabling optimization of math library calls to LLVM intrinsics. New CIR Operations (CIROps.td): - CIR_LogOp, CIR_Log10Op, CIR_Log2Op: Logarithm operations - CIR_SinOp, CIR_TanOp: Trigonometric operations - CIR_NearbyintOp, CIR_RintOp, CIR_RoundOp, CIR_TruncOp: Rounding operations - CIR_LroundOp, CIR_LLroundOp, CIR_LrintOp, CIR_LLrintOp: FP-to-int conversions - CIR_CopysignOp: Sign copy operation - CIR_FMaxNumOp, CIR_FMinNumOp: Min/max operations - CIR_FModOp, CIR_PowOp: Arithmetic operations CIRGenBuiltin.cpp changes: - Add helper templates: emitUnaryMaybeConstrainedFPToIntBuiltin, emitBinaryFPBuiltin, emitBinaryMaybeConstrainedFPBuiltin - Implement tryEmitFPMathIntrinsic cases for all new operations - Add handling for predefined library functions (fabs, lround, etc.) - Implement std::move, std::forward, std::move_if_noexcept, std::as_const builtins CIRGenExpr.cpp changes: - Fix hasAttributeNoBuiltin to default to false, matching incubator behavior - This enables builtin recognition for predefined library functions CIRGenModule.cpp changes: - Add logic to skip noinline attribute for functions containing only builtin calls that become intrinsics, allowing proper optimization LowerToLLVM.cpp changes: - Add LLVM lowering patterns for all new CIR operations - Each operation maps to its corresponding LLVM intrinsic Test updates: - builtin-floating-point.c: New comprehensive test from incubator covering all math intrinsics with CIR, LLVM, and OGCG checks - libc.c: Update to expect cir.fabs intrinsic for fabs/fabsf - builtin-fcmp-sse.c: Update CHECK patterns for new noinline behavior - builtin-isfpclass.c: Update to expect cir.is_fp_class intrinsic
This patch adds support for additional floating-point math intrinsics that were missing from the initial implementation. New CIR Operations (CIROps.td): - CIR_RoundEvenOp: Rounds to nearest integer with ties to even - CIR_FMaximumOp: IEEE 754-2019 maximum (propagates NaN) - CIR_FMinimumOp: IEEE 754-2019 minimum (propagates NaN) - CIR_ATan2Op: Two-argument arctangent CIRGenBuiltin.cpp changes: - Implement acos/acosf/acosl -> cir.acos (using existing ACosOp) - Implement asin/asinf/asinl -> cir.asin (using existing ASinOp) - Implement atan/atanf/atanl -> cir.atan (using existing ATanOp) - Implement atan2/atan2f/atan2l -> cir.atan2 (new ATan2Op) - Implement roundeven/roundevenf/roundevenl -> cir.roundeven (new RoundEvenOp) - Previously these returned RValue::getIgnored() as NYI placeholders LowerToLLVM.cpp changes: - Add LLVM lowering for RoundEvenOp -> llvm.roundeven - Add LLVM lowering for FMaximumOp -> llvm.maximum - Add LLVM lowering for FMinimumOp -> llvm.minimum - Add LLVM lowering for ATan2Op -> llvm.atan2 Test updates: - builtin-floating-point.c: Add tests for acos, asin, atan, atan2, and roundeven builtins with CIR and LLVM checks
This commit adds support for elementwise builtin intrinsics in CIR, migrating functionality from the incubator to upstream. Changes include: 1. CIROps.td: Added CIR_AbsOp for integer absolute value computation - Supports signed integers and vectors of signed integers - Includes 'poison' attribute for INT_MIN handling 2. CIRGenBuiltin.cpp: Implemented elementwise builtin emission - __builtin_elementwise_abs (integer via AbsOp, FP via FAbsOp) - __builtin_elementwise_acos, asin, atan, atan2 - __builtin_elementwise_exp, exp2 - __builtin_elementwise_log, log2, log10 - __builtin_elementwise_cos, sin, tan - __builtin_elementwise_floor, round, rint, nearbyint, trunc - __builtin_elementwise_sqrt 3. LowerToLLVM.cpp: Added LLVM lowering for AbsOp - Uses mlir::LLVM::AbsOp for lowering with poison attribute 4. Test: Added builtins-elementwise.c - Comprehensive tests for all implemented elementwise builtins - Tests scalar float, double, and vector types (vfloat4, vdouble4) - Includes CIR, LLVM, and OGCG checks for verification - Updated vector type syntax to match upstream format
Implement support for the integer absolute value builtins (abs, labs, llabs, __builtin_abs, __builtin_labs, __builtin_llabs) to emit cir.abs operations instead of library calls. The implementation handles signed overflow behavior: - SOB_Defined (-fwrapv): emit cir.abs without poison flag - SOB_Undefined: emit cir.abs with poison flag (allows optimization) - SOB_Trapping: not yet implemented (llvm_unreachable) Also fixes a deprecated builder.create<AbsOp> pattern in the __builtin_elementwise_abs handler to use AbsOp::create instead. Updates libc.c test to: - Add LLVM IR verification with llvm.abs intrinsic checks - Add -fwrapv mode tests for non-poison behavior - Verify both poison (default) and non-poison (-fwrapv) modes
- Implement __builtin_unpredictable in CIRGenBuiltin.cpp - Add pred-info-builtins.c test for expect, expect_with_probability, and unpredictable builtins - Add builtin-rotate.c test for rotate left/right builtins (8/16/32/64 bit) - All tests include CIR, LLVM, and OGCG checks to verify CIR-produced LLVM matches original codegen
5b11a61 to
b5128d7
Compare
- Rename AbsOp 'poison' attribute to 'min_is_poison' for clarity - Update AbsOp description to use 'minimum value for the type' instead of INT_MIN - Add RintOp description explaining FE_INEXACT difference from nearbyint - Rename LLroundOp/LLrintOp to LlroundOp/LlrintOp to match LLVM dialect - Add summary and description for FP-to-int ops and binary FP ops - Add explanatory comment for hasAttributeNoBuiltin setting - Use errorNYI instead of llvm_unreachable for abs overflow cases - Remove unnecessary condition around assert in unpredictable builtin - Update libc.c test to use min_is_poison syntax
| // Check if this function contains any builtin calls that will become | ||
| // intrinsics. If so, don't mark as noinline - let the optimizer handle | ||
| // it. | ||
| if (auto *fd = dyn_cast<FunctionDecl>(decl)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of doing a visitor walk here, which can get expensive, did you try a CIRGenFunction state that is flipped to true everytime if finds a builtin?
Address review feedback from bcardosolopes: Instead of walking the entire function body to find builtin calls (expensive O(n) operation), track when builtins are emitted during codegen using a flag in CIRGenFunction. This avoids the overhead of revisiting AST nodes that have already been processed during code generation.
|
Addressed @bcardosolopes's feedback about the expensive AST visitor walk. Changes in commit 0f70528:
This changes the builtin detection from O(n) AST traversal to O(1) flag check, set during the normal codegen process. |
| case Builtin::BI__builtin_fmodf16: | ||
| case Builtin::BI__builtin_fmodl: | ||
| case Builtin::BI__builtin_fmodf128: | ||
| return emitBinaryFPBuiltin<cir::FModOp>(cgf, *e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should go through emitBinaryMaybeConstrainedFPBuiltin
| case Builtin::BI__builtin_atan2f128: | ||
| case Builtin::BI__builtin_elementwise_atan2: | ||
| return RValue::getIgnored(); | ||
| return emitBinaryFPBuiltin<cir::ATan2Op>(cgf, *e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should go through emitBinaryMaybeConstrainedFPBuiltin.
| return emitUnaryFPBuiltin<cir::ATanOp>(*this, *e); | ||
| case Builtin::BI__builtin_elementwise_atan2: | ||
| case Builtin::BI__builtin_elementwise_ceil: | ||
| return emitBinaryFPBuiltin<cir::ATan2Op>(*this, *e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should go through emitBinaryMaybeConstrainedFPBuiltin, as should the other elementwise handlers for any function that could raise an FP exception. It's also not clear to me why these elementwise handlers are with the other builtins for the corresponding functions.
| case Builtin::BIforward_like: | ||
| case Builtin::BIas_const: | ||
| return RValue::get(emitLValue(e->getArg(0)).getPointer()); | ||
| case Builtin::BIforward_like: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Classic codegen has it in the same group as the others.
| } else if (codeGenOpts.getInlining() == CodeGenOptions::OnlyAlwaysInlining) { | ||
| // If inlining is disabled, force everything that isn't always_inline | ||
| // to carry an explicit noinline attribute. | ||
| // However, don't mark functions as noinline if they only contain |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not following. What does isConstant have to do with NoInline?
HendrikHuebner
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please submit smaller PRs in the future - its hard to review 3000 lines of code at once. I'd split this up into at least 3 smaller PRs personally.
| // OGCG-LABEL: rotr64 | ||
| // OGCG: call i64 @llvm.fshr.i64(i64 {{.*}}, i64 {{.*}}, i64 {{.*}}) | ||
| return __builtin_rotateright64(x, y); | ||
| } No newline at end of file |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a trailing newline here
| // RUN: %clang_cc1 -triple aarch64-apple-darwin-macho -fclangir -emit-cir %s -o %t-aarch64.cir | ||
| // RUN: FileCheck --input-file=%t-aarch64.cir %s --check-prefix=AARCH64 | ||
| // RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -emit-llvm -o %t.ll %s | ||
| // RUN: FileCheck --input-file=%t.ll %s --check-prefix=LLVM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are all missing OGCG checks
| } | ||
|
|
||
| double my_cos(double f) { | ||
| return __builtin_cos(f); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some of these are already implemented and we have tests for them: Please check out test/CIR/CodeGenBuiltins/builtins-floating-point.c. I think you can just add the ones missing ones to that test instead of adding a new file here.
| @@ -0,0 +1,118 @@ | |||
| // RUN: %clang_cc1 -triple aarch64-none-linux-android21 -fclangir -emit-cir %s -o %t.cir | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe move this test file (and the other tests) to test/CIR/CodeGenBuiltins?
| double testFabs(double x) { | ||
| return fabs(x); | ||
| // CHECK: cir.call @fabs | ||
| // CHECK: cir.fabs %{{.+}} : !cir.double |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add the other checks here as well?
| extern void __attribute__((noinline)) bar(void); | ||
|
|
||
| void expect(int x) { | ||
| if (__builtin_expect(x, 0)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is already a test for the expect builtins in CodeGenBuiltin/builtin_call.c
| case Builtin::BI__builtin_abs: | ||
| case Builtin::BI__builtin_labs: | ||
| case Builtin::BI__builtin_llabs: { | ||
| bool sanitizeOverflow = sanOpts.has(SanitizerKind::SignedIntegerOverflow); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you move this case to a separate helper function?
| [[fallthrough]]; | ||
| case LangOptions::SOB_Trapping: | ||
| cgm.errorNYI(e->getSourceRange(), "abs with overflow handling"); | ||
| return RValue::get(nullptr); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| return RValue::get(nullptr); | |
| return RValue::getIgnored(); |
| case Builtin::BI__builtin_popcountg: | ||
| return emitBuiltinBitOp<cir::BitPopcountOp>(*this, e); | ||
|
|
||
| case Builtin::BI__builtin_unpredictable: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be a separate PR
Address reviewer feedback: - Use emitBinaryMaybeConstrainedFPBuiltin for atan2 and fmod - Use emitUnaryMaybeConstrainedFPBuiltin for elementwise FP handlers - Group BIforward_like with other C++ std:: builtins (move, forward, etc.)
|
Addressed @andykaylor's feedback on using constrained FP builtins. Changes in commit f005f81:
Regarding the suggestion to move elementwise handlers to be grouped with their corresponding functions in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with Hendrik that you should move the implementation of non-FP builtins to a separate PR.
| case Builtin::BI__builtin_powf128: | ||
| return RValue::get( | ||
| emitBinaryMaybeConstrainedFPBuiltin<cir::PowOp>(cgf, *e)); | ||
| case Builtin::BI__builtin_elementwise_pow: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couldn't this be handled by the call above?
| // Track that this function contains a builtin call. This is used to avoid | ||
| // marking functions as noinline when they only contain simple builtin calls | ||
| // that will become intrinsics. | ||
| hasEmittedBuiltinCall = true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't seem right. The comment says this is handling functions that "only contain simple builtin calls" but this will have that effect for all functions that call builtins, regardless of what else the function does.
| } else if (codeGenOpts.getInlining() == CodeGenOptions::OnlyAlwaysInlining) { | ||
| // If inlining is disabled, force everything that isn't always_inline | ||
| // to carry an explicit noinline attribute. | ||
| // However, don't mark functions as noinline if they only contain |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This still doesn't seem desirable to me.
| arg.getType(), arg, false); | ||
| return RValue::get(result); | ||
| } | ||
| case Builtin::BI__builtin_elementwise_acos: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of these elementwise builtins are handled in tryEmitFPMathIntrinsic. They shouldn't be here also.
| @@ -0,0 +1,1748 @@ | |||
| // RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fclangir -emit-cir %s -o %t.cir | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please move any new tests needed to clang/test/CIR/CodeGenBuiltins/builtins-floating-point.c.
This PR adds support for various math and builtin intrinsics to CIR:
Changes
All changes include CIR, LLVM lowering, and OGCG test checks to verify correctness.