problems with fast crc support

[mdcallag]

There are a few problems with fast crc support in RocksDB:

1. LOG can claim the support is enabled even though it isn't used at run time
2. build_tools/build_detect_platform fails to compile the test code that detects SSE support and -DHAVE_SSE42 isn't added to the compiler command line. Note that RocksDB uses different methods at compile time and run time to determine whether fast crc support is compiled in vs whether it is selected at run time.

Note that an internal bug has already been opened for this.

This is a hack to get the -DHAVE_SSE42 flag added to the build command line. The problem is that the test code won't compile with -std=c++11. Adding $PLATFORM_CXXFLAGS might also fix this.

--- a/build_tools/build_detect_platform
+++ b/build_tools/build_detect_platform
@@ -456,7 +456,7 @@ elif test -z "$PORTABLE"; then
fi
fi

-$CXX $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null <<EOF
+$CXX $COMMON_FLAGS -std=c++11 -x c++ - -o /dev/null 2>/dev/null <<EOF
#include
#include <nmmintrin.h>
int main() {

Support for fast crc in RocksDB is complex:

• HAVE_SSE42 macro used at compile time to determine whether fast crc32 support is compiled
• SSE4_2 macro used at compile time to determine whether isSSE42 function will be called at run time to decide whether fast crc is used.
• The isSSE42 function decides at runtime whether fast crc is used

From the above, if the HAVE_SSE42 macro isn't defined then the fast crc function will use slow crc implementation. But LOG depends on output from isSSE42 function that depends on SSE4_2 macro which claims fast crc is supported.

[mdcallag]

I wonder whether this is an issue for the distros that bundle RocksDB?

[vadimtk]

Yes, we had problems with fast CRC support in Percona Server build.

[mdcallag]

This line in RocksDB LOG explains whether fast CRC support might be used. It can say yes when support isn't used because of this bug.
Fast CRC32 supported: 1

Code is here:
https://github.com/facebook/rocksdb/blob/master/util/crc32c.cc

To get fast crc32 support you want -DHAVE_SSE42 and probably some of -msse, -msse4.2 on compiler command line. Which mean you might need USE_SSE=1 set as an environment variable when building MyRocks.

grep for SSE in https://github.com/facebook/rocksdb/blob/master/build_tools/build_detect_platform

[bdarnell]

This has been a headache for CockroachDB too (cockroachdb/cockroach#15589). Compiling with -msse4.2 means that our binary releases don't work on older CPUs. We would prefer to compile without that flag (so that the compiler generates code for any amd64 cpu), then use runtime feature detection to enable the SSE4.2 fast CRC.

[siying]

@bdarnell you are welcome to send a PR to add a flag to achieve what you want.

[benesch]

@mdcallag can you verify that #2545 solves your second problem? I think #2513 has already fixed the first problem.

[socketpair]

possibly fixed by 55d58d9 (#10667)

in versions >= 7.9.2