Add AVX512 support to k-nn FAISS

[akashsha1]

Description

This change adds support to speed up vector search and indexing in faiss using AVX512 hardware accelerator.

Related Issues

Resolves #2056

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[naveentatikonda]

@akashsha1 Can you also update .github/workflows/CI.yml and .github/workflows/test_security.yml to detect if avx512 is supported or not and pass the flag to gradle.

Also, pls add an entry in CHANGELOG for this PR and sign off the PR to fix DCO check. Thanks!

[naveentatikonda]

Can you also please update build script to build all 3 versions of faiss lib for the distribution. Pls change this block of code like shown below

if [ "$PLATFORM" != "windows" ] && [ "$ARCHITECTURE" = "x64" ]; then
  echo "Building k-NN library after enabling AVX2"
  # Skip applying patches as patches were applied already from previous :buildJniLib task
  # If we apply patches again, it fails with conflict
  ./gradlew :buildJniLib -Davx2.enabled=true -Dbuild.lib.commit_patches=false -Dbuild.lib.apply_patches=false

  echo "Building k-NN library after enabling AVX512"
  ./gradlew :buildJniLib -Davx512.enabled=true -Dbuild.lib.commit_patches=false -Dbuild.lib.apply_patches=false
fi

[naveentatikonda]

@akashsha1 You also need to add the avx512 faiss library in plugin-security.policy to provide runtime permissions for loading the library. Pls test end to end after making all these changes. Thanks!

[naveentatikonda]

@akashsha1 Looks like some of the commits were not properly signed causing the DCO to fail. Do you want to rebase and squash everything into a single commit and sign that commit ?

[naveentatikonda]

@akashsha1 We need to add the CPU flag detection logic in these 2 workflows and pass the gradle flags to fix the failing checks
https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/backwards_compatibility_tests_workflow.yml
https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/test_security.yml

[akashsha1]

Hi @naveentatikonda - please consider this PR: #2110 in favor of the current one. The rebasing due to DCO sign-off got messy, so I created a new PR instead. It has all feedback addressed to this point, except the workflow failures which needs to be resolved.

[akashsha1]

@akashsha1 We need to add the CPU flag detection logic in these 2 workflows and pass the gradle flags to fix the failing checks https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/backwards_compatibility_tests_workflow.yml https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/test_security.yml

This needs to be addressed in #2110 ..