HDDS-10389. Implement a search feature for users to locate open keys within the Open Keys Insights section.

[ArafatKhan2198]

What changes were proposed in this pull request?

The existing search functionality for open keys in the Open Keys Insights section is limited to the dataset returned by the API endpoint, which only provides a subset of the available data. It is necessary to enhance this search feature to ensure users can accurately find the specific key information they are seeking or the closest available match.

I have implemented a new search endpoint for Recon Insights specifically for open keys:

http://localhost:9888/api/v1/insights/openKeys/search?searchPrefix=

Currently, this feature is limited to open keys to keep the initial implementation manageable and prevent the patch from becoming overly large. Since the search functionality involves querying both the KeyTable and the FileTable, we decided to introduce the feature for open keys first. The inclusion of search capabilities for deleted keys and directories will be addressed in subsequent PR's.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10389

How was this patch tested?

Unit Testing and Manual Testing

[devmadhuu]

Thanks @ArafatKhan2198 for working on this patch. Have few comments. Pls check.

[devmadhuu]

Have some more review comments. Pls check.

[devmadhuu]

Pls check , if review comments are take care. I can see some review comments are still open.

[devmadhuu]

Thanks @ArafatKhan2198 for continue working on this patch, however some old comments still open and some new comments. Pls check.

[ArafatKhan2198]

@devmadhuu @dombizita
Can you please review this!
This has been open for quite some time now.

[dombizita]

Thanks for working on this @ArafatKhan2198, overall it looks good to me, please take a look at the minor comments inline!

[devmadhuu]

Thanks @ArafatKhan2198 , one more comment.

[devmadhuu]

@ArafatKhan2198 thanks for taking care pagination scenario. few comments. Pls check.

[ArafatKhan2198]

Thanks for the initial round of reviews, @devmadhuu and @dombizita.

Could you please check the patch again? I have added more tests to cover every possible scenario that the search functionality might encounter.

Here is a short summary of various paths and scenarios tested in the test class:

1. Test Root Level Search Restriction: Ensures searching at the root level returns a bad request.
2. Test Volume Level Search Restriction: Ensures searching at the volume level returns a bad request.
3. Test Bucket Level Search: Verifies search results within different types of buckets (FSO, OBS, Legacy).
4. Test Directory Level Search: Validates searching inside specific directories.
5. Test Key Level Search: Confirms search results for specific keys within buckets.
6. Test Key Level Search Under Directory: Verifies searching for keys within nested directories.
7. Test Search Under Nested Directory: Checks search results within nested directories under dira3.
8. Test Limit Search: Tests the limit functionality of the search API.
9. Test Search Open Keys with Bad Request: Ensures bad requests with invalid parameters return appropriate responses.
10. Test Last Key in Response: Confirms the presence of the last key in paginated responses.
11. Test Search Open Keys with Pagination: Verifies paginated search results.
12. Test Search in Empty Bucket: Checks the response for searching within an empty bucket.

[ArafatKhan2198]

@devmadhuu @dombizita Please take a look

[ArafatKhan2198]

Explanation of the convertToObjectPath Method

The convertToObjectPath method converts a key prefix into an object path within the Ozone file system, based on the structure and layout of volumes, buckets, directories, and keys. Here’s a breakdown of how it works:

1. Normalize the Path and Parse Request Path:

   • The method starts by normalizing the input path and parsing it into its components (volume, bucket, directories, and key).

2. Root-Level Check:

   • If the input path is empty, it returns the original path (prevKeyPrefix).

3. Volume-Level Check:

   • If the path contains only the volume name, it fetches the volumeId and returns the path constructed with the volume ID.

4. Bucket-Level Check:

   • If the path contains volume and bucket names, it fetches the bucketId and returns the path constructed with the volume and bucket IDs, unless the bucket layout is not FILE_SYSTEM_OPTIMIZED.

5. Directory or Key-Level Check:

   • If the path contains more than two components, it checks if the last entity (directory or key) exists:
     • If it's a directory, it returns the path constructed with the directory ID.
     • If it's a key, it returns the path constructed with the key's parent directory ID and the key name.

6. Exception Handling:

   • If any exception occurs during the process, it logs the error and returns the original path.

Expected Outputs for Given Inputs:

1. Input: volume/bucket/key

   • Parsed Names: ["volume", "bucket", "key"]
   • Fetch volumeId for volume.
   • Fetch bucketId for bucket.
   • Check if key is a key in bucket.
   • If key exists, return volumeId/bucketId/parentDirectoryId/key.

2. Input: volume/bucket/dir1

   • Parsed Names: ["volume", "bucket", "dir1"]
   • Fetch volumeId for volume.
   • Fetch bucketId for bucket.
   • Check if dir1 is a directory in bucket.
   • If dir1 exists as a directory, return volumeId/bucketId/dir1Id/.

3. Input: volume/bucket/dir1/key

   • Parsed Names: ["volume", "bucket", "dir1", "key"]
   • Fetch volumeId for volume.
   • Fetch bucketId for bucket.
   • Check if dir1 is a directory in bucket.
   • Check if key is a key in dir1.
   • If key exists, return volumeId/bucketId/dir1Id/key.

4. Input: volume/bucket/dir1/dir2

   • Parsed Names: ["volume", "bucket", "dir1", "dir2"]
   • Fetch volumeId for volume.
   • Fetch bucketId for bucket.
   • Check if dir1 and dir2 are directories in bucket.
   • If dir2 exists as a directory, return volumeId/bucketId/dir2Id/.

5. Input: volume/bucket/dir1/dir2/key

   • Parsed Names: ["volume", "bucket", "dir1", "dir2", "key"]
   • Fetch volumeId for volume.
   • Fetch bucketId for bucket.
   • Check if dir1 and dir2 are directories in bucket.
   • Check if key is a key in dir2.
   • If key exists, return volumeId/bucketId/dir2Id/key.

Handling Missing Paths:

• If at any step, the path component (directory or key) does not exist, the method returns the original prevKeyPrefix.

I hope this clarifies the workings of the convertToObjectPath method!

---

[dombizita]

Thanks for working on this @ArafatKhan2198, especially on the detailed comments on the PR and in the code, it made the review much easier. I found one comment that you may not addressed, otherwise I'm fine with the changes.

[ArafatKhan2198]

Thanks for working on this @ArafatKhan2198, especially on the detailed comments on the PR and in the code, it made the review much easier. I found one comment that you may not addressed, otherwise I'm fine with the changes.

@dombizita

The IllegalArgumentException in the given code can occur due to invalid arguments passed to methods. In the context of the convertToObjectPath method and the search logic, possible scenarios include:

Invalid Volume or Bucket Names:

The validateNames method can throw an IllegalArgumentException if volume or bucket names do not meet required criteria.
Invalid Path Format:

When converting path components to IDs, if the format is incorrect or if the conversion logic encounters unexpected values, it might throw an IllegalArgumentException.

I'll mention it in the Javadoc as well.

[devmadhuu]

Thanks @ArafatKhan2198 for improving the patch. Just a nit. Pls check. Else LGTM +1

[dombizita]

Thanks for working on it @ArafatKhan2198! Thanks for the review @devmadhuu!