[DVC] Add RequestBasedMetaRepository to enable metadata retrieval directly from server

[pthirun]

[DVC] Add RequestBasedMetaRepository to enable metadata retrieval directly from server

This PR introduces the use of the /store_properties/{store_name} endpoint to retrieve and cache store metadata for the DaVinci client. The endpoint was implemented in the Venice Server in this PR: #1374.

The RequestBasedMetaRepository leverages this endpoint to maintain a cache of Store and SchemaData objects for requested stores. The cache is refreshed by the NativeMetadataRepository, which this new class extends.

By default, this feature is disabled in the ClientConfig passed to the NativeMetadataRepository. The default configuration preserves the current behavior of using the ThinClientMetaStoreBasedRepository. The relevant configuration field is ClientConfig.useRequestBasedMetaRepository. For more details, see NativeMetadataRepository.getInstance.

How was this PR tested?

Unit and Integration test included in PR.

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.

NOTE: behavior will not change by default, feature flag in ClientConfig must be turned on.

[xunyin8]

Looks good overall, thanks for the refactoring. Just some minor comments and also looks like test coverage is not passing.

[xunyin8]

Thanks for the changes and unit tests, I think we are a bit lacking on the integration tests. See comments below.

[pthirun]

Using the ContollerClients map to keep track of created stores. I think this is appropriate since the IntegrationTestPushUtils.createStoreForJob method returns a controllerClient, even though we only need one instance of the ControllerClient to create new value schemas.

Digging into the createStoreForJob method, it does look like the D2ControllerClientFactory caches the controllerClient by (clusterName, d2ServiceName,d2ZkHost). So this should result in all of the clients in this map to point to the same client instance.

[xunyin8]

Typically we share the cluster environment across tests because a cluster is expensive and the things we are testing for generally don't break the cluster environment. However, I'd recommend you stay away from sharing stores across different tests. Stores are cheaper to create and if certain test fail it might leave the store in a bad state where other tests will also fail. This will lead to scenarios where you see 10 tests fail but really only 1 test failed and caused the store to be in a bad state. The order of test execution might also have side effects on the result of the test (making it flaky). Can we refactor this class so we are not sharing stores across tests?

[pthirun]

Removing the final here so we may initialize this map in unit test mocking.

[xunyin8]

Could you point me to the unit test you are talking about? I don't see why this is necessary to achieve unit test mocking.

[pthirun]

RequestBasedMetaRepositoryTest.java and ThinClientMetaStoreBasedRepositoryTest.java.
In the unit tests, I am partially mocking the client object by using Mockito's thenCallRealMethod.

[pthirun]

The mocked clients do not seem to run any initialization on private final class variables, so these maps need to be initialized by the test.

[xunyin8]

Change looks good, just one minor concern about Thread.sleep usage in test and exposing some class field for unit test mocking.

[xunyin8]

We are already using TestUtils.waitForNonDeterministicPushCompletion. We should avoid using Thread.sleep and why do we even need it here?.

[pthirun]

The tests were not passing early on due to the stores not being ready fast enough, however removing it now and testing locally seems to work. I pushed changes to remove this sleep, waiting on CI to finish.