Skip to content

Added Domain support for stream mirroring and sourcing and KV full support for the same.#631

Merged
mtmk merged 19 commits into
nats-io:mainfrom
darkwatchuk:KV-Add-Sources-And-Mirrors
Oct 1, 2024
Merged

Added Domain support for stream mirroring and sourcing and KV full support for the same.#631
mtmk merged 19 commits into
nats-io:mainfrom
darkwatchuk:KV-Add-Sources-And-Mirrors

Conversation

@darkwatchuk

@darkwatchuk darkwatchuk commented Sep 16, 2024

Copy link
Copy Markdown
Collaborator

Added Sources and Mirror to KV Store Creation

fixes #494

PS nats-io/nats.go#1112

Comment thread src/NATS.Client.JetStream/Models/StreamSource.cs
@darkwatchuk

Copy link
Copy Markdown
Collaborator Author

Due to the difficulty of automating leaf node testing below are examples of tests/usage of mirroring and stream sourcing.

   [Fact]
   public async Task Test_LeafStreamMirrorDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);

       var mirroredStream = await jsContext.CreateStreamAsync(new StreamConfig()
       {
           Name = "mirroredStream",
           Mirror = new StreamSource
           {
               Name = "sourceStream1",
               Domain = "hub",
           },
       });
   }

   [Fact]
   public async Task Test_LeafStreamMultiSourceDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);

       var mirroredStream = await jsContext.CreateStreamAsync(new StreamConfig()
       {
           Name = "mirroredStreamMulti",
           Sources = [
                   new StreamSource { Name = "sourceStream1", Domain = "hub" },
                   new StreamSource { Name = "sourceStream2", Domain = "hub" }
               ],
       });
   }

   [Fact]
   public async Task Test_LeafKVMirrorDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);
       var kvContext = new NatsKVContext(jsContext);

       var mirroredStore = await kvContext.CreateStoreAsync(new NatsKVConfig("mirroredKvTestMirror")
       {
           Mirror = new StreamSource
           {
               Name = "kvTest",
               Domain = "hub",
           },
       });
   }

   [Fact]
   public async Task Test_LeafKVMirrorSourcesDomain()
   {
       await using var natsLeaf = new NatsConnection(new NatsOpts { Url = "nats://leaf:4223/", AuthOpts = new NatsAuthOpts { Username = "leaf", Password = "leaf" } });
       await natsLeaf.ConnectAsync();

       var jsContext = new NatsJSContext(natsLeaf);
       var kvContext = new NatsKVContext(jsContext);

       var mirroredStore = await kvContext.CreateStoreAsync(new NatsKVConfig("mirroredKvTestSources")
       {
           Sources = [
                   new StreamSource { Name = "kvTest", Domain = "hub" },
           ],
       });
   }

@darkwatchuk darkwatchuk changed the title Added Sources and Mirror to KV Store Creation Added Domain support for stream mirroring and sourcing and KV full support for the same. Sep 19, 2024
@darkwatchuk

Copy link
Copy Markdown
Collaborator Author

Object stores don't seem to support mirroring so this has not been included as previously discussed.

darkwatchuk and others added 10 commits September 19, 2024 08:44
Deleted the 'System.Diagnostics.Metrics' import as it was not being used anywhere in the file. This change helps in maintaining a clean codebase with only necessary dependencies.
Deleted an unnecessary directive for System.Xml.Schema in NatsJSContext.cs. This cleanup aids in maintaining clean and efficient code.
Clean up redundant initializations and streamline the handling of `subjects`, `mirror`, and `sources` to improve code clarity. Ensure default assignments are explicitly defined within conditional branches for better code maintainability.
Simplify the mirror object instantiation by using the ShallowCopy method, reducing code redundancy. This change improves readability and maintenance by encapsulating the cloning logic within the method.
Updated the conditional check for `config.Sources` to use pattern matching, improving readability and adhering to modern C# conventions. This change ensures cleaner and more maintainable code.
This commit updates the KeyValueStoreTest to skip the Test_CombinedSources test for NATS server versions earlier than 2.10 since some of the mirroring features are introduced with 2.10. It also removes unnecessary retry delay and timeout parameters from the test configuration.
@mtmk

mtmk commented Sep 19, 2024

Copy link
Copy Markdown
Member

Due to the difficulty of automating leaf node testing below are examples of tests/usage of mirroring and stream sourcing.

@mtmk mtmk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks @darkwatchuk

@mtmk
mtmk requested a review from Jarema September 23, 2024 14:20
@mtmk

mtmk commented Sep 23, 2024

Copy link
Copy Markdown
Member

@rickdotnet @Jarema I wouldn't mind an extra LGTM here please since I did make a few changes myself as well 😅

@rickdotnet

rickdotnet commented Sep 24, 2024

Copy link
Copy Markdown
Collaborator

Your changes LGTM, @mtmk. The CreateStreamConfig() method is a a beast. 😆

Could also get a shallow copy from records using with { }, but that's new age trickery. 😉

@mtmk

mtmk commented Sep 24, 2024

Copy link
Copy Markdown
Member

The CreateStreamConfig() method is a a beast. 😆

hehe true. we can write more tests for it maybe?

Could also get a shallow copy from records using with { }, but that's new age trickery. 😉

ah that's a good point. do you think we should use that to keep models clean? @darkwatchuk and I had a quick discussion about keeping them clean as well.

@rickdotnet

rickdotnet commented Sep 24, 2024

Copy link
Copy Markdown
Collaborator

do you think we should use that to keep models clean?

That'd be a by-product, but essentially. If we had a mixed bag of classes/records and wanted a consistent experience, I'd opt to keep the ShallowCopy() method.

mirror = config.Mirror.ShallowCopy();

vs

mirror = config.Mirror with { };

Both feel good to me.

@mtmk

mtmk commented Sep 25, 2024

Copy link
Copy Markdown
Member

do you think we should use that to keep models clean?

That'd be a by-product, but essentially. If we had a mixed bag of classes/records and wanted a consistent experience, I'd opt to keep the ShallowCopy() method.

mirror = config.Mirror.ShallowCopy();

vs

mirror = config.Mirror with { };

Both feel good to me.

I think with syntax is good especially when there are properties updated. Updated!

@rickdotnet

Copy link
Copy Markdown
Collaborator

Nice, looks good to me!

@mtmk

mtmk commented Sep 30, 2024

Copy link
Copy Markdown
Member

Looks like we're missing some of reference implementation details here:

https://github.com/nats-io/nats.go/blob/94fa0cb99411c202a4c9c2ae84f444a10f2e5ad8/kv.go#L1182-L1193

cc @Jarema

@mtmk

mtmk commented Oct 1, 2024

Copy link
Copy Markdown
Member

Double checked the implementation against nats-io/nats.go#1112 (with latest changes in main) Looks good. Only, some of the mirror bucket mapping seems to be missing which is captured in #642 and can be handled in a follow up PR

@mtmk
mtmk merged commit bb2537a into nats-io:main Oct 1, 2024
mtmk added a commit that referenced this pull request Oct 1, 2024
* Added Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Adds type-forwarders NET 5.0+ (#641)
* Run core tests on Windows and test run stability (#464)
* Add JetStream NATS Client Extensions (#598)
@mtmk mtmk mentioned this pull request Oct 1, 2024
mtmk added a commit that referenced this pull request Oct 1, 2024
* Added Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Adds type-forwarders NET 5.0+ (#641)
* Run core tests on Windows and test run stability (#464)
* Add JetStream NATS Client Extensions (#598)
mtmk added a commit that referenced this pull request Oct 9, 2024
* Add placement configuration to key-value and object store (#650)
* Bump System.Text.Json from 8.0.4 to 8.0.5 in /src/NATS.Client.Core (#649)
* Add other client extensions (#637)
* Add Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Add type-forwarders NET 5.0+ (#641)
* Add JetStream NATS Client Extensions (#598)
@mtmk mtmk mentioned this pull request Oct 9, 2024
mtmk added a commit that referenced this pull request Oct 9, 2024
* Add placement configuration to key-value and object store (#650)
* Bump System.Text.Json from 8.0.4 to 8.0.5 in /src/NATS.Client.Core (#649)
* Add other client extensions (#637)
* Add Domain support for stream mirroring and sourcing and KV full support for the same. (#631)
* Add type-forwarders NET 5.0+ (#641)
* Add JetStream NATS Client Extensions (#598)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KV Mirroring

3 participants