Skip to content

Using new windowTimeout operator with backpressure support#29714

Closed
anuchandy wants to merge 52 commits intoAzure:release/azure-messaging-eventhubs_5.12.3-beta.1from
anuchandy:eh-win-timeout
Closed

Using new windowTimeout operator with backpressure support#29714
anuchandy wants to merge 52 commits intoAzure:release/azure-messaging-eventhubs_5.12.3-beta.1from
anuchandy:eh-win-timeout

Conversation

@anuchandy
Copy link
Member

No description provided.

…zure#29711)

* [Automation] External Change

* [Automation] Generate Fluent Lite from digitaltwins#package-2022-05

* Update CHANGELOG.md

Co-authored-by: Xiaofei Cao <92354331+XiaofeiCao@users.noreply.github.com>
@anuchandy anuchandy requested a review from kasobol-msft June 30, 2022 02:44
} else {
partitionEventFlux = receiver
.windowTimeout(maxBatchSize, maxWaitTime);
}
Copy link
Member Author

@anuchandy anuchandy Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to see if the direct "reference" (i.e. non-reflective reference) to the new API here is safe? so far, my understanding is, the module gets compiled only in SDK repro and always with the reactor-core3.4.19 or above (i.e. both the new and old methods are available at compile time).

Anyone else will be using the published compiled jar (e.g., from maven), and "invocation" of one of the method overloads is protected by its discoverability at runtime, i.e., code won't try to load the new API with the older reactor-core.

But want to hear from others who have more expertise in doing this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a somewhat similar direct reference example here https://github.com/netty/netty/blob/4.1/codec/src/main/java/io/netty/handler/codec/compression/JdkZlibEncoder.java#L334-L336

private void deflate(ByteBuf out) {
    if (PlatformDependent.javaVersion() < 7) {
        deflateJdk6(out);
    }
    int numBytes;
    do {
       ...
        numBytes = deflater.deflate(
                out.array(), out.arrayOffset() + writerIndex, 
                out.writableBytes(), Deflater.SYNC_FLUSH);
       ...
    } while (numBytes > 0);
}

The Deflater.SYNC_FLUSH is directly referenced; the enum value "SYNC_FLUSH" available only from Java7, and runtime access to the value is protected by a similar runtime check.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a little bit concern for the checking. Because we may already or in the future use other new APIs in reactor-core3.4.19+, I am wondering if we need always add this checking for that code. If this is only temporary, I am ok for it.

And after users upgrade the SB version, are they better to upgrade the reactor-core version if they are using the old reactor-core and loading at runtime? we can mention that information in README.md. I am curious about the background of why they use the old reactor-core and load at runtime.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with Kun about the checking.

There is no background that users use old reactor-core I assume. If users don't need new feature for reactor-core or there is no bug in the past days, they may not upgrade it. So mention this information in README.md is enough.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anuchandy Looks like we need ReactorShim. Like we did for jackson
@lmolkova might help here.

If we pursue this way, we should also make sure this work item happens .

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kamil, Kun & Zeija for the feedback.

Let me use ObjectMapperShim as a reference implementation to understand the design-pattern we established for cross-version handling. I'll follow up with Ludmila then.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going over JacksonShim and ObjectMapperFactory; I feel like introducing ReactorShim in the azure-core for this case might be over-engineering.

We're not changing the reactor-core minor or major version. Like other patch versions, there are no breaking changes in the "existing" operators. The case here is that only Event Hubs SDK wants to use a "new" window operator overload from the latest patch, but if the user override the patch, it'll continue to use the "existing" window operator.

Today, if a user overrides with a different patch (3.4.patch) version, they won't face any linkage/method-not-found error; we continue to give the same guarantee.

So I feel we could keep it simple and limit to Event Hubs impl, Like cosmos SDK applying warmup for reactor netty Client. At least that's what I'm thinking for the EH beta, which a customer is waiting to unblock their development.

Copy link
Contributor

@kasobol-msft kasobol-msft Jul 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, if EH is the only place that needs it let's keep it local.
However, I'd suggest to create ReactorShim in the EH *.implementation.* package so that we prepare ourselves for the eventual move to azure-core with this when we get more of this or demand across SDKs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, Yes!. How is this API-shape look like?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

azure-sdk and others added 4 commits June 30, 2022 03:28
Increment package versions for digitaltwins releases
…tifier conversion (Azure#28997)

* updated api to include raw id

* CommunicationIdentifier generation from raw ID

* adding test cases

* overriding equal and hashcode

* reverting test changes

* test cases for chat identifier

* revert test changes for chat

* raw id can not be changed after manual set

* fix code review comments

* removing override hashCode

* Update CHANGELOG.md

* Remove breaking changes tag from changelog

* Revert "removing override hashCode"

This reverts commit f093447.

* fix java doc
…29309)

* Move servicebus message state key to amqp core

* update pom

* update CHANGELOG
alzimmermsft and others added 7 commits June 30, 2022 12:49
)

* Cosmos Spark Connector: Adding .Net/C# port of the NYC-Taxi-Data sample

* Adding unit test coverage for id encoding

* Update CosmosItemIdEncodingTest.java

* Update CosmosItemIdEncodingTest.java

* Update CosmosItemIdEncodingTest.java

* Fix to address that not all Mac providers are guaranteed to be cloneable

* Adding test for MacProvider not supporting cloning

* Update CHANGELOG.md

* Update CHANGELOG.md

* Update MacPool.java
* Added encryption option to perf tests

* ci fixes and pr feedback
* Cosmos Spark Connector: Adding .Net/C# port of the NYC-Taxi-Data sample

* Adding unit test coverage for id encoding

* Update CosmosItemIdEncodingTest.java

* Update CosmosItemIdEncodingTest.java

* Update CosmosItemIdEncodingTest.java

* Fix to address that not all Mac providers are guaranteed to be cloneable

* Adding test for MacProvider not supporting cloning

* Update CHANGELOG.md

* Update CHANGELOG.md

* Update MacPool.java

* Setting azure-cosmos version for hotfix release 4.32.1
Prepare Core Libraries for July 2022 Release
@anuchandy
Copy link
Member Author

Closing this, opening a new PR with clean commits to keep the beta release branch clean while crew works on other beta feature

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.