KAFKA-14960: TopicMetadata request manager by philipnee · Pull Request #14386 · apache/kafka

philipnee · 2023-09-13T19:56:47Z

TopicMetadataRequestManager is responsible for sending topic metadata requests. The manager manages API requests and build the request accordingly. All topic metadata requests are chained, if requesting the same topic, to avoid sending requests repeatedly.

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

it's in a state that all tests are passing rebase

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

it's in a state that all tests are passing rebase

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

…es are only closed once" This reverts commit 76d5c96.

Update CommitRequestManager.java clean up clean up remove idempotent references

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

it's in a state that all tests are passing rebase

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

it's in a state that all tests are passing rebase

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

…es are only closed once" This reverts commit 76d5c96.

Update CommitRequestManager.java clean up clean up remove idempotent references

…ka into cherry-pick-topic-metadata

junrao

@philipnee : Thanks for the PR. Made a pass of non-testing files. Left a few comments.

junrao · 2023-09-13T20:23:25Z


    /**
-     * Poll for the {@link OffsetFetchRequest} and {@link OffsetCommitRequest} request if there's any. The function will
+     * Poll for the {@link OffsetFetchRequest} and {@link org.apache.kafka.common.requests.OffsetCommitRequest} request if there's any. The function will


Do we need to spell out the package name for OffsetCommitRequest? It doesn't seem this is done consistently.

junrao · 2023-09-13T20:24:39Z

            }

-            OffsetCommitRequest.Builder builder = new OffsetCommitRequest.Builder(
+            org.apache.kafka.common.requests.OffsetCommitRequest.Builder builder = new org.apache.kafka.common.requests.OffsetCommitRequest.Builder(


Could we import the package?

junrao · 2023-09-13T20:27:25Z

+
+    public String topic() {
+        return topic;
+    }


Do we need to add toString, equals and hashcode?

junrao · 2023-09-13T20:28:53Z

+/**
+ * <p>
+ * Manages the state of topic metadata requests. This manager returns a
+ * {@link org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.PollResult} when a request is ready to


Do we need to spell out the package name?

junrao · 2023-09-13T20:31:13Z

+ * be sent. Specifically, this manager handles the following user API calls:
+ * </p>
+ * <ul>
+ * <li>listOffsets</li>


Is this true? OffsetsRequestManager says it handles listOffsets.

🤦 - listTopics() instead.

junrao · 2023-09-13T22:19:24Z

-            throw new RuntimeException(exception.get());
+        this.offsets = Collections.unmodifiableMap(offsets);
+
+        for (OffsetAndMetadata offsetAndMetadata : offsets.values()) {


This is an existing issue. Could we change toString to include the fields in the parent class? Do we need to add equals and hashcode for consistency?

There are quite a few events out there that need to be refactored if we do it this way. Would it be ok for you to skip this for now and I will post a MINOR patch to address after this is closed?

If your PR goes in before mine, you can make a comment and I'll add the toString changes into mine.

Yes, it's fine to clean this up separately.

@junrao - per your request. Here's the patch to address your comment: #14391

junrao · 2023-09-13T22:21:00Z

            return new NetworkClientDelegate.PollResult(Long.MAX_VALUE, Collections.emptyList());
        }

+        pendingRequests.inflightOffsetFetches.forEach(System.out::println);


Is this intended?

junrao · 2023-09-13T22:50:09Z

        public final GroupState.Generation requestedGeneration;
-        public CompletableFuture<Map<TopicPartition, OffsetAndMetadata>> future;
-
+        CompletableFuture<Map<TopicPartition, OffsetAndMetadata>> future;


Could this be private?

junrao · 2023-09-13T22:51:58Z

        List<OffsetFetchRequestState> inflightOffsetFetches = new ArrayList<>();

-        public boolean hasUnsentRequests() {
+        boolean hasUnsentRequests() {


Could this be private? Ditto below.

Sorry - I left out commit. Fixing those obvious mistakes right away.

Sorry - leaving this protected as it is used in one of the test to verify there's no more unsent request.

junrao · 2023-09-13T22:55:36Z

-            this.requestBuilder = requestBuilder;
-            this.node = node;
-            this.handler = handler;
+                             final BiConsumer<ClientResponse, Throwable> callback) {


This constructor seems unused?

I think it is according to my IDE. I think it is used in CommitRequestManager (line 219 and 256) and TopicMetadataRequestManger (line 145)

philipnee · 2023-09-14T05:19:54Z

Hey @junrao - Addressed your comments. I'm holding off on your request for Could we change toString to include the fields in the parent class? as I think it would be easier if I do that consistently for all other CompletableApplicationEvents. WDYT?

lianetm · 2023-09-14T19:44:35Z

+        CompletableFuture<Map<String, List<PartitionInfo>>> future;
+
+        public TopicMetadataRequestState(final LogContext logContext,
+                                    final Optional<String> topic,


indentation

lianetm · 2023-09-14T19:53:47Z

+    private MockTime time;
+    private TopicMetadataRequestManager topicMetadataRequestManager;
+
+    private Properties props;


This is only used in the setup so no need to keep it at the class level

junrao

@philipnee : Thanks for the updated PR. Made a pass of all files. A few more comments.

junrao · 2023-09-14T23:27:22Z

+ * </p>
+ * <ul>
+ * <li>listOffsets</li>
+ * <li>partitionsFor</li>


Yes, this is fine.

junrao · 2023-09-14T23:41:10Z

+                        future.complete(res);
+                        inflightRequests.remove(topic);
+                    } catch (RetriableException e) {
+                        if (e instanceof TimeoutException) {


This captures the case when a single request takes long on the broker and times out. However, a separate case is that individual request returns on time but with other RetriableException. In that case, TopicMetadataRequestManager will not clean up inflightRequests even when the client request has timed out.

Hey @junrao - Sorry for writing such a long reply - I really need to brush up my memory about the network module as well as the TopicMetadataRequest...

In the current code I think the following is how responses should be handled.

For the case of hard failure that the callback receives a non-null exception:
We should remove the inflight request for all non-retriable and TimeoutException because we need to fail the user requests and topic the retry. However, we should continue to retry for RetriableException until timer runs out.

For the case of soft failure (the server respond with an error code), it seems like the right thing to do is to continue to retry except for the non-retriable exception.

It is worth noting that: If the request timed-out (API level timeout), callback is completed with TimeoutException hard failure (in the network delegate module), which should eventually causes the exception to be purged from the inflight request as mentioned above.

To your question: if the responses include a retriable exception, I believe the right thing to do is to continue to retry until the timer runs out.

In fact - it seems like if (e instanceof TimeoutException) (that I added in the last commit) is actually incorrect. Because it is a RetriableException at it should be retried per:

if (error.exception() instanceof RetriableException) shouldRetry = true;

What's missing here is handling the hard RetriableException and TimeoutException. I will fix the code and add test cases to cover the scenarios above.

junrao · 2023-09-14T23:42:30Z

-            throw new RuntimeException(exception.get());
+        this.offsets = Collections.unmodifiableMap(offsets);
+
+        for (OffsetAndMetadata offsetAndMetadata : offsets.values()) {


Yes, it's fine to clean this up separately.

junrao · 2023-09-14T23:48:23Z

        verify(commitManager, times(1)).updateAutoCommitTimer(currentTimeMs);
        verify(commitManager, times(1)).maybeAutoCommit(offset);
-
-        backgroundThread.close();


It seems that we close backgroundThread in other cases?

junrao · 2023-09-14T23:55:38Z

+        try {
+            future.get();
+        } catch (Throwable e) {
+            System.out.println(e.getMessage());


In general, we should use logging instead of print.

Address PR Update TopicMetadataRequestManager.java

lianetm · 2023-09-15T14:30:27Z

+                handleException(exception, responseTimeMs);
+                return;
+            }
+            handleResponse(response, responseTimeMs);


It seems that the exception handling logic here ends up in 2 places. Apart from the initial handleException, this handleResponse could be as well end up handling exceptions, in a very similar way (catch block on handleResponse vs the handleException). If we prefer to keep them separated because of the "retryOnTimeout" part, which seems to be what makes them different, it would be helpful maybe to add a comment to describe that

thanks, @lianetm - Do you think it would be clearer to keep them in 1 place? The intention was to handle hard failures in handleException and soft failures (i.e. returning an error code) in handleResponse.

I think it would be clearer and better to maintain/troubleshoot if error handling is in one place. Maybe having a single handleException(final Throwable exception, final long responseTimeMs, boolean retryOnTimeout), called in both cases:

if (exception != null) { // Handle hard errors handleException(exception, responseTimeMs, false); ... } ... catch (Exception e) { // Handle soft errors handleException(e, responseTimeMs, true); }

junrao

@philipnee : Thanks for the updated PR. A few more comments.

junrao · 2023-09-15T17:00:00Z

+
+        private void processResponseOrException(final ClientResponse response,
+                                                final Throwable exception) {
+            long responseTimeMs = System.currentTimeMillis();


Could we pass in currentTime from the caller?

Hi Jun - understood making repeated system calls is bad. However - we do need to get the time when the response is received, so this is kind of the lazy way to do it.

I think it is possible to expose the time during the callback/future completion during the network poll (because it is computed at the top of the background thread loop and pass down to network poll.

Actually, could we use ClientResponse.receivedTimeMs?

junrao · 2023-09-15T17:01:07Z

+        }
+
+        private void handleException(final Throwable exception, final long responseTimeMs) {
+            if (exception instanceof TimeoutException || !(exception instanceof RetriableException)) {


TimeoutException is a RetriableException. Should we just treat it as other RetriableException? That seems to be what the old consumer does.

Just careful here because the current TopicMetadataFetcher handles 2 timeout cases differently regarding retries:

TimeoutException contained in a MetadataResponse as a response error is retried here

TimeoutException in getting the actual MetadataResponse throws exception here

I think for now - the manager will continue to retry all RetriableException including TimeoutException. Currently, we don't have a way to cancel the request when the user API request times-out, so this is a gap there, but a jira is filed here: https://issues.apache.org/jira/browse/KAFKA-15475

junrao · 2023-09-15T17:02:35Z

+    private final LogContext logContext;
+
+    public TopicMetadataRequestManager(final LogContext logContext, final ConsumerConfig config) {
+        this.logContext = logContext;


Could we just get rid of this? Ditto below.

sorry - got into the habit of using this. 😅

junrao · 2023-09-15T17:03:16Z

+                                         final long retryBackoffMaxMs) {
+            super(logContext, TopicMetadataRequestState.class.getSimpleName(), retryBackoffMs,
+                retryBackoffMaxMs);
+            this.future = new CompletableFuture<>();


Could we just get rid of this?

junrao · 2023-09-15T17:04:28Z

+         * {@link org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.UnsentRequest} if needed.
+         */
+        private Optional<NetworkClientDelegate.UnsentRequest> send(final long currentTimeMs) {
+            if (!this.canSendRequest(currentTimeMs)) {


It would be useful to be consistent in convention. If we want to add this in front of methods in the parent class, we want to do this consistently.

junrao · 2023-09-15T17:12:06Z

+                future.complete(res);
+                inflightRequests.remove(topic);
+            } catch (RetriableException e) {
+                onFailedAttempt(responseTimeMs);


The case that I am wondering is the following.

DefaultBackgroundThread is in a retry loop for an inflightRequest. PrototypeAsyncConsumer.partitionsFor eventually calls CompletableApplicationEvent.get(Timer timer) and times out. Do we still keep the request in inflightRequests at that point? Is that a concern?

Thanks Jun for raising this point - I think what we should do is to complete the request with a non-retriable exception to prevent the request from retrying. Otherwise - the request manager will never remove the request from the inflightRequest.

junrao · 2023-09-15T20:19:52Z

        private void retry(final long currentTimeMs) {
-            onFailedAttempt(currentTimeMs);
-            onSendAttempt(currentTimeMs);
+            this.onFailedAttempt(currentTimeMs);


currentTimeMs is when the request is sent, not when the response is receive. Could we pass along ClientResponse.receivedTimeMs?

Thanks for catching this.

Addressed, see line #259

junrao · 2023-09-15T20:54:20Z

+
+        private void processResponseOrException(final ClientResponse response,
+                                                final Throwable exception) {
+            long responseTimeMs = System.currentTimeMillis();


Actually, could we use ClientResponse.receivedTimeMs?

philipnee · 2023-09-18T20:00:39Z

 * Whether there is an existing coordinator.
 * Whether there is an inflight request.
 * Whether the backoff timer has expired.
- * The {@link org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.PollResult} contains either a wait timer


just cleaning up long references

philipnee · 2023-09-19T01:49:06Z

@junrao @lianetm - I hope I've addressed most, if not all, comments that you pointed out. So, thank you both for spending time reviewing my code.

We did have an offline discussion about user API timeout handling, and a bug report is filed here KAFKA-15475. I will submit a patch after this PR to address this small gap.

Lastly, here are the failing tests:

Build / JDK 20 and Scala 2.13 / testReplication() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest
53s
Build / JDK 17 and Scala 2.13 / testMultiWorkerRestartOnlyConnector – org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest
2m 24s
Build / JDK 17 and Scala 2.13 / testCreateTokenForOtherUserFails(String).quorum=kraft – kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest
9s
Build / JDK 11 and Scala 2.13 / testConfigurationOperations() – org.apache.kafka.controller.QuorumControllerTest
<1s
Build / JDK 11 and Scala 2.13 / executeTieredStorageTest(String).quorum=zk – org.apache.kafka.tiered.storage.integration.EnableRemoteLogOnTopicTest
43s
Build / JDK 8 and Scala 2.12 / testMultiWorkerRestartOnlyConnector – org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest
2m 25s

I believe these are flaky.

junrao

@philipnee : Thanks for the updated PR. Just a few minor comments.

junrao · 2023-09-19T17:08:58Z

+
+    public TopicMetadataRequestManager(final LogContext context, final ConsumerConfig config) {
+        logContext = context;
+        log = logContext.logger(this.getClass());


Could we remove this?

junrao · 2023-09-19T17:09:35Z

+        }
+
+        private void completeFutureAndRemoveRequest(final Throwable throwable) {
+            this.future.completeExceptionally(throwable);


Could we remove this?

junrao · 2023-09-19T17:09:59Z


    /**
-     * Poll for the {@link OffsetFetchRequest} and {@link OffsetCommitRequest} request if there's any. The function will
+     * Poll for the {@link OffsetFetchRequest} and {@link OffsetCommitRequestState} request if there's any. The function will


Hmm, OffsetCommitRequestState should be OffsetCommitRequest, right?

junrao · 2023-09-19T17:11:21Z

        List<OffsetFetchRequestState> inflightOffsetFetches = new ArrayList<>();

-        public boolean hasUnsentRequests() {
+        // Visible for teseting


typo teseting

lianetm · 2023-09-19T18:37:04Z

-         *  <p>If the request is new, it invokes a callback to remove itself from the {@code inflightOffsetFetches}
-         *  upon completion.</>
+         * <p>If the request is new, it invokes a callback to remove itself from the {@code inflightOffsetFetches}
+         * upon completion.</>


nit: wrong tag </> and other unclosed ones above

lianetm · 2023-09-19T18:39:02Z

         * Clear {@code unsentOffsetCommits} and moves all the sendable request in {@code unsentOffsetFetches} to the
         * {@code inflightOffsetFetches} to bookkeep all of the inflight requests.
-         *
+         * <p>


was the intention here to add an extra line in-between? should be <p/> if so

😓 - must be the residual from some edits..

Updates based on review comments

philipnee · 2023-09-21T04:31:26Z

@junrao @lianetm - Thanks for another round of reviews. The latest build passed with the following test failures

Build / JDK 17 and Scala 2.13 / testBecomeFollowerWhileOldClientFetchInPurgatory() – kafka.server.ReplicaManagerTest
<1s
Build / JDK 17 and Scala 2.13 / [1] enableRemoteStorage=true – kafka.server.ReplicaManagerTest
<1s
Build / JDK 20 and Scala 2.13 / testRackAwareRangeAssignor() – integration.kafka.server.FetchFromFollowerIntegrationTest
46s
Build / JDK 20 and Scala 2.13 / testBumpTransactionalEpoch(String).quorum=zk – kafka.api.TransactionsTest
13s
Build / JDK 20 and Scala 2.13 / testCommitTransactionTimeout(String).quorum=kraft – kafka.api.TransactionsTest
9s
Build / JDK 20 and Scala 2.13 / executeTieredStorageTest(String).quorum=kraft – org.apache.kafka.tiered.storage.integration.DeleteTopicTest
41s
Build / JDK 11 and Scala 2.13 / testCloseDuringRebalance() – kafka.api.ConsumerBounceTest
10s
Build / JDK 11 and Scala 2.13 / [2] Type=Raft-Isolated, Name=testDescribeQuorumReplicationSuccessful, MetadataVersion=3.7-IV0, Security=PLAINTEXT – org.apache.kafka.tools.MetadataQuorumCommandTest
1m 8s
Build / JDK 8 and Scala 2.12 / testMultiWorkerRestartOnlyConnector – org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest
2m 31s
Build / JDK 8 and Scala 2.12 / testBalancePartitionLeaders() – org.apache.kafka.controller.QuorumControllerTest

I believe they are irrelevant.

junrao

@philipnee : Thanks for the latest PR. LGTM

philipnee and others added 24 commits September 11, 2023 07:51

KAFKA-14960: [Part I]TopicMetadataRequestManager Implementation (#7)

33a221e

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

rebase

a350729

it's in a state that all tests are passing rebase

KAFKA-14274, #3: Introduce IdempotentCloser to ensure resources are o…

76d5c96

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

refactor existing code

d33188b

Fixing uncompleted future

ceb8694

KAFKA-14960: [Part I]TopicMetadataRequestManager Implementation (#7)

3b7400b

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

rebase

1ac4148

it's in a state that all tests are passing rebase

KAFKA-14274, #3: Introduce IdempotentCloser to ensure resources are o…

875abb0

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

refactor existing code

a8be7ba

Revert "KAFKA-14274, #3: Introduce IdempotentCloser to ensure resourc…

227d9b6

…es are only closed once" This reverts commit 76d5c96.

Clean up sad unused code

ae44539

Update CommitRequestManager.java clean up clean up remove idempotent references

KAFKA-14960: [Part I]TopicMetadataRequestManager Implementation (#7)

4b2a7f7

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

rebase

74b4960

it's in a state that all tests are passing rebase

KAFKA-14274, #3: Introduce IdempotentCloser to ensure resources are o…

c89958c

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

refactor existing code

d4e874a

Fixing uncompleted future

b86bc87

KAFKA-14960: [Part I]TopicMetadataRequestManager Implementation (#7)

d236f1b

TopicMetadataRequestManager implementation --------- Co-authored-by: Lianet Magrans <lmagrans@confluent.io>

rebase

09463d2

it's in a state that all tests are passing rebase

KAFKA-14274, #3: Introduce IdempotentCloser to ensure resources are o…

6a00f7e

…nly closed once Added IdempotentCloser to ensure resources are only closed once. While this class is broadly useful, it will be constrained to consumer internals until a later date.

refactor existing code

381730f

Revert "KAFKA-14274, #3: Introduce IdempotentCloser to ensure resourc…

530535d

…es are only closed once" This reverts commit 76d5c96.

Clean up sad unused code

8a0687c

Update CommitRequestManager.java clean up clean up remove idempotent references

Update DefaultBackgroundThreadTest.java

26d59dd

Merge branch 'cherry-pick-topic-metadata' of github.com:philipnee/kaf…

d1bada5

…ka into cherry-pick-topic-metadata

philipnee marked this pull request as ready for review September 13, 2023 19:57

philipnee added the ctr Consumer Threading Refactor (KIP-848) label Sep 13, 2023

junrao reviewed Sep 13, 2023

View reviewed changes

Address comments on reviews.

ffc66e9

philipnee mentioned this pull request Sep 14, 2023

MINOR: Implement toStringBase to avoid duplicated code #14391

Closed

lianetm reviewed Sep 14, 2023

View reviewed changes

junrao reviewed Sep 14, 2023

View reviewed changes

PR comments

6c06137

Address PR Update TopicMetadataRequestManager.java

lianetm reviewed Sep 15, 2023

View reviewed changes

junrao reviewed Sep 15, 2023

View reviewed changes

PR comment - passing responseTime from ClientResponse

ec600b3

philipnee force-pushed the cherry-pick-topic-metadata branch from 920d63b to ec600b3 Compare September 18, 2023 04:09

Philip Nee added 3 commits September 18, 2023 08:07

addressed more comments

403e122

Merge branch 'trunk' into cherry-pick-topic-metadata

269cbcb

clean up

179cd5d

philipnee commented Sep 18, 2023

View reviewed changes

clean up

628cb4c

junrao reviewed Sep 19, 2023

View reviewed changes

lianetm reviewed Sep 19, 2023

View reviewed changes

Update CommitRequestManager.java

6473597

Updates based on review comments

junrao approved these changes Sep 21, 2023

View reviewed changes

junrao merged commit 7d89bdc into apache:trunk Sep 21, 2023

Uh oh!

Conversation

philipnee commented Sep 13, 2023

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philipnee commented Sep 14, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philipnee Sep 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lianetm Sep 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junrao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

philipnee Sep 15, 2023 •

edited

Loading

lianetm Sep 15, 2023 •

edited

Loading