Skip to content

HDDS-8129. ContainerStateMachine allows two different tasks with the same container id running in parallel. #4370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
arp7 merged 1 commit into from
Mar 10, 2023
Merged

HDDS-8129. ContainerStateMachine allows two different tasks with the same container id running in parallel. #4370

arp7 merged 1 commit into from
Mar 10, 2023

Conversation

@szetszwo
Copy link
Contributor

@szetszwo szetszwo commented Mar 10, 2023

What changes were proposed in this pull request?

Add a new TaskQueueMap with synchronized submit(..) and removeIfEmpty(..) methods.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8129

How was this patch tested?

Using ozone-1.3...szetszwo:ozone:multipartUpload (may include the test here or in a separated JRIA.)

@szetszwo szetszwo requested a review from errose28 March 10, 2023 04:41
sumitagrawl
sumitagrawl approved these changes Mar 10, 2023
View reviewed changes
Copy link
Contributor

@sumitagrawl sumitagrawl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@szetszwo
Copy link
Contributor Author

szetszwo commented Mar 10, 2023

Without this change, TestMultipartUpload can have "Unexpected block count changed" and "updateBlockCommitSequenceId from higher value to a lower value"

$search.sh ../multipartUpload_out/ testMultipartParallel.\* "Unexpected block count changed"

root dir    : ../multipartUpload_out/
file pattern: testMultipartParallel.*
search regex: Unexpected block count changed

-- ../multipartUpload_out/testMultipartParallel.17.log ------------------------
3469: 2023-03-10 00:27:24,590 [ContainerOp-5064d27f-d9ac-4ab6-af4e-f1be11d1be0b-1] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 2: Unexpected block count changed from 123 to 124
   1 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.18.log ------------------------
3661: 2023-03-10 00:33:18,304 [ContainerOp-1867ee36-ca62-4378-a490-a6f3419292d5-6] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 3: Unexpected block count changed from 82 to 83
   1 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.19.log ------------------------
3452: 2023-03-10 00:39:17,711 [ContainerOp-62e6617d-6792-44e7-a292-b6a96c01b64e-3] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 1: Unexpected block count changed from 148 to 149
3477: 2023-03-10 00:39:17,716 [ContainerOp-62e6617d-6792-44e7-a292-b6a96c01b64e-1] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 1: Unexpected block count changed from 149 to 150
30451: 2023-03-10 00:41:27,282 [ContainerOp-62e6617d-6792-44e7-a292-b6a96c01b64e-6] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 125: Unexpected block count changed from 28 to 29
34055: 2023-03-10 00:41:41,227 [ContainerOp-0ea95297-ac7b-4b74-9d2f-2d00f92d4bfa-0] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 135: Unexpected block count changed from 45 to 46
   4 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.20.log ------------------------
3684: 2023-03-10 00:45:10,749 [ContainerOp-ffc30796-c178-4a2e-bc85-86a26695052d-8] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 7: Unexpected block count changed from 8 to 9
3746: 2023-03-10 00:45:12,256 [ContainerOp-ed472d22-d206-4383-b556-4cece7af4793-1] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 8: Unexpected block count changed from 51 to 52
   2 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.6.log -------------------------
3635: 2023-03-09 23:20:19,005 [ContainerOp-3ff18daf-f5be-47e7-ac67-58569d6f6fc5-3] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 2: Unexpected block count changed from 99 to 100
3660: 2023-03-09 23:20:19,014 [ContainerOp-3ff18daf-f5be-47e7-ac67-58569d6f6fc5-7] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 2: Unexpected block count changed from 100 to 101
   2 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.9.log -------------------------
3753: 2023-03-09 23:39:40,482 [ContainerOp-7aca9b62-897b-43e7-8bd8-1ac4f9294dc8-4] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 2: Unexpected block count changed from 44 to 45
14755: 2023-03-09 23:40:38,913 [ContainerOp-0c7e9e30-9170-481a-8c80-369a446ffa4a-1] WARN  impl.BlockManagerImpl (BlockManagerImpl.java:persistPutBlock(200)) - Container 54: Unexpected block count changed from 106 to 108
   2 occurrence(s)
===============================================================================
Totally 12 occurrence(s) in 20 file(s).

$search.sh ../multipartUpload_out/ testMultipartParallel.\* "updateBlockCommitSequenceId from higher value"

root dir    : ../multipartUpload_out/
file pattern: testMultipartParallel.*
search regex: updateBlockCommitSequenceId from higher value

-- ../multipartUpload_out/testMultipartParallel.18.log ------------------------
3662: 2023-03-10 00:33:18,305 [ContainerOp-1867ee36-ca62-4378-a490-a6f3419292d5-6] ERROR keyvalue.KeyValueContainerData (KeyValueContainerData.java:updateBlockCommitSequenceId(205)) - Container 3 updateBlockCommitSequenceId from higher value 648 to a lower value 646
   1 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.19.log ------------------------
3453: 2023-03-10 00:39:17,712 [ContainerOp-62e6617d-6792-44e7-a292-b6a96c01b64e-3] ERROR keyvalue.KeyValueContainerData (KeyValueContainerData.java:updateBlockCommitSequenceId(205)) - Container 1 updateBlockCommitSequenceId from higher value 1143 to a lower value 1139
   1 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.20.log ------------------------
3685: 2023-03-10 00:45:10,750 [ContainerOp-ffc30796-c178-4a2e-bc85-86a26695052d-8] ERROR keyvalue.KeyValueContainerData (KeyValueContainerData.java:updateBlockCommitSequenceId(205)) - Container 7 updateBlockCommitSequenceId from higher value 919 to a lower value 915
   1 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.6.log -------------------------
3636: 2023-03-09 23:20:19,005 [ContainerOp-3ff18daf-f5be-47e7-ac67-58569d6f6fc5-3] ERROR keyvalue.KeyValueContainerData (KeyValueContainerData.java:updateBlockCommitSequenceId(205)) - Container 2 updateBlockCommitSequenceId from higher value 729 to a lower value 726
   1 occurrence(s)
-- ../multipartUpload_out/testMultipartParallel.9.log -------------------------
14756: 2023-03-09 23:40:38,914 [ContainerOp-0c7e9e30-9170-481a-8c80-369a446ffa4a-1] ERROR keyvalue.KeyValueContainerData (KeyValueContainerData.java:updateBlockCommitSequenceId(205)) - Container 54 updateBlockCommitSequenceId from higher value 13697 to a lower value 13687
   1 occurrence(s)
===============================================================================
Totally 5 occurrence(s) in 20 file(s).

@szetszwo
Copy link
Contributor Author

szetszwo commented Mar 10, 2023

After this fix, both problem are gone.

$search.sh ../multipartUpload_with_HDDS-8129_out/ testMultipartParallel.\* "updateBlockCommitSequenceId from higher value"

root dir    : ../multipartUpload_with_HDDS-8129_out/
file pattern: testMultipartParallel.*
search regex: updateBlockCommitSequenceId from higher value

===============================================================================
Totally 0 occurrence(s) in 20 file(s).

$search.sh ../multipartUpload_with_HDDS-8129_out/ testMultipartParallel.\* "Unexpected block count changed"

root dir    : ../multipartUpload_with_HDDS-8129_out/
file pattern: testMultipartParallel.*
search regex: Unexpected block count changed

===============================================================================
Totally 0 occurrence(s) in 20 file(s).

@arp7
Copy link
Contributor

arp7 commented Mar 10, 2023

+1 lgtm

@arp7 arp7 merged commit b1695e3 into apache:master Mar 10, 2023
errose28 added a commit to errose28/ozone that referenced this pull request Mar 16, 2023
@errose28
Merge branch 'master' into improve-upgrade-acc-tests
de7d66c 
* master: (262 commits)
  HDDS-8153. Integrate ContainerBalancer with MoveManager (apache#4391)
  HDDS-8090. When getBlock from a datanode fails, retry other datanodes. (apache#4357)
  HDDS-8163 Use try-with-resources to ensure close rockdb connection in SstFilteringService (apache#4402)
  HDDS-8065. Provide GNU long options (apache#4394)
  HDDS-7930. [addendum] input stream does not refresh expired block token.
  HDDS-7930. input stream does not refresh expired block token. (apache#4378)
  HDDS-7740. [Snapshot] Implement SnapshotDeletingService (apache#4244)
  HDDS-8076. Use container cache in Key listing API. (apache#4346)
  HDDS-8091. [addendum] Generate list of config tags from ConfigTag enum - Hadoop 3.1 compatibility fix (apache#4374)
  HDDS-8144. TestDefaultCertificateClient#testTimeBeforeExpiryGracePeriod fails as we approach DST. (apache#4382)
  HDDS-8151. Support fine grained lifetime for root CA certificate (apache#4386)
  HDDS-8150. RpcClientTest and ConfigurationSourceTest not run due to naming convention (apache#4388)
  HDDS-8131. Add Configuration for OM Ratis Log Purge Tuning Parameters. (apache#4371)
  HDDS-8133. Create ozone sh key checksum command (apache#4375)
  HDDS-8142. Check if no entries in Block DB for a container on container delete (apache#4379)
  HDDS-8118. Fail container delete on non empty chunks dir (apache#4367)
  HDDS-8028. JNI for RocksDB SST Dump tool (apache#4315)
  HDDS-8129. ContainerStateMachine allows two different tasks with the same container id running in parallel. (apache#4370)
  HDDS-8119. Remove loosely related AutoCloseable from SendContainerOutputStream (apache#4368)
  close db connection (apache#4366)
  ...
k5342 pushed a commit to pfnet/ozone that referenced this pull request May 10, 2023
@szetszwo @k5342
HDDS-8129. ContainerStateMachine allows two different tasks with the …
15b4122 
…same container id running in parallel. (apache#4370)

(cherry picked from commit b1695e3)
kuenishi pushed a commit to pfnet/ozone that referenced this pull request Aug 14, 2023
@szetszwo @kuenishi
HDDS-8129. ContainerStateMachine allows two different tasks with the …
e3d09a2 
…same container id running in parallel. (apache#4370)

(cherry picked from commit b1695e3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sumitagrawl sumitagrawl sumitagrawl approved these changes

@errose28 errose28 Awaiting requested review from errose28

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@szetszwo @arp7 @sumitagrawl