Skip to content

HDDS-7426. (superseded) #3991

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
szetszwo wants to merge 44 commits into from
Closed

HDDS-7426. (superseded) #3991

szetszwo wants to merge 44 commits into from

Conversation

@szetszwo
Copy link
Contributor

@szetszwo szetszwo commented Nov 22, 2022

What changes were proposed in this pull request?

Add a new acceptance test for Streaming Pipeline.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7426

How was this patch tested?

NA

captainzmc and others added 30 commits November 7, 2022 10:03
@captainzmc @szetszwo
HDDS-5366. [Ozone-Streaming] Implement stream method to ContainerStat…
d4bf89a 
…eMachine. (#2358).  Contributed by mingchao zhao

(cherry picked from commit 16bafc6)
(cherry picked from commit b3495e8d39ab6ce09637f9ef05acfcf66e3087b8)
(cherry picked from commit e15e987)
@kaijchen @szetszwo
HDDS-5452. Add link method to ContainerStateMachine for Ratis streami…
92a8081 
…ng (#2422)

(cherry picked from commit b6e2143)
(cherry picked from commit 425bc95539a8de334125f4bfe38aff93f1730f5d)
(cherry picked from commit 5626566)
@kaijchen @szetszwo
HDDS-5481. Fix stream() and link() method in ContainerStateMachine. (#…
28d3035 
…2451)

(cherry picked from commit bda7004)
(cherry picked from commit 444b7b38406fbcc4c6ec957a10273ff1ee58cdfb)
(cherry picked from commit 01beaf8)
@captainzmc @szetszwo
HDDS-5480. [Ozone-Streaming] Client and server should support stream …
c69955e 
…setup. (#2452)

(cherry picked from commit 5230630)
(cherry picked from commit 0a4bbf5c7b1d7190df818c3828110fcba4261704)
(cherry picked from commit 0568725)
@kaijchen @szetszwo
HDDS-5488. [Ozone-Streaming] Add a new BlockOutputStream/KeyOutputStr…
e6be97e 
…eam to support streaming api (#2495)

(cherry picked from commit d33d9cb)
(cherry picked from commit 18db09d7d4e20f020b9fe2a73d9daac97b2ce51c)
(cherry picked from commit 0952195)
@captainzmc @szetszwo
HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …
db89077 
…buffer copying (#2557)

(cherry picked from commit b1f1051)
(cherry picked from commit 6b51811d1f0605e95d52441579190865064b9d05)
(cherry picked from commit ae462a6)
@kaijchen @szetszwo
HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to ByteBuffer…
8853c8a 
…StreamOutput (#2603)

(cherry picked from commit 3ea9af3)
(cherry picked from commit 74c5f0f5fca4475bb4369fd89ab07424fa87008b)
(cherry picked from commit 81b9f3d)
@captainzmc @szetszwo
HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
72bc750 
(cherry picked from commit bb45118)
(cherry picked from commit b7b16c69778c63ebd08a2e029a1917893c7d2c4f)
(cherry picked from commit b2133a5)
@captainzmc @szetszwo
HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…
8124a1c 
…de (#2682)

(cherry picked from commit 2e85eeb)
(cherry picked from commit 10b14300d274cf224f56763c66dea9c2d8a6f731)
(cherry picked from commit 2791680)
@captainzmc @szetszwo
HDDS-5849. [Ozone-Streaming]Write exceptions occur after checksum is …
a6968d5 
…enabled (#2729)

(cherry picked from commit aea1d84)
(cherry picked from commit 79507fe72f98a0f02a0cad88066437fbb7d63bad)
(cherry picked from commit b1f06e8)
@sadanand48 @szetszwo
HDDS-5674.[Ozone-Streaming] Handle client retries on exception (#2701)
3958708 
(cherry picked from commit 3daf3d7)
(cherry picked from commit 508fda8744a9bd04bb1cb6c4ee12df5581e4dded)
(cherry picked from commit 5b73b51)
@captainzmc @szetszwo
HDDS-5895. [Ozone-Streaming] Make raft.server.data-stream.client.pool…
483eae8 
….size configurable (#2766)

(cherry picked from commit 47ae948)
(cherry picked from commit ca7a926165aa585195aa1f277cbb52f659ddbb29)
(cherry picked from commit 11fb9bf)
@szetszwo
HDDS-5763. Provide an Executor for each LocalStream in ContainerState…
e4a0f00 
…Machine (#2782)

(cherry picked from commit e186bfc)
(cherry picked from commit 311b36652697155b052cd9781fd1590d22256660)
(cherry picked from commit 7cbb3fa)
@guohao-rosicky @szetszwo
HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream config (#2841
029055b 
)

(cherry picked from commit 9204b35)
(cherry picked from commit bb4eef2f7c90f1f919574307af05dc70a07f57dd)
(cherry picked from commit 3cf0877)
@guohao-rosicky @szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
7290ff9 
…the stream write (#2833)

(cherry picked from commit ddc8a2c)
(cherry picked from commit 65ec5817e22a666a4fa4fe76ac5c0c1c3263865a)
(cherry picked from commit 8cbadbc)
@guohao-rosicky @szetszwo
HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStrea…
2054fe4 
…mKey method (#2760)

(cherry picked from commit be9641f)
(cherry picked from commit 9a16e41e1f29373e0e0f2a186237ded0dd0b53b4)
(cherry picked from commit 8de50af)
@sadanand48 @szetszwo
HDDS-5743. [Ozone-Streaming] Add option to write files via streaming …
d81ca2a 
…api in ofs and o3fs. (#2770)

(cherry picked from commit f3a2451)
@sadanand48 @szetszwo
HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer fixed bounda…
9d5ccb3 
…ry for streaming writes. (#2866)

(cherry picked from commit 7a6ea80)
(cherry picked from commit 3b9141e18d42555ccc62f7c9718b96fd8b88e55f)
(cherry picked from commit eb441a8)
@sadanand48 @szetszwo
HDDS-6039. Define a minimum packet size during streaming writes. (#2883)
28fdecd 
(cherry picked from commit 1f56a45)
(cherry picked from commit 750c5f475a4a03cdb078699d5f5a2254f1f491b0)
(cherry picked from commit 7e61b1f)
@captainzmc @szetszwo
HDDS-6130. [Ozone-Streaming] When releaseBuffers will get “Couldn 't …
58da69f 
…find the required future” (#2939)

(cherry picked from commit 2aacc7a)
(cherry picked from commit 1ecb774a39971f264325b6ce2ab0d4cbe4025ee6)
(cherry picked from commit ac981a7)
@sadanand48 @szetszwo
HDDS-6139. [Ozone-Streaming] Fix incorrect computation of totalAckDat…
156cd46 
…aLength. (#2978)

(cherry picked from commit 0db32a0)
(cherry picked from commit 6bd6cc7f2d605168432930f55138cba5b582f984)
(cherry picked from commit 25c3495)
@sadanand48 @szetszwo
HDDS-6178. [Ozone-Streaming] Fix NPE in HDDS-6139. (#2984)
ddf9f84 
(cherry picked from commit 72e1e02)
(cherry picked from commit 151ee53cd71442a272688840d0a5bfc01816aa11)
(cherry picked from commit 33c0260)
@guohao-rosicky @szetszwo
HDDS-6281. Update ratis version to 2.3.0-94db58b-SNAPSHOT version (#3059
dd62684 
)

(cherry picked from commit 84728e0)
(cherry picked from commit fdd7b641562377404c2bfe4080d2b56c789b261e)
(cherry picked from commit b3ec548)
@sadanand48 @szetszwo
HDDS-6138.[Ozone-Streaming] Define a limit on the size of the retry b…
340f959 
…ufferList. (#2946)

(cherry picked from commit 30f8311)
(cherry picked from commit 54361e5a4ded763d83c5b3116203deece0303960)
(cherry picked from commit 693eea7)
@guohao-rosicky @szetszwo
HDDS-6298. Add XceiverServerRatis stream config (#3070)
20f1682 
(cherry picked from commit 113a56e)
(cherry picked from commit f59d30da451fe29b0486b4e9507ae74bd184f440)
(cherry picked from commit 1a31d6b)
@captainzmc @szetszwo
HDDS-5487. [Ozone-Streaming] BlockDataStreamOutput support FlushDelay. (
9c6904d 
#3002)

(cherry picked from commit 366a0d2)
(cherry picked from commit 576c359114aba78d203520a5edf62ff831cf5919)
(cherry picked from commit 434d1fc)
@guohao-rosicky @szetszwo
HDDS-6282. Fix BlockDataStreamOutput#doFlushIfNeeded NPE (#3060)
fce0dfa 
(cherry picked from commit 8f8f166)
(cherry picked from commit 9d24f5870443ad3a9f8b6a508bcc1426e3c8bc4f)
(cherry picked from commit cca5e3b)
@guohao-rosicky @szetszwo
HDDS-6229. [Ozone-Streaming] Data Channel abstraction on datanode (#3023
4ce474c 
)

(cherry picked from commit 3cc2a7e)
(cherry picked from commit de690f6c259437bdce4389c7b06b189fa4cd20c0)
(cherry picked from commit 3dde9e2)
@guohao-rosicky @szetszwo
HDDS-6355. [Ozone-Streaming] Fix CheckStyle problem (#3119)
328c389 
(cherry picked from commit e635393)
(cherry picked from commit c1bc998af667eae6a404c127f017ce724f5415a3)
(cherry picked from commit b2ec47d)
@captainzmc @szetszwo
HDDS-6388. [Ozone-Streaming] Streaming write support both pipeline mo…
1f097e1 
…del and star model (#3145)

(cherry picked from commit ab61b71)
(cherry picked from commit caa3b30a1e9336dcbae0600180c550a4806bcf3e)
(cherry picked from commit 8bbe943)
szetszwo and others added 14 commits November 7, 2022 10:07
@szetszwo
HDDS-5798. [Ozone-Streaming] Setup TlsConf parameters. (#3207)
b42829c 
(cherry picked from commit 181a17e)
(cherry picked from commit 22e77a7b3d963bae49b93ab26a4eedb615cd1511)
(cherry picked from commit 8e6be78)
@guohao-rosicky @szetszwo
HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)
06bc968 
(cherry picked from commit 88c4d59)
(cherry picked from commit 0a432aa0eb57bd25d63e656d0ec6e96a05689ac7)
(cherry picked from commit 99b414d)
@szetszwo
HDDS-6500. [Ozone-Streaming] Buffer the PutBlockRequest at the end of…
d0407c1 
… the stream. (#3229)

(cherry picked from commit d72a8a0)
(cherry picked from commit 1dddbec2524ca3463aa6140f33c6f16f0468fce8)
(cherry picked from commit f2d574c)
@sadanand48 @szetszwo
HDDS-5666. Add option to createKey via streaming api in Freon (#2574)
d1b7a92 
(cherry picked from commit 51b5395)
(cherry picked from commit a2501c3de3052a6bfc9729a73fbb99453dbfdd72)
(cherry picked from commit 4a5b683)
@guohao-rosicky @szetszwo
HDDS-6592. [Ozone-Streaming] Fix ContainerStateMachine#applyTransacti…
a7eb40e 
…on assert error (#3315)

(cherry picked from commit 9d0b07e)
(cherry picked from commit f850883767b306b7065ad690acec87bb7b8a24cb)
(cherry picked from commit 7805efb)
@szetszwo
HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in S…
317053d 
…treamCommitWatcher. (#3492)

(cherry picked from commit 8315f08)
(cherry picked from commit 4c064f42aaa027baa0ca0c7ed373a143a26a6cec)
(cherry picked from commit a5980ce)
@captainzmc @szetszwo
HDDS-6867. [Ozone-Streaming] PutKeyHandler should not use streaming t…
b3a6cfc 
…o put EC key. (#3516)

(cherry picked from commit 2227d33)
(cherry picked from commit 04ebe898435bbef40228751590cd26f1d1e2f051)
(cherry picked from commit 29a8c15)
@kaijchen @szetszwo
HDDS-6955. [Ozone-streaming] Add explicit stream flag in ozone shell (#…
6319ce8 
…3559)

* Add stream option in ozone sh key

(cherry picked from commit 4c7151b)
(cherry picked from commit c84da64fbcd34e62b17f1bd2a52238b0497952e8)
(cherry picked from commit 653a120)
@szetszwo
HDDS-7431. [Ozone-Streaming] Disable data steam by default. (#3900)
53af908 
(cherry picked from commit 88e1d1e)
@szetszwo
HDDS-7438. [Ozone-Streaming] Add a createStreamKey method to OzoneBuc…
cfdf9cd 
…ket. (#3914)

(cherry picked from commit eedd983)
@szetszwo
HDDS-7425. Add documentation for the new Streaming Pipeline feature. (#…
7fc85ee 
…3913)
@szetszwo
HDDS-7478. [Ozone-Streaming] NPE in when creating a file with o3fs. (#…
c17be70 
…3949)
@szetszwo
HDDS-7426. Add a new acceptance test for Streaming Pipeline.
0400358
@szetszwo
Enable Streaming in ozonesecure.
eb775c2
@kerneltime
Copy link
Contributor

kerneltime commented Nov 28, 2022

@duongkame

@kerneltime
Copy link
Contributor

kerneltime commented Nov 28, 2022

cc @swamirishi

@szetszwo szetszwo mentioned this pull request Nov 29, 2022
Merged
@szetszwo
Copy link
Contributor Author

szetszwo commented Nov 29, 2022

The test for secure ozone clusters currently is failing. It needs apache/ratis#791 .

duongkame
duongkame approved these changes Nov 30, 2022
View reviewed changes
Copy link
Contributor

@duongkame duongkame left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test looks good, yet, it's failing Test timeout 5 minutes exceeded.. Can you look into it?

@adoroszlai adoroszlai marked this pull request as draft November 30, 2022 05:53
@szetszwo szetszwo mentioned this pull request Dec 1, 2022
Merged
@szetszwo
Copy link
Contributor Author

szetszwo commented Dec 1, 2022

Sorry that I have accidentally used apache/ozone git repo instead of my fork. Submitted #4019 to supersedes this.

@szetszwo szetszwo changed the title HDDS-7426. Add a new acceptance test for Streaming Pipeline. HDDS-7426. (superseded) Dec 1, 2022
@szetszwo szetszwo closed this Dec 1, 2022
@szetszwo
Copy link
Contributor Author

szetszwo commented Dec 1, 2022 •
edited
Loading

The test looks good, yet, it's failing Test timeout 5 minutes exceeded.. Can you look into it?

@duongkame , thank for reviewing this. The secure fs test needs apache/ratis#791 .

BTW, moved this to #4019 .

@szetszwo szetszwo deleted the HDDS-7426 branch December 5, 2022 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@duongkame duongkame duongkame approved these changes

@adoroszlai adoroszlai Awaiting requested review from adoroszlai

@kerneltime kerneltime Awaiting requested review from kerneltime

@neils-dev neils-dev Awaiting requested review from neils-dev

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@szetszwo @kerneltime @duongkame @captainzmc @kaijchen @sadanand48 @guohao-rosicky