EC GET #3497

cthulhu-rider · 2025-08-01T12:45:19Z

based on Implement EC logic in PUT #3457 in terms of format

codecov · 2025-08-11T12:15:30Z

Codecov Report

❌ Patch coverage is 60.30534% with 156 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.01%. Comparing base (dd83657) to head (dffbbc8).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/services/object/get/ec.go	66.04%	67 Missing and 24 partials ⚠️
pkg/services/object/get/util.go	7.50%	36 Missing and 1 partial ⚠️
cmd/neofs-node/object.go	0.00%	9 Missing ⚠️
pkg/services/object/get/container.go	56.25%	5 Missing and 2 partials ⚠️
pkg/services/object/get/exec.go	73.91%	4 Missing and 2 partials ⚠️
internal/ec/ec.go	50.00%	2 Missing and 1 partial ⚠️
pkg/services/object/get/service.go	0.00%	2 Missing ⚠️
cmd/neofs-node/policy.go	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3497      +/-   ##
==========================================
+ Coverage   25.71%   26.01%   +0.29%     
==========================================
  Files         659      660       +1     
  Lines       49399    49766     +367     
==========================================
+ Hits        12705    12945     +240     
- Misses      35685    35784      +99     
- Partials     1009     1037      +28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

carpawell

Will there be issues for TODOs or some separate PR that fixes them? We once had work that was only related to going through every TODO to fix it or create a new issue for it.

pkg/services/object/get/ec.go

carpawell · 2025-08-29T08:28:44Z

pkg/services/object/get/ec.go

+		return object.Object{}, nil, errors.New("all nodes failed")
+	}
+
+	defer rc.Close()


can this panic in theory, if for above is empty?

only in theory, if some expectations of behavior are violated

pkg/services/object/get/ec.go

carpawell · 2025-08-29T08:31:01Z

pkg/services/object/get/ec.go

+		return *parentHdr, nil, nil
+	}
+
+	buf := make([]byte, partHdr.PayloadSize())


can it be gigabytes?

it can be up to MaxObjectSize net setting

oh, missed that there are part and parent headers (confusing naming btw, at least, i misunderstood)

it can be up to MaxObjectSize net setting

ok, but even in this case, is it ok to always allocate so much memory? found TODOs about streaming but isn't it necessary to solve ASAP?

isn't it necessary to solve ASAP

no. And not only due to not yet resolved streaming issue, but cuz decoder can now work in-mem only (#3501)

pkg/services/object/get/ec.go

pkg/services/object/get/service.go

cthulhu-rider · 2025-08-29T11:18:43Z

Will there be issues for TODOs or some separate PR that fixes them? We once had work that was only related to going through every TODO to fix it or create a new issue for it.

all TODOs will be resolved later. There are many of them, so i dont wanna create issue for each. Prefer to just have them in code

carpawell · 2025-08-29T16:23:51Z

Prefer to just have them in code

Dont mind, just wanted to hear that it is not a final solution and the next iteration will be soon.

roman-khimov

Unaddressed TODOs should have corresponding issues.

cmd/neofs-node/policy_test.go

internal/ec/ec.go

pkg/services/object/get/get.go

pkg/services/object/get/service.go

pkg/services/object/get/util.go

pkg/services/object/get/ec.go

roman-khimov · 2025-09-05T11:48:55Z

pkg/services/object/get/ec.go

+			return object.Object{}, err
+		}
+
+		fmt.Println(err)


carpawell

Except fmt.Println.

roman-khimov · 2025-09-08T07:46:54Z

Needs to be rebased now.

Follow cd3e5d3 for GET. Closes #3422. Signed-off-by: Leonard Lyubich <[email protected]>

Signed-off-by: Leonard Lyubich <[email protected]>

cthulhu-rider · 2025-09-08T08:14:46Z

pkg/services/object/get/ec.go

+	//  and parity servers work fast, availability can still be provided. Right now, for example, if one server
+	//  responds after the context deadline, whole operation fails. Think how this can be accurately improved.
+	// TODO: explore streaming options. See reedsolomon.NewStream. Cmp performance. https://github.com/nspcc-dev/neofs-node/issues/3501
+	// TODO: if routine in the loop fails, we may already be in a situation when we have to get parity parts.


cthulhu-rider · 2025-09-08T08:17:59Z

pkg/services/object/get/util.go

+func (c *clientCacheWrapper) InitGetObjectStream(ctx context.Context, node netmap.NodeInfo, pk ecdsa.PrivateKey,
+	cnr cid.ID, id oid.ID, sTok *session.Object, bTok *bearer.Token, local, verifyID bool, xs []string) (object.Object, io.ReadCloser, error) {
+	// TODO: code is copied from pkg/services/object/get/container.go:63. Worth sharing?
+	// TODO: we may waste resources doing this per request. Make once on network map change instead.


issues this

cthulhu-rider · 2025-09-08T08:23:19Z

pkg/services/object/get/util.go

+		return object.Object{}, nil, err
+	}
+
+	// TODO: SkipChecksumVerification() turns off checking all object checksums. Better to keep checking


cthulhu-rider · 2025-09-08T08:26:42Z

pkg/services/object/get/get.go

+			// TODO: track https://github.com/nspcc-dev/neofs-api/issues/269.
+			return errors.New("invalid request: EC part requested in container without EC policy")
+		}
+		// TODO: deny if node is not in the container?


cthulhu-rider · 2025-09-08T08:44:53Z

rebased and created issues for meaningful TODOs. Will open new PR for refactoring ones

cthulhu-rider force-pushed the ec-get branch 7 times, most recently from 1a25190 to 340a149 Compare August 11, 2025 12:12

cthulhu-rider force-pushed the ec-get branch 4 times, most recently from 17e09c9 to 7f286bf Compare August 12, 2025 12:29

This was referenced Aug 14, 2025

services/object: use stream API for Get operation #3466

Merged

Metabase EC #3530

Merged

cthulhu-rider force-pushed the ec-get branch 7 times, most recently from 689dbc4 to 243ea9c Compare August 26, 2025 14:19

cthulhu-rider marked this pull request as ready for review August 26, 2025 14:50

cthulhu-rider requested review from End-rey, carpawell and roman-khimov as code owners August 26, 2025 14:50

cthulhu-rider force-pushed the ec-get branch from 243ea9c to c468302 Compare August 28, 2025 15:07

carpawell reviewed Aug 29, 2025

View reviewed changes

cthulhu-rider force-pushed the ec-get branch from c468302 to 9c520f3 Compare August 29, 2025 11:14

cthulhu-rider force-pushed the ec-get branch from d8c4555 to 3779e74 Compare September 1, 2025 09:24

roman-khimov reviewed Sep 5, 2025

View reviewed changes

cthulhu-rider mentioned this pull request Sep 5, 2025

Split context in EC GET #3560

Open

cthulhu-rider force-pushed the ec-get branch 6 times, most recently from 64bd139 to db69124 Compare September 5, 2025 10:51

roman-khimov reviewed Sep 5, 2025

View reviewed changes

carpawell approved these changes Sep 5, 2025

View reviewed changes

cthulhu-rider force-pushed the ec-get branch from db69124 to 8f47d72 Compare September 8, 2025 07:34

cthulhu-rider force-pushed the ec-get branch 2 times, most recently from d4a62f0 to c94c0c4 Compare September 8, 2025 07:58

cthulhu-rider added 2 commits September 8, 2025 11:03

sn/object: Initial support for erasure coding policies by GET server

88497d1

Follow cd3e5d3 for GET. Closes #3422. Signed-off-by: Leonard Lyubich <[email protected]>

sn/object: Fix interface docs

dffbbc8

Signed-off-by: Leonard Lyubich <[email protected]>

cthulhu-rider force-pushed the ec-get branch from c94c0c4 to dffbbc8 Compare September 8, 2025 08:05

cthulhu-rider mentioned this pull request Sep 8, 2025

Sort EC rules by complexity and try simpler ones first #3563

Open

cthulhu-rider commented Sep 8, 2025

View reviewed changes

cthulhu-rider mentioned this pull request Sep 8, 2025

Optimize receiving parity parts when GET #3564

Open

cthulhu-rider commented Sep 8, 2025

View reviewed changes

cthulhu-rider mentioned this pull request Sep 8, 2025

Do not repeat conversion of SN netmap addresses #3565

Open

cthulhu-rider commented Sep 8, 2025

View reviewed changes

cthulhu-rider mentioned this pull request Sep 8, 2025

Check OID and payload checksum in EC SN-SN GET #3566

Open

cthulhu-rider commented Sep 8, 2025

View reviewed changes

cthulhu-rider mentioned this pull request Sep 8, 2025

Deny local-only read object requests when SN outside the container #3567

Open

roman-khimov merged commit 1a9aedd into master Sep 8, 2025
21 of 22 checks passed

roman-khimov deleted the ec-get branch September 8, 2025 10:04

EC GET #3497

EC GET #3497

Uh oh!

Conversation

cthulhu-rider commented Aug 1, 2025

Uh oh!

codecov bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

carpawell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cthulhu-rider commented Aug 29, 2025

Uh oh!

carpawell commented Aug 29, 2025

Uh oh!

roman-khimov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carpawell left a comment

Choose a reason for hiding this comment

Uh oh!

roman-khimov commented Sep 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cthulhu-rider commented Sep 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Aug 11, 2025 •

edited

Loading