Skip to content

Commit 7de00d5

Browse files
committed
container: reduce the number of fetched iterator items
An attempt to list 2K+ containers ended up this way: error handler/util.go:34 call method {"status": 500, "request_id": "0cedec2b-935d-4575-a0c2-94ed4650c978", "method": "ListBuckets", "bucket": "", "object": "", "description": "something went wrong", "error": "list user containers via connection pool: status: code = 1024 message = json error: too big: size"} We're trying to get 2K items from the iterator in a single call, each item is base64 of CID, so it's 42 bytes, but then it's wrapped into VM type/value struct like this: {"value":"","type":"ByteString"} which adds another 32 bytes. For 2K elements this yields 144K JSON (not including other data which is also present) which NeoGo refuses to return given that the maximum size is 128K (nspcc-dev/neo-go#3185). Fetching in smaller chunks should fix the problem (for other iterator-based methods as well). Signed-off-by: Roman Khimov <[email protected]>
1 parent a5ffd32 commit 7de00d5

File tree

2 files changed

+5
-2
lines changed

2 files changed

+5
-2
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Changelog for NeoFS Node
99
- Send on closed channel panic in node's new epoch handler (#3529)
1010
- Tomstoned objects revival not working (#3542)
1111
- Negative logic object counters in metabase (#3555)
12+
- Inability to list 2K+ containers via API (#3558)
1213

1314
### Changed
1415
- Stream payload without buffering to reduce memory usage in CLI `Get/Put` operations (#3535)

pkg/morph/client/container/client.go

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,10 @@ const (
4141

4242
// iteratorPrefetchNumber is a number of stack items to prefetch in the
4343
// first call of iterator-based methods. neo-go's stack elements default
44-
// limit is 2048, make it less a little.
45-
iteratorPrefetchNumber = 2000
44+
// limit is 2048, but JSON output is then limited to 128K, given that
45+
// each item has ~30 bytes overhead for type/value structure using large
46+
// values is dangerous here.
47+
iteratorPrefetchNumber = 512
4648
)
4749

4850
var (

0 commit comments

Comments
 (0)