Skip to content

Implement production of redundant split objects carrying parity Reed-Solomon shards #3419

@cthulhu-rider

Description

@cthulhu-rider

Is your feature request related to a problem? Please describe.

to support EC with RS scheme, parity blocks (or shards) should be produced and stored along with the data ones. The system is not capable of doing this now

Describe the solution you'd like

for each K data objects produced from N bytes of original payload according to split scheme, R extra objects are created, making [1, ..., K, K+1, ..., K+R] list so that:

  1. all objects have L = N / K bytes of payload except K-th one which may have N mod K size
  2. having access to any of K+R objects, it is possible to resolve full [1, ..., K+R] list
  3. parity objects remain outside queries which is currently used to assemble objects without redundancy (split v2)

to comply with these requirements

  • each object has IDX: i attribute
  • each object has DATA: K attribute
  • each object has PARITY: R attribute
  • each object except 1st has SHARDS_FIRST: ID(1) attribute
  • each parity object has SHARDS_LAST: ID(K) attribute
  • each parity object has no split field

all attributes are __NEOFS__EC_RS_-prefixed

DATA and PARITY preserve list size, IDX - ordering. FIRST groups all shards, controls placement and distribution across container. LAST will help when last object is inaccessible (here)

Describe alternatives you've considered

unset split field may be wrong for GC. Parity objects should be removed along with data split members. To achieve this now, split.parent must be set to some chain member. Not root ID cuz we have 3rd requirement. Maybe the 1st?

Metadata

Metadata

Assignees

Labels

featureCompletely new functionalityneofs-storageStorage node application issues

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions