Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support car-scope param #33

Closed
olizilla opened this issue Apr 12, 2023 · 1 comment
Closed

Support car-scope param #33

olizilla opened this issue Apr 12, 2023 · 1 comment

Comments

@olizilla
Copy link
Contributor

olizilla commented Apr 12, 2023 •
edited
Loading

?car-scope=block|file|all lets the user state if they want a single block , or the subset of blocks needed for a file or complete dir listing (could be single unixfs directory block or all hamt shards required to construct dir list) or all the blocks for the dag (traditional flavour, how freeway works today for CAR responses)... note that #32 still applies and CARs should also contain all additional blocks required to verify any path provided, regardless of car-scope.

?car-scope=block|file|all supersedes ?depth=0|1|all. Both may need to be supported in the short term but car-scope is where the work is going.

see: ipfs-inactive/bifrost-gateway#80
see: filecoin-saturn/L1-node#324 (comment) (note lassie has implemented block as root currently.

For context, from the Lassie car spec (which then goes on to describe the params in terms of depthType which seems to be another previous iteration on this)

All traversals will include the content at the termination of the path specifier, as well as all blocks from the root CID to the path terminus where a path is provided. At the termination of a provided path, or from the root CID where no path is provided, the depth of the DAG returned may vary depending on the request or available data.
– https://github.com/filecoin-project/lassie/blob/main/docs/CAR.md#dag-depth

### Tasks
- [ ] https://github.com/web3-storage/dagula/pull/8
- [ ] https://github.com/web3-storage/gateway-lib/pull/21
- [x] update freeway
- [x] deploy freeway
@olizilla olizilla mentioned this issue Apr 12, 2023
Open
@heyjay44 heyjay44 mentioned this issue Apr 12, 2023
Closed
@olizilla olizilla mentioned this issue Apr 14, 2023
Merged
olizilla added a commit to storacha/dagula that referenced this issue Apr 17, 2023
@olizilla
feat: getPath with carScope
6235baf 
add getPath method as a generator that returns blocks for the targeted dag and all blocks traversed while resolving a cid+path string

supports carScope to specify what blocks to return for the resolved dag
- 'all': return the entire dag starting at path. (default)
- 'block': return the block identified by the path.
- 'file': Mimic gateway semantics: Return All blocks for a multi-block file or just enough blocks to enumerate a dir/map but not the dir contents.

see: storacha/freeway#33
see: storacha/freeway#34

TODO:
- [] find out how to identify the boundaries of a unixfs hamt (unixfs-exported seems to define it as "not having an empty or null Link.Name after the first 2 chars are stripped, which seems risky... what happens if the actual dir listing has 2 char long link names? see: https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42

License: MIT
Signed-off-by: Oli Evans <[email protected]>
@olizilla olizilla mentioned this issue Apr 17, 2023
Merged
1 task
olizilla added a commit to storacha/dagula that referenced this issue May 1, 2023
@olizilla
feat: getPath with carScope (#8)
a613b45 
add getPath method as a generator that returns blocks for the targeted
dag and all blocks traversed while resolving a cid+path string

supports carScope to specify what blocks to return for the resolved dag
- `'all'`: return the entire dag starting at path. (default)
- `'block'`: return the block identified by the path.
- `'file'`: Mimic gateway semantics: Return All blocks for a multi-block
file or just enough blocks to enumerate a dir/map but not the dir
contents.

see: storacha/freeway#33
see: storacha/freeway#34
see: ipfs/specs#402

TODO:
- [x] find out how to identify the boundaries of a unixfs hamt 

...unixfs-exporter seems to define it as "not having an empty or null
Link.Name after the first 2 chars are stripped, which seems loose...
what happens if the actual dir listing has 2 char long link names? see:
https://github.com/ipfs/js-ipfs-unixfs/blob/e853049bd63d6773442e1540ae49b6a443ca8672/packages/ipfs-unixfs-exporter/src/resolvers/unixfs-v1/content/hamt-sharded-directory.ts#L20-L42

License: MIT

---------

Signed-off-by: Oli Evans <[email protected]>
Co-authored-by: Alan Shaw <[email protected]>
olizilla added a commit to storacha/gateway-lib that referenced this issue May 1, 2023
@olizilla
feat!: support ?car-scope and verfiable paths for format=car
c711076 
- update dagula to get `getPath` with carScope support https://github.com/web3-storage/dagula/releases/tag/v6.0.0
- update handleCar to extract ?car-scope query and use `dagula.getPath`

BREAKING CHANGE: CARs returned for cid+path will now be rooted at the root cid rather than the resovled cid for the end of the path and include all blocks needed to verify the path was traveresed correctly.

see: storacha/freeway#33
see: storacha/freeway#34

License: MIT
Signed-off-by: Oli Evans <[email protected]>
@olizilla olizilla mentioned this issue May 1, 2023
Merged
olizilla added a commit to storacha/gateway-lib that referenced this issue May 2, 2023
@olizilla
feat!: support ?car-scope and verifiable paths for format=car (#21)
7593a2f 
- update dagula to get `getPath` with carScope support
https://github.com/web3-storage/dagula/releases/tag/v6.0.0
- update handleCar to extract ?car-scope query and use `dagula.getPath`

BREAKING CHANGE: CARs returned for cid+path will now be rooted at the
root cid rather than the resovled cid for the end of the path and
include all blocks needed to verify the path was traveresed correctly.

see: storacha/freeway#33
see: storacha/freeway#34

License: MIT

---------

Signed-off-by: Oli Evans <[email protected]>
@olizilla
Copy link
Contributor Author

olizilla commented May 3, 2023

Completed in #35
Deployed as #36

@olizilla olizilla closed this as completed May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@olizilla