Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] feat: Basic IPNS support #539

Closed
wants to merge 704 commits into from
Closed

Conversation

ppodolsky
Copy link
Contributor

@ppodolsky ppodolsky commented Nov 22, 2022

I've found planned refactorings in GetRecords libp2p/rust-libp2p#2712 (CC: @dignifiedquire) to make it streaming. In this PR I've supported DhtGet queries to iroh-p2p and gateway. Proposed changes take into account further possible changes in libp2p::GetRecords.

All implemented APIs return Stream<PeerRecord> but streaming is not used yet, I'm just doing usual Quorum query, wait for response and send all received records through stream. It works, but there is also a lot of unwraps() and todo!() that I'll eliminate if general approach would be reviewed and accepted here.

dignifiedquire and others added 30 commits October 14, 2022 17:41
This focuses on creating tests for the `iroh get` CLI command. The Iroh API is also improved along the way.

This required more work in the underlying API to make it more testable. What's tested here is everything *except* the actual IPFS behavior -- the focus is on the behavior of the CLI and API and its interactions with the filesystem.

 ## trycmd tests

In `iroh/tests/cmd` you can see quite a few `.trycmd` files. These describes the command-line interaction. Stuff behind `$ is a command, there's a special `? failed` indicator if the command is considered to have failed, and the output of the command is shown.

New are the `.out` and `.in` directories with the same names as the `.trycmd` files. These describe the filesystem before the command runs (may be missing), and the filesystem after the command has run. This way we can describe the effects of the `iroh get` command - directories and files are supposed to be created. We can also test failure scenarios where we refuse to overwrite a directory that already exists. #269 tracks various test cases.

## `get_stream`

The `get` high level CLI method has been removed from the mockable `Api` trait (read on to see where it went). Instead, a more low level but still useful API method `get_stream` has been added to the `Api` trait. This gets a stream of relative paths and `OutType`, describing the directories and files you can create. 

The big difference with what was there before is that it returns relative paths and doesn't calculate final destination paths -- that's up to the user of the API.

## No `async_trait` macro for `Api` trait

The interactions between the `Api` trait, `mockall` macro and `async_trait` were getting so hairy I couldn't figure out how to express things anymore once I wanted to add `get_stream`. For my sanity and also to learn better how this really works underneath, I've rewritten the `Api` trait to describe itself explicitly in terms of `(Local)BoxFuture` and `(Local)_BoxStream`. It's more verbose but functionally equivalent and I could express what I wanted. 

## `ApiExt` trait

The `ApiExt` trait is a trait that implements the high level `get` command. It puts everything together: it handles various error conditions, accesses the stream and then writes the stream to the filesystem. It's basically `iroh get`.

The `ApiExt` trait is solely intended to contain default trait methods, and is automatically available when the `Api` contract is fulfilled (if you `use` it).

Factored out `save_get_stream` from `getadd.rs` to be solely concerned with turning a stream into files and directories on the files system. That makes it possible to test its behavior in isolation.

## test fixture

The `get` test fixture now mocks `get_stream` and returns a fake stream made from a `Vec`. This defines the actual stuff that the CLI writes to disk.

## relative_path

Now depend on the [`relative-path` crate](https://crates.io/crates/relative-path) because what the stream returns are clearly relative paths, and we want to force the user to do something with them before being able to actually write stuff.
* feat: range request support

* cr

* tests & fixes

* fix etag range headers

* rebase cleanup
…iroh` commands (n0-computer#348)

* docs: add full help text for `lookup`, `connect`, `get`, `p2p`, and
`iroh` commands

* move long descriptions into constants

Co-authored-by: b5 <[email protected]>
dignifiedquire and others added 18 commits December 21, 2022 17:40
…ature

fix(deps): We use the serde feature from Url
Newer clippy wants the identifiers in the format string when
possible.  Boring change but why not.
This makes sure to await the future from async_channel::Sender::send.
When not awaiting this only makes a struct and noting is ever sent.

Caught by clippy really.
…puter#656)

Updates the requirements on [sysinfo](https://github.com/GuillaumeGomez/sysinfo) to permit the latest version.
- [Release notes](https://github.com/GuillaumeGomez/sysinfo/releases)
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/commits)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Floris Bruynooghe <[email protected]>
…er#657)

Updates the requirements on [rlimit](https://github.com/Nugine/rlimit) to permit the latest version.
- [Release notes](https://github.com/Nugine/rlimit/releases)
- [Changelog](https://github.com/Nugine/rlimit/blob/master/CHANGELOG.md)
- [Commits](Nugine/rlimit@v0.8.3...v0.8.3)

---
updated-dependencies:
- dependency-name: rlimit
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Floris Bruynooghe <[email protected]>
…uter#659)

Updates the requirements on [base64](https://github.com/marshallpierce/rust-base64) to permit the latest version.
- [Release notes](https://github.com/marshallpierce/rust-base64/releases)
- [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md)
- [Commits](marshallpierce/rust-base64@v0.13.1...v0.13.1)

---
updated-dependencies:
- dependency-name: base64
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Floris Bruynooghe <[email protected]>
The store does not use the RPC client at all.  Remove it.
We used to use the same config for the service and server (aka the
binary).  This is confusing when creating configs to use with
e.g. iroh-one, iroh-embed or iroh-share because some fields are not
used.

This splits off the config structs to avoid this problem, services now
have a Config and binaries a ServerConfig.  This allows creating the
services standalone without all the baggage a server needs.

While this isn't many fields yet, this will get worse as we add more
features (this is split off from another PR where this seemed useful).
This removes the use of mockall which is causing a lot of trouble with
the cargo features intricacies it brings with it.  A lot of the
tooling struggles with these different versions of the structs which
also behave differently in surprising ways.

The tests affected by this will be converted into end-to-end tests as part of
is now unused.  This is possibly only temprorary and that might come
back when the end-to-end tests are added.
* feat: Add pub ctor to Directory

* fix: reduce logging level

* feat: Pretty printing for addrs

* fix: clippy

* pr fixes
)

* fix(iroh-share): force close gossip task on finish/done

* fmt

* fix
@ppodolsky ppodolsky force-pushed the feature/ipns branch 5 times, most recently from 3473d39 to 3c5ff3f Compare January 28, 2023 17:29
@lidel
Copy link

lidel commented Feb 5, 2023

Hi, just a heads-up that if you are implementing things, and signatureV2 is present, focus on validating values from data CBOR in V2 records, so it is less painful to refactor when we switch to V2-only and stop duplicating values (V2-only will only have data and signatureV2).

Wrote initial cleanup plan in ipfs/specs#376, there will be IPIP with details, but wanted to give you early ping on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.