Implemented experimental ptp(corenet) interface #3943

magik6k · 2017-05-26T21:27:43Z

This is built on #3518.

TODO:

Write some more tests/get good coverage
Test in some small real-world use
Cleanup the code a bit more

whyrusleeping · 2017-05-26T23:12:04Z

core/commands/corenet.go

+
+var CorenetCmd = &cmds.Command{
+	Helptext: cmds.HelpText{
+		Tagline: "Application network streams.",


tagline: "libp2p stream mounting"
shortdescription: "expose a local application to remote peers over libp2p

Note: this command is experimental and subject to change as usecases and APIs are refined"

cc @Kubuxu @lgierth

whyrusleeping · 2017-05-26T23:13:26Z

Lets add a config option Experimental.Libp2pStreamMounting and only enable this command if that variable is set to true.

Kubuxu · 2017-05-28T22:00:03Z

test/sharness/t0180-corenet.sh

@@ -26,7 +26,16 @@ test_expect_success "test ports are closed" '
  (! (netstat -ln | grep "LISTEN" | grep ":10102 "))
 '

-test_expect_success 'start ipfs listener' '
+test_expect_success 'fail without config option being enabled' '
+  ! ipfsi 0 exp corenet ls


Use test_must_fail instead of ! here.

whyrusleeping · 2017-05-30T00:02:18Z

Looking good. The one last thing to consider now is whether or not we should put it under the 'exp' subcommand, or just keep it in the normal namespace. The other experimental commands (filestore and pubsub) exist in the top level, so i'm leaning towards doing that. Also not sure on the naming of 'corenet'.

What do you think @Kubuxu @lgierth @hsanjuan ? (also, @hsanjuan you might be able to rework ipfs cluster to use this feature once its merged :D )

whyrusleeping · 2017-05-30T04:53:58Z

I'm leaning towards having this in the top level namespace (not under an exp command). The whole exp idea came before the config setting anyways.

magik6k · 2017-05-30T08:47:15Z

I still want/need to move all the registry logic to a root package, and make it have instance in node instead of 1 global now. I don't like the naming too, but after trying to come up with something short that would make sense (other than p2pnet, but I'm not sure) I decided to leave corenet, though if you can think of anything that would fit I'll happily change it.

Kubuxu · 2017-05-30T10:32:34Z

I am also not really attached to the name.

ghost · 2017-05-31T01:12:39Z

I'll look at this tomorrow -- awesome work! :)

License: MIT Signed-off-by: Tom Swindell <[email protected]>

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

whyrusleeping · 2017-06-03T00:35:54Z

ptp/apps.go

+}
+
+// AppRegistry is a collection of local application protocol listeners.
+type AppRegistry struct {


probably want to add some locking here to make it threadsafe. I can definitely see people running these commands in parallel

whyrusleeping · 2017-06-03T00:36:13Z

ptp/apps.go

+}
+
+// Deregister deregisters protocol handler from this registry
+func (c *AppRegistry) Deregister(proto string) error {


will also want to add a listing method that does proper locking

ghost · 2017-06-03T02:30:00Z

Awesome work :):) 🌴 I think the use cases here are a bit underexplained, I don't see any showstoppers with the approach taken, but I'm just not clear enough on the use cases :) Is it just a simple 1-to-1 netcat? Or 1-to-n? What is the UX for accepting more than one connection?

Design questions:

Why involve listeners and dialers at all? This feels like the layer where you wouldn't deal with IP addresses anymore, only with PeerIDs (or with whatever ipfs swarm connect would accept). And, if you bind an App to a listener here, wouldn't that also involve all other stream handlers too?
It's worth considering generalizing this more, to something that just simply manages and exposes stream handlers, connections, and streams. Identify, Bitswap, and the others can be just special cases of that, where go-ipfs itself already has the code to mount them. That'd mean you could adhoc enable or disable the core protocols of IPFS, kill individual streams with individual peers, etc.
A propos Identify, will Identify announce what's called Apps here?

I haven't thoroughly CRed the code itself yet, but I think we can keep this whole package free of dependencies on go-ipfs code. There are a few instances where you depend on core.IpfsNode but actually just use its go-libp2p-host.Host and go-libp2p-peer.ID.

magik6k · 2017-06-03T10:35:56Z

For now it's more of a tunneling thing (like ssh port forwarding). For more than one connection I'm thinking of a dial option to make it forward more than one connection (so behave like ssh forwarding), perhaps even making it the default behavior and have option for one-time dial.

I decided to use dialers as there are no easily muxable API systems yet (websocket/custom proto/etc.). I'm not really sure what you mean by other stream handlers being involved, but the answer is probably no - have you seen ptp/net/net.go?
With this code you shouldn't be able to manage built-in stream handlers - the ones used here are prefixed with /app/ prefix(which I will likely change to /ptp/). For now it's only tunnels over libp2p.
I don't really know what do you mean by Identify here.

Other than that, some naming conventions are still going to change to make things more clear (app was left from the base PR).

whyrusleeping · 2017-06-03T16:30:51Z

@lgierth identify should announce these protocols as it just grabs whatever is registered on the host. I think we can keep designing this as we move forward. It's clearly marked as experimental so making api and design changes post merge is perfectly acceptable. Once people start using it and requesting different use cases, we will have a much better idea of what exactly is needed to generalize it better.

…

On Sat, Jun 3, 2017, 03:36 Łukasz Magiera ***@***.***> wrote: For now it's more of a tunneling thing (like ssh port forwarding). For more than one connection I'm thinking of a dial option to make it forward more than one connection (so behave like ssh forwarding), perhaps even making it the default behavior and have option for one-time dial. - I decided to use dialers as there are no easily muxable API systems yet (websocket/custom proto/etc.). I'm not really sure what you mean by other stream handlers being involved, but the answer is probably no - have you seen ptp/net/net.go? - With this code you shouldn't be able to manage built-in stream handlers - the ones used here are prefixed with /app/ prefix(which I will likely change to /ptp/). For now it's only tunnels over libp2p. - I don't really know what do you mean by Identify here. Other than that, some naming conventions are still going to change to make things more clear (app was left from the base PR). — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#3943 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABL4HKIUr1e62cfF4ADDYs2U6lQF4mWLks5sATcOgaJpZM4NoE8K> .

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

whyrusleeping

This LGTM. like i said earlier, I want to get this in so people who have cool use cases for it can start playing with it. Once we get their feedback we can refine how things work better, and also once we get websockets in the api we can use that as an option.

@lgierth @Kubuxu sound good to you? I'd love to get this into the next release quite soon

Kubuxu · 2017-06-07T13:08:39Z

core/commands/ptp.go

+		if !n.OnlineMode() {
+			res.SetError(errNotOnline, cmds.ErrClient)
+			return
+		}


IMO I would compress code above into once function (returning node or error in case those checks fail) as it is used everywhere.

Kubuxu · 2017-06-07T13:12:11Z

core/commands/ptp.go

+
+			handlerID, err = strconv.ParseUint(req.Arguments()[0], 10, 64)
+			if err != nil {
+				proto = "/ptp/" + req.Arguments()[0]


What if someone creates protocol that is identified by decimal numbers?

If this is a problem right now, please fix it and create sharness test for it.

Kubuxu · 2017-06-07T13:14:08Z

core/commands/ptp.go

+					break
+				}
+			}
+		}


The logic here is quite complex, anyway to make it much simpler to read?

Probably it would be better to completly split off the close all, with handler, and with protocol into separate functions. Maybe even place them in the ptp package.

Kubuxu · 2017-06-07T13:29:36Z

test/dependencies/ma-pipe-unidir/main.go

+	case "send":
+		io.Copy(conn, os.Stdin)
+	default:
+		//TODO: a bit late


Resolve this todo.

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

whyrusleeping · 2017-06-10T02:55:59Z

Here goes! :)

Implemented experimental ptp(corenet) interface

magik6k force-pushed the feat/corenet2 branch 3 times, most recently from 283965d to 672f69d Compare May 26, 2017 22:23

whyrusleeping reviewed May 26, 2017

View reviewed changes

whyrusleeping closed this May 26, 2017

whyrusleeping reopened this May 26, 2017

whyrusleeping added the status/in-progress In progress label May 26, 2017

Kubuxu reviewed May 28, 2017

View reviewed changes

magik6k force-pushed the feat/corenet2 branch 2 times, most recently from 795f028 to 7fcb9d0 Compare May 29, 2017 17:32

whyrusleeping requested review from a user and hsanjuan May 30, 2017 00:02

magik6k force-pushed the feat/corenet2 branch 3 times, most recently from 24d5f04 to 42e9040 Compare May 30, 2017 20:21

Tom Swindell and others added 8 commits May 31, 2017 11:33

Experimental corenet application support.

2f17a67

License: MIT Signed-off-by: Tom Swindell <[email protected]>

Corenet API: Fixed tests

a55255f

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Corenet API: Use simple util instead of socat for tests

ad1dc5f

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Corenet API: Split list subcmd into ls/streams

9414e73

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Corenet API: Some more tests

69d3e08

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Corenet API: Apply suggestions, cleanups

fc08d53

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Corenet API: Store state in node

8948519

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Corenet API: Drop 'exp' namespace

71bda3d

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k force-pushed the feat/corenet2 branch from 42e9040 to 71bda3d Compare May 31, 2017 09:33

magik6k force-pushed the feat/corenet2 branch from 2adc639 to eac7184 Compare June 2, 2017 19:44

magik6k changed the title ~~Implemented experimental corenet interface~~ Implemented experimental ptp(corenet) interface Jun 2, 2017

whyrusleeping reviewed Jun 3, 2017

View reviewed changes

magik6k force-pushed the feat/corenet2 branch from 4816a6c to 55e656a Compare June 3, 2017 11:13

PTP API: Rename Corenet to PTP

ad9ae35

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k force-pushed the feat/corenet2 branch 2 times, most recently from 43549df to 712a522 Compare June 3, 2017 21:43

PTP API: Make code more object oriented, use less node

104268f

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k force-pushed the feat/corenet2 branch from 712a522 to 104268f Compare June 4, 2017 01:08

whyrusleeping approved these changes Jun 7, 2017

View reviewed changes

Kubuxu reviewed Jun 7, 2017

View reviewed changes

magik6k force-pushed the feat/corenet2 branch from e8a5bb3 to aab4303 Compare June 7, 2017 17:47

PTP API: Address review comments

043aa5f

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k force-pushed the feat/corenet2 branch from aab4303 to 043aa5f Compare June 7, 2017 17:51

Kubuxu approved these changes Jun 9, 2017

View reviewed changes

whyrusleeping merged commit 18d2a2d into ipfs:master Jun 10, 2017

whyrusleeping removed the status/in-progress In progress label Jun 10, 2017

whyrusleeping added this to the Ipfs 0.4.10 milestone Jun 10, 2017

This was referenced Jun 13, 2017

Expose corenet interface #3128

Closed

Corenet exposure ipfs/notes#160

Closed

Kubuxu mentioned this pull request Jun 13, 2017

Implemented experimental corenet application protocol support. #3518

Closed

laurentsenta pushed a commit to laurentsenta/kubo that referenced this pull request Feb 25, 2022

Merge pull request ipfs#3943 from magik6k/feat/corenet2

3cbf801

Implemented experimental ptp(corenet) interface

laurentsenta pushed a commit to laurentsenta/kubo that referenced this pull request Feb 25, 2022

go-ipfs-config: Merge pull request ipfs#3943 from magik6k/feat/corenet2

4634d3a

Implemented experimental ptp(corenet) interface

laurentsenta pushed a commit to laurentsenta/kubo that referenced this pull request Mar 4, 2022

go-ipfs-config: Merge pull request ipfs#3943 from magik6k/feat/corenet2

90b6d80

Implemented experimental ptp(corenet) interface

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented experimental ptp(corenet) interface #3943

Implemented experimental ptp(corenet) interface #3943

magik6k commented May 26, 2017 •

edited

Loading

whyrusleeping May 26, 2017

whyrusleeping commented May 26, 2017

Kubuxu May 28, 2017

whyrusleeping commented May 30, 2017

whyrusleeping commented May 30, 2017

magik6k commented May 30, 2017 •

edited

Loading

Kubuxu commented May 30, 2017

ghost commented May 31, 2017

whyrusleeping Jun 3, 2017

whyrusleeping Jun 3, 2017

ghost commented Jun 3, 2017

magik6k commented Jun 3, 2017

whyrusleeping commented Jun 3, 2017 via email

whyrusleeping left a comment

Kubuxu Jun 7, 2017

Kubuxu Jun 7, 2017

Kubuxu Jun 7, 2017

Kubuxu Jun 7, 2017

whyrusleeping commented Jun 10, 2017

Implemented experimental ptp(corenet) interface #3943

Implemented experimental ptp(corenet) interface #3943

Conversation

magik6k commented May 26, 2017 • edited Loading

Choose a reason for hiding this comment

whyrusleeping commented May 26, 2017

Choose a reason for hiding this comment

whyrusleeping commented May 30, 2017

whyrusleeping commented May 30, 2017

magik6k commented May 30, 2017 • edited Loading

Kubuxu commented May 30, 2017

ghost commented May 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Jun 3, 2017

magik6k commented Jun 3, 2017

whyrusleeping commented Jun 3, 2017 via email

whyrusleeping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping commented Jun 10, 2017

magik6k commented May 26, 2017 •

edited

Loading

magik6k commented May 30, 2017 •

edited

Loading