pub/sub - publish / subscribe #64
Comments
|
Cc #42 |
|
It might be easier start with a basic [slow] implementation before doing the high performance multicast P2P thing. For example, this paper (https://www.cs.utexas.edu/~yzhang/papers/mad-info09.pdf) has two modes depending on how active the group is. The basic implementation would be as simple as:
Encrypted streams with group-specific encryption, multicast and multiple publishers can then be deferred to the more advanced implementation. |
|
I guess the basic could be made even simpler by using the backlink ideas (ipfs/ipfs#31) instead of a new interface to the routing. You then only need a new P2P message to notify a node of changes. |
|
I stumbled upon this "Decentralized Reddit using a DHT to store content and a blockchain to rank it" The first comment seems particularly relevent: [1] http://liamz.co/wp-content/uploads/2015/03/Computer-Science-Extended-Essay_Liam-Edwards-Playne.pdf I'm reading the papers, but it seems like some very interesting discussion for pubsub. In particular Poldercast's finding a subset of nodes with a particular interest and only those nodes host that interest reminds me of IPFS's policy of not downloading anything unless you ask for it. I'll read the papers before I make any specific pub/sub recommendations. |
|
@bharrisau agreed on starting simple to get something working and moving to more efficient constructions later. @spikebike [1] link is broken. |
|
The xmpp stuff proved really hard in comparison to federation doing pubsubhubub. |
|
Maybe not new to folks here, but new to me: |
|
Now I see that discussion about the use of Bloom filters is already underway: |
|
See also http://www.w3.org/TR/push-api/ (HT @nicola) |
|
Came across this (IPFS) while looking for Its easy to setup and highly performant. However, the major problem with WAMP is: it is 'routed' / 'brokered'. If there is a way this WAMP can be integrated with webRTC/bittorrent (webTorrent), for P2P, then it would pave way for next generation IOT apps. For PubSUB - here is one sample functionality what we are trying to achieve with IOT:
Similarly for RPC - here is one sample functionality that we are looking to achieve:
Is it possible to achieve above kind of functionality with IPFS? If not yet, I would suggest to strongly consider integrating WAMP as pubsub+rpc part of the protocol (rather than reinventing the wheel). Autobahn implementations of WAMP comes with clients for many languages and performance metrics are also very good. |
|
@KrishnaPG as soon as pub/sub is implemented, you should be able to do all that! You can use IPNS to expose each node's chunks. You can aggregate the data by having a list of the nodes on each node. For RPC calls, you can't do that using IPFS, but you can have the nodes download the code to execute from somewhere using IPNS, publish their results, then you'd have to aggregate them. You can't really do the RPC stuff you described but your usecase would work really well using only IPFS for all your networking (assuming it has pub/sub implemented) you just need to implement that functionality in a different way. Also keep in mind that IPFS has global deduplication 👍 But pub/sub is not implemented yet. |
|
Thank you @fazo96 . Yes, you are right - Doing RPC (moving the code to the data and executing it locally on each chunk) may require an additional layer on top of IPFS (which involves additional functionality that involves job queues, schedulers, retry mechanism and result aggregators). If we look at it, fundamentally RPC is kind of opposite to the basic file-read or pub/sub (in terms of data flow direction). For example, in a simple file-read/copy, data goes from the disk/chunk --> the client/reader. Whereas, in RPC the data (the code to be executed) has to go from the client --> the disk/chunk and get executed, and the results should either go back to the client (if the size is small) or get stored as additional files/chunks on the local machine (if the results data size is large). This requirement to be able to create additional chunks/files on the host locally may need support from the base IPFS, though. On the other-hand, there is another radical way to look at this. That is, treating every operation (including file_read, copy, delete etc.) as RPC call, and allowing transformations over basic operations. For example, consider this In a normal read operation, the And when we need to execute, say
This model treats RPC as first-class citizen (where all the regular operations, such as file CRUD operations and notifications are implemented on top of RPC). Not sure how difficult/easy it would be to do this with present architecture, though. For the pubsub, wondering how easy/complex it would be to reuse/integrate the Autobahn. If it can be done, then RPC comes for free on top of it (Demos). |
|
AFAICT, autobahn requires websockets, that's too much, we need something more basic that can run over any transport. we also need a pub/sub that can scale to millions of subscribers -- this isnt going to cut it: http://wamp-proto.org/why/#unified_routing -- we need a protocol that creates spanning trees/dags with fan out on its own, using measured latencies/bw in the network, etc. basically, a serious protocol from the multicast literature. |
|
@KrishnaPG you should read more into how IPFS works and how the protocols work. suggest also looking at: |
|
Thanks @jbenet I was looking for the spec info, your pointers are helpful. As for WAMP, yes - it started out as websocket based initially. Now, it is decoupled and works with any message based transport (http://wamp-proto.org/faq/#is_websocket_necessary_for_wamp) However, my intention in pointing to WAMP was not to use it as is, but rather to adapt its pubsub+RPC part of the spec while removing the broker part (replacing it with whatever routing the ipfs uses, dht etc..) WAMP is a perfect protocol for IOT requirements, but the strong dependency on router/broker is a deal-breaker. |
|
@KrishnaPG ah ok, good they generalized it |
|
Hi, or I am missing something here? |
|
I think the bare minimum pub/sub should expose this kind of interface: ipfs.sub(target_node_id, handler)
CLI version: $ ipfs sub $target_node_id
/ipfs/...
/ipfs/...
# A new line is emitted every time a new record is found |
|
Re RPC: we've discussed this briefly before, but I'm not sure what the actual plans are |
|
@fazo96 |
It's not really an arbitrary get/put. It puts a record mapping
You're free to do so but it may break in the future. There's a reason this is marked experimental (although we will try to avoid breaking things too badly).
You do need the topic to subscribe to it (there's no "list all topics" command). However, your node will tell all of its peers which topics it's interested in so they're hardly secret.
Yes. We explicitly do not want to allow arbitrary data on the DHT (too easy to abuse). The correct way to do that is with IPNS (IPNS record points to a "profile" stored in IPFS). IPNS is currently a bit... finicky in practice. |
|
Thank you for all the details @Stebalien! From the README, it looks like the JS version of the DHT isn't totally implemented. How does pubsub work in the browser if the DHT support is iffy? |
|
As I said, that particular peer-finding mechanism is a bit of a hack. I doubt that JS uses it. @diasdavid? |
|
Is there any progression on the pubsub front? |
|
I've been thinking about the state of pubsub, was wondering if I could get comments on the following: I think that constructing a minimum spanning tree (MST) for groups is the way to go.
|
|
Distributed minimum spanning tree does look like the right approach for this, I have to say. |
|
@RangerMauve interesting! About the "distance" function, I'm thinking that poorly connected nodes could impact the network if the latency or failure rate is not taken into account.
Perhaps this could bring some type of decentralised measure of "health", which could then be used to calculate distance? |
|
@pgte The distance function is there to determine which nodes are logically "close" to each other in order to determine who should connect together to form the minimum spanning tree. The main benefit is that it's computationally cheap to determine the logical distance since all you need is the ID of the peer, a node can calculate it's distance to any peer without connecting to it or knowing anything else about it. I'm taking this concept directly from the kademlia DHT design (which is the DHT used in IPFS) I think peers will only bother learning about the nodes connected to their immediate neighbors, so some sort of gossip protocol will be useful here. I'm still reading up about the different strategies out there for distributed MSPs so I'm not 100% sure what the requirements are yet. I like the idea of adding weights based on what a node things the "health" of their peers is, I'd be worried about spreading that information though, because two nodes may have a poor connection between each other, but another node may have a good connection to both based on what their physical relationship is. I think that for an MVP of a MSP type pub-sub protocol, adding in health measures will complicate the implementation, but it should definitely be investigated once we can actually construct a MSP. |
|
@pgte <https://github.com/pgte> The distance function is there to determine which nodes are logically "close" to each other in order to determine who should connect together to form the minimum spanning tree. The main benefit is that it's computationally cheap to determine the logical distance since all you need is the ID of the peer, a node can calculate it's distance to any peer without connecting to it or knowing anything else about it. I'm taking this concept directly from the kademlia DHT <https://en.wikipedia.org/wiki/Kademlia#System_details> design (which is the DHT used in IPFS)
Gotcha.
I think peers will only bother learning about the nodes connected to their immediate neighbors, so some sort of gossip protocol will be useful here. I'm still reading up about the different strategies out there for distributed MSPs so I'm not 100% sure what the requirements are yet.
I like the idea of adding weights based on what a node things the "health" of their peers is, I'd be worried about spreading that information though, because two nodes may have a poor connection between each other, but another node may have a good connection to both based on what their physical relationship is.
I think that it's exactly what both the Swim.js protocol and the Accrual failure detection model solve: it gives a cooperative measure of health, not a one-sided measure (the swim.js less so).
… I think that for an MVP of a MSP type pub-sub protocol, adding in health measures will complicate the implementation, but it should definitely be investigated once we can actually construct a MSP.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#64 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC7Jhr2UhfYsk9_lC0xA1SOKC6uVS2Lks5tQyI5gaJpZM4GL9fG>.
|
|
I've been tinkering around and here's a demo of an algorithm I've made for building a MSP.
I've made a demo with 32 nodes which boostrap with a list of 6 random nodes. I'm not sure how viable this is yet, so I'll work on other visualizations in order to catch possible partitions forming. |
|
I've updated my demo to better visualize what's happening. I've found that having nodes find a random list of other node IDs, then sorting them my logical distance and connecting to the closes one, I always get a fully-connected tree. I'll explore logic for visualizing messages being sent down the tree (for pub sub) and dropped connections. |
|
@RangerMauve what you're describing really is just a network topology mesh, where your route priority metric is driven only by logical distance. Look at the Babel routing algorithm, there is a lot we can learn from that. Batman-adv as well. |
|
@paralin Yes. I'm not focusing on routing data across the mesh at the moment. I'm focusing on forming the mesh in a way where individual nodes know every little about other nodes or the shape of the graph itself. The existing pubsub connects to nodes at random and has a chance of having "supernodes" which have too much bandwidth going to them and network partitions which prevent groups of peers from sending events between each other. Once an efficient and self-healing mesh is set up, it can be a basis for the protocols that will be built on top (pubsub to start, but other things can probably make use of it). My initial goal is to make a more efficient pubsub. I love what the Kademlia DHT does for discovering peers that have a resource, but there needs to be something for forming a persistent network for distributed applications that need to get data to all nodes interested in it. |
|
@RangerMauve Wrt. Babel: I am suggesting we use it as a reference for more reasons -
The same goes for batman-adv, except that project focuses more on radio transmission optimizations. This matches with what you're talking about with mesh balancing. It's the same problem we are solving. Routing in the typical internet sense is the same thing as routing here. It might be a good idea to look at generalizing the concept of routing a bit, and applying it in pub-sub as well as content and network routing. |
|
@paralin Sounds good. Is the RFC a good place to start? Babel looks like a protocol that should be used after the actual connections are made between nodes. I'm still at the stage where I'm figuring out which nodes should connect to which, but after that this will be great for making the actual traffic over the mesh be more efficient. |
|
@RangerMauve I see what you're saying, they are actually two different problems actually when you consider the structure of the open internet, and don't limit connections to link-local. In a way though it is actually the same thing. Consider this: all of the peers online in the world are actually just next-hop peers in the babel algorithm. What you're doing is trying to figure out what subset of that global set to connect to, and then prioritize further based on that. But it's still the same problem. The 'brute-force' solution would rank all of the known peers based on available information about them, used to compute a routing metric. You then sort by the routing metric, chose the top N nodes, connect to those, etc. Babel fits here as a mechanism of generating that routing metric. It's one of many. I think maybe a pluggable system for deriving routing metrics might be a good approach to this. You can run the fast route scoring algorithms first, before exploring a more advanced plan. It might be possible to map this to some sort of opportunistic graph solver. |
|
@paralin I get what you're saying, but I think that's a bit heavy and requires a node to know a lot more peers. With my current algorithm, a node can search the DHT for the first 20 random nodes it finds, and then connect to the mesh without focusing the load too much in once place. All it needs to do to decide who to connect to is their ID without having to somehow measure latency or any other metrics about them. Otherwise a node would need to connect to the network, learn about potential peers and how they might relate to themselves, and then actually connect to the network. Though this will be more efficient in the long run, the initial setup is likely going to be too long for the time it takes for a typical web app to load. |
|
@RangerMauve We are in agreement, I'm just generalizing what you're describing a bit. You chose a next peer to send traffic to, then you build a connection to that peer, then you send the packet to the peer. The question of where is best to send the traffic is the same as the question of where to connect to next. What you are proposing, is to start with no information about the peers, and to randomly connect to some from the DHT, and then optimize from there. This is the same as what I described in my last comment - you start with no information, you make a decision at random, and then as you learn more information about the network, the routing can be optimized further. Babel is just one example of a protocol to quickly exchange information about available routes in a closed network. It does not apply to this issue directly, but it is an example of an efficient way of exchanging information about the network between peers that happen to already be connected. If we structure the code in such a way that these routing metrics, and the way they are prioritized and balanced together, are pluggable, then we can actually start to make use of the traffic metrics in LibP2P to make better routing decisions. |
|
There must be some way to make a incentive-based system like BitSwap, but for incentivizing routing packets through the network. (Different problem, of course :)) @RangerMauve It's a little bit like a particle filter in a way... If you make a measurement of the ping to a peer, for example, the confidence and derived weight that information has on routing decisions can decay over time (temporal decay) as well as when you have other detected changes in the network topology (event-based invalidation). If we have the structure described above for pluggable metrics, you could get really fancy really quickly with algorithms to make, probably. |
|
The BitSwap ledger is already based on bytes sent and received, so I'd assume it should be relatively simple (and I do mean relatively) to extend it to also account for routed bytes (maybe as a separate category, or with some weight etc. since message sizes are likely negligible compared to blocks) |
|
Oh, and I don't know if someone's already looked into Chord, but there's been some work on how to build pub/sub systems with it:
There are also a couple of different pub/sub-like schemes over Pastry: I'd argue that mimicing their approaches might be a good way to go. There's likely similar systems on top of Kademlia, so it might be worth while to look into them. Chord, for example, is provably correct and functions well even with high churn. edit: for an example of a wildly different approach, there's Secure Scuttlebutt that builds on the Scuttlebutt gossip protocol. Having a gossip protocol in libp2p doesn't sound bad at all. revenge of the edit: @spikebike talked about PolderCast some years ago, and as he said, it seems like an extremely good choice. |
|
I'm curious if you folks are aware of what these folks are up to. |
|
Submitting aeron for us to read about and investigate:
It seems to be a publisher/subscriber system that uses efficient UDP multicast or inter-process communication to get the job done. |
|
What's the relationship between this discussion and floodsub? Is floodsub the "official" pub/sub for IPFS, or is this discussion completely independent? |
|
For everyone still following this thread, you will want to check the latest on libp2p PubSub with Gossipsub v1.1 |
and others
We've known for some time we need to layer a pub/sub system on IPFS. We should try to reuse other things. The least work, the better. But it should be a simple protocol, easy to implement, well-layered, and meshing well with the rest of IPFS abstractions.
Requirements
We need to:
I likely won't have time to do a proper survey until late Nov or Dec. If you'd like to speed this along, post links to great {papers, systems} here for me.
Relevant to research:
#ipfsIRC logs.The text was updated successfully, but these errors were encountered: