Sprint: Browser accesses go-ipfs content

[daviddias]

Hi! If you are coming to this issue for the first time, this might be new to you. We are going to try something new, instead of having milestones and tackling through them during the quarter, we are going to have teams focused for 2 week periods on getting some parts of IPFS that fulfill a user story, so that things can be completed incrementally and in a discrete way instead of continuously spreading our attention too thin.

What is this sprint going to achieve

Browser accesses go-ipfs content

• As a user, be able to cat files from a browser ipfs node that the user added through go-ipfs CLI
• User story:
  • 1. User follows an example on js-ipfs README and spawns a node in the browser
  • 2. User sees that node has already some peers (bootstrappers)
  • 3. User adds a file with go-ipfs CLI
  • 4. User cats the file in the browser node and sees the file

Tasks

• libp2p-webrtc-direct transport (@diasdavid)

  • Update js-multiaddr to support this new transport
  • Update js-mafmt to support this new transport
  • complete webrtc-direct implementation
  • Integrate it with libp2p-webrtc-star signalling server Aborted, see update: Sprint: Browser accesses go-ipfs content #310 (comment)
  • Test the dial behind https
  • Deploy the new signalling server to signal.libp2p.io

• https://signal.libp2p.io @lgierth + @diasdavid

  • Need a cert for signal.libp2p.io
  • Deploy a signalling server that exposes https endpoint

• stream multiplexer, two options

  • multiplex route @victorbjelkholm + @diasdavid
    • Finish implementation of multiplex (find better name) [WIP] pull-multiplex - pull-streams all the way down libp2p/js-libp2p-mplex#44 @victorbjelkholm
    • Fix and finish pull-wrap version of multiplex @victorbjelkholm
    • multiplex spec - Move to libp2p/specs - identify if there is anything missing
      • https://github.com/whyrusleeping/go-multiplex/blob/master/spec.md @victorbjelkholm
    • interop stream muxer assured Solving the Stream Muxer interoperability between Go and JS implementations js-ipfs#721 @victorbjelkholm

• multiaddr

  • /wss/ - Add wss support go-ipfs More granular address announcements kubo#3613 @lgierth
  • /dns/
    • spec @lgierth /dns in multiaddrs multiformats/multiaddr#22
    • impl in go-ipfs @lgierth Add /p2p, /dns4, /dns6, /wss, /p2p-circuit multiformats/go-multiaddr#46
    • impl in js-ipfs @diasdavid Implementation of DNS Support multiformats/js-multiaddr#37

• Deploy bootstraper nodes

  • go-ipfs with /ws/ and /wss/ @lgierth Websockets bootstrappers infra#194
  • js-ipfs with /ws/, /wss/, /libp2p-webrtc-star/ and /tcp/ @lgierth

• Upgrade IPFS Config @lgierth More granular address announcements kubo#3613

  • addresses to listen
  • addresses to announce
    • add /dns addrs here
  • addresses to not announce
  • This needs essentially a lot of tests to make sure we don't screw up (nodes starting to send through Identify, multiaddrs that they are not accesible through). Have a node dedicated just for the tests we will run in modules

• Tutorials with Demos @haad

  • Every example should be ready by working with js-ipfs Node.js to js-ipfs Browser.
  • list of examples to be made
    • file Explorer (like what Whyrusleeping did)
    • cat a file in the browser that was added through go-ipfs tutorial: Transfer Files  js-ipfs#714
      • UI/UX
      • Demo done and complete
      • Tutorial explaning step by step how to create the same demo
    • files feed tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db js-ipfs#716
      • UI/UX
      • Demo done and complete
      • Tutorial explaning step by step how to create the same demo

• Documentation, from 0 to hero - We need to make it very easy for someone that opens github.com/ipfs/js-ipfs, to go from nothing to having a Node running in the browser, that is the whole goal, preferably with a tutorial that is not massive @haad @diasdavid -

  • Update documentation, readme and examples docs/improve newcomers experience js-ipfs#723
  • Organize README so that the first impression is more pleasant docs: organise badges js-ipfs#722

• Circuit relay (Basic Turn, no relay discovery) Implement (finish) circuit switching relay libp2p/go-libp2p#82

  • review current implementation (understand what it actually does) @lgierth
  • spec @lgierth
    • usage fix duration spelling #1: a flag on the swarm/connect command Not going to happen anymore
    • usage Sprint 1 #2: encapsulate relay info in multiaddr
  • implementation
    • in JS @diasdavid
    • in Go @lgierth

• discovery

  • make sure that having mdns discovery and webrtc-star discovery doesn't freat out js-ipfs (write tests for it) test: multiple discovery systems js-ipfs#717
  • fix mdns interop between go-ipfs and js-ipfs mdns is squating on host hostname libp2p/go-libp2p#176

• DHT as a service @lgierth

  • expose findProovs in the gateways
  • expose findPeers in the gateways

Outside of the sprint (Time on this should not be invested unless all the above is done

• Improving Init endeavour @haad

Timeline

We will be running this sprint for 2 weeks, starting on Week 3 (15 Jan) and ending on the final day of Week 4 (22 Jan). During these period, we know that:

• @haadcode unavailable the second week
• @lgierth unavailable from 19th to 24th

We will be using this issue to do daily check in, please avoid commenting if you are not check in in, if you have a question, consider opening another issue or asking through the #IPFS channel in freenode.

[daviddias]

David's check in Mon 16

• Flying out to Lisbon today during the morning ✈️
• Available for the calls later
• Will use the remaining available time to focus on working on:
  • Update js-multiaddr to support this new transport
  • Update js-mafmt to support this new transport
  • complete libp2p-webrtc-direct
  • Integrate webrtc-direct in webrtc-star
  • test behind https
  • deploy it as a new signalling server

PS: #first :D

Update: Mon 16: got some hiccups with webpacking libp2p-webrtc-direct (random segfaults, similat to ones seen in js-ipfs-api), not a blocker, should haven't invested so much time trying to solve them, will compensate tomorrow.

[haadcode]

I'll be working on the following today:

• Design the interop examples
• Implement skeletons for the examples
• Open issues and PRs for tracking the work
• Weekly calls
• Update Waffle

I don't have any blockers atm. Will need feedback for examples soon.

[victorb]

2016-01-16

Gonna be working on finishing @dignifiedquire's implementation of multiplex. To track: libp2p/js-libp2p-mplex#44

Update end-day: Still working on getting the .destroy stuff to work correctly, looking at spdy implementation for inspiration and making the interface-stream-muxer tests to pass.

2016-01-17

Starting from where I left off yesterday, getting .destroy to work in js-libp2p-multiplex branch pull-streams

Update end-day: Ended up having a call with Friedel and David about what to do with pull-streams not establishing connections unless reading, adding test case for closing connections if read/not read.

2016-01-18

Following the todo at https://hackmd.io/KYdmGMAYQVgTgLQgMwCYAcCAsJ2PQGzgBmCAhsQIyXhbCqRwAmMQA===?view#

Update end-day: Managed to get tons of help from our dear and beloved @diasdavid and we made lots of progress on getting streams to close properly in multiplex. Fork here: https://github.com/diasdavid/multiplex/tree/fix/close-muxed-streams

2016-01-19

Continuing with the todos at https://hackmd.io/KYdmGMAYQVgTgLQgMwCYAcCAsJ2PQGzgBmCAhsQIyXhbCqRwAmMQA===?view#

Update end-day: Got all the tests from interface-stream-muxer to pass, including the mega-stress test (even though it takes ~20 minutes to finish) with Davids help to close the streams probably. Gonna continue today with the two items of testing with go-ipfs for interop.

2016-01-20

Gonna look into getting interop to work between js-libp2p and go-libp2p, identify any issues and make it easy to reproduce, probably gonna spend some time talking to the go-ipfs team as well.

Update middle-day: Got a new TODO list on what I'm working on.

• Update js-libp2p-multiplex documentation Update readme.md's example and added files for it libp2p/js-libp2p-mplex#54
• Enable these tests again - https://github.com/libp2p/js-libp2p-swarm/blob/master/test/04-muxing-multiplex.node.js Enable multiplex tests libp2p/js-libp2p-switch#153
• Swarm tests are not passing with multiplex + identify
• can we have both in js-libp2p-ipfs-nodejs & js-libp2p-ipfs-browser, both muxers supported and something that enables us to disable them using env variables (ie. SPDY=false), so that it makes interop tests way easier
• Note in js-ipfs-libp2p-nodejs and js-ipfs how you can run webrtc stuff on Ubuntu 16.04
• See if js-libp2p-ipfs-nodejs to go-libp2p fails with the same behaviour
  • Example:
    • https://github.com/ipfs/js-libp2p-ipfs-nodejs/tree/master/examples/echo
    • https://github.com/libp2p/go-libp2p/blob/master/examples/echo/main.go
  • Break by changing to A LOT of data
• Stress-test go-ipfs with tests go-libp2p-spdy has problems - Can not send objects to go, if they are larger than 64k libp2p/js-libp2p-spdy#48 (comment)
  • Make it easy to reproduce
  • https://github.com/ipfs/js-ipfs/pull/715/files
  • Run with go-ipfs master
• Learn how to run the stream-muxer tests that jbenet did for Stream Muxers in Go
  • Tests are available here: https://github.com/jbenet/go-stream-muxer
  • Document how to test go-multiplex, go-yamux and go-spdystream with them (verify that go-multiplex is actually passing those tests)
    • Instructions in the repo, spdystream tests are not passing, everything else is passing: spdystream get stuck for second or third stream libp2p/go-stream-muxer#5 (comment)
  • Disable all tests except multiplex

Update end-of-day: Got stuck when node-webrtc doesn't work properly on Ubuntu 16.04 systems, leading to segfaults.

2016-01-23

Catching up with notifications of stuff happening during the weekend, continuing with the todo-list above and making sure ipfs/js-ipfs#721 gets solved.

2016-01-24

Got stuck at .identify not working in .swarm because of multiplex. David submitted a fix today libp2p/js-libp2p-mplex#55 and I'll continue with the todo items above, basing my work of that branch.

Update end-of-day: Not the most productive day, needed lots of help but lots of help was not available to me. Enabled some more testing and continued to test the interop between go-libp2p and js-libp2p.

2016-01-25

Plan for today is to finish the multistream-select tests and continue to debug the problems with interop between go-libp2p and js-libp2p

2016-01-26

Two remaining issues for interop, hangup doesn't work properly in swarm and somehow the streams closes before finishing when using the echo example.

Also, probably gonna have to spend some time these last two days on helping data.gov sprint with setting up kubernetes/test-cluster.

Middle of the day: Switching to take over the documentation/examples effort and David moving to focusing on the muxers compatibility.

[ghost]

Damn I was pretty sure I posted earlier.

• No blockers
• Current: looking into relay in go-ipfs, what needs fixing, and how to actually make use of it.
• Next: /wss prespec

[victor]

I don't care about this, sorry, but if you feel this way you'll work better, then go for it 👍🏼 Enviat des del meu iPhone El 16 gen 2017, a les 19:40, Lars Gierth <[email protected]> va escriure:

…

Damn I was pretty sure I posted earlier. No blockers Current: looking into relay in go-ipfs, what needs fixing, and how to actually make use of it. Next: /wss prespec — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

@victor uhm, are you trying to tell use anything related to this sprint?

[daviddias]

@victor my apologies for bothering you, apparently we tagged you instead of @victorbjelkholm. Fixed now, you can unsubscribe to this issue.

[ghost]

Ah that makes sense, sorry victor!

[daviddias]

David's check in Mon 17

• Integrate webrtc-direct in webrtc-star
• test behind https
• deploy it as a new signalling server
• review haads first example
• fix interop tests in js-ipfs
• fix http-api tests in js-ipfs
• release 0.21 release 0.21 js-ipfs#712 (comment)

Update:

• Unfortunately my idea to bypass wss is not viable, the browser will block HTTP GET requests to non https if a page is loaded through https. The only thing that passes through are things like CSS and images, but it does gives warnings. It would probably still be doable if I hacked around an <img> tag, but that is at the level of XSS injection to make a thing work. What we really need is https and wss.
• @lgierth I've updated with the list of tasks with a https://signal.libp2p.io @lgierth + @diasdavid section. This has become one of the priorities, since the webrtc-direct hack won't do the trick :(

[haadcode]

Yesterday:

• Implemented the browser side of things for the first example ("basic"), and with it you can .cat pictures between browser nodes (try it from tutorial: Transfer Files  js-ipfs#714).
• Drafted a skeleton for a tutorial for the basic example.

Today:

• Implement first version of the second example ("advanced")
• Continue writing the tutorial
• Test the actual interop with the basic example (if available today in js-ipfs).

Blockers:
none

[daviddias]

Notes from pull-multiplex extravaganza discussion: https://hackmd.io/KYdmGMAYQVgTgLQgMwCYAcCAsJ2PQGzgBmCAhsQIyXhbCqRwAmMQA===?edit

[ghost]

Yesterday I examined the relay impl in go-ipfs. It's not wired up with anything and doesn't provide end-to-end encryption. I've described a way forward in libp2p/go-libp2p#82

• Blockers: feedback on relay-as-a-transport (see above).
• Current: adding multiaddr codes for /relay, /wss, /dns to go-ipfs
• Next: implement relay-as-a-transport and wire it up

[daviddias]

David's check in Wed 18

• make sure that having mdns discovery and webrtc-star discovery doesn't freat out js-ipfs (write tests for it)
• Chase @whyrusleeping to get back to me on "fix mdns interop between go-ipfs and js-ipfs mdns is squating on host hostname libp2p/go-libp2p#176"
• Work on DNS for JS Implementation of DNS Support multiformats/js-multiaddr#37

[haadcode]

End of the day check in:

• Yesterday I started working on the "advanced" (second) example for interop.
• Today I continued on it and made a WIP PR: tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db js-ipfs#716. The UI and browser side is pretty much done, next step is to interopopopop.
• Will be traveling rest of today --> afk

Tomorrow:

• Make the interop work
• Continue writing the tutorials

Blockers:

• I couldn't find documentation on how do I connect js-ipfs (node.js) and js-ipfs (browser) nodes, so I need instructions on links to documentation to get started. cc @diasdavid

[daviddias]

I couldn't find documentation on how do I connect js-ipfs (node.js) and js-ipfs (browser) nodes, so I need instructions on links to documentation to get started. cc @diasdavid

@haadcode by using webrtc-star, they will find each other automatically, however it would be good, since go-ipfs won't have that feature, that you use websockets addr in the js-ipfs node and that in the demo page, you have a box saying "multiaddr to dial". Which reminds me, it would be ideal to have a list of peers connected visual in the webdemo

[ghost]

• No blockers
• Yesterday: added /dns, /libp2p-relay, /wss to the protocols list in multiaddr (Add /dns, /wss, /relay, /p2p protocol codes multiformats/multiaddr#34), and to go-multiaddr (Add /p2p, /dns4, /dns6, /wss, /p2p-circuit multiformats/go-multiaddr#46).
• Current: working on turning go-libp2p's RelayService into a transport (libp2p/go-libp2p-relay).
• Next: add config options to enable relay.

[daviddias]

David's check in Thu 19

• Continue working on DNS for JS Implementation of DNS Support multiformats/js-multiaddr#37 (see list of tasks inside this issue)

[haadcode]

After yesterday's check in:

• Tried js-ipfs(nodejs) and js-ipfs(browser) interop without success. Getting segfaults sometimes and .cat is not working either way, but I do get the peers connected. I need to discuss this in sync today, will ping relevant people on IRC:
• Added a list of connected peers to the UI of the second example (file manager)

Today:

• Will add the missing UI elements for manually connecting to peers ("Connect to multiaddr" inputs)
• Continue figuring out how to make the interop work
• Document the interop steps as I discover what is needed

Blockers:
I have 2 days left for this sprint and while the browser side of things should be good to go in terms of finishing them, leaving them for others to work, I'm running low on time for writing the tutorials and documentation. I'm not convinced I can finish all of that in 2 days (and I'll be gone next week). I know you're not in this sprint @RichardLitt, but would you be able to help out next week with the tutorials (I can work with you on those).

I'll bring these up in our sync today so we can discuss the steps forward.

[daviddias]

I've noticed that folks aren't giving love to the waffle board, what's up?

[daviddias]

Group Check In Jan 19, Thu

Notes: https://hackmd.io/GYTgJmCMDsDMwFoBMBTJAOBAWADAI0gTwGMwkF0A2LMSnYPdYaEIA===?both

Top priorities before the weekend:

• Get cert for signal.libp2p.io (@lgierth)
• Get signal.libp2p.io deployed with http and https (no HSTS header) (@lgierth)
• Review both examples (@diasdavid)
  • tutorial: Transfer Files  js-ipfs#714
  • tutorial: File Exchange - Transfer files between desktop and browser using go-ipfs, js-ipfs and orbit-db js-ipfs#716
• circuit relay spec draft for review (@lgierth)

[daviddias]

David's check in Mon 23

Plan:

• Review @haadcode examples
• Participate in @haadcode handoff to @RichardLitt
• Catch up with @pgte (he is coming to Lisbon today), plan the rest of the week
• Attends All Hands calls
• Help out speed up the Stream Muxer situation Solving the Stream Muxer interoperability between Go and JS implementations js-ipfs#721
  • swarm tests with identify passing!

[ghost]

Lars monday

• Blockers
  • CR, but no real "blocker" blockers
• Last
  • Deployed two hosts running go-ipfs with /ws listeners (not wired to nginx yet)
  • SSL cert for star-signal.cloud.ipfs.team
• Today
  • Circuit relay proposal for review before implementation

[daviddias]

David's check in Tue 24

• Continue speeding up Pedro on things
• Continue helping @victorbjelkholm with the Stream Muxer situation (made a lot of progress yesterday, still ongoing)
  • Fix multiplex 'clean itself out' tests
  • Understand why the heck go-multiplex limits to 1024

Update:

• This keeps on giving.. https://github.com/ipfs/js-libp2p-ipfs-nodejs/issues/59

[ghost]

Lars tuesday

• Blockers
  • no real "blocker" blockers
• Yesterday
  • circuit-relay prespec and ethereum meetup
• Today
  • circuit-relay prespec
  • /ws bootstrappers

[daviddias]

David's check in Wed 25

• Focus of this day will be doing an last pass on the examples and doing the 'documentation from 0 to hero'

[RichardLitt]

@RichardLitt today:

• Worked on node-github-issue-bot, hoping to find some time today to work on this.

[ghost]

Lars wednesday

• No blockers
• Yesterday
  • Wanted to get circuit-relay prespec done, but got distracted by 0.4.5 release prep and pinbot/gateway debugging.
• Today
  • circuit-relay prespec
  • Provisioned a host for the 300TB sprint

[ghost]

circuit-relay prespec: libp2p/go-libp2p-circuit#1 -- feedback much appreciated, sorry this document gets written so late in the sprint.

[daviddias]

David's check in Thu 26

• Review circuit-relay prespec from @lgierth
• Continue working on the examples
• last sync with Pedro before he leaves Lisbon
• Group check in
• Prep session for next sprint

[ghost]

Lars thursday

• No real blockers
• Yesterday
  • posted circuit-relay prespec Here's a pre-spec libp2p/go-libp2p-circuit#1
• Today
  • go-ipfs multiaddr update
  • 0.4.5 release prep

[daviddias]

The last mile 🏃🏽💨

We just had the second group check in to discuss the state of tasks and decide what are the things which we should spend our time for the last day and showcase Monday. Notes: https://hackmd.io/GYEwxiCMkBwKwFowENgAYEBYBsAjbCyAnGogKYy5lFjC65EBMYQA?edit

• Host WebSockets Secure Bootstrapers @lgierth
• Tutorials
  • Transfers Files
    • DX/UX - @victorbjelkholm
    • Tutorial Steps @RichardLitt
  • File Feed
    • Review it @diasdavid
    • DX/UX - @victorbjelkholm
    • Tutorial Steps @RichardLitt
• @lgierth will check mdns is squating on host hostname libp2p/go-libp2p#176
• Update IPFS Config with addrs to listen, addrs to announce, addrs to not announce @lgierth
• DHT as a service @lgierth (expose it in the gateway)
• Continue the multiplex battle @diasdavid
• "Pack the bags", make sure every outstanding issue is well documented for other sprints

[ghost]

Here are two /ws bootstrappers which also have the gateway on the same port:

/ip4/159.203.166.189/tcp/80/ws/ipfs/QmWyLSnMHW2H6bmCG9e9PQq4ARve94JduvGjbutUuzx4a8
/ip4/178.62.215.134/tcp/80/ws/ipfs/QmVcj9MATxGTAFoQSbrJvZ9Fbs4Jzvrxy9hyJeRwbW8NeA

These should also work on /tcp/443/wss but I haven't tested it, as I have yet to patch /dns and /wss support into go-ipfs. So I'm not yet able to dial the same with go-ipfs.

[ghost]

The respective /dns+/wss addresses:

/dns4/strawberry.i.ipfs.io/tcp/443/wss/ipfs/QmWyLSnMHW2H6bmCG9e9PQq4ARve94JduvGjbutUuzx4a8
/dns4/blueberry.i.ipfs.io/tcp/443/wss/ipfs/QmVcj9MATxGTAFoQSbrJvZ9Fbs4Jzvrxy9hyJeRwbW8NeA

and https://strawberry.i.ipfs.io + https://blueberry.i.ipfs.io for the gateway.

[ghost]

For deferring routing to the gateway:

• dht/findprovs: https://strawberry.i.ipfs.io/api/v0/dht/findprovs?arg=QmaYRrBpZQRjonxBpBBSLvYmy893ySPpXpHCakhomaQtkU
• dht/findpeer: https://blueberry.i.ipfs.io/api/v0/dht/findpeer?arg=QmWyLSnMHW2H6bmCG9e9PQq4ARve94JduvGjbutUuzx4a8

And you can use api/v0/refs?r=true to warm up the /ws bootstrapper/gateway with a stuff added on other go-ipfs nodes. That way we cheat our way around the lack of relaying.

[daviddias]

You know something cool, now we have multiplex working woooo \o/ :D Once go-ipfs 0.4.7 is release, no user will have to worry about it again :D

[flyingzumwalt]

@diasdavid can we call this "done"?

[daviddias]

Yes! :D