second draft of vision doc process

rfc-drafts/stream.md

@@ -247,7 +247,7 @@ trait Stream {
 
 Unfortunately, async methods in traits are not currently supported,
 and there [are a number of challenges to be
-resolved](https://rust-lang.github.io/wg-async-foundations/design_notes/async_fn_in_traits.html)
+resolved](https://rust-lang.github.io/wg-async-foundations/design_docs/async_fn_in_traits.html)
 before they can be added. 
 
 Moreover, it is not clear yet how to make traits that contain async
@@ -309,7 +309,7 @@ design issues] to be resolved before they are added. Therefore, we've
 decided to enable progress on the stream trait by stabilizing a core,
 and to come back to the problem of extending it with combinators.
 
-[outstanding design issues]: https://rust-lang.github.io/wg-async-foundations/design_notes/async_closures.html
+[outstanding design issues]: https://rust-lang.github.io/wg-async-foundations/design_docs/async_closures.html
 
 This path does carry some risk. Adding combinator methods can cause
 existing code to stop compiling due to the ambiguities in method

src/SUMMARY.md

@@ -1,20 +1,29 @@
 # Summary
 
-- [Welcome](./welcome.md)
-- [Design notes](./design_notes.md)
-    - [Yield-safe lint](./design_notes/yield_safe.md)
-    - [Stream trait](./design_notes/stream.md)
-    - [Generator syntax](./design_notes/generator_syntax.md)
-    - [AsyncRead, AsyncWrite traits](./design_notes/async_read_write.md)
-    - [Async fn in traits](./design_notes/async_fn_in_traits.md)
-    - [Mutex (future-aware)](./design_notes/mutex.md)
-    - [Async aware channels](./design_notes/channels.md)
-    - [Async closures](./design_notes/async_closures.md)
-    - [Join combinator](./design_notes/join.md)
-    - [Select combinator](./design_notes/select.md)
-    - [Sink trait](./design_notes/sink_trait.md)
-    - [Async main](./design_notes/async_main.md)
-    - [Async drop](./design_notes/async_drop.md)
-       - [Async lifecycle](./design_notes/async_lifecycle.md)
-    - [Completion-based futures](./design_notes/completion_based_futures.md)
-
+- [👋🏽 Welcome](./welcome.md)
+- [🔮 The vision](./vision.md)
+   - [✏️ Design tenets for async](./vision/tenets.md)
+   - [🙋‍♀️ Cast of characters](./vision/characters.md)
+   - [😱 Status quo](./vision/status_quo.md)
+   - [✨ Shiny future](./vision/shiny_future.md)
+   - [📅 Roadmap for 2021](./vision/roadmap.md)
+   - [❓How to vision doc](./vision/how_to_vision_doc.md)
+- [🔍 Triage meetings](./triage.md)
+- [🔬 Design docs](./design_docs.md)
+    - [⚠️ Yield-safe lint](./design_docs/yield_safe.md)
+    - [☔ Stream trait](./design_docs/stream.md)
+    - [⚡ Generator syntax](./design_docs/generator_syntax.md)
+    - [📝 AsyncRead, AsyncWrite traits](./design_docs/async_read_write.md)
+    - [🧬 Async fn in traits](./design_docs/async_fn_in_traits.md)
+    - [🔒 Mutex (future-aware)](./design_docs/mutex.md)
+    - [📺 Async aware channels](./design_docs/channels.md)
+    - [📦 Async closures](./design_docs/async_closures.md)
+    - [🤝 Join combinator](./design_docs/join.md)
+    - [🤷‍♀️ Select combinator](./design_docs/select.md)
+    - [🚰 Sink trait](./design_docs/sink_trait.md)
+    - [🎇 Async main](./design_docs/async_main.md)
+    - [🗑️ Async drop](./design_docs/async_drop.md)
+       - [♻️ Async lifecycle](./design_docs/async_lifecycle.md)
+    - [⏳ Completion-based futures](./design_docs/completion_based_futures.md)
+- [💬 Conversations](./conversations.md)
+   - [🐦 2021-02-12 Twitter thread](./conversations/2021-02-12-Twitter-Thread.md)

src/conversations.md

@@ -0,0 +1,9 @@
+# 💬 Conversations
+
+This section contains notes and summaries from conversations that we have had with people are using Rust and async and describing their experiences. These conversations and links are used as "evidence" when building [the "status quo" section][sq].
+
+[sq]: vision/status_quo.md
+
+## Not exhaustive nor mandatory
+
+This section is not meant to be an "exhaustive list" of all sources. That would be impossible. Many conversations are short, not recorded, and hard to summaize. Others are subject to NDA. We certainly don't require that all claims in the [status quo][sq] section are backed by evidence found here. Still, it's useful to have a place to dump notes and things for future reference.

src/conversations/2021-02-12-Twitter-Thread.md

@@ -0,0 +1,50 @@
+# 🐦 2021-02-12 Twitter thread
+
+Notes taken from the thread in response to [Niko's tweet](https://twitter.com/nikomatsakis/status/1359454255971770372).
+
+* [Enzo](https://twitter.com/enzo_mdd/status/1359544617121820676)
+    * A default event loop. "choosing your own event loop" takes time, then you have to understand the differences between each event loop etc.
+    * Standard way of doing `for await (variable of iterable)` would be nice.
+    * Standard promise combinators.
+* [creepy_owlet](https://twitter.com/creepy_owlet/status/1359649695103131649)
+    * https://github.com/dtantsur/rust-osauth/blob/master/src/sync.rs
+* async trait --
+    * https://twitter.com/jcsp_tweets/status/1359820431151267843
+    * "I thought async was built-in"?
+    * nasty compiler errors
+    * [ownership puzzle](https://www.fpcomplete.com/blog/ownership-puzzle-rust-async-hyper/) blog post
+* [rubdos](https://twitter.com/rubdos/status/1359462402702606336)
+    * [blog post](https://www.rubdos.be/corona/qt/rust/tokio/actix/2020/05/23/actix-qt.html) describes integrating two event loops
+    * mentions desire for runtime independent libraries
+    * qt provides a mechanism to integrate one's own event loop
+    * [llvm bug](https://github.com/rust-lang/rust/issues/60605) generates invalid arm7 assembly
+* [alexmiberry](https://twitter.com/alexmiberry/status/1359559299161325581)
+    * kotlin/scala code, blocked by absence of async trait
+* helpful blog post
+    * [jamesmcm](http://jamesmcm.github.io/blog/2020/05/06/a-practical-introduction-to-async-programming-in-rust/)
+        * note that `join` and `Result` play poorly together
+            * [async-std version](https://github.com/jamesmcm/async-rust-example/blob/async-std/client_async/src/main.rs#L50-L59)
+            * [tokio version](https://github.com/jamesmcm/async-rust-example/blob/master/client_async/src/main.rs#L40-L61) has some wild "double question marks" -- I guess that spawn must be adding a layer of `Result`?
+    * the post mentions rayon but this isn't really a case where one ought to use rayon -- still, Rayon's APIs here are SO much nicer :)
+    * [rust aws and lambda](http://jamesmcm.github.io/blog/2020/04/19/data-engineering-with-rust-and-aws-lambda/#en)
+* [issue requiring async drop](https://github.com/jamesmcm/s3rename/issues/16)
+* [fasterthanlime](https://fasterthanli.me/articles/getting-in-and-out-of-trouble-with-rust-futures) -- 
+    * this post is amazing
+    * the discussion on Send bounds and the ways to debug it is great
+* [bridging different runtimes using GATs](https://github.com/thanethomson/async-channel-abs/blob/master/src/runtime.rs)
+* first server app, [great thread with problems](https://twitter.com/richardsabow/status/1345815109201842178)
+    * "I wasn't expecting that it will be easy but after Go and Node.js development it felt extremely hard to start off anything with Rust."
+    * "felt like I have to re-learn everything from scratch: structuring project and modules, dependency injection, managing the DB and of course dealing with concurrency"
+    * common thread: poor docs, though only somewhat in async libraries
+        * I had enums in the DB and it was a bit more complex to map them to my custom Rust enums but I succeeded with the help of a couple of blog posts – and not with Diesel documentation
+        * I used Rusoto for dealing with AWS services. It's also pretty straightforward and high quality package – but again the documentation was sooo poor. 
+* [implaustin](https://t.co/4rlyfUlFES?amp=1) wrote a [very nice post](https://t.co/4rlyfUlFES?amp=1) but it felt more like a "look how well this worked" post than one with actionable feedback
+    * "Async has worked well so far.  My top wishlist items are Sink and Stream traits in std.  It's quite difficult to abstract over types that asynchronously produce or consume values."
+    * "AsyncRead/AsyncWrite work fine for files, tcp streams, etc.  But once you are past I/O and want to pass around structs, Sink and Stream are needed.  One example of fragmentation is that Tokio channels used to implement the futures Sink/Stream traits, but no longer do in 1.0."
+    * "I usually use Sink/Stream to abstract over different async channel types.  Sometimes to hide the details of external dependencies from a task (e.g. where is this data going?).  And sometimes to write common utility methods."
+    * "One thing I can think of: there are still a lot of popular libraries that don't have async support (or are just getting there).  Rocket, Criterion, Crossterm's execute, etc."
+* [EchoRior](https://twitter.com/EchoRior/status/1359964691305496579):
+    * "I've written a bit of rust before, but rust is my first introduction to Async. My main gripes are that it's hard to figure our what the "blessed" way of doing async is. I'd love to see async included in the book, but I understand that async is still evolving too much for that."
+    * "Adding to the confusion: theres multiple executors, and they have a bit of lock in. Async libraries being dependent on which executor version I use is also confusing for newcomers. In other langs, it seems like one just uses everything from the stdlib and everything is compatible"
+    * "That kind of gave me a lot of hesitation/fomo in the beginning, because it felt like I had to make some big choices around my tech stack that I felt I would be stuck with later. I ended up chatting about this in the discord & researching for multiple days before getting started."
+    * "Also, due to there not being a "blessed" approach, I don't know if I'm working with some misconceptions around async in rust, and will end up discovering I will need to redo large parts of what I wrote."

src/design_docs.md

@@ -0,0 +1,19 @@
+# 🔬 Design documents
+
+The **design documents** (or "design docs", more commonly) describe potential designs. These docs vary greatly in terms of their readiness to be implemented:
+
+* Early on, they describe a vague idea for a future. Often this takes the shape of capturing constraints on the solution, rather than the solution itself.
+* When a feature is getting ready to ship, they can evolve into a full blown RFC, with links to tracking issues or other notes.
+
+## Early stage design docs
+
+In the early stages, design docs are meant to capture interesting bits of "async design space". They are often updated to capture the results of a fruitful conversation or thread which uncovered contraints or challenges in solving a particular problem. They will capture a combination of the following:
+
+* use cases;
+* interesting aspects to the design;
+* alternatives;
+* interactions with other features.
+
+## Late stage design docs
+
+As a design progresses, the doc should get more and more complete, until it becomes something akin to an RFC. (Often, at that point, we will expand the design document into a directory, adding an actual RFC draft and other contents; those things can live in this repo or elsewhere, depending.) Once we decide to put a design doc onto the roadmap, it will also contain links to tracking issues or other places to track the status.

src/design_docs/async_closures.md

@@ -0,0 +1 @@
+# 📦 Async closures

src/design_docs/async_drop.md

@@ -0,0 +1 @@
+# 🗑️ Async drop

src/design_notes/async_fn_in_traits.md → src/design_docs/async_fn_in_traits.md

@@ -1,4 +1,4 @@
-# Async fn in traits
+# 🧬 Async fn in traits
 
 * [Why async fn in traits are hard][wafth]
 
@@ -81,3 +81,8 @@ trait MyMethod {
     async fn foo(&self);
 }
 ```
+
+## 🤔 Frequently Asked Questions
+
+* **What do people say about this to their friends on twitter?**
+    * (Explain your key points here)

src/design_docs/async_lifecycle.md

@@ -0,0 +1 @@
+# ♻️ Async lifecycle

src/design_docs/async_main.md

@@ -0,0 +1,7 @@
+# 🎇 Async main
+
+## What is it?
+
+## Motivation
+
+## Frequently Asked Questions

src/design_docs/async_read_write.md

@@ -0,0 +1 @@
+# 📝 AsyncRead, AsyncWrite traits

src/design_docs/channels.md

@@ -0,0 +1 @@
+# 📺 Async aware channels

src/design_notes/completion_based_futures.md → src/design_docs/completion_based_futures.md

@@ -1,4 +1,4 @@
-# Completion-based futures
+# ⏳ Completion-based futures
 
 [Notes on io_uring][withoutboats-blog]
 

src/design_notes/generator_syntax.md → src/design_docs/generator_syntax.md

@@ -1,4 +1,4 @@
-# Generator syntax
+# ⚡  Generator syntax
 
 * It would be useful to be able to write a function to return an iterator or (in the async context) a generator
 * The basic shape might be (modulo bikeshedding) `gen fn` that contains `yield`

src/design_docs/join.md

@@ -0,0 +1 @@
+# 🤝 Join combinator

src/design_notes/mutex.md → src/design_docs/mutex.md

@@ -1,4 +1,4 @@
-# Mutex (future-aware)
+# 🔒 Mutex (future-aware)
 
 [Description of various challenges with async mutexes][blog]
 

src/design_docs/select.md

@@ -0,0 +1 @@
+# 🤷‍♀️ Select combinator

src/design_docs/sink_trait.md

@@ -0,0 +1 @@
+# 🚰 Sink trait

src/design_notes/stream.md → src/design_docs/stream.md

@@ -1,4 +1,4 @@
-# Stream trait
+# ☔ Stream trait
 
 * [Current definition](https://docs.rs/futures/0.3/futures/stream/trait.Stream.html)
 

src/design_notes/yield_safe.md → src/design_docs/yield_safe.md

@@ -1,4 +1,4 @@
-# Yield-safe lint
+# ⚠️ Yield-safe lint
 
 ## Use-case
 

src/triage.md

@@ -0,0 +1,59 @@
+# 🔍 Triage meetings
+
+## When, where
+
+The weekly triage meeting is held on [Zulip] at 13:00 US Eastern time on Fridays ([google calendar event for meeting](https://calendar.google.com/event?action=TEMPLATE&tmeid=M2VhYjRjczZxanE5ODcwbzR1bnZsNTV0MGFfMjAyMTAyMjZUMTgwMDAwWiA2dTVycnRjZTZscnR2MDdwZmkzZGFtZ2p1c0Bn&tmsrc=6u5rrtce6lrtv07pfi3damgjus%40group.calendar.google.com&scp=ALL)).
+
+[Zulip]: ./welcome.md#zulip
+
+## So you want to fix a bug?
+
+If you're interested in fixing bugs, there is no need to wait for the triage meeting.
+Take a look at the [mentored async-await bugs that have no assignee][bugs].
+Every mentored bug should have a few comments.
+If you see one you like, you can add the `@rustbot claim` comment into the bug and start working on it!
+Feel to reach out to the mentor on [Zulip] to ask questions.
+
+[bugs]: https://github.com/rust-lang/rust/issues?q=is%3Aopen+label%3AE-mentor+label%3AA-async-await+no%3Aassignee
+
+## Project board
+
+The [project board] tracks various bugs and other work items for the async foundation group.
+It is used to drive the triage process.
+
+[project board]: https://github.com/orgs/rust-lang/projects/2
+
+## Triage process
+
+In our weekly triage meetings, we take new issues assigned [`A-async-await`] and categorize them. 
+The process is:
+
+- Review the [project board], from right to left:
+  - Look at what got **Done**, and celebrate! :tada:
+  - Review **In progress** issues to check we are making progress and there is a clear path to finishing (otherwise, move to the appropriate column)
+  - Review **Blocked** issues to see if there is anything we can do to unblock
+  - Review **Claimed** issues to see if they are in progress, and if the assigned person still intends to work on it
+  - Review **To do** issues and assign to anyone who wants to work on something
+- Review [uncategorized issues]
+  - Mark `P-low`, `P-medium`, or `P-high`
+  - Add `P-high` and _assigned_ `E-needs-mentor` issues to the [project board]
+  - Mark `AsyncAwait-triaged`
+- If there's still a shortage of **To do** issues, review the list of [`P-medium`] or [`P-low`] issues for candidates
+
+## Mentoring
+
+If an issue is a good candidate for mentoring, mark `E-needs-mentor` and try to find a mentor.
+
+Mentors assigned to issues should write up mentoring instructions. 
+**Often, this is just a couple lines pointing to the relevant code.** 
+Mentorship doesn't require intimate knowledge of the compiler, just some familiarity and a willingness to look around for the right code.
+
+After writing instructions, mentors should un-assign themselves, add `E-mentor`, and remove `E-needs-mentor`. 
+On the project board, if a mentor is assigned to an issue, it should go to the **Claimed** column until mentoring instructions are provided. 
+After that, it should go to **To do** until someone has volunteered to work on it.
+
+[`A-async-await`]: https://github.com/rust-lang/rust/labels/A-async-await
+[uncategorized issues]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AA-async-await+is%3Aopen+-label%3AAsyncAwait-Triaged&type=Issues
+[`P-high`]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AAsyncAwait-Triaged+label%3AP-high+is%3Aopen&type=Issues
+[`P-medium`]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AAsyncAwait-Triaged+label%3AP-medium+is%3Aopen&type=Issues
+[`P-low`]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AAsyncAwait-Triaged+label%3AP-low+is%3Aopen&type=Issues

src/vision.md

@@ -0,0 +1,42 @@
+# 🔮 The vision
+
+## What is this
+
+This section lays out a general vision and set of goals for Async I/O in Rust. This document is being written in January of 2021. The core pieces of async-await are stable, but there remains a lot of work for it to feel like a "first class" feature of Rust. 
+
+This section describes three things:
+
+* The **design tenets**, or principles, that we are using to drive our work on Async I/O in Rust;
+* Descriptions of how writing Async I/O code in Rust **feels today**;
+* Descriptions of how we think writing Async I/O code in Rust **should feel eventually**.
+
+## Status and schedule
+
+**This document is currently being formed.** 
+We expect to publish the first version on March 31.
+Once that version is published, we intend to revisit the document once per quarter to keep it up to date.
+
+| Phase | Date |
+| --- | --- |
+| **Drafting initial stories and proposals** | **until March 31** |
+| 2021 Q1 revision published | March 31 |
+| 2021 Q2 revision process begins | June 14 - June 30 |
+| 2021 Q2 revision published | June 30 |
+| 2021 Q3 revision process begins | Sep 13 - Sep 30 |
+| 2021 Q3 revision published | Sep 30 |
+| 2021 Q4 revision process begins | Nov 29 - Dec 10 |
+| 2021 Q4 revision published | Dec 10 |
+
+## Think big -- too big, if you have to
+
+You'll notice that the ideas in this document are **maximalist and ambitious**. They stake out an opinionated position on how the ergonomics of Async I/O should feel. This position may not, in truth, be attainable, and for sure there will be changes along the way. Sometimes the realities of how computers actually work may prevent us from doing all that we'd like to. That's ok. This is a dream and a goal.
+
+## The vision drives the work
+
+The vision is not just idle speculation. It is the central document that we use to organize ourselves. When we think about our [roadmap](./roadmap.md) for any given year, it is always with the aim of moving us closer to the vision we lay out here. 
+
+## This is a group effort
+
+As the leads of the Async Foundation group, Niko and Tyler are driving and organizing this document. But writing it and shaping it is a group effort. If you think there is a part of the async experience that is not reflected here, we want to hear from you! Check out the [How to vision doc] for more details on how to contribute.
+
+[How to vision doc]: ./vision/how_to_vision_doc.md