Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

benchmark to test how expensive mapping into/out of joins is #478

Draft
wants to merge 588 commits into
base: main
Choose a base branch
from
Draft

benchmark to test how expensive mapping into/out of joins is #478

wants to merge 588 commits into from

Conversation

zzlk
Copy link
Contributor

@zzlk zzlk commented Mar 16, 2023

jhellerstein and others added 30 commits September 19, 2022 13:20
@jhellerstein
Book edits, fix hydro-project#183 (hydro-project#190)
5a17779 
* replace refs in book: "Core API" -> "Surface API"

* remove dead examples/book

* split Graph Reachability into two subsections

* explain vec![0] and remove refs to pull

* explain vec![0]

* fix link to design doc

* fix bug/typo where pull should have been push

* more careful discussion of the space of partitionings

* restore refs to Core API, other small edits

* clarify that example_4_1 illustrates multiple outputs
@MingweiSamuel
Update recv_stream to handle all Streams instead of just `tokio::…
8b68c64 
…mpsc::unbounded_channel`

Breaking change
@MingweiSamuel
Enable partitioned output on hydroflow_parser!
4936198
@MingweiSamuel
Add surface diamond tests
7ba4321 
To check subgraph partitioning bug in linear chains
@MingweiSamuel
Surface syntax fix handling of wildcard linear chains which might cau…
3559fbf 
…se later pull-push conflicts
@MingweiSamuel
Fix handling of empty merge()/tee(), add tests
3a0ab8a
@MingweiSamuel
Remove internal runtime, use tokio::spawn mechanism (requires tokio c…
302b213 
…ontext)
@MingweiSamuel
Add tcp echo example test
9362edf
@MingweiSamuel
Rename send_async -> write_async to match trait names
666d14e
@MingweiSamuel
Restructor operator constraints into single write_fn
4a36e1b
@MingweiSamuel
Emit better, more local, more useful error messages in surface syntax
bba512f 
Make sure to set the correct spans in codegen
@MingweiSamuel
Implement and add test for sink_async
19424cf
@MingweiSamuel
Add #[allow(clippy::map_identity)] to some tests
1bf8642
@jhellerstein
mechanical merge of fork (hydro-project#197)
c2fc4b2 
* replace refs in book: "Core API" -> "Surface API"

* remove dead examples/book

* split Graph Reachability into two subsections

* explain vec![0] and remove refs to pull

* explain vec![0]

* fix link to design doc

* fix bug/typo where pull should have been push

* more careful discussion of the space of partitionings

* restore refs to Core API, other small edits

* clarify that example_4_1 illustrates multiple outputs
@MingweiSamuel
Add UDP networking & test
c9529c5
@MingweiSamuel
Remove awkward testing side-effects from surface syntax async tests
84d7406
@MingweiSamuel
Add assertiosn to surface tcp test
dfc22e8
@MingweiSamuel
Fix surface syntax port ordering bug
c241c05
@shadaj
Use Rust Sitter release from crates.io (hydro-project#204)
83ab8a5
@MingweiSamuel
Fix hydro-project#201 run_async(), wait when ALL strata have no work
fa20c12 
Instead of waiting when ANY stratum has no work
@MingweiSamuel
Add apache-2.0 license
7854971
@MingweiSamuel
Add type guards, better spans to surface syntax codegen
09953f7
@MingweiSamuel
Update datalog snapshots
6d9616e
@MingweiSamuel
Add cross join surface syntax operator, update tests, fix hydro-proje…
c526f9a 
…ct#200 (hydro-project#211)
@MingweiSamuel
Implement inspect() surface syntax operator, fix hydro-project#208
7797c6c
@jhellerstein
add flatten op to surface syntax (hydro-project#213)
f802b95 
* add flatten op to surface syntax

* handle push case in flatten and test it
@jhellerstein
start porting examples to surface syntax (hydro-project#207)
01ff0c6 
* first draft of hydroflow chat

* share a single UDP port among all message channels

* remove bad comments

* remove stray keystroke

* more examples ported to surface syntax

* remove two_pc from Cargo.toml

* make clippy happy

* remove unfinished two_pc example

* make clippy happy

* use cross_join in chat

* pretty up the chat code

* move chat helpers out of protocol

* hydroflow for 2PC

* address clippy and PR feedback

* make clippy happy with a flatten op

* persistent aggregation in 2PC

* remove leftover comments from surface API in README
@jhellerstein
use sequential numbers to index tee output
ea99408
@MingweiSamuel
Fix example graph printing (hydro-project#222)
f141609 
Seems there are two code paths. For surface syntax:
```rust
hf.serde_graph().unwrap().to_mermaid()
```
For deprecated surface API. And core API, kinda
```rust
hf.generate_mermaid()
```
ref hydro-project#219
@MingweiSamuel
Remove surface API, fix hydro-project#224
7b75f5e 
Related: hydro-project#59, hydro-project#219

Also add
```rust
// TODO(mingwei): Need rust-analyzer support
```
As needed for new clippy lint. New syntax doesn't work with rust-analyzer yet.
MingweiSamuel and others added 13 commits March 13, 2023 12:39
@MingweiSamuel
Update subgraph handoff algorithm (hydro-project#441 9/14)
577071a
@MingweiSamuel
Remove subgraph_send/recv_handoffs from PartitionedGraph, compute…
a1efedc 
… on the fly (hydro-project#441 10/14)
@MingweiSamuel
Remove FlatGraph, unify under PartitionedGraph (hydro-project#441 …
b640b53 
…11/14)
@MingweiSamuel
Rename PartitionedGraph -> HydroflowGraph (hydro-project#441 12/14)
f95b325
@MingweiSamuel
Make HydroflowGraph fields private (hydro-project#441 13/14)
3ddb10a
@MingweiSamuel
Cleanup graph docs, organize method names (hydro-project#441 14/14)
09d3b57 
- Cleanup (`DiMulGraph`) graph docs
- Unify/organize `HydroflowGraph` edge methods
- Update `HydroflowGraph` subgraph method naming
- Unify/organize `HydroflowGraph` node methods
- Add `ExactSizeIterator` to predecessor/successor methods
@shadaj
Allow specifying args to launch HydroflowCrate with (hydro-project#460
3575fd3 
)
@MingweiSamuel
Add generic arg to identity(), add tests, close hydro-project#392 (h…
09dd190 
…ydro-project#465)
@shadaj
Support networking topologies that mix local and cloud through SSH tu…
0ec6d88 
…nneling (hydro-project#461)
@shadaj
Directly expose Rust bindings as Python APIs (hydro-project#462)
b94413a
@shadaj
Add APIs for sending data to a Hydroflow service from Python (hydro-p…
c2203a1 
…roject#466)
@MingweiSamuel
Add scalar persist() operator, hydro-project#438 (hydro-project#468)
688026b
@zzlk
benchmark to test how expensive mapping into/out of joins is
82fb79a
jhellerstein
jhellerstein approved these changes Mar 16, 2023
View reviewed changes
Copy link
Collaborator

@jhellerstein jhellerstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@zzlk
Copy link
Contributor Author

zzlk commented Apr 14, 2023

I wonder if this has changed now that we have LTO turned on, maybe now there is no difference (although I'm not sure why that would be?)

@MingweiSamuel MingweiSamuel force-pushed the main branch 2 times, most recently from b0c3916 to 5d48544 Compare May 30, 2023 02:27
@MingweiSamuel MingweiSamuel force-pushed the main branch 2 times, most recently from e9a87eb to 6039078 Compare July 18, 2024 15:55
@MingweiSamuel MingweiSamuel force-pushed the main branch 4 times, most recently from 4f84634 to 0e64791 Compare July 22, 2024 21:30
@MingweiSamuel MingweiSamuel force-pushed the main branch 2 times, most recently from fb8b3b3 to a2ec110 Compare August 30, 2024 20:07
@MingweiSamuel MingweiSamuel force-pushed the main branch 2 times, most recently from b569212 to fb8393a Compare March 15, 2025 01:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhellerstein jhellerstein jhellerstein approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zzlk @jhellerstein @shadaj @MingweiSamuel @tylerhou @alexras