Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate Timestamps and Time Windowed Dataflows #351

Open
AshleySchaeffer opened this issue Jan 10, 2022 · 0 comments
Open

Consolidate Timestamps and Time Windowed Dataflows #351

AshleySchaeffer opened this issue Jan 10, 2022 · 0 comments

Comments

@AshleySchaeffer
Copy link

AshleySchaeffer commented Jan 10, 2022

I've not quite got around to trying this stuff out but from reading the docs I suspect this may covered by the empty chapter found here:

https://timelydataflow.github.io/differential-dataflow/chapter_6/chapter_6.html

I have two main questions/problems I'm aiming solve.

Firstly, I'm unsure how to go about consolidating differences from higher resolution timestamps to a single, lower resolution difference. E.g. perhaps 5 separate differences occurred within a minute but I only care about knowing what happened within the minute, not exactly when within the minute. I think the same logic could apply for any kind of timestamp that was a numeric type.

Secondly, I'd like to have a dataflow retain changes and reverse them upon a specified time window passing. Essentially forming a temporal window.

In both cases, I think it would be solvable to have logic be applied to the data prior to inputting it into the dataflow. However, I'd quite like a reusable operator for each. I don't think I can achieve either outcome with existing operators but I could well be wrong. Does anyone have any ideas how this could be approached? Are operators for these outcomes going to result in some unforeseen issues?

Thanks for your time regardless of whether you can help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant