[Hackathon] Allow sourcing events from stdin#11619
Merged
matthinz merged 2 commits intologin-hackathon-2024-user-narrativefrom Dec 11, 2024
Merged
[Hackathon] Allow sourcing events from stdin#11619matthinz merged 2 commits intologin-hackathon-2024-user-narrativefrom
matthinz merged 2 commits intologin-hackathon-2024-user-narrativefrom
Conversation
It may be useful sometimes to take a local cache of cloudwatch events and pipe them into this command. [skip changelog]
This is required for `complete` to work
Comment on lines
+68
to
+72
| def stdin_source(&block) | ||
| $stdin.each_line do |line| | ||
| next if line.blank? | ||
| event = JSON.parse(line) | ||
| block.call(event) |
Contributor
There was a problem hiding this comment.
micro-optimization! don't assign a variable to the block argument, it saves an allocation on the heap, use the yield keyword which uses the stack version
Also added some docs to indicate that it takes a block
Suggested change
| def stdin_source(&block) | |
| $stdin.each_line do |line| | |
| next if line.blank? | |
| event = JSON.parse(line) | |
| block.call(event) | |
| # @yield calls block for each parsed event | |
| # @yieldparam [Hash] event | |
| def stdin_source | |
| $stdin.each_line do |line| | |
| next if line.blank? | |
| event = JSON.parse(line) | |
| yield event |
mdiarra3
approved these changes
Dec 11, 2024
eileen-nava
approved these changes
Dec 11, 2024
Contributor
eileen-nava
left a comment
eileen-nava
left a comment
There was a problem hiding this comment.
Approved! Go, team, go.
matthinz
added a commit
that referenced
this pull request
Jan 8, 2025
* Initial cloudwatch query script to summarize events * Query cloudwatch and get user events * add timestamp remove limit * [Hackathon] Allow sourcing events from stdin (#11619) * Allow sourcing events from stdin It may be useful sometimes to take a local cache of cloudwatch events and pipe them into this command. [skip changelog] * Add 'limit: 10000' to CW query This is required for `complete` to work * [Hackathon] Add ExampleMatcher (#11622) * Add ExampleMatcher Add an example matcher that just counts events and outputs how many it saw. [skip changelog] * Remove excess whitespace * Add frozen_string_literal: true * use optparse to allow command options/defaults (#11627) * [HACKATHON] Initial crack at an IdV matcher (#11624) * Initial crack at an IdV matcher Matcher is a state machine that collects IDV "attempts" as they happen and tries to suss out interesting things about them. [skip changelog] * removed unused method --------- Co-authored-by: Douglas Price <douglas.price@gsa.gov> * [HACKATHON] Output formatting tweaks (#11635) * Normalize @timestamp to UTC for each event Pre-parse it in the script so that matchers don't have to worry about it * Slightly improve output - Include timestamps where possible [skip changelog] * [HACKATHON] Minor tweaks (#11637) * Don't crash if no events found * Tweak handling of --end-date - Use a dash rather than underscore - Make sure we respect it if it's passed in * Sort events on stdin before processing Events from Cloudwatch queries will be sorted, but stdin is not guaranteed. Processing unsorted events can lead to weird, weird, outcomes * report on TrueID success/failure (#11638) * Try to identify IDV abandonment (#11639) If the user: - Has not completed the initial workflow and - Does not have an idv-related event new that 1 hour Call their attempt abandoned * Login hackathon 2024 user narrative account deletion (#11629) * include timestamp * add account deletion narrative matcher * remove unneeded matcher requirement * add deletion matcher * lint * rename account deletion * read events from file without changing stdin * remove ipp from gpo code submission event * update example documentation in script * Update lib/event_summarizer/vendor_result_evaluators/aamva.rb Co-authored-by: Zach Margolis <zachmargolis@users.noreply.github.com> * Start writing a spec * Tidy up logic in IV result evaluator * Set event['name'] if not already set * Fix typo * Use Eastern time zone by default * Update pluralization code + add spec * Start on spec for summarize-user-events command * Protect rubocop's delicate sensibilities * Add more specs Add some specs around option parsing, time parsing, and actually running the program * Look at banner michael --------- Co-authored-by: Malick Diarra <malick.diarra@gsa.gov> Co-authored-by: Doug Price <douglas.price@gsa.gov> Co-authored-by: Eileen <eileenmcfarland@navapbc.com> Co-authored-by: eileen-nava <80347702+eileen-nava@users.noreply.github.com> Co-authored-by: Zach Margolis <zachmargolis@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Context for anyone not working on this project can be found in Slack)
Sometimes it is convenient to source events locally rather than having to make a Cloudwatch query every time.
This PR checks to see if stdin is a tty, and if it's not, tries to read newline-delimited JSON (ndjson) events from it.