Add --deflate option to data-pull and action-account scripts#9424
Merged
zachmargolis merged 4 commits intomainfrom Oct 20, 2023
Merged
Add --deflate option to data-pull and action-account scripts#9424zachmargolis merged 4 commits intomainfrom
zachmargolis merged 4 commits intomainfrom
Conversation
changelog: Internal, Scripts, Add compression option to optimize script output
| stdout.puts Base64.encode64( | ||
| Zlib::Deflate.deflate( | ||
| (result.json || result.table).to_json, | ||
| Zlib::BEST_COMPRESSION, |
Contributor
There was a problem hiding this comment.
Did BEST_COMPRESSION make a significant difference from the defaults?
Contributor
There was a problem hiding this comment.
We also have MessagePack if we'd rather avoid the Base64 encoding
Contributor
Author
There was a problem hiding this comment.
re: BEST_COMPRESSION, I didn't try others
re: messagepack, I will give it a shot! I thought it was a binary format for some reason
Contributor
Author
There was a problem hiding this comment.
Just confirmed that message pack is binary, I think it'll run into the same issue that the binary DEFLATE data had, which is that SSM command seems to eat it.
The original reason I chose JSON was this blog post that seemed to indicate JSON + GZIP yielded smaller payloads than MSGPACK
Contributor
There was a problem hiding this comment.
ah got it, sounds good!
n1zyy
approved these changes
Oct 20, 2023
Contributor
n1zyy
left a comment
n1zyy
left a comment
There was a problem hiding this comment.
Nice! This compresses so much better than I would have expected.
Merged
mdiarra3
added a commit
that referenced
this pull request
Oct 24, 2023
* LG-11083: Enable USPS Public Endpoint (#9355) * changelog: Internal, In-Person Proofing, Enable public USPS post office search * Use EnrollmentHelper to switch between mock/real thing * Try behaves_like * Revert shared examples for now * Use full name * Update report mailer preview to be more realistic (#9419) **How**: stubs CloudwatchClient changelog: Internal, Reporting, Updates report preview to use live code * Add analytics section to frontend documentation (#9421) * Add analytics section to frontend documentation changelog: Internal, Documentation, Add analytics frontend documentation * link to correct javascript package * LG-11101: Support multiple valid MFA to satisfy authentication request (#9335) changelog: User-Facing Improvements, MFA, Avoid prompting for MFA in some scenarios where a recent MFA satisfies the requirement * LG-11148 | Adds monthly report on total verified users (#9376) changelog: Internal, Reporting, Monthly report now includes total verified users Also incorporates LG-11150 Co-authored-by: Zach Margolis <zachary.margolis@gsa.gov> * Remove second MFA prompt exception for strict MFA requirement (#9422) changelog: User-Facing Improvements, MFA Setup, Add second MFA reminder screen for single-MFA accounts when signing in at AAL2 * LG-11126 Update Start over verifying your identity screen (#9313) * change text for start over verify screen * add translations for page * add changelog changelog: User-Facing Improvements, IdV By Mail, update text in start over verifying identity screen * remove unused i18n * create new translation with question mark added * current step indicator for user not in gpo flow yet * a missing period * Restore deleted translations, and rename start_over to start_over_new_address Co-authored-by: Doug Price <douglas.price@gsa.gov> * New template for confirm start over from request_letter Add source param to indicate whether referer is request_letter * Update specs to check for correct template Co-authored-by: Doug Price <douglas.price@gsa.gov> * Add before_letter route for new screen, don't use it yet And analytics * Lint, unused arg in analytics_events * alphabetization lint * Add suggested comment Co-authored-by: Matt Hinz <matt.hinz@gsa.gov> * lints --------- Co-authored-by: Douglas Price <douglas.price@gsa.gov> Co-authored-by: Sonia Connolly <sonia.connolly@gsa.gov> Co-authored-by: Matt Hinz <matt.hinz@gsa.gov> * LG-11198: Update address text (#9420) Update address text changelog: User-Facing Improvements, IdV, Update text for address * LG-10922: Display new headings for Hybrid Handoff page on AB test (#9316) * changelog: User-Facing Improvements, Doc Auth, Display new headings for Hybrid Handoff page on AB test Adds: * Conditional headers depending on which flag is on * Hybrid handoff show view test * Translations * LG-11235: Rename double address verification as ipp_enrollment_in_progress (#9390) * Removed double address verification replaced with ipp_enrollment_in_progress * changelog: Internal, In-person Proofing, change DAV references to reflect reality * Change test description to be closer to what is being changed in the controller * Addressing 50/50 state concerns in proofer and adjudicator * Addressing linter issues * Set missing initial value for dav * Moving arg with default value to end of list * Apply suggestions from code review Adding proper input to job_arguments hash. Co-authored-by: Zach Margolis <zachmargolis@users.noreply.github.com> * Adding note about existing ticket for work post 50/50 state * Resolving Shannon's comments * Adding back in test for dav, need reader on adjudicator * Adding back in test for dav, need reader on adjudicator --------- Co-authored-by: jack.ryan@gsa.gov <johnaryan@fcoh2j-f4t79kf4.myfiosgateway.com> Co-authored-by: Zach Margolis <zachmargolis@users.noreply.github.com> * Add --deflate option to data-pull and action-account scripts (#9424) changelog: Internal, Scripts, Add --deflate option to data-pull and action-account scripts --------- Co-authored-by: Matt Gardner <wilburnforce@gmail.com> Co-authored-by: Zach Margolis <zachmargolis@users.noreply.github.com> Co-authored-by: Andrew Duthie <andrew.duthie@gsa.gov> Co-authored-by: Matt Wagner <mattwagner@navapbc.com> Co-authored-by: Zach Margolis <zachary.margolis@gsa.gov> Co-authored-by: Alex Bradley <alexander.bradley@gsa.gov> Co-authored-by: Douglas Price <douglas.price@gsa.gov> Co-authored-by: Sonia Connolly <sonia.connolly@gsa.gov> Co-authored-by: Matt Hinz <matt.hinz@gsa.gov> Co-authored-by: jc-gsa <104452882+jc-gsa@users.noreply.github.com> Co-authored-by: Brittany Greaner <35475380+night-jellyfish@users.noreply.github.com> Co-authored-by: Jack Ryan <jackryan@navapbc.com> Co-authored-by: jack.ryan@gsa.gov <johnaryan@fcoh2j-f4t79kf4.myfiosgateway.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary of changes
SSM commands have a max output of about 63kb, and some of our export tasks output more data than that, so they get truncated.
This adds DEFLATE compression which seems to help pack down output, so we can export more. Some notes from example testing (Slack link):
Other notes: