-
Notifications
You must be signed in to change notification settings - Fork 166
Adding attempt events packaging batch job. #7259
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
c4f2fc7
Adding attempt events packaging batch job.
Rwolfe-Nava 8d0a15b
cleaned up comments and added tests for attempts_events_batch_job
Rwolfe-Nava 171ea70
cleanup per draft PR feedback
Rwolfe-Nava 746c4b9
linter fixes
Rwolfe-Nava ce8749f
Update app/jobs/irs_attempts_events_batch_job.rb
Rwolfe-Nava 0fd6178
Update app/jobs/irs_attempts_events_batch_job.rb
Rwolfe-Nava d853a07
lint fixes
Rwolfe-Nava 5a37484
fixed unit test after merging suggested changes
Rwolfe-Nava 0e268e0
lint fix
Rwolfe-Nava 46f1966
Implemented test changes based on feedback
Rwolfe-Nava File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| class IrsAttemptsEventsBatchJob < ApplicationJob | ||
| queue_as :default | ||
|
|
||
| def perform(timestamp: Time.zone.now - 1.hour, dir_path: './attempts_api_output') | ||
| return nil unless IdentityConfig.store.irs_attempt_api_enabled | ||
|
|
||
| events = IrsAttemptsApi::RedisClient.new.read_events(timestamp: timestamp) | ||
| event_values = events.values.join("\r\n") | ||
|
|
||
| decoded_key_der = Base64.strict_decode64(IdentityConfig.store.irs_attempt_api_public_key) | ||
| pub_key = OpenSSL::PKey::RSA.new(decoded_key_der) | ||
|
|
||
| result = IrsAttemptsApi::EnvelopeEncryptor.encrypt( | ||
| data: event_values, timestamp: timestamp, public_key: pub_key, | ||
| ) | ||
|
|
||
| # write to a file and store on the disk until S3 is setup | ||
| FileUtils.mkdir_p(dir_path) | ||
|
|
||
| file_path = "#{dir_path}/#{result.filename}" | ||
|
|
||
| File.open(file_path, 'wb') do |file| | ||
| file.write(result.encrypted_data) | ||
| end | ||
|
|
||
| return { encryptor_result: result, file_path: file_path } | ||
|
|
||
| # Write the file to S3 instead of whatever dir_path winds up being | ||
| end | ||
| end | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| require 'rails_helper' | ||
|
|
||
| RSpec.describe IrsAttemptsEventsBatchJob, type: :job do | ||
| describe '#perform' do | ||
| context 'IRS attempts API is enabled' do | ||
| let(:start_time) { Time.new(2020, 1, 1, 12, 0, 0, 'UTC') } | ||
| let(:private_key) { OpenSSL::PKey::RSA.new(4096) } | ||
| let(:events) do | ||
| [ | ||
| { | ||
| event_key: 'key1', | ||
| jwe: 'some_event_data_encrypted_with_jwe', | ||
| timestamp: start_time + 10.minutes, | ||
| }, | ||
| { | ||
| event_key: 'key2', | ||
| jwe: 'some_other_event_data_encrypted_with_jwe', | ||
| timestamp: start_time + 15.minutes, | ||
| }, | ||
| ] | ||
| end | ||
|
|
||
| before do | ||
| allow(IdentityConfig.store).to receive(:irs_attempt_api_enabled).and_return(true) | ||
| allow(IdentityConfig.store).to receive(:irs_attempt_api_public_key). | ||
| and_return(Base64.strict_encode64(private_key.public_key.to_der)) | ||
|
|
||
| Dir.mktmpdir do |dir| | ||
| @dir_path = dir | ||
| end | ||
Rwolfe-Nava marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| travel_to start_time + 1.hour | ||
|
|
||
| redis_client = IrsAttemptsApi::RedisClient.new | ||
| events.each do |event| | ||
| redis_client.write_event( | ||
| event_key: event[:event_key], jwe: event[:jwe], | ||
| timestamp: event[:timestamp] | ||
| ) | ||
| end | ||
| end | ||
|
|
||
| it 'batches and writes attempt events to an encrypted file' do | ||
| result = IrsAttemptsEventsBatchJob.perform_now(timestamp: start_time, dir_path: @dir_path) | ||
| expect(result[:file_path]).not_to be_nil | ||
|
|
||
| file_data = File.open(result[:file_path], 'rb') do |file| | ||
| file.read | ||
| end | ||
|
|
||
| final_key = private_key.private_decrypt(result[:encryptor_result].encrypted_key) | ||
|
|
||
| decrypted_result = IrsAttemptsApi::EnvelopeEncryptor.decrypt( | ||
| encrypted_data: file_data, | ||
| key: final_key, iv: result[:encryptor_result].iv | ||
| ) | ||
|
|
||
| events_jwes = events.pluck(:jwe) | ||
|
|
||
| expect(decrypted_result).to eq(events_jwes.join("\r\n")) | ||
| end | ||
| end | ||
|
|
||
| context 'IRS attempts API is not enabled' do | ||
| before do | ||
| allow(IdentityConfig.store).to receive(:irs_attempt_api_enabled).and_return(false) | ||
| end | ||
|
|
||
| it 'returns nil' do | ||
| file_path = IrsAttemptsEventsBatchJob.perform_now | ||
| expect(file_path).to eq(nil) | ||
| end | ||
| end | ||
| end | ||
| end | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in hindsight, this
mkdir_pwas masking the bug with the tmpdir being cleaned up beforehand, maybe we should remove it. that prevents the job from writing files where it's not supposed toThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we actually need to get rid of this? I can see an argument for doing so, that whatever calls this shouldn't pass in a directory that doesn't exist. Is there some other way we can ensure safety / error handling if we remove this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that this code will evolve over time. Eventually we'll write a temp file and upload that to S3. So in that case, the code could make it's own tmpdir and clean that up.
However in the meantime, this code hid a bug so I'd vote to remove it.