Adding attempt events packaging batch job.

[Rwolfe-Nava]

changelog: Internal, Attempts API, Add batch job to package events.

🎫 Ticket

Link to the relevant ticket.

🛠 Summary of changes

Added a batch job to read attempt events from the previous hour from redis. Writes those events to a file and encrypts the file. File is then stored in a placeholder location until we get an S3 bucket.

Want to get some early eyes on this. WIP and still needs lots of cleanup.

[zachmargolis]

another idea: let's make this write to the /tmp directory?

We can use ruby's Tempfile class to help make a file in there

[n1zyy]

I'd suggested that we wanted to avoid Tempfile because it's eager to delete the file we just wrote. This will probably work fine once we're writing to S3, but for now it was just an annoyance and I figured we should treat what we're writing as if it persists.

[n1zyy]

I'd suggested that we wanted to avoid Tempfile because it's eager to delete the file we just wrote. This will probably work fine once we're writing to S3, but for now it was just an annoyance and I figured we should treat what we're writing as if it persists.

[zachmargolis]

this will get run in all environments regardless of if attempts api is enabled or not, which seems not good. let's either:

1. add a check inside the job to bail if the api is not enabled
2. only add the job to the schedule if the api enabled

and I know we have a lot of configs, but it seems like we may want to have a separate feature flag for just the attempts API background job so that we can smoothly transition to it if/when we are ready

[Rwolfe-Nava]

I think the first line inside the job checks if IRS attempts API is enabled and returns nil if it isn't, so it should only continue if it is enabled.

return nil unless IdentityConfig.store.irs_attempt_api_enabled

With that said, I do like the idea of ALSO not scheduling the job unless it is enabled, since I don't think that makes much sense. Plus, even though its probably not a huge resource tax, it's one less job the scheduler has to deal with if it doesn't need to.

Are there any examples of other jobs implementing similar functionality I can look at?

[zachmargolis]

in hindsight, this mkdir_p was masking the bug with the tmpdir being cleaned up beforehand, maybe we should remove it. that prevents the job from writing files where it's not supposed to

Suggested change

	# write to a file and store on the disk until S3 is setup
	FileUtils.mkdir_p(dir_path)

[Rwolfe-Nava]

Do we actually need to get rid of this? I can see an argument for doing so, that whatever calls this shouldn't pass in a directory that doesn't exist. Is there some other way we can ensure safety / error handling if we remove this line?

[zachmargolis]

I think that this code will evolve over time. Eventually we'll write a temp file and upload that to S3. So in that case, the code could make it's own tmpdir and clean that up.

However in the meantime, this code hid a bug so I'd vote to remove it.