DevOps issues from slack/mr0grog

[lightandluck]

Thought this should be archived here before it gets lost in Slack history

mr0grog APP [12:34 PM]
In the meeting today, @dcwalk asked about ops/devops issues we need help with. Sorry I haven’t ever enumerated all that super clearly, so here’s a quick run down of what’s been simmering in the back of my mind that we could really use help with (I should probably make some more GitHub issues, too):

• We need to have sane tools for log handling, error tracking. This issue is on DB but we need the same thing for all 4 code projects: Set up error tracking service web-monitoring-db#47
• Upgrading our CI usage: Upgrade to CircleCI 2.0 web-monitoring-db#99 and Upgrade to CircleCI 2.0 web-monitoring-ui#91
• (No issue for this) ^ related to the above, it would be lovely to have some kind of continuous deployment setup based on passing/failing CI (Update 2019-03-10: we explicitly decided to punt on this for now)
• This documentation for deployment is about to merge: Add documentation for current deployment process web-monitoring-processing#72, but we’ll be creating other issues for improvements to the process it describes
• We should have similar documentation for all projects (-versionista-scraper has it, but -ui and -db do not) (Update 2019-03-10: all this now has a home in edgi-govdata-archiving/web-monitoring-ops)
• We should have scripted deploy tools for all (we have only for -ui)
• If the AWS stuff comes through, we probably want to move everything over to EC2. Would love expertise and help in how best to set things up (I can do this, but am 100% sure someone who is an actual ops or devops person would do something approximately 10,000% better) (Update 2019-03-10: this has long been done.)
• Make the deploy/test/etc. process and tools for all 4 projects reasonably similar and consistent (Update 2019-03-10: see web-monitoring-ops)
• Automated DB exports/dumps Set up automated process to export pages/versions web-monitoring-db#45
• Adding caching tools (e.g. Varnish) or integrating memcached/redis with Rails for better caching in the API and the other services (related, but smaller scope: Implement caching to reduce loading time while working with diff files #59)
• We are likely going to eventually get to adding a queueing service in order to do Determine how to trigger processing/analysis when new versions are added to DB #62 . That will make deployment more complicated (Update 2019-03-10: this was done)

(UPDATED 2018-07-27 by @Mr0grog to cross out stuff that’s since been done)
(UPDATED 2019-03-10 by @Mr0grog)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.

[lightandluck]

I'm not sure if the undone stuff is still relevant, but imagine what still is would be better broken up into separate issues

[Mr0grog]

Pretty much everything on this list has either been done, is tracked in an issue or better managed in web-monitoring-ops, or has been canceled/punted on.

I’ve updated the original post with some cross-outs and pointers to newer info, but I think it’s time to close this issue.