LG-9449: Prevent ArcGIS Token Expiration

[NavaTim]

🎫 Ticket

• LG-9449

🛠 Summary of changes

• Create scheduled job to refresh ArcGIS token regularly before it expires
• Configure API so the PO search can be configured with the following strategies:
  • No token refresh
    • The call to ArcGIS will fail rather than attempt to renew an expired token.
  • Refresh token on expiration (current)
    • The application will renew the token after it has expired.
    • Note: this strategy can result in failed ArcGIS calls if triggered shortly before expiration.
  • Sliding window expiration
    • Uses a "sliding" expiration time that occurs before the real expiration
    • The first server to observe that the sliding expiration has elapsed will:
      • Update the sliding expiration to a later time
      • Attempt to renew the token
      • Update the token in cache
    • Reduces redundant attempts to renew the token
• Remove code for /suggest endpoint
  • This endpoint is no longer in use, and it would require updates related to this change in order for the associated code to remain functional.
• Remove manual Redis TTL usage
  • This is no longer required to workaround Rails cache behavior.
  • See the following:
    • LG-9086: Preemptively refresh USPS auth tokens #8035 (comment)
    • Set ttl for redis and memcache cache stores when using expires_at rails/rails#45047

This PR addresses key pieces of feedback from the original PR (#8579):

• The strategies for handling the token renewal are explicitly broken out into modules.
• The original token format will be preserved until the sliding expiration flag is enabled.
  • Prevents issues when previous servers access the token during rollout.
• Improves the documentation around the usage of custom Faraday middleware

📜 Testing Plan

Provide a checklist of steps to confirm the changes.

• Token renewal via job with job enabled
• Token renewal via job prevented with job disabled
• Token renewal via UI with original behavior
• Token renewal via UI with sliding window
• Token renewal via UI prevented when sync token refresh is disabled
• Location search page works with token in new format
• Location search page works with token in original format
• ArcGIS API error handling during token renewal
• ArcGIS API error handling during geocoding

[NavaTim]

After seeing the amount of refactoring required in order to properly modularize the functions for the sliding window, I've decided to start from scratch without the sliding window and without the automatic retries. I plan to include some of the same cleanup, though.