Support Multi-Instance Dispatching

[liam-middlebrook]

Ideally we should be able to run n instances of Conditional without any issue. This would tie in well with our current WSGI setup allowing us to have multiple WSGI "threads".

This also would be useful if we ever wanted to containerize Conditional and would allow for load balancing between different instances.

[stevenmirabito]

@liam-middlebrook Have you identified any specific issues that would prevent this from just working? Aside from some inconsequential data consistency issues if someone loads a page while something is being updated by evals, I don't see any reason this wouldn't just work.

[liam-middlebrook]

Yeah it looks like it should just be an issue with ldap queries, but I'd like to do some more research.

[liam-middlebrook]

I just poked around with disabling the lru_cache functions and still encountered this issue.
I've attached a log below that should be helpful with later debugging.

[Fri Sep 09 14:08:24 2016] [error] [client 172.21.61.218] request_id='82e71b15-ceb1-4978-adec-e088573549c8' user_name='loothelion' action='display housing' evnt='frontend'
[Fri Sep 09 14:08:25 2016] [error] [client 172.21.61.218] request_id='ad3eea07-c961-4a63-8770-5edf8f62c8e9' user_name='loothelion' action='display membership valuations listing' event='frontend'
Fatal Python error: saving thread twice?

Current thread 0x00007f9280f52700:
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 263 in _ldap_call
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 757 in search_ext
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 761 in search_ext_s
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 1071 in _apply_method_s
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 1134 in search_ext_s
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 768 in search_s
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 75 in __ldap_get_members__
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 34 in wrapped_func
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 175 in ldap_get_current_students
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 154 in ldap_get_active_members
  File "/var/www/priv/conditional-dev/conditional/blueprints/spring_evals.py", line 39 in display_spring_evals
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 1625 in dispatch_request
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 1639 in full_dispatch_request
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 1988 in wsgi_app
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 2000 in __call__

Thread 0x00007f9282f56700:
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 263 in _ldap_call
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 687 in result4
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 680 in result3
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 673 in result2
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 669 in result
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 762 in search_ext_s
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 1071 in _apply_method_s
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 1134 in search_ext_s
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/ldap/ldapobject.py", line 768 in search_s
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 75 in __ldap_get_members__
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 34 in wrapped_func
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 175 in ldap_get_current_students
  File "/var/www/priv/conditional-dev/conditional/util/ldap.py", line 169 in ldap_get_onfloor_members
  File "/var/www/priv/conditional-dev/conditional/blueprints/housing.py", line 30 in display_housing
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 1625 in dispatch_request
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 1639 in full_dispatch_request
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 1988 in wsgi_app
  File "/var/www/priv/conditional-dev/.venv/lib/python3.3/site-packages/flask/app.py", line 2000 in __call__

Thread 0x00007f929d2ee740:

[stevenmirabito]

As @audiolion pointed out in #112, our caching strategy will have to be re-evaluated as lru_cache will not work with multiple threads/instances.

[liam-middlebrook]

I just tested this out again with mod_wsgi but instead of increasing the threads count I increased the processes count. I was not able to replicate this issue when doing so.

[liam-middlebrook]

Since there isn't anything left to change for this software side I'm going to close the issue.