bug: repeatedly broken assertion

[consideRatio]

When using hubtraf to spawn pods faster than a JupyterHub could handle, I ended up crashing hubtraf two or three times like through this assertion being broken. This only happened after a lot of other errors that made it clear where the limit was for the JupyterHub deployment.

hubtraf/hubtraf/user.py

Lines 172 to 189 in 78ab8b5

	async def stop_server(self):
	assert self.state == User.States.SERVER_STARTED
	self.debug('server-stop', phase='start')
	start_time = time.monotonic()
	try:
	resp = await self.session.delete(
	self.hub_url / 'hub/api/users' / self.username / 'server',
	headers=self.headers
	)
	except Exception as e:
	self.failure('server-stop', exception=str(e), duration=time.monotonic() - start_time)
	return False
	if resp.status != 202 and resp.status != 204:
	self.failure('server-stop', exception=str(resp), duration=time.monotonic() - start_time)
	return False
	self.success('server-stop', duration=time.monotonic() - start_time)
	self.state = User.States.LOGGED_IN
	return True

Failure: kernel-stop hubtraf-32 exception:<ClientResponse(https://hub.neurohackademy.org/user/hubtraf-32/api/kernels/339cf55c-85c8-481a-81d3-00a414164acd) [403 Forbidden]>
<CIMultiDictProxy('Content-Length': '40', 'Content-Security-Policy': "frame-ancestors 'self';report-uri /hub/security/csp-report; default-src 'none'", 'Content-Type': 'application/json', 'Date': 'Tue, 21 Jul 2020 05:23:33 GMT', 'Server': 'TornadoServer/6.0.3', 'Strict-Transport-Security': 'max-age=15724800', 'X-Content-Type-Options': 'nosniff', 'X-Jupyterhub-Version': '1.1.0')>
 duration:0.49191117999725975
Traceback (most recent call last):
  File "/home/erik/miniforge3/bin/hubtraf-simulate", line 11, in <module>
    load_entry_point('hubtraf', 'console_scripts', 'hubtraf-simulate')()
  File "/home/erik/dev/contrib/yuvipanda/hubtraf/hubtraf/simulate.py", line 98, in main
    loop.run_until_complete(run(args))
  File "/home/erik/miniforge3/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "/home/erik/dev/contrib/yuvipanda/hubtraf/hubtraf/simulate.py", line 44, in run
    outputs = await asyncio.gather(*awaits)
  File "/home/erik/dev/contrib/yuvipanda/hubtraf/hubtraf/simulate.py", line 29, in simulate_user
    await u.stop_server()
  File "/home/erik/dev/contrib/yuvipanda/hubtraf/hubtraf/user.py", line 173, in stop_server
    assert self.state == User.States.SERVER_STARTED
AssertionError

This issue is created as a followup to neurohackademy/nh2020-jupyterhub#44 (comment)

[yuvipanda]

Curious what state it was in, if not SERVER_STARTED. Maybe the kernel stop failed but we still tried to stop it anyway?