Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

readonly_standy attempts to write to db #861

Closed
schelcj opened this issue Sep 30, 2023 · 4 comments · Fixed by #862
Closed

readonly_standy attempts to write to db #861

schelcj opened this issue Sep 30, 2023 · 4 comments · Fixed by #862

Comments

@schelcj
Copy link
Contributor

schelcj commented Sep 30, 2023

Describe the bug
Starting irrd with readonly_standby: true fails to start with error "Exception: Attempted to write to SQL database from readonly database handler".

To Reproduce
Set readonly_standby: true and enabled the http server.

Expected behaviour
irrd to start.

IRRd version you are running
4.4.0

Additional context
Traced this out and the decorator on irrd.webui.helpers.secret_key_derive always attempts to commit to the database irrespective of the readonly flag.

Debug logs when starting irrd:

2023-09-30 12:15:23,345 irrd[463440]: [irrd.utils.process_support#CRITICAL] Essential IRRd subprocess encountered a fatal error, traceback follows, shutting down: Attempted to write to SQL database from readonly database handler
Traceback (most recent call last):
  File "/home/irrd/srv/releases/4.4.0/venv/lib/python3.10/site-packages/irrd/utils/process_support.py", line 21, in run
    super().run()
  File "/home/irrd/.pyenv/versions/3.10.10/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/irrd/srv/releases/4.4.0/venv/lib/python3.10/site-packages/irrd/server/http/server.py", line 28, in run_http_server
    secret_key_derive("scope", thread_safe=False)
  File "/home/irrd/srv/releases/4.4.0/venv/lib/python3.10/site-packages/irrd/storage/orm_provider.py", line 96, in endpoint_wrapper
    provider.commit_close()
  File "/home/irrd/srv/releases/4.4.0/venv/lib/python3.10/site-packages/irrd/storage/orm_provider.py", line 38, in commit_close
    self.database_handler.commit()
  File "/home/irrd/srv/releases/4.4.0/venv/lib/python3.10/site-packages/irrd/storage/database_handler.py", line 164, in commit
    self._check_write_permitted()
  File "/home/irrd/srv/releases/4.4.0/venv/lib/python3.10/site-packages/irrd/storage/database_handler.py", line 885, in _check_write_permitted
    raise Exception(msg)
Exception: Attempted to write to SQL database from readonly database handler
@mxsasha
Copy link
Collaborator

mxsasha commented Oct 11, 2023

This is a confusing error message, but an unavoidable situation when starting a standby instance against an empty database.

This is fixed if you set up the SQL replication first, and run a non-standby IRRD at any time against the PostgreSQL primary, the secret key is generated and stored, and standby instances will not attempt to generate their own.

An improvement would be for the standby IRRD instance to detect this and issue a clearer error. It can not resolve this situation, because it can not write to a replica database. Even if IRRD's database handler would allow it, PostgreSQL would not.

@schelcj
Copy link
Contributor Author

schelcj commented Oct 11, 2023

This was against a running standby instance. I was an upgrade after an upgrade from 4.3.1 to 4.4.0. The master is runnng 4.4.0 already and replicated those changes. I stepped through secret_key_derive in a debugger and it is seeing that setting in the database but the decorator is trying to write back to the db.

@mxsasha
Copy link
Collaborator

mxsasha commented Oct 12, 2023

Ah right, misread the traceback. Definite bug and no workaround. I'll try to get a fixed release out in the next few days, or otherwise early next week.

mergify bot pushed a commit that referenced this issue Oct 12, 2023
mxsasha added a commit that referenced this issue Oct 12, 2023
mxsasha added a commit that referenced this issue Oct 12, 2023
@mxsasha
Copy link
Collaborator

mxsasha commented Oct 12, 2023

4.4.1 is out with the fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants