Automatically set PRAGMAs using connection query params #85
Conversation
Introduce the flexibility to adjust certain PRAGMAs of the SQLite3 connection without having to hardcode those in your codebase (and wait for compilation). This allows applications to use `DATABASE_URL` to dynamically fine tune their SQLite3 configuration. The change complements `#setup_connection` that offers, via code, the option to perform queries on setup of each connection. Only a few PRAGMAs necessary to allow more performant concurrent reads and reduce write locking. These pragmas are detected and combined in a single SQL string to reduce to 1 the number of calls to `sqlite3_exec` C function. There is no validation of supplied values as SQLite3 automatically ignores incorrect values for these pragmas. Closes crystal-lang#84 References: - https://www.sqlite.org/pragma.html
|
Hello @bcardiff, didn't think it fully as I was following a bit what mattn/go-sqlite3 did, but also to differentiate those from connection pool options (Eg. ALLOWED_PRAGMAS = {"busy_timeout", "cache_size", "foreign_keys", "journal_mode", ...}
# ...
URI::Params.parse(query) do |key, value|
next unless key.in?(ALLOWED_PRAGMAS)
pragmas[key] = value
endre: I also did that in case someone had the same pragma multiple times in the query parameters and that resulted in multiple SQL queries 😁 Thank you for your promptly response and let me know about the other changes. |
|
Starting with the hash is fine, no need to change. In crystal-mysql we recently added an I'm not sure what would be a good convention going forward. If there is no clear better alternative we can go ahead and keep |
|
Hello @bcardiff, I had some time to think about the PR, some notes:
If is OK with you, I would suggest:
Let me know what you think of this plan and I will introduce the changes. Thank you in advance for your time! ❤️ ❤️ ❤️ |
Sounds good. My motivation to have some prefix is that we could validate the options names and instead of ignoring when they don't match a known pragma we could tell the user that a typo was made. So far we've not been doing that because we add one option at a time and it didn't occur to me. So again, sounds good to drop the prefix since we are not going to start validating all the other options probably.
Unless we have a shared memory cache I don't see how this can be done. crystal-db passes only the connection string.
These would be validation of values, not keys right? I think we can defer for another PR if needed. It seems that the main thing to decide is whether or not to keep the prefix. I'm fine dropping them if you feel they are too verbose. |
|
Btw, to avoid the hash in a neat way I was thinking of require "uri"
query = "foo=1&bar=first&bar=second&quz=ignore"
params = extract(query, foo: nil, bar: nil)
puts typeof(params) # => NamedTuple(foo: String | Nil, bar: String | Nil)
puts params # => {foo: "1", bar: "second"}
def extract(query, **default : **T) forall T
res = default
URI::Params.parse(query) do |key, value|
{% begin %}
case key
{% for key in T %}
when {{ key.stringify }}
res = res.merge({{key.id}}: value)
{% end %}
end
{% end %}
end
res
endsome copy of the values could be avoided, but it does the work. |
No longer prefix PRAGMAS with `_`, so the mapping between the real SQLite3 pragmas and the usage in the URI is more direct. Use macros instead of case to detect pragmas from URI params and return those as NamedTuple.
|
Hello @bcardiff, thanks for your patience coming back to this.
Apologies for not making it clear, thought it was but re-reading my response seems didn't explain it. The reason I was using the Hash was so under certain scenarios with double variables in the URI (Eg. This was done to avoid having the same PRAGMA multiple times when performing the call to SQLite3. Very naive approach that looking at your suggestion clearly addresses. On that subject, your macro approach uses less memory and performs a bit faster than the Hash approach: Source coderequire "benchmark"
require "uri"
def extract(query, **default : **T) forall T
res = default
URI::Params.parse(query) do |key, value|
{% begin %}
case key
{% for key in T %}
when {{ key.stringify }}
res = res.merge({{key.id}}: value)
{% end %}
end
{% end %}
end
res
end
def hash_extract(query)
res = Hash(String, String).new
URI::Params.parse(query) do |key, value|
case key
when "busy_timeout"
res["busy_timeout"] = value
when "cache_size"
res["cache_size"] = value
when "foreign_keys"
res["foreign_keys"] = value
when "journal_mode"
res["journal_mode"] = value
end
end
res
end
params = "busy_timeout=1000&garbage=foo&cache_size=-1000&foreign_keys=1&journal_mode=wal&busy_timeout=5000"
Benchmark.ips do |x|
x.report("extract (hash)") { hash_extract(params) }
x.report("extract (macro)") {
extract(params,
busy_timeout: nil,
cache_size: nil,
foreign_keys: nil,
journal_mode: nil,
synchronous: nil,
wal_autocheckpoint: nil,
)
}
endAs for the validation of input, we might want to introduce something that provides that support part of crystal-db for others to use, but I would say we can talk about that in a different PR. 😉 I decided to push additional changes to:
I hope is all good. Thank you again for your time and your suggestions. |
Ah, great catch! missed when I was doing the search and replace! 🤦🏽 Thank you! |
Introduce the flexibility to adjust certain PRAGMAs of the SQLite3 connection without having to hardcode those in your codebase (and wait for compilation).
This allows applications to use
DATABASE_URLto dynamically fine tune their SQLite3 configuration.The change complements
#setup_connectionthat offers, via code, the option to perform queries on setup of each connection.Only a few PRAGMAs necessary to allow more performant concurrent reads and reduce write locking.
These pragmas are detected and combined in a single SQL string to reduce to 1 the number of calls to
sqlite3_execC function.There is no validation of supplied values as SQLite3 automatically ignores incorrect values for these pragmas.
Closes #84
References: