Add SQLite to Godot.

[madmiraal]

Provide built-in support for SQLite databases.

Based on the work by @fire and @TGRCdev in godot-sqlite, which in turn is based on the work by @khairul169 in gdsqlite-native and @TGRCdev's branch. The third party library also includes @spsoft's spmemvfs to also support read-only access to game resource databases.

• SQLite is in the public domain.
• spmemvfs is provided under the BSD 2-Clause License and included in the COPYRIGHT.txt
• Khairul Hidayat is the original author of the modules/sqlite/sqlite.h and modules/sqlite/sqlite.cpp and I've tried to make this clear in the files themselves, because they are provided under the MIT License.

[Geequlim]

What is the use case for a game engine to intergarate data base ?

[avril-gh]

What is the use case for a game engine to integrate data base ?

storing, fast accessing and managing well arranged game data like items data, monsters data, player inventory, and many other things.
(Of course i can see the point that simple game may don't need this functionality)

I remember its been discussed earlier for a few times, to not include it in core by default, but to use optional module when necessary.

Because i like to have things integrated tightly (compiled in) rather than using separate modules,
im using @khairul169 's gdsqlite module which i can confirm, works well with godot 3.x as well.
https://github.com/avril-gh/gdsqlite/tree/godot-3.x

...im neither for nor against, if it will get merged i see it useful, if not there are good modules with this functionality for the ones who need it.

[madmiraal]

@Geequlim A database is needed anytime you need to store any significant amount of data; especially if you need to search the data to find what you need using indices; and even more so, if the resultant data is obtained by combining data from multiple tables via lookups.

My personal use case is the creation of levels. Instead of trying to manually create and save each level as a separate scene, the levels are created procedurally by combing the data stored in multiple property tables. This allows me to create over 3,000 pre-defined different levels, without needing to store all the details of each of those 3,000 levels. This is done by only storing the code (unique id) associated with each component of the level, and then having separate tables for the details of the components. Without a SQLite database, just looking up the level's codes in the 3,000 row Array of Arrays (or Dictionaries), is noticeably slow.

[avril-gh]

my use case is managing game items, where each item might have its characteristics, and ability to contain another items within, which then might contain yet another ones inside (item enhancing / forging / disassembling etc.) which then influence characteristics of main item, therefore possibly player characteristics and so on.
With SQL its easy and fast to 'compile' result of all these relations and display final form of item inventory page, player characteristics page etc. Such thing could be troublesome without SQL features.

There are many use cases, where it is absolutely necessary and it depends on game type and its design, but of course i agree that there are simple games that don't need it, or it should be only on server side (eg. when game is multiplayer and its recommended to NOT do/keep, such things on player side where it can be altered/cheated)

[swarnimarun]

I generally agree with the idea that there's a need for some kind of Database integration but I am not really sure how tight the integration should be.
GDNative modules might work just well for this. Or do they not?

And there is also the question of why sqlite?
I believe it's perfectly valid to ask which all databases we should support and which we might not want to. Clearly integrating sqlite today and alienating others isn't a pretty solution.
But we can't support everything.

I think this is the reason we haven't added them into the engine yet.

TBH I would much rather have a DatabaseAbstraction API of sorts which would allow you to be completely backend agnostic(yes just like an ORM), but the implementation specific integration would have to be imported in. Just like how VCS works at the moment.

But it's not something that's easy to maintain either way.

[Xrayez]

GDNative modules might work just well for this. Or do they not?

As for now, usability-wise GDNative is not on par with plain C++ module development. Also, feature-wise it might never provide what C++ modules can due to its dynamic nature: godotengine/godot-proposals#565 (comment). All depends on your actual use cases though.

Because of this, people want to have their features already available within Godot as built-in, which probably goes against design philosophy regarding what goes into core (still not sure about the exact criteria).

To help overcome some issues regarding C++ modules usage within projects, I've come up with #36922, but this still might be in vain because of godotengine/godot-proposals#565 (comment):

given the idea is to discourage the use of modules as much as possible towards the future.

which I'm still not sure what the rationale behind this.

[avril-gh]

@Xrayez

given the idea is to discourage the use of modules as much as possible towards the future. (...)

which I'm still not sure what the rationale behind this.

i believe rationale might be different, but real life is tough, so It will quickly turn into situation where godot will be 'like open source' engine with few basic functionalities, then there will be many pay'd and closed source, licensed third party GDNative DLL's for any / each serious feature, without alternative of open source modules...

[NHodgesVFX]

I think this should be integrated into the engine it can be incredibly useful for not just games but apps as well. SQLite also is file based rather then server based, which is useful if you dont need high performance concurrent acsses. If an abstraction was created it would make sense to support MySQL and Postgres. SQLIte is also better for apps and games that dont need any type of server. For server stuff it's easy enough to create a python rest api that interacts with a DB then call it through godots http classes. If you want a simple DB, that's on the client to store stuff it would be way easier if it's just integrated into the engine.

[girng]

AFAIK, item, level, npc data is stored in a database, and exported out as csv or json (or whatever format the developer prefers), and then parsed and used locally on the client. In Godot's case, it would be parsed into a dictionary or array. And that data is exported when you export your game.

Every developer does it differently, however, I can't imagine performing SQL queries locally for that would be advantageous..

[TGRCdev]

TBH I would much rather have a DatabaseAbstraction API of sorts which would allow you to be completely backend agnostic(yes just like an ORM), but the implementation specific integration would have to be imported in. Just like how VCS works at the moment.

I actually really like this idea. Lots of classes in Godot are designed to be inherited for different implementation purposes (Shape, Texture, AudioStream, etc.), which makes Godot a really modular and toolable engine.

I think we could create a Database class that can be implemented for different database backends (DatabaseSQLite, DatabasePostgres, etc.). We could base it on Python's Database API.

I'm currently doing some basic experiments on how this would work, and I'm looking into what backends I could implement at this stage.

• SQLite would be trivial to implement.
• I've gotten Postgres partially working, but not without significant effort in the build stage, and not portably.
• The community version of MySQL's C API is licensed under GPL. It has a FOSS exception, but the wording is hard to understand, and I can't say for sure whether or not it could be distributed with Godot.

Thoughts?

[kuruk-mm]

You can put those on the implementation, and declare:

class sqlite3;
class sqlite3_stmt;

For hide the headers from who includes this file.

[madmiraal]

Unfortunately, the spmemvfs.h header also includes the sqlite3.h header; so it will still be included. And we can't use the same trick for the spmemvfs.h header, because the p_db member variable is not a pointer of spmemvfs_db_t.

[TheDuriel]

Also in favor. It eases over integration of databases created outside of Godot, and provides a very nice feature for tool developers. Content creation tools, like modding kits, also benefit heavily.

[Skalli]

I'm using SQLite in my project already, but would much prefer it to be integrated in the engine to have a clean interface.
I use it also for various things, currently in my dialog system where it's much faster than XML or JSON based lookups, also will use it for faster saving and loading, levels and other things.
Would like it if the integration doesn't abstract away the SqliteCommand and such but exposes them.

[ghmart]

Would be even greater if SQLite were integrated at much deeper level as a data binding mechanism for some of nodes' properties.
Also SQLite's R*Tree module can be used for "streaming worlds" feature I suppose?

[slapin]

Databases for saves might be interesting idea. Need to check risks and gains. Have anybody checked data corruption cases in case of io errors on user side? also in case of safe data scheme changes?

[slapin]

I guess there are GIS extentions to sqlite which might be useful for world annotations in open world games.

[ghmart]

What is the use case for a game engine to intergarate data base ?

Baldur's Gate 1/2, Icewind Dale "enhanced editions" use SQL for storing game options for example.

[fire]

There has been no response to this pr, so yes your suggestion would improve code, but it won't improve the approval process. So I'm waiting for some decision on approval.

It's the difference between working and working but better.

Also, this module has been used in a shipped game.

Edited:

Also I believe both limits are exactly the same. The type of size() on a byte array.

The expected usecase of spmemvfs is for the use of sending prepackaged data that is merged with player data inside of user://. There is no ability to modify res:// via a godot based VFS, but the one via spmemvfs can be modified since it's a variation of an in memory database. I only touch on the fact that a Godot class for in-memory streams exist. However, spmemvfs is less work.

Godot VFS	spmemvfs
❌ PackedByteArray().size() limit	❌ PackedByteArray().size() limit
❌ Cannot modify res://	❌ Cannot modify res://
✔️ Read as used	❌ In memory
❌ Cannot modify after load from res://	✔️ Can modify after loading from res://
❌ 🔥 Does not exist	✔️ Already coded
✔️ Can modify user://	✔️ Can modify user://

[LikeLakers2]

PackedByteArray().size() limit

I mean... we don't need to load the whole file in. SQLite is already built around seeking through the file and only picking out the parts we need -- or at least, that's how I understood it. Godot also even exposes that type of functionality (seeking and reading only the parts they need) to the user through the File class, which means there must already be underlying APIs that allow Godot to avoid fully loading a opened file into memory.

Cannot modify res://

I never said we needed to modify res://?

Cannot modify after load from res:// || Can modify after loading from res://

I'm confused why this is being brought up. If someone really needs to modify a database that's been loaded from res://, they're going to have to save to something other than res://, regardless of whether they're using a Godot VFS or spmemvfs.

Additionally, we can offer a "make a in-memory copy of this database" method that uses SQLite's Backup API (and we might want to anyways, even with spmemvfs, to make it easy to duplicate for the purposes of save files), so this point seems kinda moot.

🔥 Does not exist || Already coded

We already code abstractions between Godot and third-party library... so what's one more if it makes the third-party library a bit more tightly integrated with Godot?

[fire]

I don't understand, are you offering to do this work? The problem is it won't improve the chances of this being accepted and it's a lot of work.

[LikeLakers2]

I am not offering, unfortunately. While I've examined the engine's internals before, I don't really have the know-how on how to use whatever file APIs that Godot might be using under the hood -- and even if I did, I don't feel like I'd be comfortable messing with something like that.

Still, I honest-to-god feel like spmemvfs is a super-hacky solution to the problem of reading files from res://. That's why I've been responding so much and pushing for a Godot-specific VFS within this PR -- because I feel like it would be a billion times better in every way besides some maintenance cost. But if that isn't worth the effort (and you don't seem to think it is), then whatever. Forget I brought up the idea, I guess.

[WilliamTambellini]

The note from @swarnimarun is wise : an DB agnostic API (ORM or not) would be futur proof, up to implement, atm, only 1 backend (SQLite).

The best practices for engine contributors recommend avoiding future-proofing unless a real need has been demonstrated in real-world projects.

@Calinou At least 1 real-world project needing godot to communicate with mysql:
https://godotforums.org/discussion/24586/small-job-offering-connect-to-mysql-database#latest

[Calinou]

@Calinou At least 1 real-world project needing godot to communicate with mysql:
https://godotforums.org/discussion/24586/small-job-offering-connect-to-mysql-database#latest

I don't think a single project popping up from time to time is sufficient evidence to add this as a core feature. A MySQL client can be provided by an add-on for those who actually need it (and can't use a REST API or similar to interact with a database).

Either way, this is getting off-topic as this pull request is about integrating SQLite to Godot (which has different use cases from MySQL and other client-server databases).

[aaronfranke]

This PR needs a proposal, please open one. In particular, the justification for why this has to be core needs to be laid out - what prevents people from integrating SQLite (or others) via GDNative? @Xrayez mentioned usability issues and limitations with GDNative, but those are separate issues, as GDNative usability should be improved regardless.

[akien-mga]

@Mohsen7s Feedback on the proposal is welcome, but please refrain from calling contributors "random guys".

And this feature proposal is not specific to GDScript.

I do think database integration is more GDExtension material than core modules, but either way an integration would be usable by GDScript, C#, any language bound to GDExtension, C++ modules, potential future visual scripting solutions, etc.

[aaronfranke]

@Mohsen7s Arguments about how this could be in an extension, or that there are multiple database choices so why prefer one, or that there are a ton of PRs and issues and we can't get to them all or else Godot 4 will take ten years, are all valid arguments. However, telling people who want to use a database to go to C# is not a solution, so please stop. The idea behind this PR is very valid, even if I disagree with this being an engine feature.

[akien-mga]

@Mohsen7s If you came here to pick up fights with contributors, this is not welcome. You're breaching our Code of Conduct, and given some follow-up interactions on Twitter which were extremely inappropriate, the Code of Conduct team decided to restrict your access to the @godotengine GitHub organization.

[Zireael07]

Bump. I find that I either repeat data or need hacks. An SQLite database would be the solution except... there isn't one in Godot?

[Calinou]

Bump. I find that I either repeat data or need hacks. An SQLite database would be the solution except... there isn't one in Godot?

I'd say it's the perfect fit for GDExtension. SQLite isn't exactly a small library after all 🙂

Godot 4 binaries will already be a fair bit larger than Godot 3 – let's not make the problem worse than it currently is.

[SimplyPhy]

SQLite isn't exactly a small library

Isn't is generally between 500 and 1100 KB? https://www.sqlite.org/footprint.html (note: this is from 5 years ago)
I'm not an expert on what constitutes a small library, but for such a powerful resource, it seems very small to me.

I guess what I'm really trying to say is, whether or not SQLite belongs in Godot is unlikely to be determined by its size. Perhaps I'm mistaken.

[Calinou]

I'm not an expert on what constitutes a small library, but for such a powerful resource, it seems very small to me.

That's pretty large compared to most libraries embedded in Godot, especially for something that will only be used by a small portion of all projects. In comparison, some libraries embedded in Godot are used in every project out there (such as FreeType or libpng).

[aaronfranke]

This PR (still) needs a proposal before it can be considered, please open one. In particular, the justification for why this has to be core needs to be laid out - what prevents people from integrating SQLite (or others) via GDExtension?

[Scoppio]

Yeah, unless the addition of the SQLite comes with an integration like Unreal has, where resources can be table like structures where you can add hundreds or thousands of stuff in it and just instantiate directly from there, and it keeps a tight coupling with the table, so when you change something in there it also changes the coupled instance in the world.

Even for saving games, like the new Mewgenics which Tyler Glaiel is making that after me and some other people suggested him to use SQLite now utilizes it to persist the characers, the save games, it even has anti-save-editing measures in place and since it can do atomic operations, prevents from corrupting a save file during save for "ironman mode" and this kind of stuff.

I don't see what functionality it brings to the engine that can't be currently solved with the addon, since the only argument pro adding it currently is "because it is neater not to have to import more things". The price has to be paid somewhere and here it would be in the engine and therefore in the final executable.

[MMUTFX2053]

the library size isnt big, and looking at the other comments here, i see the claim that without additional features sqlite on its own isn't that useful, but wouldn't you still need sqlite implemented as a core library to implement those other features on top of sqlite ?

[aaronfranke]

but wouldn't you still need sqlite implemented as a core library to implement those other features on top of sqlite ?

@MMUTFX2053 No. The entire package, including the library and high-level interfaces, can be done in GDExtension.

[MMUTFX2053]

but wouldn't you still need sqlite implemented as a core library to implement those other features on top of sqlite ?

@MMUTFX2053 No. The entire package, including the library and high-level interfaces, can be done in GDExtension.

can this be done as an official plugin maintained by the Godot team ?

[SimplyPhy]

can this be done as an official plugin maintained by the Godot team ?

@MMUTFX2053 as noted, discussion regarding your question requires a proposal. Likewise, for clarity, the question isn't "can" but "should/will". If you're invested in this proposal and believe you have solid groups for advocating for it, I recommend following the proposal guidelines and creating one.