Option to export git log to sqlite

[amitu]

When this command runs it creates an in memory sqlite database with all tables and throws us in the sqlite3 shell (possibly with --dump, which simply stores the tables in a sqlite db disc based file). This will allow much richer query.

[filhodanuvem]

Hey @amitu ,
I think that the main goal of gitql is make queries on git repositories. If we put a feature about export to sqlite, it will loss that goal because the query language would be from sqlite. Maybe create other project
looks better.
When I started to code this, I thought to follow that strategy, but one of the problems is that you always would need to sync the database before run queries and maybe it would causes a bad experience.

What do you thought about this problem?

[amitu]

The in memory db was to avoid the problem of resyncing. With in memory db its more like git log | less, the content displayed in pager won't change, it will be a snapshot, and on next call it would be the latest snapshot.

The very first thing I tried after installing gitql was to do a group by author, and realised group by doesn't work.

It would be vary challenging task to keep the query language be full featured, so passing to sqlite would be a good idea instead of reimplementing every SQL nuance.

[luizperes]

how long would gitql spend to fill in the database in the case of in memory db, @amitu?

[amitu]

It will depend on the size of the repository. It will take almost the same time as git log does.

[luizperes]

We could also try to implement memoization together with taking the latest snapshot. Sounds promising, but we gotta study its viability and speed

[jcubic]

I've create small script in python that export git log to sqlite database file:

https://gist.github.com/jcubic/0c7ccae5d764ff61788fc626096e5688

so maybe you can use it instead of implementing option to export to sqlite in gitql.

[jcubic]

and it's pretty fast, it was only slow when I print the log when debugging so if you will import export in gitql it shouldn't be slow.

[luizperes]

That is cool! I will take a look at that later, as I think that our priority for now would be fixing the small bugs on gitql. Only then we could start working on that. What do you think @jcubic ?

[jcubic]

No rush and great work on the tool.

[filhodanuvem]

Do you guys think that this is something needed in the tool? As we can show the results as a json, if someone need to that, they can use the json as an input to another script.
If we move on, our domain will change a lot, we will start thinking in different database drivers, for example.

[filhodanuvem]

Well. I'm closing this for now based that gitql should not connects with real databases, we can reopen it to think in other tool like jcubic did. Thank you for all the suggestions.

[jsixface]

I agree. But do you think, we can have export to a CSV file. Most databases support exporting a query to a csv file; easier for sharing data to humans. JSON is good for sharing data with other apps, which we already have.