Nonblockingdb Status

GSoC-2013 Status page for project "Non-Blocking Database Access"

Student: Nazar Gerasymchuk (email: nazar.gerasymchuk at gmail.com, IRC: tr0)

Mentor: Daniel Schürmann (email: daschuer at mixxx.org, IRC: daschuer)

My blogs

You can find me there, and contact me through blogs comments:

• https://nonblockingdb.wordpress.com/ (EN) -- blog for news on my developing process at GSoC 2013.
• http://neval8.wordpress.com/ (EN, UA) -- my personal blog with tips'n'tricks, hacks and so on.

My fork of Mixxx at GitHub: https://github.com/troyane/mixxx

---

What plans do I have for this summer?

1. Improve my Git skills
2. Get involved in developing of Mixxx
3. Solve different bugs on Mixxx
4. Build new scheme for database access in Mixxx
5. Test that scheme, try to find bottlenecks, possible deadlocks...
6. Code that scheme separate from Mixxx
7. Write documentation on scheme
8. Apply that scheme to Mixxx
9. Test Mixxx with my scheme
10. PROFIT :)

Weeks schedule / Tasks

In general, I see two milestones:

1. Before midterm: prepare separate working prototype for db access.
2. Before finalterm: merge prototype into Mixxx.

1. Week (17.06 -- 23.06)
   1. Write down this schedule.
   2. Write article "Overview of all what we have in Mixxx (Qt, SQLite, threads, …). What to do with it?": https://nonblockingdb.wordpress.com/2013/06/19/overview-of-all-what-we-have-in-mixxx-qt-sqlite-threads-what-to-do-with-it/.
   3. Fork Mixxx on GitHub.
2. Week (24.06 -- 30.06)
   1. Dig into Sqlite and define or find some suitable test cases.
   2. Dig into Mixxx and find all usage of db access, analyze it, write brief description of different types of DB usage in Mixxx and think on what to aware of when designing solution.
   3. Prepare UML class (and probably, sequence) dias for nowadays Mixxx DB accesss. // For this purposes I use Dia (http://projects.gnome.org/dia/), ArgoUML (http://argouml-stats.tigris.org/) or Umrello (http://uml.sourceforge.net/)//
   4. Find all related bugs.
3. Week (01.07 -- 07.07)
   1. Begin writing documentation.
   2. Writing the specs for a possible solution.
   3. Prepare UML class (and probably, sequence) dias for new DB accesss in Mixxx.
   4. Create separate repo on GitHub for prototype.
   5. Start implementing of prototype database access class with separate test project.
   6. Start Open discussion to involve other developers to review prototype.
   7. (Try to) Fix some bugs.
4. Week (08.07 -- 14.07)
   1. Implement prototype database access class with separate test project.
   2. (Try to) Fix some bugs.
5. Week (15.07 -- 21.07)
   1. Get acquainted with "Google Test" framework.
   2. Write tests using "Google Test" framework.
   3. (Try to) Fix some bugs.
6. Week (22.07 -- 28.07)
   1. Add and test all possible ways of database usage present in Mixxx handled by DAO.
   2. (Try to) Fix some bugs.
7. Week (29.07 -- 04.08)
   1. Finish prototype that is able to handle asynchronous db access, ..., flexible and extendable.

• Important date 29.07 -- Begins midterm deadline
• Important date 20.08 -- Midterm deadline
• For midterm I'll prepare separate prototype project on separate repo.

1. Week (05.08 -- 11.08)
   1. Plan (re-plan) schedule of work for finalterm.
   2. ...
   3. (Try to) Fix some bugs.
2. Week (12.08 -- 18.08)
   1. ...
   2. (Try to) Fix some bugs.
3. Week (19.08 -- 25.08)
   1. ...
   2. (Try to) Fix some bugs.
4. Week (26.08 -- 01.09)
   1. ...
   2. (Try to) Fix some bugs.
5. Week (02.09 -- 08.09)
   1. ...
   2. (Try to) Fix some bugs.

1. Week (09.09 -- 15.09)
   1. Iron out remaining bugs.
   2. Merge my branch with master.
2. Week (16.09 -- 22.09)

• Important date 16.09 -- Pencils down begins

1. Week (23.09 -- 29.09) :

• Important date 27.09 -- Finalterm deadline

1. 01.10 : end

Current status

Currently I'm working on:

1. Extended article to collect as much as possible information about project and how to solve it: http://nonblockingdb.wordpress.com/2013/06/19/overview-of-all-what-we-have-in-mixxx-qt-sqlite-threads-what-to-do-with-it/
2. My very approximate roadmap for this summer:

Roadmap:

• Learn SQLite deeper as Daniel pointed (learn sources of SQLite multi-threading).
• Fork Mixxx on GitHub
• Create empty class DBAccessor
• add own class inherited from QThread, named, for example, DBAccessor
• create handmade DBAccessor's event-loop (like while(1)...)
• make class DBAccessor Singletone and all-source-wide
• make DBAccessor instantiate on Mixxx startup
• code freeing DBAccessor on Mixxx exit
• Code DBAccessor's queue manager
• code placing query into queue
• code controlling on executed queries
• code removing from queue
• ...
• Code DBAccessor's minimal access to SQLite db
• (Think first on how to) Organize DAO's names, priorities, and so on so we can send it as parameter to DBAccessor

---

Tasks

Original task: Non-Blocking Database Access

Currently some database transactions are stalling the GUI. This is because some database queries are preformed from the GUI thread.

This should be solved during the project by something like a standard non blocking interface to sqlite. All Database queries should be issued though this new interface.

This project would make Mixxx more reliable by this new concept how to deal with database actions.

Your proposal should include a draft proposal how do you will achieve the goal. You should have already have experience in parallel processing and sqlite.

My proposal

Let’s use RPC-style pure callbacks. I can propose to create class DBManager. DBManager will be a singleton and it will be instance in separate thread (inherited from QThread). This class will have own event loop. Also it will have queue of queries and pointers to QFuture objects to control (in own event loop) what queries have been executed.

We need one thread (DBManager) for management database access (and if needed -- solve probably conflicts), other (there can be more than one, so we can use QtConcurrent::run and pointers to QFuture objects to handle states and results) is for executing queries to database.

The task of processing a short queue of queries to database is a probably long time procedure, so we need to have separate thread for it. By the way we don’t miss possibility to execute multiple queries in the same time (if it is possible).

Executing queries from GUI thread is not good idea -- we’ll avoid it. After DBManager will be applied, we’ll got only one unified entry point to access database.

As this class will be singleton, it can be reachable from every module. All queries will be applied through DBManager this way:

• On module side (from concrete class):
• prepare query,
• set query parameters (such as type of needed DAO, priority or transaction recommendations),
• prepare callback function (function to apply after successfully executing query),
• call DBManager’s function to apply query with own callback functions(s) as parameter(s).
• On DBManager side:
• got new query into own queue of queries,
• scan current active queries,
• solve conflicts on how to place this query into queue of queries (to avoid blocking and other moments -- must discuss it),
• create new thread using QtConcurrent::run and run query from queue (if possible) using respective DAO-object in it and control it using QFuture,
• check what queries executed and execute respective callbacks.

Of course, there are lot of moments to discuss on.

More information will be here soon.