Fix error when creating node queries for large datasets

[zhuoyuan-liu]

Implement batch creation of node queries to handle more than 10,000 nodes efficiently.

It's a quick fix and I think it's a good idea to applied the batchsize as a global configuration for DB cofig.

It can be done by:

db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
    CreateBatchSize: 1000, 
})

[Copilot]

Pull Request Overview

This PR addresses an error encountered when creating node queries for large datasets by applying batch creation to prevent performance issues.

• Replaces q.DB.Create with q.DB.CreateInBatches using a hard-coded batch size (1000).
• Aims to improve performance for datasets with more than 10,000 nodes.

Comments suppressed due to low confidence (1)

pkg/queries/queries.go:411

• [nitpick] Consider using a global configuration variable for the batch size instead of hardcoding 1000, in order to align with the intended DB configuration.

if err := q.DB.CreateInBatches(&nodeQueries, 1000).Error; err != nil {

[javuto]

Is there something missing in this PR?

[javuto]

Nevermind, I found the documentation https://gorm.io/docs/create.html#Batch-Insert

[javuto]

If we apply the configuration using the gorm.Config statement, then every Create will be affected by that batch size. Not sure if that could have a negative impact or not.

[zhuoyuan-liu]

Here is the source code:

For a single value creation, it functions the same as a regular create operation. As for the rest of the database operations, I don't see any batch creation involved, so this change should have no impact on other operations.