extract data using datetimemin and datetimemax parameters

[bchiavarini]

Using Python API on Dballe 8.6 i am trying to extract data in certain date range. To define the date range i am using the datetimemin and datetimemax parameters but i have some problems in queries results.
The following queries are getting some results:

• dballe query: {'level': (1, None, None, None), 'var': 'B13011', 'trange': (1, 0, 86400), 'datetimemin': datetime.datetime(2006, 1, 9, 0, 0, tzinfo=tzlocal())}
  result: result_with_datetimemin.zip
• dballe query: {'level': (1, None, None, None), 'var': 'B13011', 'trange': (1, 0, 86400), 'datetimemax': datetime.datetime(2006, 1, 12, 9, 8, 57, 986000, tzinfo=tzlocal())}
  result: result_with_datetimemax.zip

But i don't manage to get any result if i add the rep_memo parameter to the query with any value
ex. dballe query: {'level': (1, None, None, None), 'var': 'B13011', 'rep_memo': 'agrmet', 'trange': (1, 0, 86400), 'datetimemax': datetime.datetime(2006, 1, 12, 9, 8, 57, 986000, tzinfo=tzlocal())}
or if i use in the same query both the datetimemin and datetimemax parameters
ex. dballe query: {'level': (1, None, None, None), 'var': 'B13011', 'trange': (1, 0, 86400), 'datetimemin': datetime.datetime(2006, 1, 9, 0, 0, tzinfo=tzlocal()), 'datetimemax': datetime.datetime(2006, 1, 12, 9, 8, 57, 986000, tzinfo=tzlocal())}

Is this the correct way to extract data in certain date range? why the queries using both datetime parameters or using datetime parameters combined with report parameter are not getting any result?

[edigiacomo]

Sorry, maybe I'm missing something but I can't reproduce your issue.

I have imported the first JSON file in a dbadb

$ rpm -qi dballe | grep -i version
Version     : 8.9
$ dbadb import --wipe-first --dsn=sqlite:buttami.sqlite3 -t json result_with_datetimemin.json

And then, I tried different queries, using this Python script

import dballe
import datetime

db = dballe.DB.connect("sqlite:buttami.sqlite3")

basequery = {
    'level': (1, None, None, None),
    'var': 'B13011',
    'trange': (1, 0, 86400),
}


queries = {
    "min_only": {
        'level': (1, None, None, None),
        'var': 'B13011',
        'trange': (1, 0, 86400),
        'datetimemin': datetime.datetime(2006, 1, 9, 0, 0),
    },
    "min_and_rep_memo": {
        'level': (1, None, None, None),
        'var': 'B13011',
        'trange': (1, 0, 86400),
        'datetimemin': datetime.datetime(2006, 1, 9, 0, 0),
        'rep_memo': 'agrmet',
    },
    "min_and_max": {
        'level': (1, None, None, None),
        'var': 'B13011',
        'trange': (1, 0, 86400),
        'datetimemin': datetime.datetime(2006, 1, 9, 0, 0),
        'datetimemax': datetime.datetime(2006, 1, 12, 9, 8, 57, 986000),
    },
    "max_max_and_rep_memo": {
        'level': (1, None, None, None),
        'var': 'B13011',
        'trange': (1, 0, 86400),
        'datetimemin': datetime.datetime(2006, 1, 9, 0, 0),
        'datetimemax': datetime.datetime(2006, 1, 12, 9, 8, 57, 986000),
        'rep_memo': 'agrmet',
    }
}

with db.transaction() as tr:
    for name, query in queries.items():
        count = tr.query_data(query).remaining
        print("Number of result for {}: {}".format(name, count))

And the output is:

Number of result for min_only: 20374
Number of result for min_and_rep_memo: 1967
Number of result for min_and_max: 3473
Number of result for max_max_and_rep_memo: 270

[bchiavarini]

i attach the original bufr file i have in my dballe on which i have tested the previous queries
meteo-2006-01.zip

[edigiacomo]

I'm sorry but I still can't reproduce the issue:

$ rm buttami.sqlite3 && dbadb import --wipe-first --dsn=sqlite:buttami.sqlite3 meteo-2006-01.bufr

And the output is the same:

Number of result for min_only: 20374
Number of result for min_and_rep_memo: 1967
Number of result for min_and_max: 3473
Number of result for max_max_and_rep_memo: 270

Can you reproduce the bug with a minimal script and attach it here? Thanks!

[bchiavarini]

i think the problem is with the Explorer object. I'm using an Explorer object to check if the query gives back some results: i set the query as filter for the explorer and using explorer.levels (or whatever field) i check if the explorer is empty (query does not give results) or not.
here the script i have used
script with explorer.zip

and my output is:

levels for query min_only: [dballe.Level(1,None,None,None)]
levels for query min_and_rep_memo: []
levels for query min_and_max: []
levels for query max_max_and_rep_memo: []

[edigiacomo]

Thank you!

[spanezz]

Ho cambiato lo script aggiungendo una prova con una query tradizionale e ho confermato il problema:

for name, query in queries.items():
    explorer.set_filter(query)
    # check if query is giving a result
    levels_in_query = explorer.levels
    print('levels for query {}: {}'.format(name, levels_in_query))
    
    with db.transaction() as tr:
        count = 0
        for rec in tr.query_data(query):
            count += 1
        print(f'results for query {name}: {count}')

Output:

levels for query min_only: [dballe.Level(1,None,None,None)]
results for query min_only: 20374
levels for query min_and_rep_memo: []
results for query min_and_rep_memo: 1967
levels for query min_and_max: []
results for query min_and_max: 3473
levels for query max_max_and_rep_memo: []
results for query max_max_and_rep_memo: 270

Ora guardo a cosa stia succedendo

[spanezz]

Le query che intersecavano, invece di contenere, i range di datetime, per errore non accettavano il range: ora l'ho corretto. Grazie per aver trovato e riprodotto questa magagna!