BUG: unexpected behavior on dataset deletion

[jotelha]

Currently, I see the following behavior:

In [1]: from dtool_lookup_api import get_datasets, get_dataset, register_dataset, delete_dataset, get_base_uris, get_base_uri, register_base_uri, delete_base_uri

In [2]: get_base_uris()
Out[2]: 
[{'base_uri': 's3://test-1'},
 {'base_uri': 's3://test-bucket'},
 {'base_uri': 'smb://test-share'}]

In [3]: get_dataset("s3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677")
Out[3]: {'code': 404, 'status': 'Not Found'}

In [4]: get_datasets(base_uris=["s3://test-1"])
Out[4]: 
[{'base_uri': 's3://test-1',
  'created_at': 1604860720.736,
  'creator_username': 'testuser',
  'frozen_at': 1604864525.691,
  'name': 'testuser',
  'number_of_items': 0,
  'size_in_bytes': 0,
  'uri': 's3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe677'}]

In [5]: delete_dataset("s3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677")
Out[5]: True

In [6]: get_datasets(base_uris=["s3://test-1"])
Out[6]: 
[{'base_uri': 's3://test-1',
  'created_at': 1604860720.736,
  'creator_username': 'testuser',
  'frozen_at': 1604864525.691,
  'name': 'testuser',
  'number_of_items': 0,
  'size_in_bytes': 0,
  'uri': 's3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe677'}]

This looks like some inconsistency in the database / server functionality.

[jotelha]

Apparently, the deletion of the dataset is not properly propagated to the search database in the search plugin.

[jotelha]

The database entry apparently persist even after having deleted and recreated the base URI:

In [48]: delete_base_uri('s3://test-1')
Out[48]: True

In [49]: get_user('testuser')
Out[49]: 
{'is_admin': True,
 'register_permissions_on_base_uris': [],
 'search_permissions_on_base_uris': ['smb://test-share', 's3://test-bucket'],
 'username': 'testuser'}

In [50]: get_base_uris()
Out[50]: [{'base_uri': 's3://test-bucket'}, {'base_uri': 'smb://test-share'}]

In [51]: get_datasets(base_uris=["s3://test-1"])
Server returned no pagination information. Server version outdated.
Out[51]: []

In [52]: get_dataset("s3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677")
Out[52]: {'code': 403, 'status': 'Forbidden'}

In [53]: register_base_uri(**{
    ...:             "base_uri": "s3://test-1",
    ...:             "users_with_search_permissions": ["testuser"],
    ...:             "users_with_register_permissions": ["testuser"],
    ...:         })
Out[53]: True

In [54]: get_base_uris()
Out[54]: 
[{'base_uri': 's3://test-1'},
 {'base_uri': 's3://test-bucket'},
 {'base_uri': 'smb://test-share'}]

In [55]: get_user('testuser')
Out[55]: 
{'is_admin': True,
 'register_permissions_on_base_uris': ['s3://test-1'],
 'search_permissions_on_base_uris': ['smb://test-share',
  's3://test-bucket',
  's3://test-1'],
 'username': 'testuser'}

In [56]: get_datasets(base_uris=["s3://test-1"])
Out[56]: 
[{'base_uri': 's3://test-1',
  'created_at': 2604860720.736,
  'creator_username': 'another-testuser',
  'frozen_at': 2604864525.691,
  'name': 'updated_test_dataset',
  'number_of_items': 2,
  'size_in_bytes': 3,
  'uri': 's3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe677'}]

In [57]: get_dataset("s3://test-1/1a1f9fad-8589-413e-9602-5bbd66bfe677")
Out[57]: {'code': 404, 'status': 'Not Found'}