uncorrect resumtion token

[rerowep]

Tried to update on of our servers to invenio version 3.2.
Now I can not ListRecordsamy more. I get following error:

  File "/XXX/src/invenio-oaiharvester/invenio_oaiharvester/cli.py", line 87, in harvest
    setspecs
  File "/XXX/src/invenio-oaiharvester/invenio_oaiharvester/api.py", line 103, in list_records
    for record in request.ListRecords(**params):
  File "/XXX/lib/python3.6/site-packages/sickle/iterator.py", line 52, in __next__
    return self.next()
  File "/XXX/lib/python3.6/site-packages/sickle/iterator.py", line 151, in next
    self._next_response()
  File "/XXX/lib/python3.6/site-packages/sickle/iterator.py", line 138, in _next_response
    super(OAIItemIterator, self)._next_response()
  File "/XXX/lib/python3.6/site-packages/sickle/iterator.py", line 84, in _next_response
    self.oai_response = self.sickle.harvest(**params)
  File "/XXX/lib/python3.6/site-packages/sickle/app.py", line 120, in harvest
    http_response.raise_for_status()
  File "/XXX/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: UNPROCESSABLE ENTITY for url: http://localhost/oai2d?resumptionToken=.eJyNjU9vgjAchr_L70wWiiJ_bjJTiJkECC2Mi6mlKKEM0pJBNH73eZgel723N3nyPDfQQtTgm2_OxrVdz7HNtWO7G2TAyM4CfMuAbmbqrMG_QS8mVrOJJUo07QI-9ExxC4EBjRr6xx-F6lut228BdwM0V4OUx_bhh90XRlW5t2l4kaeuknUYD0G6_V1TYBkn9FqTE8YBuVbjobwsOdpOothjHvEnGKGlynPEVzRcspQeLJbjNSde-lHKNiuD1RMMvK4gplfGKNOE4Dzuu0VEOCHvyP7s6FzPrzb7b9sK_m7D_QfDQWlb.XkkFkw.ldY9EVX1H7W3LOgZv2goXLJ4M0Y&verb=ListRecords

During normal installation the version 1.1.1 of invenio-oaiserver gets installed. Acording to the logs of git there should be a version 1.1.2.

With version 1.0.3 of invenio-oaiserver every thing works as expected.

[lnielsen]

Seems like the release wasn't tagged and pushed. I've pushed the tag now to release v1.1.2. I however doubt it will fix the problem.

[rerowep]

The tag is correctly at 1.1.2 now but the token is still not working.

[rerowep]

I have still following error if i try to harvest with invenio-harvester:

Traceback (most recent call last):
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/bin/invenio", line 8, in <module>
    sys.exit(cli())
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/src/invenio-oaiharvester/invenio_oaiharvester/cli.py", line 87, in harvest
    setspecs
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/src/invenio-oaiharvester/invenio_oaiharvester/api.py", line 103, in list_records
    for record in request.ListRecords(**params):
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/sickle/iterator.py", line 52, in __next__
    return self.next()
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/sickle/iterator.py", line 151, in next
    self._next_response()
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/sickle/iterator.py", line 138, in _next_response
    super(OAIItemIterator, self)._next_response()
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/sickle/iterator.py", line 84, in _next_response
    self.oai_response = self.sickle.harvest(**params)
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/sickle/app.py", line 120, in harvest
    http_response.raise_for_status()
  File "/Users/wep/devel/virtualenvs/rero-ils-nkNkJRmS/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: UNPROCESSABLE ENTITY for url: http://localhost:5005/oai2d?resumptionToken=.eJw1zbsKgzAYQOF3-WcpGjVewMEiLXSqtt6ylKDxQgxKItZWfPe69KzfcDZQjNXg6yfDc7GDselZ2EW2gy0NJtoy8JEG_E1lq8DfoJGjAB8mJkWvVL8w0ECwmdZ0pnfJmn49VFBZIQN2DVQlx2F49ccBouJikOJmx7nNE5HpJWrP4T8S5hXPlmfUdeS6pkSfo5i3n-RhpKVZftM4CGD_Ac0WNfI.XphYNA.aRg37ECuKQ0I4o1Fy6tYq_lZXDo&verb=ListRecords

[rerowep]

Ok it is working with marshmallow = 2.21.0 it is not working with marshmallow = 3.5.1
Following line from resumtion_token.py
result['kwargs'] = self.root.load(result['kwargs'], partial=True).data
is crashing with 3.51with following error:
{'from': ['Unknown field.']}

[rerowep]

What is the status of this issue. Will the problem be solved in the near futur?

[rerowep]

Still not solved. Manually the resumption token is working but if I harvest with: https://github.com/inveniosoftware/invenio-oaiharvester
I got following error: requests.exceptions.HTTPError: 422 Client Error: UNPROCESSABLE ENTITY for url: http://localhost:5000/oai2d?resumptionToken=.eJw1jU8LgjAcQL_L7zzCTQdO8JCEQcRqkf-6xNAp5oaySQXid89L7_Au7_AWcEo1EHk7RgMa-uFmRhjDDMEkOwURQTB8pO0cRAu0djQQwaSs6Z3r3woQGDXLRs7yalXbf7dqpK0JhhWBq-2o9bPfBnAoU_woT1QUdLiZ3KtIl-z_nIPikjNdpzopXvzeeJxmR6y54aLKsC9EHMP6A5A4NIk.X5E2Ow.NESNSuQO14reGp1_66rpiby2uzY&verb=ListRecords

[rerowep]

The following harvest with Sickle works:
records = sickle.ListRecords(**{'metadataPrefix': 'oai_dc'})
but if I try to use the from parameter like this:
records = sickle.ListRecords(**{'metadataPrefix': 'oai_dc', 'from': '2020-01-01'})
the request errors out with 422 Client Error: UNPROCESSABLE ENTITY for url:
http://localhost:5000/oai2d?resumptionToken=.eJw1zU1vgjAcgPHv8j-TpS3CJokHkZXFRDKYlJeL6Wxh2HaSFqfT-N3nZc_td3pu4KQUEKEnFPoBeg4xDn0fI_8FezDyXkJEPFBnbnsH0Q2MnLjgE3-3shsuEIHhdk8weNDZo3l4lNYMzg0_Eu4euL09ar0bHgegqT415ILblHUNmZ9EymaCzrGotEpqitt6HeRVoArDUEP6mL7FBWdi-qBsxRj73l7Vb0bGhCVfm-V_m7IqjK6z16wsq3XxeW3VVi9nPG3P2SE-5PliAfc_m6hIQQ.X6P9cg.z7bzup2NK_r_glsgFo4ppW6GcmQ&verb=ListRecords

Is there a fix planed for the near futur?

[rerowep]

If I pop from and until in invenio_oaiserver/resumption_token.py::serialize like this:

def serialize(pagination, **kwargs):
    """Return resumption token serializer."""

    if not pagination.has_next:
        return

    token_builder = URLSafeTimedSerializer(
        current_app.config['SECRET_KEY'],
        salt=kwargs['verb'],
    )
    schema = _schema_from_verb(kwargs['verb'], partial=False)
    data = dict(seed=random.random(), page=pagination.next_num,
                kwargs=schema.dump(kwargs).data)
    data['kwargs'].pop('from', None)
    data['kwargs'].pop('until', None)

    scroll_id = getattr(pagination, '_scroll_id', None)
    if scroll_id:
        data['scroll_id'] = scroll_id

    return token_builder.dumps(data)

Its working correctly.
I do not understand why the kwargs=schema.dump(kwargs).data changes the dates from from and until to permissive?