DynamoDB Tracker Store does not pull all sender_ids

[kearnsw]

Rasa Version : 2.8.0
Minimum Compatible Version: 2.8.0
Rasa SDK Version : 2.8.0
Rasa X Version : None
Python Version : 3.7.10
Operating System : Darwin-20.5.0-x86_64-i386-64bit

Issue:
The dynamodb tracker store only pulls the first batch of sender_ids. It needs to loop in order to pull all sender_ids.

[sara-tagger]

Thanks for raising this issue, @wochinge will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[wochinge]

Which code / command are you using to cause the error @kearnsw ?

[kearnsw]

Hi @wochinge,

I am using the Exporter, which has the following code that is causing the error:

    def _get_conversation_ids_in_tracker(self) -> Set[Text]:
        """Fetch conversation IDs in `self.tracker_store`.
        Returns:
            A set of conversation IDs in `self.tracker_store`.
        Raises:
            `NoConversationsInTrackerStoreError` if
            `conversation_ids_in_tracker_store` is empty.
        """
        conversation_ids_in_tracker_store = set(self.tracker_store.keys())

        if conversation_ids_in_tracker_store:
            return conversation_ids_in_tracker_store

        raise NoConversationsInTrackerStoreError(
            "Could not find any conversations in connected tracker store. "
            "Please validate your `endpoints.yml` and make sure the defined "
            "tracker store exists. Exiting."
        )

This, in turn, calls the following when the endpoints.yml file is set to use DynamoDB tracker store:

    def keys(self) -> Iterable[Text]:
        """Returns sender_ids of the `DynamoTrackerStore`."""
        return [
            i["sender_id"]
            for i in self.db.scan(ProjectionExpression="sender_id")["Items"]
        ]

This is the incorrect way to pull all values from DynamoDB the correct way is implemented in #9196.