model is retrained everytime due to dynamic checkpoint names

[JEM-Mosig]

Rasa version:
Rasa 1.3.8
Commit 39300da

Python version:
3.6.9

Operating system (windows, osx, ...):
osx

Issue:
I trained my model, and running rasa shell everything works fine. But once I start rasa interactive, it starts training a model, even though I didn’t change anything. After Your Rasa model is trained and saved at ... and Type a message and press enter (press 'Ctr-c' to exit)., it continues with processing story blocks, which takes longer than training the model. (I aborted at this point.)

If I want do rasa train and rasa shell after that, the action server seems not accessible.

Error (including full traceback):

First, no error message, just unexpected behaviour. Here is a screenshot:

Then, If I want do rasa train and rasa shell after that, I get this message:

rasa shell --debug                                                                                                                                       [16:59:55]
2019-10-15 17:00:53 DEBUG    rasa.model  - Extracted model to '/var/folders/yg/vcndsyk94tl492272h0x_wlm0000gp/T/tmpn8j0x0jm'.
2019-10-15 17:00:53 DEBUG    rasa.cli.utils  - Parameter 'endpoints' not set. Using default location 'endpoints.yml' instead.
2019-10-15 17:00:53 DEBUG    rasa.cli.utils  - Parameter 'credentials' not set. Using default location 'credentials.yml' instead.
2019-10-15 17:00:53 DEBUG    rasa.model  - Extracted model to '/var/folders/yg/vcndsyk94tl492272h0x_wlm0000gp/T/tmp2lsre1di'.
2019-10-15 17:00:54 INFO     root  - Connecting to channel 'cmdline' which was specified by the '--connector' argument. Any other channels will be ignored. To connect to all given channels, omit the '--connector' argument.
2019-10-15 17:00:54 DEBUG    sanic.root  - CORS: Configuring CORS with resources: {'/*': {'origins': [''], 'methods': 'DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT', 'allow_headers': ['.*'], 'expose_headers': None, 'supports_credentials': True, 'max_age': None, 'send_wildcard': False, 'automatic_options': True, 'vary_header': True, 'resources': {'/*': {'origins': ''}}, 'intercept_exceptions': True, 'always_send': True}}
2019-10-15 17:00:54 DEBUG    rasa.core.utils  - Available web server routes: 
/webhooks/rest                                     GET                            custom_webhook_CmdlineInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_CmdlineInput.receive
/                                                  GET                            hello
2019-10-15 17:00:54 INFO     root  - Starting Rasa server on http://localhost:5005
2019-10-15 17:00:54 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2019-10-15 17:00:54 INFO     root  - Enabling coroutine debugging. Loop id 5558218776.
2019-10-15 17:00:54 DEBUG    rasa.model  - Extracted model to '/var/folders/yg/vcndsyk94tl492272h0x_wlm0000gp/T/tmp6h47_0xh'.
2019-10-15 17:00:58 DEBUG    rasa.core.tracker_store  - Connected to InMemoryTrackerStore.
2019-10-15 17:00:58 DEBUG    rasa.core.lock_store  - Connected to lock store 'InMemoryLockStore'.
2019-10-15 17:00:58 DEBUG    rasa.model  - Extracted model to '/var/folders/yg/vcndsyk94tl492272h0x_wlm0000gp/T/tmp72_fr08t'.
2019-10-15 17:00:58 DEBUG    pykwalify.compat  - Using yaml library: /Users/johannes/miniconda3/envs/rasa/lib/python3.6/site-packages/ruamel/yaml/__init__.py
Bot loaded. Type a message and press enter (use '/stop' to exit): 
Your input ->  hi                                                                                                                                                                                                
Unable to start server
Traceback (most recent call last):
  File "/Users/johannes/miniconda3/envs/rasa/lib/python3.6/site-packages/sanic/server.py", line 778, in serve
    http_server = loop.run_until_complete(server_coroutine)
  File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1686, in create_server
OSError: [Errno 48] error while attempting to bind on address ('0.0.0.0', 5005): address already in use

Restarting my machine fixes this issue until I run rasa interactive again.

Command or request that led to error:

rasa train
rasa shell
rasa interactive  <-- problem here --/>

Content of configuration file (config.yml) (if relevant):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
- name: "DucklingHTTPExtractor"
  # url of the running duckling server
  url: "http://localhost:8000"
  # dimensions to extract
  dimensions: ["duration", "number"]
  # allows you to configure the locale, by default the language is
  # used
  locale: "en_EN"
  # if not set the default timezone of Duckling is going to be used
  # needed to calculate dates from relative expressions like "tomorrow"
  timezone: "Europe/Berlin"
- name: "WhitespaceTokenizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "CountVectorsFeaturizer"
  analyzer: "char_wb"
  min_ngram: 1
  max_ngram: 4
- name: "EmbeddingIntentClassifier"

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: EmbeddingPolicy
    max_history: 5  # default: 5
    batch_strategy: "balanced" # alternative: "sequence"
    epochs: 30 # default: 1
    random_seed: 2019  # default: null
    evaluate_on_num_examples: 0 # default: 100
  - name: MappingPolicy
  - name: FormPolicy

Content of domain file (domain.yml) (if relevant):

%YAML 1.1
---
actions:
- rating_inform
- travel_inform
- utter_acknowledge_problem_badlink
- utter_ask_eval_binary_natural
- utter_ask_eval_binary_helpful
- utter_ask_eval_binary_recommend
- utter_ask_bot_eval
- utter_ask_bot_rating
- utter_ask_helpdevelopers
- utter_ask_passengercount
- utter_ask_rephrase
- utter_ask_reqmore
- utter_ask_addtravel
- utter_ask_should_have_done
- utter_ask_should_restart
- utter_ask_travel
- utter_ask_travelmode
- utter_ask_traveltime
- utter_ask_whattodo
- utter_ask_why_here
- utter_explain_why_offset_travel
- utter_general_about
- utter_general_affirm
- utter_general_deny
- utter_general_emotional_happy
- utter_general_emotional_resigned
- utter_general_emotional_surprised
- utter_general_emotional_wow
- utter_general_goodbye
- utter_general_greet
- utter_general_oos
- utter_general_oos_future
- utter_general_thankyou
- utter_general_yourewelcome
- utter_meta_acknowledge_privacy
- utter_meta_iamabot
- utter_meta_list_features
- utter_meta_privacy
- utter_sorry
entities:
- duration
- number
- travelmode
forms:
- rating_form
- travel_form
intents:
- general_convey_affirmation
- travel_convey_affirmation
- general_convey_denial
- general_convey_express_uncertainty
- general_convey_farewell
- general_convey_greeting
- general_convey_thankyou
- general_express_emotion_happy
- general_express_emotion_unhappy
- general_express_emotion_surprised
- general_inform_affirmation_all
- general_inform_alreadydoing
- general_inform_problem_badlink:
    triggers: utter_acknowledge_problem_badlink
- general_inform_problem_badreply
- general_inquire_clarification
- meta_inquire_capabilities:
    triggers: utter_meta_list_features
- meta_inquire_botchallenge:
    triggers: utter_meta_iamabot
- oos_inform:
    triggers: utter_general_oos
- oos_inquire:
    triggers: utter_general_oos
- oos-future_inform:
    triggers: utter_general_oos_future
- oos-future_inquire:
    triggers: utter_general_oos_future
- travel_inform
- travel_inform_distance
- travel_inform_duration
- travel_inquire_clarification
- travel_inquire_howto_offset:
    triggers: travel_form
- general_placeholder
slots:
  eval_binary_helpful:
    type: unfeaturized
  eval_binary_natural:
    type: unfeaturized
  eval_binary_recommend:
    type: unfeaturized
  passengercount:
    type: unfeaturized
  requested_slot:
    type: unfeaturized
  travelmode:
    type: unfeaturized
  traveltime:
    type: unfeaturized
  traveldistance:
    type: unfeaturized
templates:
  utter_acknowledge_problem_badlink:
  - text: Oh no! I am sorry the link doesn't work anymore. I'll let my developers
      know.
  utter_ask_bot_eval:
  - text: Anything you'd like my developers to know?
  utter_ask_bot_rating:
  - buttons:
    - payload: you are dreadful
      title: dreadful
    - payload: you are mediocre
      title: mediocre
    - payload: you are good
      title: good
    - payload: you must be a human!
      title: human-like
    text: How do you think my performance will be rated?
  utter_ask_eval_binary_helpful:
  - text: Was our conversation helpful to you / did you learn something by chatting
      with me?
  utter_ask_eval_binary_natural:
  - text: Would you say our conversation was natural / intuitive?
  utter_ask_eval_binary_recommend:
  - text: Would you ask others to chat with me?
  utter_ask_helpdevelopers:
  - text: Could you help me with something, too?
  utter_ask_passengercount:
  - text: How many people were in the car?
  utter_ask_rephrase:
  - text: Not sure if I got that right. Could you please rephrase this?
  utter_ask_reqmore:
  - text: What would you like to do now?
  - text: Can I help you with anything else?
  utter_ask_addtravel:
  - text: Would you like to add another travel mode?
  utter_ask_should_have_done:
  - text: Oh no. I am utterly confused now. What should I have said?
  utter_ask_should_restart:
  - text: Thanks. I will send this to my developers so they can improve me. For now,
      I suggest I wipe my memory and restart our conversation. I won't remember anything
      you've said. Is that ok?
  utter_ask_travel:
  - text: Did you travel recently, or are you planning to travel?
  utter_ask_travelmode:
  - buttons:
    - payload: By car
      title: car
    - payload: By plane
      title: airplane
    - payload: By train
      title: train
    text: By what means did you travel?
  utter_ask_traveltime:
  - text: And how long did you / will you travel with the {travelmode}?
  utter_ask_whattodo:
  - text: What would you like to do?
  utter_ask_why_here:
  - text: I see. Don't you want to reduce your greenhouse gas emissions?
  utter_explain_why_offset_travel:
  - text: Traveling, especially by airplane, usually emits greenhouse gases which
      are causing the ongoing climate change. If you cannot avoid these emissions,
      you can buy 'offsets', i.e. donations to projects that reduce greenhouse gases,
      e.g. by planting trees.
  utter_general_about:
  - text: I was built to help you reduce or offset your greenhouse gas emissions.
  utter_general_affirm:
  - text: 'Yes'
  utter_general_deny:
  - text: 'No'
  utter_general_emotional_happy:
  - text: :-)
  utter_general_emotional_resigned:
  - text: Too bad :-/
  utter_general_emotional_surprised:
  - text: Seriously?!
  utter_general_emotional_wow:
  - text: Wow!
  utter_general_goodbye:
  - text: Goodbye then!
  - text: Alright, bye!
  utter_general_greet:
  - text: Hello!
  - text: Hi
  utter_general_oos:
  - text: I am afraid I can't help you with that.
  - text: That's beyond the scope of my programming.
  utter_general_oos_future:
  - text: Good question! A future version of me might be able to answer this.
  utter_general_thankyou:
  - text: Thank you.
  - text: Thanks
  utter_general_yourewelcome:
  - text: You're welcome!
  - text: No problem.
  utter_meta_acknowledge_privacy:
  - text: Data collection is set to {allow_datacollection}.
  utter_meta_iamabot:
  - text: I am a bot, powered by Rasa.
  utter_meta_list_features:
  - text: At the moment, I can only help you offset CO2 emissions of your travels.
  utter_meta_privacy:
  - buttons:
    - payload: You can collect my data
      title: Yes, you can record and share my conversation.
    - payload: Please forget my data
      title: No, I'd like to keep this private.
    text: Can we record and publish the conversation you are about to have?
  utter_sorry:
  - text: Oh no, sorry!

Enable Squad Definition of Done:

rasa train will not retrain model if stories have not changed, even if they contain OR steps.

[Ghostvv]

you forgot to kill previous action server, so it cannot start the new one

[wochinge]

The problem are the OR statements. They get different names:

## happy path
* greet
    - utter_greet
* mood_great
    - utter_happy

## sad path 1
* greet
    - utter_greet
* mood_great
> GENR_OR_9b3d9

## sad path 1
* greet
    - utter_greet
* mood_unhappy
> GENR_OR_9b3d9

## sad path 1
> GENR_OR_9b3d9 . <--
    - utter_cheer_up
    - utter_did_that_help
* affirm
    - utter_happy

## sad path 2
* greet
    - utter_greet
* mood_unhappy
    - utter_cheer_up
    - utter_did_that_help
* deny
    - utter_goodbye

## say goodbye
* goodbye
    - utter_goodbye

## bot challenge
* bot_challenge
    - utter_iamabot

ffcabb810d5df18f86f3fafee2a6206f

[JEM-Mosig]

@btotharye @wochinge Right now, at commit 6980a71 on master, when I do rasa init and then rasa interactive, I get

Traceback (most recent call last):
  File "/Users/johannes/miniconda3/envs/rasa/bin/rasa", line 11, in <module>
    load_entry_point('rasa', 'console_scripts', 'rasa')()
  File "/Users/johannes/rasa/rasa/rasa/__main__.py", line 76, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/johannes/rasa/rasa/rasa/cli/interactive.py", line 60, in interactive
    zipped_model = train.train(args)
  File "/Users/johannes/rasa/rasa/rasa/cli/train.py", line 75, in train
    persist_nlu_training_data=args.persist_nlu_data,
AttributeError: 'Namespace' object has no attribute 'persist_nlu_data'

[btotharye]

@JEM-Mosig I think I know what I might have missed, gimme a few and I'll test fixing this

[btotharye]

@wochinge I don't see anything obvious yet with the change I made I'll look into this further but I didn't even think about testing the interactive piece with this before

[btotharye]

so I feel pretty confident this is not related to my commit if someone wants to double check my work I only changed the info the /status route gives back so not sure how that broke this

[wochinge]

I'm . tackling it 👍

[erohmensing]

@wochinge to be clear you tackled the comment #4612 (comment)

but not the overall issue of this thread right? cause i just had a customer run into it as well. AFAIK they have checkpoints but not OR statements, but I can verify that

[Ghostvv]

it'll happen as well if they have cycles due to checkpoints

[wochinge]

Yes, I only tackled the comment #4612 (comment) .

[btotharye]

hey @wochinge I don't believe there is anything for me to do on this issue right? If so I can probably unassign myself from it.

Thanks

[akelad]

@btotharye i just figured out what tobias meant - he fixed only the part in the comment here: #4612 (comment)
The original issue is still remaining, which you can work on (even if it's not necessarily to do with your original change)

[btotharye]

@btotharye i just figured out what tobias meant - he fixed only the part in the comment here: #4612 (comment)
The original issue is still remaining, which you can work on (even if it's not necessarily to do with your original change)

ok makes sense now so the original issue that this was filed for is what actually needs to be looked at by me then, gotcha.

[btotharye]

so is the original issue not related to the amount of OR and/or checkpoints used in the stories which slows it down and causes the issue? thoughts @wochinge ? I was going to add some of these into my test bot and see if I can reproduce this issue

I haven't been able so far to reproduce the action server issue as I think that might have just been related to it still running or something before being stopped.

I do agree there is an issue with the fact it runs rasa train again with the rasa interactive even though the model is already there and trained.

[btotharye]

Looks like it happens to me with a basic setup without any OR or checkpoints it just processes super fast so I never noticed it before:

Nothing changed. You can use the old model stored at '/models/20191031-104526.tar.gz'.
2019-10-31 10:45:42 INFO     absl  - Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]
Bot loaded. Visualisation at http://localhost:5006/visualization.html.
Type a message and press enter (press 'Ctr-c' to exit).
Processed Story Blocks: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 1230.72it/s, # trackers=1]
? Your input ->

[btotharye]

Looking into why this happens I've been able to reproduce this with the rasa-demo code as well and you can see the issue there since it has more data and takes a few with the story blocks before you can talk to it.

Should have updates shortly on it. Thanks

[wochinge]

@btotharye I unassigned it since you haven't started yet. Feel free to pick it up again :-)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[erohmensing]

@wochinge do checkpoints and ORs exist still in 2.0 stories? wondering if this may have been incidentally fixed

[Ghostvv]

if they are not, they should be

[akelad]

they still exist sadly, we don't have a better solution yet

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[wochinge]

related other issue: #7581