Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix data validator ignore_warnings logic #11844

Merged
merged 17 commits into from
Feb 22, 2023
Merged

Conversation

timidri
Copy link
Contributor

@timidri timidri commented Dec 2, 2022

Proposed changes:

  • Fixes the logic for the ignore_warnings parameter inside validator.py
  • Explicitly specifies ignore_warnings=False in test_validator.py
  • Fixes failing tests
  • Adds a test test_verify_utterances_does_not_error_when_no_utterance_template_provided to make sure that utterances specified without a template are not considered a validation error

Status (please check what you already did):

  • added some tests for the functionality
  • updated the documentation
  • updated the changelog (please check changelog for instructions)
  • reformat files using black (please check Readme for instructions)

This PR resolves 11843

Fix failing tests logic
Add test_verify_utterances_does_not_error_when_no_utterance_template_provided
@timidri timidri requested a review from a team as a code owner December 2, 2022 15:19
@timidri timidri requested review from znat and removed request for a team December 2, 2022 15:19
@CLAassistant
Copy link

CLAassistant commented Dec 2, 2022

CLA assistant check
All committers have signed the CLA.

Copy link
Member

@tmbo tmbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for that bug fix and adding a test for it 🔥

@tmbo tmbo enabled auto-merge (squash) December 7, 2022 09:05
auto-merge was automatically disabled December 8, 2022 08:18

Head branch was pushed to by a user without write access

@timidri
Copy link
Contributor Author

timidri commented Dec 8, 2022

Thanks a lot for that bug fix and adding a test for it 🔥

My pleasure. I fixed one linter issue but I see some other validation errors in CI which I am not aware of...

@timidri
Copy link
Contributor Author

timidri commented Dec 26, 2022

@tmbo What needs to happen for this PR to get merged?

@m-vdb
Copy link
Collaborator

m-vdb commented Jan 11, 2023

I've just approved the CI runs. Once they go green the PR should be automatically merged. Thanks for your contribution 🙏🏻

@m-vdb
Copy link
Collaborator

m-vdb commented Jan 12, 2023

@timidri it looks like the integration tests failed (build), can you look into it?

@github-actions
Copy link
Contributor

github-actions bot commented Jan 12, 2023

Could not update branch. Most likely this is due to a merge conflict. Please update the branch manually and fix any issues. Error: HttpError: user doesn't have permission to update head repository

@timidri timidri requested a review from a team as a code owner February 17, 2023 23:45
auto-merge was automatically disabled February 18, 2023 00:47

Head branch was pushed to by a user without write access

@timidri
Copy link
Contributor Author

timidri commented Feb 18, 2023

@m-vdb I think I've fixed the issue in the integration test.

@m-vdb
Copy link
Collaborator

m-vdb commented Feb 20, 2023

it seems that the test-nlu-predictors now consistently fail on Ubuntu, can you look into it?

Screenshot 2023-02-20 at 15 44 59

@timidri
Copy link
Contributor Author

timidri commented Feb 20, 2023

@m-vdb

it seems that the test-nlu-predictors now consistently fail on Ubuntu, can you look into it?

This is strange, if I run the tests locally they pass:

$ cat /etc/issue.net
Ubuntu 20.04.5 LTS

$ make test-nlu-predictors
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f  {} +
rm -rf build/
rm -rf .mypy_cache/
rm -rf dist/
rm -rf docs/build
rm -rf docs/.docusaurus
# TF_CPP_MIN_LOG_LEVEL=2 sets C code log level for tensorflow to error suppressing lower log events
OMP_NUM_THREADS=1 TF_CPP_MIN_LOG_LEVEL=2 poetry run pytest tests -n 1 --dist loadscope -m "category_nlu_predictors and (not flaky)" --cov rasa --ignore tests/integration_tests/
================================================= test session starts ==================================================
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/dimitri/Projects/idtlsm/rasa, configfile: pytest.ini
plugins: anyio-3.6.2, timeout-2.1.0, cov-4.0.0, asyncio-0.20.3, xdist-3.1.0, typeguard-2.13.3, sanic-1.9.1
timeout: 60.0s
timeout method: signal
timeout func_only: True
asyncio: mode=strict
gw0 [174]
................................................................................................................ [ 64%]
..............................................................                                                   [100%]
=================================================== warnings summary ===================================================
rasa/engine/caching.py:152
rasa/engine/caching.py:152
  /home/dimitri/Projects/idtlsm/rasa/rasa/engine/caching.py:152: MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
    Base: DeclarativeMeta = declarative_base()

.venv/lib/python3.10/site-packages/sanic_cors/extension.py:39
.venv/lib/python3.10/site-packages/sanic_cors/extension.py:39
  /home/dimitri/Projects/idtlsm/rasa/.venv/lib/python3.10/site-packages/sanic_cors/extension.py:39: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SANIC_VERSION = LooseVersion(sanic_version)

.venv/lib/python3.10/site-packages/sanic_cors/extension.py:40
.venv/lib/python3.10/site-packages/sanic_cors/extension.py:40
  /home/dimitri/Projects/idtlsm/rasa/.venv/lib/python3.10/site-packages/sanic_cors/extension.py:40: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SANIC_21_9_0 = LooseVersion("21.9.0")

.venv/lib/python3.10/site-packages/future/standard_library/__init__.py:65
.venv/lib/python3.10/site-packages/future/standard_library/__init__.py:65
  /home/dimitri/Projects/idtlsm/rasa/.venv/lib/python3.10/site-packages/future/standard_library/__init__.py:65: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

.venv/lib/python3.10/site-packages/xdist/plugin.py:227
.venv/lib/python3.10/site-packages/xdist/plugin.py:227
  /home/dimitri/Projects/idtlsm/rasa/.venv/lib/python3.10/site-packages/xdist/plugin.py:227: DeprecationWarning: The --rsyncdir command line argument and rsyncdirs config variable are deprecated.
  The rsync feature will be removed in pytest-xdist 4.0.
    config.issue_config_time_warning(warning, 2)

.venv/lib/python3.10/site-packages/pytest_sanic/utils.py:168
  /home/dimitri/Projects/idtlsm/rasa/.venv/lib/python3.10/site-packages/pytest_sanic/utils.py:168: PytestCollectionWarning: cannot collect test class 'TestClient' because it has a __init__ constructor (from: tests/core/test_agent.py)
    class TestClient:

tests/core/test_evaluation.py:352
  /home/dimitri/Projects/idtlsm/rasa/tests/core/test_evaluation.py:352: PytestUnknownMarkWarning: Unknown pytest.mark.skip_on_ci - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.skip_on_ci

tests/core/test_processor.py:1484
  /home/dimitri/Projects/idtlsm/rasa/tests/core/test_processor.py:1484: PytestUnknownMarkWarning: Unknown pytest.mark.skip_on_ci - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.skip_on_ci

tests/core/test_test.py:110
  /home/dimitri/Projects/idtlsm/rasa/tests/core/test_test.py:110: PytestUnknownMarkWarning: Unknown pytest.mark.skip_on_ci - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.skip_on_ci

<frozen importlib._bootstrap>:283: 3 warnings
tests/nlu/classifiers/test_diet_classifier.py: 46 warnings
tests/nlu/classifiers/test_sklearn_classifier.py: 3 warnings
tests/nlu/extractors/test_crf_entity_extractor.py: 7 warnings
tests/nlu/extractors/test_extractor.py: 5 warnings
tests/nlu/selectors/test_selectors.py: 43 warnings
  <frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead

rasa/shared/core/slot_mappings.py:221
  /home/dimitri/Projects/idtlsm/rasa/rasa/shared/core/slot_mappings.py:221: UserWarning: Slot auto-fill has been removed in 3.0 and replaced with a new explicit mechanism to set slots. Please refer to https://rasa.com/docs/rasa/domain#slots to learn more.
    rasa.shared.utils.io.raise_warning(

tests/core/test_tracker_stores.py:733
  /home/dimitri/Projects/idtlsm/rasa/tests/core/test_tracker_stores.py:733: SADeprecationWarning: Calling URL() directly is deprecated and will be disabled in a future release.  The public constructor for URL is now the URL.create() method.
    (URL(PGDialect.name), True),

tests/core/test_tracker_stores.py:734
  /home/dimitri/Projects/idtlsm/rasa/tests/core/test_tracker_stores.py:734: SADeprecationWarning: Calling URL() directly is deprecated and will be disabled in a future release.  The public constructor for URL is now the URL.create() method.
    (URL(SQLiteDialect.name), False),

tests/engine/test_caching.py:30
  /home/dimitri/Projects/idtlsm/rasa/tests/engine/test_caching.py:30: PytestCollectionWarning: cannot collect test class 'TestCacheableOutput' because it has a __init__ constructor (from: tests/engine/test_caching.py)
    @dataclasses.dataclass

rasa/core/tracker_store.py:998
rasa/core/tracker_store.py:998
  /home/dimitri/Projects/idtlsm/rasa/rasa/core/tracker_store.py:998: SADeprecationWarning: Calling URL() directly is deprecated and will be disabled in a future release.  The public constructor for URL is now the URL.create() method.
    return sa.engine.url.URL(

tests/utils/tensorflow/test_data_generator.py:70
  /home/dimitri/Projects/idtlsm/rasa/tests/utils/tensorflow/test_data_generator.py:70: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.array(

tests/utils/tensorflow/test_data_generator.py:86
  /home/dimitri/Projects/idtlsm/rasa/tests/utils/tensorflow/test_data_generator.py:86: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.array(

tests/utils/tensorflow/test_data_generator.py:93
  /home/dimitri/Projects/idtlsm/rasa/tests/utils/tensorflow/test_data_generator.py:93: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.array(

tests/utils/tensorflow/test_data_generator.py:84
  /home/dimitri/Projects/idtlsm/rasa/tests/utils/tensorflow/test_data_generator.py:84: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.array(

tests/utils/tensorflow/test_data_generator.py:149
  /home/dimitri/Projects/idtlsm/rasa/tests/utils/tensorflow/test_data_generator.py:149: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.array(

tests/nlu/classifiers/test_diet_classifier.py::test_compute_default_label_features
  /home/dimitri/Projects/idtlsm/rasa/rasa/utils/io.py:69: DeprecationWarning: There is no current event loop
    "Enabling coroutine debugging. Loop id {}.".format(id(asyncio.get_event_loop()))

tests/nlu/classifiers/test_diet_classifier.py: 30 warnings
tests/nlu/selectors/test_selectors.py: 18 warnings
  /home/dimitri/Projects/idtlsm/rasa/rasa/utils/train_utils.py:528: UserWarning: constrain_similarities is set to `False`. It is recommended to set it to `True` when using cross-entropy loss.
    rasa.shared.utils.io.raise_warning(

tests/nlu/classifiers/test_diet_classifier.py: 15 warnings
tests/nlu/selectors/test_selectors.py: 14 warnings
  /home/dimitri/Projects/idtlsm/rasa/rasa/shared/utils/io.py:98: UserWarning: 'evaluate_every_number_of_epochs=20' is greater than 'epochs=1'. No evaluation will occur.
    warnings.warn(message, category=category, **kwargs)

tests/nlu/classifiers/test_diet_classifier.py: 27 warnings
tests/nlu/selectors/test_selectors.py: 10 warnings
  /home/dimitri/Projects/idtlsm/rasa/rasa/utils/tensorflow/model_data.py:774: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.concatenate(np.array(f)),

tests/nlu/classifiers/test_diet_classifier.py::test_train_persist_load_with_different_settings
tests/nlu/classifiers/test_diet_classifier.py::test_margin_loss_is_not_normalized
tests/nlu/selectors/test_selectors.py::test_margin_loss_is_not_normalized[classifier_params0]
  /home/dimitri/Projects/idtlsm/rasa/rasa/shared/utils/io.py:98: UserWarning: Overriding defaults by setting model_confidence to auto as loss_type is set to margin in the configuration. This means that model's confidences will be computed as cosine similarities. Users are encouraged to shift to cross entropy loss by setting `loss_type=cross_entropy`.
    warnings.warn(message, category=category, **kwargs)

tests/nlu/classifiers/test_diet_classifier.py::test_train_persist_load_with_only_entity_recognition
tests/nlu/classifiers/test_diet_classifier.py::test_train_persist_load_with_composite_entities[classifier_params0]
tests/nlu/classifiers/test_diet_classifier.py::test_train_persist_load_with_composite_entities[classifier_params1]
  /home/dimitri/Projects/idtlsm/rasa/rasa/utils/tensorflow/model_data_utils.py:400: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    np.array([v[0] for v in values]), number_of_dimensions=3

tests/nlu/classifiers/test_keyword_classifier.py: 16 warnings
tests/nlu/extractors/test_crf_entity_extractor.py: 2 warnings
  /home/dimitri/Projects/idtlsm/rasa/rasa/shared/nlu/training_data/formats/rasa.py:58: FutureWarning: NLU data in Rasa JSON format is deprecated and will be removed in Rasa Open Source 4.0.0. Please convert your JSON NLU data to the Rasa YAML format.
    rasa.shared.utils.io.raise_deprecation_warning(

tests/nlu/classifiers/test_keyword_classifier.py::test_valid_data
  /home/dimitri/Projects/idtlsm/rasa/.venv/lib/python3.10/site-packages/_pytest/python.py:195: PytestRemovedIn8Warning: Passing None has been deprecated.
  See https://docs.pytest.org/en/latest/how-to/capture-warnings.html#additional-use-cases-of-warnings-in-tests for alternatives in common use cases.
    result = testfunction(**testargs)

tests/nlu/selectors/test_selectors.py::test_train_selector[config_params1]
tests/nlu/selectors/test_selectors.py::test_transformer_size_gets_corrected
  /home/dimitri/Projects/idtlsm/rasa/rasa/nlu/selectors/response_selector.py:377: UserWarning: You have enabled a transformer inside ResponseSelector by setting a positive value for `number_of_transformer_layers`, but you left the hidden layer sizes at their default value: `hidden_layers_sizes={'text': [256, 128], 'label': [256, 128]}`. We recommend to disable the hidden layers when using a transformer, by specifying `hidden_layers_sizes={'text': [], 'label': []}`.
    rasa.shared.utils.io.raise_warning(

tests/nlu/selectors/test_selectors.py::test_transformer_size_gets_corrected
  /home/dimitri/Projects/idtlsm/rasa/rasa/nlu/selectors/response_selector.py:397: UserWarning: `transformer_size` is set to `None` for ResponseSelector, but a positive size is required when using `number_of_transformer_layers > 0`. ResponseSelector will proceed, using `transformer_size=256`. Alternatively, specify a different value in the component's config.
    rasa.shared.utils.io.raise_warning(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.10.8-final-0 -----------
Name                                                                     Stmts   Miss  Cover
--------------------------------------------------------------------------------------------
rasa/__init__.py                                                             5      0   100%
rasa/__main__.py                                                            57     36    37%
rasa/api.py                                                                 26     20    23%
rasa/cli/__init__.py                                                         2      0   100%
rasa/cli/arguments/__init__.py                                               0      0   100%
rasa/cli/arguments/data.py                                                  21     13    38%
rasa/cli/arguments/default_arguments.py                                     29     16    45%
rasa/cli/arguments/evaluate.py                                              16     10    38%
rasa/cli/arguments/export.py                                                 9      5    44%
rasa/cli/arguments/interactive.py                                           33     24    27%
rasa/cli/arguments/run.py                                                   41     32    22%
rasa/cli/arguments/shell.py                                                  8      3    62%
rasa/cli/arguments/test.py                                                  55     41    25%
rasa/cli/arguments/train.py                                                 65     45    31%
rasa/cli/arguments/visualize.py                                              8      5    38%
rasa/cli/arguments/x.py                                                      9      5    44%
rasa/cli/data.py                                                            98     65    34%
rasa/cli/evaluate.py                                                        55     37    33%
rasa/cli/export.py                                                          78     53    32%
rasa/cli/interactive.py                                                     56     34    39%
rasa/cli/run.py                                                             59     46    22%
rasa/cli/scaffold.py                                                        87     68    22%
rasa/cli/shell.py                                                           57     41    28%
rasa/cli/telemetry.py                                                       35     23    34%
rasa/cli/test.py                                                            95     70    26%
rasa/cli/train.py                                                           78     58    26%
rasa/cli/utils.py                                                           70     49    30%
rasa/cli/visualize.py                                                       16      8    50%
rasa/cli/x.py                                                               77     52    32%
rasa/constants.py                                                           26      0   100%
rasa/core/__init__.py                                                        4      0   100%
rasa/core/actions/__init__.py                                                0      0   100%
rasa/core/actions/action.py                                                444    325    27%
rasa/core/actions/constants.py                                               2      0   100%
rasa/core/actions/forms.py                                                 194    143    26%
rasa/core/actions/loops.py                                                  36     21    42%
rasa/core/actions/two_stage_fallback.py                                     69     40    42%
rasa/core/agent.py                                                         202    132    35%
rasa/core/brokers/__init__.py                                                0      0   100%
rasa/core/brokers/broker.py                                                 55     34    38%
rasa/core/brokers/file.py                                                   30     16    47%
rasa/core/brokers/kafka.py                                                 122     92    25%
rasa/core/brokers/pika.py                                                  126     88    30%
rasa/core/brokers/sql.py                                                    40     15    62%
rasa/core/channels/__init__.py                                              19      0   100%
rasa/core/channels/botframework.py                                         174    127    27%
rasa/core/channels/callback.py                                              46     23    50%
rasa/core/channels/channel.py                                              132     76    42%
rasa/core/channels/console.py                                              116     81    30%
rasa/core/channels/facebook.py                                             190    127    33%
rasa/core/channels/hangouts.py                                             155    106    32%
rasa/core/channels/mattermost.py                                            97     64    34%
rasa/core/channels/rasa_chat.py                                             71     43    39%
rasa/core/channels/rest.py                                                  79     48    39%
rasa/core/channels/rocketchat.py                                            80     51    36%
rasa/core/channels/slack.py                                                254    187    26%
rasa/core/channels/socketio.py                                             120     85    29%
rasa/core/channels/telegram.py                                             139     98    29%
rasa/core/channels/twilio.py                                                88     60    32%
rasa/core/channels/twilio_voice.py                                         115     79    31%
rasa/core/channels/webexteams.py                                            67     41    39%
rasa/core/constants.py                                                      36      0   100%
rasa/core/evaluation/__init__.py                                             0      0   100%
rasa/core/evaluation/marker.py                                             124     44    65%
rasa/core/evaluation/marker_base.py                                        324    183    44%
rasa/core/evaluation/marker_stats.py                                        91     61    33%
rasa/core/evaluation/marker_tracker_loader.py                               43     27    37%
rasa/core/exceptions.py                                                      9      2    78%
rasa/core/exporter.py                                                      100     74    26%
rasa/core/featurizers/__init__.py                                            0      0   100%
rasa/core/featurizers/precomputation.py                                    120     79    34%
rasa/core/featurizers/single_state_featurizer.py                           115     82    29%
rasa/core/featurizers/tracker_featurizers.py                               296    213    28%
rasa/core/http_interpreter.py                                               36     24    33%
rasa/core/jobs.py                                                           25     16    36%
rasa/core/lock.py                                                           65     35    46%
rasa/core/lock_store.py                                                    140     81    42%
rasa/core/migrate.py                                                       181    154    15%
rasa/core/nlg/__init__.py                                                    3      0   100%
rasa/core/nlg/callback.py                                                   34     19    44%
rasa/core/nlg/generator.py                                                  34     21    38%
rasa/core/nlg/interpolator.py                                               29     23    21%
rasa/core/nlg/response.py                                                   85     65    24%
rasa/core/policies/__init__.py                                               0      0   100%
rasa/core/policies/ensemble.py                                              84     54    36%
rasa/core/policies/memoization.py                                          184    130    29%
rasa/core/policies/policy.py                                               176     94    47%
rasa/core/policies/rule_policy.py                                          417    311    25%
rasa/core/policies/ted_policy.py                                           542    422    22%
rasa/core/policies/unexpected_intent_policy.py                             234    144    38%
rasa/core/processor.py                                                     332    246    26%
rasa/core/run.py                                                            94     64    32%
rasa/core/test.py                                                          438    334    24%
rasa/core/tracker_store.py                                                 513    315    39%
rasa/core/train.py                                                          34     20    41%
rasa/core/training/__init__.py                                              21     17    19%
rasa/core/training/converters/__init__.py                                    0      0   100%
rasa/core/training/converters/responses_prefix_converter.py                 41     41     0%
rasa/core/training/interactive.py                                          697    570    18%
rasa/core/training/story_conflict.py                                       117     80    32%
rasa/core/training/training.py                                              38     26    32%
rasa/core/utils.py                                                         132     98    26%
rasa/core/visualize.py                                                      26     26     0%
rasa/engine/__init__.py                                                      0      0   100%
rasa/engine/caching.py                                                     180    110    39%
rasa/engine/constants.py                                                     8      0   100%
rasa/engine/exceptions.py                                                    4      0   100%
rasa/engine/graph.py                                                       193    113    41%
rasa/engine/loader.py                                                        9      3    67%
rasa/engine/recipes/__init__.py                                              0      0   100%
rasa/engine/recipes/default_components.py                                   31      0   100%
rasa/engine/recipes/default_recipe.py                                      391    292    25%
rasa/engine/recipes/graph_recipe.py                                         29     14    52%
rasa/engine/recipes/recipe.py                                               26     11    58%
rasa/engine/runner/__init__.py                                               0      0   100%
rasa/engine/runner/dask.py                                                  43     24    44%
rasa/engine/runner/interface.py                                             11      0   100%
rasa/engine/storage/__init__.py                                              0      0   100%
rasa/engine/storage/local_model_storage.py                                 116     52    55%
rasa/engine/storage/resource.py                                             37     19    49%
rasa/engine/storage/storage.py                                              56      7    88%
rasa/engine/training/__init__.py                                             0      0   100%
rasa/engine/training/components.py                                          49     21    57%
rasa/engine/training/fingerprinting.py                                      19      5    74%
rasa/engine/training/graph_trainer.py                                       66     46    30%
rasa/engine/training/hooks.py                                               52     29    44%
rasa/engine/validation.py                                                  194    147    24%
rasa/exceptions.py                                                          22      3    86%
rasa/graph_components/__init__.py                                            0      0   100%
rasa/graph_components/converters/__init__.py                                 0      0   100%
rasa/graph_components/converters/nlu_message_converter.py                   14      2    86%
rasa/graph_components/providers/__init__.py                                  0      0   100%
rasa/graph_components/providers/domain_for_core_training_provider.py        24     10    58%
rasa/graph_components/providers/domain_provider.py                          32     15    53%
rasa/graph_components/providers/forms_provider.py                           21      4    81%
rasa/graph_components/providers/nlu_training_data_provider.py               28     13    54%
rasa/graph_components/providers/responses_provider.py                       21      4    81%
rasa/graph_components/providers/rule_only_provider.py                       23      8    65%
rasa/graph_components/providers/story_graph_provider.py                     18      4    78%
rasa/graph_components/providers/training_tracker_provider.py                20      5    75%
rasa/graph_components/validators/__init__.py                                 0      0   100%
rasa/graph_components/validators/default_recipe_validator.py               139     90    35%
rasa/graph_components/validators/finetuning_validator.py                    92     56    39%
rasa/jupyter.py                                                             31     31     0%
rasa/model.py                                                               47     33    30%
rasa/model_testing.py                                                      147    114    22%
rasa/model_training.py                                                     130     93    28%
rasa/nlu/__init__.py                                                         4      0   100%
rasa/nlu/classifiers/__init__.py                                             1      0   100%
rasa/nlu/classifiers/classifier.py                                           2      0   100%
rasa/nlu/classifiers/diet_classifier.py                                    601     85    86%
rasa/nlu/classifiers/fallback_classifier.py                                 72      2    97%
rasa/nlu/classifiers/keyword_intent_classifier.py                           85      3    96%
rasa/nlu/classifiers/logistic_regression_classifier.py                      94     11    88%
rasa/nlu/classifiers/mitie_intent_classifier.py                             72     19    74%
rasa/nlu/classifiers/regex_message_handler.py                               19      0   100%
rasa/nlu/classifiers/sklearn_intent_classifier.py                          134      8    94%
rasa/nlu/constants.py                                                       31      0   100%
rasa/nlu/convert.py                                                         17     10    41%
rasa/nlu/emulators/__init__.py                                               0      0   100%
rasa/nlu/emulators/dialogflow.py                                            11     11     0%
rasa/nlu/emulators/emulator.py                                              16      9    44%
rasa/nlu/emulators/luis.py                                                  23     16    30%
rasa/nlu/emulators/no_emulator.py                                            5      1    80%
rasa/nlu/emulators/wit.py                                                   14     14     0%
rasa/nlu/extractors/__init__.py                                              0      0   100%
rasa/nlu/extractors/crf_entity_extractor.py                                272     26    90%
rasa/nlu/extractors/duckling_entity_extractor.py                            81     11    86%
rasa/nlu/extractors/entity_synonyms.py                                      76     13    83%
rasa/nlu/extractors/extractor.py                                           148      7    95%
rasa/nlu/extractors/mitie_entity_extractor.py                              117     14    88%
rasa/nlu/extractors/regex_entity_extractor.py                               73      0   100%
rasa/nlu/extractors/spacy_entity_extractor.py                               39      3    92%
rasa/nlu/featurizers/__init__.py                                             0      0   100%
rasa/nlu/featurizers/dense_featurizer/__init__.py                            0      0   100%
rasa/nlu/featurizers/dense_featurizer/convert_featurizer.py                160     97    39%
rasa/nlu/featurizers/dense_featurizer/dense_featurizer.py                   22      4    82%
rasa/nlu/featurizers/dense_featurizer/lm_featurizer.py                     201    147    27%
rasa/nlu/featurizers/dense_featurizer/mitie_featurizer.py                   65     26    60%
rasa/nlu/featurizers/dense_featurizer/spacy_featurizer.py                   62      4    94%
rasa/nlu/featurizers/featurizer.py                                          34      5    85%
rasa/nlu/featurizers/sparse_featurizer/__init__.py                           0      0   100%
rasa/nlu/featurizers/sparse_featurizer/count_vectors_featurizer.py         302     88    71%
rasa/nlu/featurizers/sparse_featurizer/lexical_syntactic_featurizer.py     170     20    88%
rasa/nlu/featurizers/sparse_featurizer/regex_featurizer.py                 108     18    83%
rasa/nlu/featurizers/sparse_featurizer/sparse_featurizer.py                  5      0   100%
rasa/nlu/model.py                                                           10      3    70%
rasa/nlu/persistor.py                                                      122     84    31%
rasa/nlu/run.py                                                             18     18     0%
rasa/nlu/selectors/__init__.py                                               0      0   100%
rasa/nlu/selectors/response_selector.py                                    262     85    68%
rasa/nlu/test.py                                                           461    344    25%
rasa/nlu/tokenizers/__init__.py                                              0      0   100%
rasa/nlu/tokenizers/jieba_tokenizer.py                                      73     39    47%
rasa/nlu/tokenizers/mitie_tokenizer.py                                      32      2    94%
rasa/nlu/tokenizers/spacy_tokenizer.py                                      33      4    88%
rasa/nlu/tokenizers/tokenizer.py                                           102     20    80%
rasa/nlu/tokenizers/whitespace_tokenizer.py                                 42      3    93%
rasa/nlu/utils/__init__.py                                                  15      8    47%
rasa/nlu/utils/bilou_utils.py                                              147     61    59%
rasa/nlu/utils/hugging_face/__init__.py                                      0      0   100%
rasa/nlu/utils/hugging_face/registry.py                                     11     11     0%
rasa/nlu/utils/hugging_face/transformers_pre_post_processors.py             67     67     0%
rasa/nlu/utils/mitie_utils.py                                               39      4    90%
rasa/nlu/utils/pattern_utils.py                                             53     15    72%
rasa/nlu/utils/spacy_utils.py                                              116      9    92%
rasa/plugin.py                                                              27      9    67%
rasa/server.py                                                             599    519    13%
rasa/shared/__init__.py                                                      0      0   100%
rasa/shared/constants.py                                                    82      0   100%
rasa/shared/core/__init__.py                                                 0      0   100%
rasa/shared/core/constants.py                                               68      2    97%
rasa/shared/core/conversation.py                                            15      4    73%
rasa/shared/core/domain.py                                                 727    348    52%
rasa/shared/core/events.py                                                 682    324    52%
rasa/shared/core/generator.py                                              322    264    18%
rasa/shared/core/slot_mappings.py                                           82     49    40%
rasa/shared/core/slots.py                                                  184    106    42%
rasa/shared/core/trackers.py                                               307    174    43%
rasa/shared/core/training_data/__init__.py                                   0      0   100%
rasa/shared/core/training_data/loading.py                                   35     20    43%
rasa/shared/core/training_data/story_reader/__init__.py                      0      0   100%
rasa/shared/core/training_data/story_reader/story_reader.py                 54     32    41%
rasa/shared/core/training_data/story_reader/story_step_builder.py           83     63    24%
rasa/shared/core/training_data/story_reader/yaml_story_reader.py           373    222    40%
rasa/shared/core/training_data/story_writer/__init__.py                      0      0   100%
rasa/shared/core/training_data/story_writer/story_writer.py                 13      4    69%
rasa/shared/core/training_data/story_writer/yaml_story_writer.py           157    123    22%
rasa/shared/core/training_data/structures.py                               351    260    26%
rasa/shared/core/training_data/visualization.py                            245    203    17%
rasa/shared/data.py                                                         75     23    69%
rasa/shared/exceptions.py                                                   38     17    55%
rasa/shared/importers/__init__.py                                            0      0   100%
rasa/shared/importers/importer.py                                          213    115    46%
rasa/shared/importers/multi_project.py                                     110     77    30%
rasa/shared/importers/rasa.py                                               44     16    64%
rasa/shared/importers/utils.py                                              12      3    75%
rasa/shared/nlu/__init__.py                                                  0      0   100%
rasa/shared/nlu/constants.py                                                38      0   100%
rasa/shared/nlu/interpreter.py                                               4      4     0%
rasa/shared/nlu/training_data/__init__.py                                    0      0   100%
rasa/shared/nlu/training_data/entities_parser.py                            74     14    81%
rasa/shared/nlu/training_data/features.py                                  120     71    41%
rasa/shared/nlu/training_data/formats/__init__.py                            5      0   100%
rasa/shared/nlu/training_data/formats/dialogflow.py                         97     61    37%
rasa/shared/nlu/training_data/formats/luis.py                               38     26    32%
rasa/shared/nlu/training_data/formats/rasa.py                               43      9    79%
rasa/shared/nlu/training_data/formats/rasa_yaml.py                         253    120    53%
rasa/shared/nlu/training_data/formats/readerwriter.py                       90     52    42%
rasa/shared/nlu/training_data/formats/wit.py                                33     23    30%
rasa/shared/nlu/training_data/loading.py                                    69      8    88%
rasa/shared/nlu/training_data/lookup_tables_parser.py                        9      7    22%
rasa/shared/nlu/training_data/message.py                                   151     31    79%
rasa/shared/nlu/training_data/schemas/__init__.py                            0      0   100%
rasa/shared/nlu/training_data/schemas/data_schema.py                        13      0   100%
rasa/shared/nlu/training_data/synonyms_parser.py                            11      0   100%
rasa/shared/nlu/training_data/training_data.py                             270    140    48%
rasa/shared/nlu/training_data/util.py                                       77     30    61%
rasa/shared/utils/__init__.py                                                0      0   100%
rasa/shared/utils/cli.py                                                    22     13    41%
rasa/shared/utils/common.py                                                 80     48    40%
rasa/shared/utils/io.py                                                    217     73    66%
rasa/shared/utils/pykwalify_extensions.py                                    9      2    78%
rasa/shared/utils/schemas/__init__.py                                        0      0   100%
rasa/shared/utils/schemas/events.py                                         29      0   100%
rasa/shared/utils/validation.py                                            111     53    52%
rasa/telemetry.py                                                          311    193    38%
rasa/utils/__init__.py                                                       0      0   100%
rasa/utils/common.py                                                       222    156    30%
rasa/utils/converter.py                                                     13     13     0%
rasa/utils/endpoints.py                                                    117     68    42%
rasa/utils/io.py                                                            96     44    54%
rasa/utils/plotting.py                                                     131    106    19%
rasa/utils/tensorflow/__init__.py                                            0      0   100%
rasa/utils/tensorflow/callback.py                                           44      5    89%
rasa/utils/tensorflow/constants.py                                          82      0   100%
rasa/utils/tensorflow/crf.py                                                79     57    28%
rasa/utils/tensorflow/data_generator.py                                    110     30    73%
rasa/utils/tensorflow/environment.py                                        61     46    25%
rasa/utils/tensorflow/exceptions.py                                          2      0   100%
rasa/utils/tensorflow/layers.py                                            355    171    52%
rasa/utils/tensorflow/layers_utils.py                                       15      9    40%
rasa/utils/tensorflow/model_data.py                                        302     56    81%
rasa/utils/tensorflow/model_data_utils.py                                  154     10    94%
rasa/utils/tensorflow/models.py                                            281     26    91%
rasa/utils/tensorflow/rasa_layers.py                                       235     51    78%
rasa/utils/tensorflow/transformer.py                                       211    138    35%
rasa/utils/tensorflow/types.py                                               3      0   100%
rasa/utils/train_utils.py                                                  162     46    72%
rasa/validator.py                                                          176    141    20%
rasa/version.py                                                              1      0   100%
--------------------------------------------------------------------------------------------
TOTAL                                                                    27533  15211    45%

==================================== 174 passed, 275 warnings in 1218.15s (0:20:18) ====================================

@timidri
Copy link
Contributor Author

timidri commented Feb 20, 2023

@m-vdb One change I observed is that I am using Python 3.10 and the failing test uses 3.8. Not sure if that is intended, but different tests in the set use different Python versions (integration tests are using 3.10, so I installed that locally).

@m-vdb
Copy link
Collaborator

m-vdb commented Feb 21, 2023

but it seems these tests are failing on Ubuntu for all the python versions we support: 3.7, 3.8, 3.9 and 3.10 😬

@timidri
Copy link
Contributor Author

timidri commented Feb 21, 2023

but it seems these tests are failing on Ubuntu for all the python versions we support: 3.7, 3.8, 3.9 and 3.10 😬

Right, then I don't know what is happening. The tests pass on my local Ubuntu system. Furthermore, this seems to be unrelated to my PR since I only changed validation logic.

@timidri
Copy link
Contributor Author

timidri commented Feb 21, 2023

@m-vdb I've re-tested on Ubuntu 22.04.1 LTS / Python 3.10 and the tests still succeed:

===================== 174 passed, 275 warnings in 739.26s (0:12:19) ======================

@timidri
Copy link
Contributor Author

timidri commented Feb 21, 2023

@m-vdb I just found similarly failing tests in PR #12074, for example: https://github.com/RasaHQ/rasa/actions/runs/4232026432/jobs/7351360693

@m-vdb
Copy link
Collaborator

m-vdb commented Feb 22, 2023

@timidri I'm going to go ahead and merge this PR, thanks for putting up with our flaky CI checks. We'll make sure to address these pains in the near future

@m-vdb m-vdb merged commit 56d2a0d into RasaHQ:main Feb 22, 2023
@timidri
Copy link
Contributor Author

timidri commented Feb 22, 2023

@timidri I'm going to go ahead and merge this PR, thanks for putting up with our flaky CI checks. We'll make sure to address these pains in the near future

Thank you!

@timidri timidri deleted the fix-validator branch February 24, 2023 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rasa data validate does not properly ignore warnings
6 participants