Add separate non-TensorFlow test run in Travis CI #2075

orionr · 2019-03-31T22:12:03Z

Continuation of work for #2027 and ultimately removing the TENSORBOARD_NO_TF=1 environment variable.

Plugins tests aren't run yet, but this confirms that build and limited tests run without TensorFlow installed.

wchargin

Mostly looks good; thanks!

How did you determine which tests can be run without TensorFlow? If I
create a new virtualenv with only the dependencies specified in the new
no-TF section of the Travis build script, then try to run all the tests,
I see that a number of additional tests pass:

//tensorboard:lazy_test                                                  PASSED in 0.2s
//tensorboard:lib_test                                                   PASSED in 0.8s
//tensorboard/compat/tensorflow_stub:gfile_test                          PASSED in 0.8s
//tensorboard/components/tf_backend/test:test_chromium                   PASSED in 6.8s
//tensorboard/components/tf_categorization_utils/test:test_chromium      PASSED in 6.7s
//tensorboard/components/tf_color_scale/test:test_chromium               PASSED in 5.9s
//tensorboard/components/tf_data_selector/test:test_chromium             PASSED in 7.0s
//tensorboard/components/tf_paginated_view/test:test_chromium            PASSED in 6.1s
//tensorboard/components/tf_storage/test:test_chromium                   PASSED in 5.4s
//tensorboard/components/vz_line_chart2/test:test_chromium               PASSED in 6.7s
//tensorboard/components/vz_sorting/test:test_chromium                   PASSED in 5.2s
//tensorboard/plugins/graph/tf_graph_common/test:test_chromium           PASSED in 7.5s
//tensorboard/plugins/histogram/tf_histogram_dashboard/test:test_chromium PASSED in 6.6s
//tensorboard/plugins/hparams/tf_hparams_google_analytics_tracker:test_chromium PASSED in 6.9s
//tensorboard/plugins/hparams/tf_hparams_parallel_coords_plot:test_chromium PASSED in 7.6s
//tensorboard/plugins/hparams/tf_hparams_query_pane:test_chromium        PASSED in 8.1s
//tensorboard/plugins/hparams/tf_hparams_scale_and_color_controls:test_chromium PASSED in 6.5s
//tensorboard/plugins/hparams/tf_hparams_utils:test_chromium             PASSED in 5.7s
//tensorboard/summary:summary_test                                       PASSED in 0.8s

Also: when running

bazel run //tensorboard/pip_package:build_pip_package -- --tf-version ''

in such a virtualenv as of this PR (d9c66b5a), I see:

+ python -c '
import tensorboard as tb
assert tb.__version__ == tb.version.VERSION
tb.summary.scalar_pb('\''test'\'', 42)
from tensorboard.plugins.projector import visualize_embeddings
from tensorboard.plugins.beholder import Beholder, BeholderHook
tb.notebook.start  # don'\''t invoke; just check existence
'
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/tmp/tensorboard-pip.bjgl3EqxDD/smoke-venv2/local/lib/python2.7/site-packages/tensorboard/plugins/scalar/summary.py", line 89, in pb
    import tensorflow.compat.v1 as tf
ImportError: No module named tensorflow.compat.v1

in the smoke test (which then fails). This happens regardless of whether
I include TENSORBOARD_NO_TF=1. My virtualenv contents are as follows:

pip freeze output

$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
absl-py==0.7.1
asn1crypto==0.24.0
aws-xray-sdk==0.95
backports.ssl-match-hostname==3.7.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
boto==2.49.0
boto3==1.9.86
botocore==1.12.126
certifi==2019.3.9
cffi==1.12.2
chardet==3.0.4
configparser==3.7.4
cookies==2.2.1
cryptography==2.6.1
docker==3.7.2
docker-pycreds==0.4.0
docutils==0.14
ecdsa==0.13
enum34==1.1.6
flake8==3.5.0
funcsigs==1.0.2
future==0.17.1
futures==3.1.1
grpcio==1.6.3
idna==2.8
ipaddress==1.0.22
Jinja2==2.10
jmespath==0.9.4
jsondiff==1.1.1
jsonpickle==1.1
MarkupSafe==1.1.1
mccabe==0.6.1
mock==2.0.0
moto==1.3.7
numpy==1.16.2
pbr==5.1.3
pkg-resources==0.0.0
protobuf==3.7.1
pyaml==18.11.0
pycodestyle==2.3.1
pycparser==2.19
pycryptodome==3.8.0
pyflakes==1.6.0
python-dateutil==2.8.0
python-jose==2.0.2
pytz==2018.9
PyYAML==5.1
requests==2.21.0
responses==0.10.6
s3transfer==0.1.13
six==1.12.0
urllib3==1.24.1
websocket-client==0.56.0
Werkzeug==0.15.1
wrapt==1.11.1
xmltodict==0.12.0
yamllint==1.5.0

Can you reproduce this, or does the script work for you?

tensorboard/pip_package/build_pip_package.sh

wchargin · 2019-04-01T22:19:53Z

tensorboard/main.py

-    # TENSORBOARD_NO_TF environment variable, we check for TensorFlow here so that if it's
-    # missing we generate a clear and immediate error rather than partial functionality.
-    # TODO(#2027): Remove environment check once we have placeholder UI
-    if os.getenv('TENSORBOARD_NO_TF') is None:


If this removal is included in this PR, then we can’t merge this PR
until #2032 is merged. If you don’t want to block on that PR, then it
looks to me like you should be able to add TENSORBOARD_NO_TF=1 to the
Travis CI script and/or Pip package smoke test as appropriate. Or, feel
free to keep this as is if you don’t mind waiting for #2032.

For now I'll keep it, but we can remove it if this is ready to land before the UI one. Thanks!

orionr · 2019-04-02T00:28:05Z

The script works for me, but I'm not sure if I'm running it correctly. From a conda environment:

(notf) orionr-mbp:tensorboard orionr$ bazel run //tensorboard/pip_package:build_pip_package -- --tf-version ''
INFO: Analysed target //tensorboard/pip_package:build_pip_package (2 packages loaded, 760 targets configured).
INFO: Found 1 target...
Target //tensorboard/pip_package:build_pip_package up-to-date:
  bazel-bin/tensorboard/pip_package/build_pip_package
INFO: Elapsed time: 0.747s, Critical Path: 0.10s
INFO: 0 processes.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions
+ command -v curl
+ command -v perl
+ command -v virtualenv

wchargin · 2019-04-02T02:20:03Z

It looks like it’s failing on Travis with the same error:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/tmp/tensorboard/smoke-venv/local/lib/python2.7/site-packages/tensorboard/plugins/scalar/summary.py", line 89, in pb
    import tensorflow.compat.v1 as tf
ImportError: No module named tensorflow.compat.v1
The command "bazel run //tensorboard/pip_package:build_pip_package -- --tf-version "${TF_VERSION_ID}" --smoke" exited with 1.

Build: https://travis-ci.com/tensorflow/tensorboard/jobs/189495702

Your Bazel invocation is correct. Maybe something’s the matter in your
environment (either Conda or POSIX). I can reproduce the problem with

$ tmpdir="$(mktemp -d)"
$ virtualenv -q -p python2 "${tmpdir}/ve"
$ . "${tmpdir}/ve/bin/activate"
(ve) $ pip install -q "absl-py>=0.7.0" "numpy<2.0,>=1.14.5" boto3==1.9.86 flake8==3.5.0 futures==3.1.1 grpcio==1.6.3 mock==2.0.0 moto==1.3.7 yamllint==1.5.0
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
(ve) $ git status 
HEAD detached at orionr/run-notf-travis-tests
nothing to commit, working tree clean
(ve) $ git rev-parse --verify HEAD
3046da351a13e90b979227d564b287e878e9e41d
(ve) $ bazel run //tensorboard/pip_package:build_pip_package -- --tf-version ''
<snip; same traceback as in Travis>

in case that helps. Can you look into this when you get a chance?

I’m not sure what’s up with the assertItemsEqual Python 3 failures in
your Travis build, though. I can’t reproduce them, and they don’t make
much sense to me.

.travis.yml

wchargin

I’m not sure what’s up with the assertItemsEqual Python 3 failures in
your Travis build, though. I can’t reproduce them, and they don’t make
much sense to me.

Ah, I was at an older commit: I can reproduce them at 45ba2589, but
not at d9c66b5a. This PR’s modifications to change these tests to use
unittest.TestCase instead of tf.test.TestCase are not sound, because
the unittest method has a different name in Python 3 vs. Python 2.

We can fix this by using six.assertCountEqual, but could you please
revert the changes to these tests regardless? The PR that adds testing
support for notf in our CI shouldn’t also have a functional change to
make more tests work in notf. (That is: enabling tests is fine;
changing them is less so.) I’d gladly welcome a working version of this
change as a separate PR, which wouldn’t need to depend on anything.

tensorboard/manager_test.py

tensorboard/backend/application_test.py

orionr · 2019-04-02T03:59:54Z

Rebased and removed test changes, etc. However, I still can't repro the ImportError: No module named tensorflow.compat.v1 error locally, which is weird. Going to look at that tomorrow. Thanks @wchargin for the review.

orionr · 2019-04-03T17:42:14Z

@wchargin CI passes now! Let me know your thoughts and thanks.

wchargin

Looks good! Verified that smoke tests work properly on all three configs
(nightly, preview, and notf), and that a failure in summary.scalar_pb
is still caught by the smoke tests.

Please address two small comments, then we can merge.

.travis.yml

tensorboard/pip_package/build_pip_package.sh

wchargin · 2019-04-03T19:01:34Z

Thanks!

wchargin self-requested a review April 1, 2019 22:07

wchargin reviewed Apr 1, 2019

View reviewed changes

wchargin reviewed Apr 2, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

wchargin reviewed Apr 2, 2019

View reviewed changes

tensorboard/manager_test.py Outdated Show resolved Hide resolved

tensorboard/backend/application_test.py Outdated Show resolved Hide resolved

orionr added 6 commits April 1, 2019 20:48

Add separate non-TensorFlow test run in Travis CI

65f2846

Install absl and numpy for non-TensorFlow CI

7d70e7b

Correctly quote pip installs

41b90ba

Code review changes

af6bd62

Separate test from check in .travis.yml

0e49ac4

Set TENSORBOARD_NO_TF in smoke test

c851333

orionr force-pushed the run-notf-travis-tests branch from 3046da3 to c851333 Compare April 2, 2019 03:54

orionr added 2 commits April 3, 2019 09:57

Restrict API tests without TensorFlow

00b54f3

Move more API tests to just TensorFlow

20befa0

wchargin approved these changes Apr 3, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

tensorboard/pip_package/build_pip_package.sh Outdated Show resolved Hide resolved

Code review changes

9b0b026

wchargin merged commit 762e4bb into tensorflow:master Apr 3, 2019

orionr mentioned this pull request Apr 3, 2019

Enable more tests in notf case for Travis CI #2089

Merged

wchargin mentioned this pull request May 25, 2019

Improve travis.yml config for faster CI #2278

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add separate non-TensorFlow test run in Travis CI #2075

Add separate non-TensorFlow test run in Travis CI #2075

orionr commented Mar 31, 2019

wchargin left a comment

wchargin Apr 1, 2019

orionr Apr 2, 2019

orionr commented Apr 2, 2019

wchargin commented Apr 2, 2019

wchargin left a comment

orionr commented Apr 2, 2019

orionr commented Apr 3, 2019

wchargin left a comment

wchargin commented Apr 3, 2019

Add separate non-TensorFlow test run in Travis CI #2075

Add separate non-TensorFlow test run in Travis CI #2075

Conversation

orionr commented Mar 31, 2019

wchargin left a comment

Choose a reason for hiding this comment

wchargin Apr 1, 2019

Choose a reason for hiding this comment

orionr Apr 2, 2019

Choose a reason for hiding this comment

orionr commented Apr 2, 2019

wchargin commented Apr 2, 2019

wchargin left a comment

Choose a reason for hiding this comment

orionr commented Apr 2, 2019

orionr commented Apr 3, 2019

wchargin left a comment

Choose a reason for hiding this comment

wchargin commented Apr 3, 2019