Test with geonode-selenium #4240

frafra · 2019-02-18T15:07:17Z

I tested the modifications alone, but not the merged .travis-ci.yml file because I have issue with Travis CI (it says that my geonode repository is legacy, they do not support migration to the new system at the moment, so it seems that I cannot use it nor disable it).

This pull request fixes #4234.

frafra · 2019-02-18T15:26:55Z

Travis is failing because I am using Ubuntu 16.04 instead of the default 14.04.
Ubuntu 14.04 will be EOL in about 2 months. It seems reasonable to me to update .travis-ci.yml to a more recent LTS version.

frafra · 2019-02-20T09:58:19Z

Ok, we are getting closer to it. 5 jobs out of 9 are now working as expected; 3 GeoServer jobs are full of Java errors (could someone please have a look at them?), while the new geonode-selenium job is not working because nginx tries to bind to a port that is in use, but that should be easy to fix.

afabiani · 2019-02-20T10:07:27Z

Seems like a GeoFence initialization issue

Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/annotation/adapters/XmlAdapter
	at org.geoserver.geoserver.authentication.GeoFenceXStreamPersisterInitializer.init(GeoFenceXStreamPersisterInitializer.java:40)
	at org.geoserver.config.util.XStreamPersisterFactory.buildPersister(XStreamPersisterFactory.java:58)

trying to restart the job in order to see if this is a temporary issue. I will dig into it otherwise.

afabiani · 2019-02-20T11:24:23Z

I guess those addings to .travis.yml

script: paver run_tests --coverage --local false

are causing issues.
GeoServer (and the test environment in general) is setup by pavement.py at runtime accordingly to the .travis setting.
By running the tests directly most lickely causes issues due to misconfiguration.

frafra · 2019-02-20T13:06:37Z

Moving from having a script script for every job to having one single script with an if-statement didn't fix the issue.

afabiani · 2019-02-20T13:08:24Z

weird... let me take a look!

afabiani · 2019-02-20T13:58:50Z

@frafra I suspect the issue is the Java version it is using.
Probably by changing the distribution it uses a more recent JDK (OpenJDK 11 from the logs).

We should install the Oracle JDK 8 instead.
Let me check around for the scripts on how to setup this one.

afabiani · 2019-02-20T14:02:23Z

Ideally we should do something like the following:

sudo apt-add-repository -y ppa:webupd8team/java; \
  sudo apt update; sudo apt upgrade -y; sudo apt autoremove --purge; \
  sudo apt install -y oracle-java8-set-default

sudo apt install -y tomcat8; \
  sudo sed -i -e 's/xom-\*\.jar/xom-\*\.jar,bcprov\*\.jar/g' /var/lib/tomcat8/conf/catalina.properties

echo 'JAVA_HOME=/usr/' | sudo tee --append /etc/default/tomcat8

frafra · 2019-02-20T14:36:56Z

You are right, it is using a different JVM:

java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode

The current code seems very similar to what you proposed, and almost identical to the one in master (I just call debconf to accept the license):

sudo add-apt-repository -y ppa:webupd8team/java;
sudo apt-get update;
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections;
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections;
sudo apt-get install -y oracle-java8-set-default ant maven --no-install-recommends;
sudo update-java-alternatives --set java-8-oracle;

It seems that installing oracle-java8-set-default and using update-java-alternatives is not enough to update the java symlink.

…ode-selenium

afabiani · 2019-02-21T15:32:51Z

@frafra almost there... I see this error on geonode-selenium test

ERROR: for spcgeonode_nginx_1_716eb1f0ea87  Cannot start service nginx: driver failed programming external connectivity on endpoint spcgeonode_nginx_1_4bdcf89d1c19 (ba85edaab462d4a3d1ca1553a049caa22fcc646f7226dfce111258c5bab5249e): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
ERROR: for nginx  Cannot start service nginx: driver failed programming external connectivity on endpoint spcgeonode_nginx_1_4bdcf89d1c19 (ba85edaab462d4a3d1ca1553a049caa22fcc646f7226dfce111258c5bab5249e): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
ERROR: Encountered errors while bringing up the project.
The command "if [ "$TEST_RUN_SELENIUM" = "True" ]; then geonode-selenium/test-docker.sh; else paver run_tests --coverage --local false; fi" exited with 1.

not sure why it is trying to run two nginx instances

frafra · 2019-02-21T16:36:08Z

@afabiani I updated geonode-selenium to use a different port by default. Feel free to restart the Travis CI pipeline.

frafra · 2019-02-21T18:40:40Z

Please try one last time.

afabiani · 2019-02-22T09:09:09Z

@frafra few progresses but still errors

/home/travis/virtualenv/python3.5_with_system_site_packages/bin/python3: No module named fades
+python3 -m pytest
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/manager.py", line 268, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2345, in load
    self.require(*args, **kwargs)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2368, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pkg_resources/__init__.py", line 789, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (chardet 2.3.0 (/usr/lib/python3/dist-packages), Requirement.parse('chardet<3.1.0,>=3.0.2'), {'requests'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pytest.py", line 89, in <module>
    raise SystemExit(pytest.main())
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/_pytest/config/__init__.py", line 61, in main
    config = _prepareconfig(args, plugins)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/_pytest/config/__init__.py", line 196, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/hooks.py", line 284, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/_pytest/config/__init__.py", line 655, in pytest_cmdline_parse
    self.parse(args)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/_pytest/config/__init__.py", line 871, in parse
    self._preparse(args, addopts=addopts)
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/_pytest/config/__init__.py", line 817, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/travis/virtualenv/python3.5_with_system_site_packages/lib/python3.5/site-packages/pluggy/manager.py", line 274, in load_setuptools_entrypoints
    message="Plugin %r could not be loaded: %s!" % (ep.name, e),
pluggy.manager.PluginValidationError: Plugin 'seleniumbase' could not be loaded: (chardet 2.3.0 (/usr/lib/python3/dist-packages), Requirement.parse('chardet<3.1.0,>=3.0.2'), {'requests'})!
The command "if [ "$TEST_RUN_SELENIUM" = "True" ]; then geonode-selenium/test-docker.sh; else paver run_tests --coverage --local false; fi" exited with 1.

afabiani · 2019-02-22T10:19:21Z

:(

+fades -r requirements.txt -x python -m pytest
*** fades ***  2019-02-22 10:16:52,863  INFO     Hi! This is fades 8.1, automatically managing your dependencies
*** fades ***  2019-02-22 10:16:52,863  ERROR    fades is running from inside a virtualenv ('/home/travis/virtualenv/python3.5_with_system_site_packages'), which is not supported
The command "if [ "$TEST_RUN_SELENIUM" = "True" ]; then geonode-selenium/test-docker.sh; else paver run_tests --coverage --local false; fi" exited with 255.

frafra · 2019-02-22T12:03:23Z

Ok, tests are now running, but it looks that there is a timing issue. I am going to fix it. I propose to squash the "fix" commits after that and merge. Thanks for your help.

frafra force-pushed the geonode-selenium branch 25 times, most recently from dee27a0 to 023da23 Compare February 20, 2019 08:37

frafra force-pushed the geonode-selenium branch from a3e1877 to f9e1ae3 Compare February 20, 2019 12:00

Test with geonode-selenium

9c8d1b2

frafra force-pushed the geonode-selenium branch from f9e1ae3 to 9c8d1b2 Compare February 20, 2019 12:33

afabiani added 7 commits February 21, 2019 12:08

Fixes Travis geonode-selenium build

d8b05a3

Merge branch 'master' of https://github.com/GeoNode/geonode into geon…

32aab87

…ode-selenium

Fixes Travis geonode-selenium build

79a4da9

Fixes Travis geonode-selenium build

04ce54c

Fixes Travis geonode-selenium build

09cc694

Merge branch 'master' of https://github.com/GeoNode/geonode into geon…

c4ae963

…ode-selenium

Fixes Travis geonode-selenium build

fca593a

Fixes Travis geonode-selenium build

9ae8535

frafra force-pushed the geonode-selenium branch from 083c7aa to 9ae8535 Compare February 22, 2019 10:43

afabiani mentioned this pull request Feb 22, 2019

[Closes #4249] Contribute back upstream menu management from IGAD #4250

Merged

Disable system_site_packages for geonode-selenium

6f1bb7d

frafra force-pushed the geonode-selenium branch from 188598d to 6f1bb7d Compare February 25, 2019 08:39

frafra changed the title ~~[WIP] Test with geonode-selenium~~ Test with geonode-selenium Feb 25, 2019

afabiani approved these changes Feb 26, 2019

View reviewed changes

afabiani merged commit d75b1bf into GeoNode:master Feb 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test with geonode-selenium #4240

Test with geonode-selenium #4240

frafra commented Feb 18, 2019 •

edited

Loading

frafra commented Feb 18, 2019

frafra commented Feb 20, 2019

afabiani commented Feb 20, 2019

afabiani commented Feb 20, 2019

frafra commented Feb 20, 2019 •

edited

Loading

afabiani commented Feb 20, 2019

afabiani commented Feb 20, 2019

afabiani commented Feb 20, 2019

frafra commented Feb 20, 2019

afabiani commented Feb 21, 2019

frafra commented Feb 21, 2019

frafra commented Feb 21, 2019

afabiani commented Feb 22, 2019

afabiani commented Feb 22, 2019

frafra commented Feb 22, 2019

Test with geonode-selenium #4240

Test with geonode-selenium #4240

Conversation

frafra commented Feb 18, 2019 • edited Loading

frafra commented Feb 18, 2019

frafra commented Feb 20, 2019

afabiani commented Feb 20, 2019

afabiani commented Feb 20, 2019

frafra commented Feb 20, 2019 • edited Loading

afabiani commented Feb 20, 2019

afabiani commented Feb 20, 2019

afabiani commented Feb 20, 2019

frafra commented Feb 20, 2019

afabiani commented Feb 21, 2019

frafra commented Feb 21, 2019

frafra commented Feb 21, 2019

afabiani commented Feb 22, 2019

afabiani commented Feb 22, 2019

frafra commented Feb 22, 2019

frafra commented Feb 18, 2019 •

edited

Loading

frafra commented Feb 20, 2019 •

edited

Loading