Unable to upload layers anymore

[frafra]

Running SPCGeonode, GeoNode commit 287708d (Jan 15 2019). I am unable to upload layers. Data comes from a 2.8 instance. I tried to generate and run migrations to see it that could be the issue, but it is not (and the makemigrations warning is shown every time). GeoNode and GeoServer can talk each other and the configuration seems just fine.

It worked fine, now I have the same issue on two different servers. The only thing that happens is that the containers get restarted from time to time.

The first relevant error is the following one (I am logged in as administrator):

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/oauth2_provider/oauth2_validators.py", line 380, in validate_bearer_token
    access_token = AccessToken.objects.select_related("application", "user").get(token=token)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
DoesNotExist: AccessToken matching query does not exist.

Then there is a database error and an additional error trying to manage the previous one.

Full logs below:

-----------------------------------------------------
STARTING DJANGO ENTRYPOINT Fri Feb 15 09:47:29 UTC 2019
-----------------------------------------------------
Running initialize.py...
-----------------------------------------------------
1. Running the migrations
Operations to perform:
  Apply all migrations: account, actstream, admin, agon_ratings, announcements, auth, avatar, base, contenttypes, dialogos, documents, geonode_client, geonode_themes, groups, guardian, invitations, layers, maps, oauth2_provider, people, pinax_notifications, services, sessions, sites, socialaccount, taggit, tastypie, upload, user_messages
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
-----------------------------------------------------
2. Creating/updating superuser
superuser successfully updated
-----------------------------------------------------
3. Create/update an OAuth2 provider to use authorisations keys
oauth2 provider successfully updated
-----------------------------------------------------
4. Loading fixtures
Installed 300 object(s) from 1 fixture(s)
-----------------------------------------------------
5. Running updatemaplayerip
-----------------------------------------------------
 Collecting static files
-----------------------------------------------------
7. Securing GeoServer
Master password was already changed. No changes made.
-----------------------------------------------------
FINISHED DJANGO ENTRYPOINT --------------------------
-----------------------------------------------------
*** Starting uWSGI 2.0.17 (64bit) on [Fri Feb 15 09:47:33 2019] ***
compiled with version: 6.3.0 20170516 on 17 January 2019 13:42:37
os: Linux-4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019
nodename: 22222f1dd06a
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /spcgeonode
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /spcgeonode
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 127.0.0.1:8001 fd 3
spawned uWSGI http 1 (pid: 15)
uwsgi socket 0 bound to TCP address :8000 fd 6
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 2.7.14 (default, May  1 2018, 23:38:34)  [GCC 6.3.0 20170516]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x5561a4a4b290
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 364520 bytes (355 KB) for 5 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x5561a4a4b290 pid: 1 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (pid: 1, cores: 1)
spawned uWSGI worker 2 (pid: 23, cores: 1)
spawned uWSGI worker 3 (pid: 24, cores: 1)
spawned uWSGI worker 4 (pid: 25, cores: 1)
spawned uWSGI worker 5 (pid: 26, cores: 1)
[pid: 26|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 294 bytes} [Fri Feb 15 10:48:29 2019] GET / => generated 29556 bytes in 1586 msecs (HTTP/1.1 200) 8 headers in 393 bytes (1 switches on core 0)
[pid: 1|app: 0|req: 1/2] 10.255.0.2 () {46 vars in 817 bytes} [Fri Feb 15 10:48:31 2019] GET / => generated 29556 bytes in 1400 msecs (HTTP/1.1 200) 9 headers in 476 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 2/3] 10.255.0.2 () {46 vars in 729 bytes} [Fri Feb 15 10:48:33 2019] GET /jsi18n/ => generated 11591 bytes in 7 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 25|app: 0|req: 1/4] 10.255.0.2 () {46 vars in 729 bytes} [Fri Feb 15 10:48:33 2019] GET /lang.js => generated 13361 bytes in 1273 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 1|app: 0|req: 2/5] 10.255.0.2 () {44 vars in 760 bytes} [Fri Feb 15 10:48:34 2019] GET /favicon.ico => generated 18681 bytes in 32 msecs (HTTP/1.1 404) 8 headers in 400 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 3/6] 10.255.0.2 () {46 vars in 771 bytes} [Fri Feb 15 10:48:34 2019] GET /api/featured/ => generated 132 bytes in 25 msecs (HTTP/1.1 200) 8 headers in 262 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 4/7] 10.255.0.2 () {46 vars in 821 bytes} [Fri Feb 15 10:48:36 2019] GET /layers/?limit=20 => generated 56933 bytes in 97 msecs (HTTP/1.1 200) 8 headers in 393 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 5/8] 10.255.0.2 () {44 vars in 714 bytes} [Fri Feb 15 10:48:37 2019] GET /jsi18n/ => generated 11591 bytes in 3 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 6/9] 10.255.0.2 () {44 vars in 714 bytes} [Fri Feb 15 10:48:37 2019] GET /lang.js => generated 13361 bytes in 4 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 1|app: 0|req: 3/10] 10.255.0.2 () {44 vars in 797 bytes} [Fri Feb 15 10:48:37 2019] GET /api/thesaurus/keywords/?type=layer => generated 139 bytes in 62 msecs (HTTP/1.1 200) 8 headers in 262 bytes (1 switches on core 0)
[pid: 25|app: 0|req: 2/11] 10.255.0.2 () {44 vars in 773 bytes} [Fri Feb 15 10:48:37 2019] GET /api/owners/?type=layer => generated 2474 bytes in 98 msecs (HTTP/1.1 200) 8 headers in 263 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 7/12] 10.255.0.2 () {44 vars in 781 bytes} [Fri Feb 15 10:48:37 2019] GET /api/categories/?type=layer => generated 12658 bytes in 919 msecs (HTTP/1.1 200) 8 headers in 264 bytes (1 switches on core 0)
[pid: 1|app: 0|req: 4/13] 10.255.0.2 () {44 vars in 787 bytes} [Fri Feb 15 10:48:37 2019] GET /api/layers/?limit=20&offset=0 => generated 38240 bytes in 1088 msecs (HTTP/1.1 200) 8 headers in 264 bytes (1 switches on core 0)
[pid: 23|app: 0|req: 1/14] 10.255.0.2 () {44 vars in 779 bytes} [Fri Feb 15 10:48:37 2019] GET /h_keywords_api?type=layer => generated 4363 bytes in 1482 msecs (HTTP/1.1 200) 7 headers in 238 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 1/15] 10.255.0.2 () {44 vars in 775 bytes} [Fri Feb 15 10:48:37 2019] GET /api/regions/?type=layer => generated 66452 bytes in 1820 msecs (HTTP/1.1 200) 8 headers in 264 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 2/16] 10.255.0.2 () {46 vars in 841 bytes} [Fri Feb 15 10:48:40 2019] GET /layers/upload => generated 0 bytes in 3 msecs (HTTP/1.1 302) 8 headers in 316 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 3/17] 10.255.0.2 () {46 vars in 882 bytes} [Fri Feb 15 10:48:40 2019] GET /account/login/?next=/layers/upload => generated 20730 bytes in 194 msecs (HTTP/1.1 200) 8 headers in 393 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 8/18] 10.255.0.2 () {44 vars in 732 bytes} [Fri Feb 15 10:48:40 2019] GET /lang.js => generated 13361 bytes in 4 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 4/19] 10.255.0.2 () {44 vars in 732 bytes} [Fri Feb 15 10:48:40 2019] GET /jsi18n/ => generated 11591 bytes in 5 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/oauth2_provider/oauth2_validators.py", line 380, in validate_bearer_token
    access_token = AccessToken.objects.select_related("application", "user").get(token=token)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
DoesNotExist: AccessToken matching query does not exist.
[pid: 1|app: 0|req: 5/20] 10.0.7.17 () {40 vars in 713 bytes} [Fri Feb 15 10:48:45 2019] POST /api/o/v4/tokeninfo/?access_token=e2dddcea310611e9ab3a02420a00073c => generated 13 bytes in 11 msecs (HTTP/1.1 403) 7 headers in 235 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 5/21] 10.255.0.2 () {50 vars in 969 bytes} [Fri Feb 15 10:48:45 2019] POST /account/login/ => generated 0 bytes in 138 msecs (HTTP/1.1 302) 11 headers in 651 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 9/22] 10.255.0.2 () {46 vars in 1021 bytes} [Fri Feb 15 10:48:45 2019] GET /layers/upload => generated 42816 bytes in 235 msecs (HTTP/1.1 200) 9 headers in 475 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 6/23] 10.255.0.2 () {44 vars in 755 bytes} [Fri Feb 15 10:48:46 2019] GET /jsi18n/ => generated 11591 bytes in 5 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 23|app: 0|req: 2/24] 10.255.0.2 () {44 vars in 755 bytes} [Fri Feb 15 10:48:46 2019] GET /lang.js => generated 13361 bytes in 466 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 26|app: 0|req: 10/25] 10.255.0.2 () {42 vars in 773 bytes} [Fri Feb 15 10:48:46 2019] GET /favicon.ico => generated 19764 bytes in 32 msecs (HTTP/1.1 404) 8 headers in 400 bytes (1 switches on core 0)
[pid: 24|app: 0|req: 7/26] 10.255.0.2 () {46 vars in 853 bytes} [Fri Feb 15 10:48:53 2019] GET /upload/progress => generated 17 bytes in 3 msecs (HTTP/1.1 200) 7 headers in 228 bytes (1 switches on core 0)
/usr/local/lib/python2.7/site-packages/owslib/iso.py:117: FutureWarning: the .identification and .serviceidentification properties will merge into .identification being a list of properties.  This is currently implemented in .identificationinfo.  Please see https://github.com/geopython/OWSLib/issues/38 for more information
  FutureWarning)
/usr/local/lib/python2.7/site-packages/owslib/iso.py:482: FutureWarning: The .keywords and .keywords2 properties will merge into the .keywords property in the future, with .keywords becoming a list of MD_Keywords instances. This is currently implemented in .keywords2. Please see https://github.com/geopython/OWSLib/issues/301 for more information
  FutureWarning)
>>> Step 2. Make sure we are not trying to overwrite a existing resource named [g250_90] with the wrong type
./geonode/geoserver/helpers.py:1216: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
  if store:
>>> Step 3. Identifying if [g250_90] is vector or raster and gathering extra files
>>> Step 4. Starting upload of [g250_90] to GeoServer...
>>> Step 5. Generating the metadata for [g250_90] after successful import to GeoSever
>>> Step 6. Making sure [g250_90] has a valid projection
>>> Step 7. Creating style for [g250_90]
>>> Step 10. Creating Django record for [g250_90]
[pid: 26|app: 0|req: 11/27] 10.255.0.2 () {46 vars in 853 bytes} [Fri Feb 15 10:48:56 2019] GET /upload/progress => generated 17 bytes in 3 msecs (HTTP/1.1 200) 7 headers in 228 bytes (1 switches on core 0)
/usr/local/lib/python2.7/site-packages/geoserver/style.py:73: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
  if not user_style:
/usr/local/lib/python2.7/site-packages/geoserver/style.py:77: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(elem)' or 'elem is not None' test instead.
  if user_style:
Exception while publishing message: Traceback (most recent call last):
  File "./geonode/messaging/producer.py", line 71, in sync_if_local_memory
    worker.run(timeout=broker_socket_timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume
    conn.drain_events(timeout=safety_interval)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
    get(self._deliver, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 1001, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, in _poll
    return cycle.get(callback)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, in _get_and_deliver
    callback(message, queue)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver
    callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, in _callback
    return callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks]
  File "./geonode/messaging/consumer.py", line 104, in on_geoserver_messages
    geoserver_post_save_local(layer)
  File "./geonode/geoserver/signals.py", line 290, in geoserver_post_save_local
    if any(instance.keyword_list()):
  File "./geonode/base/models.py", line 915, in keyword_list
    return [kw.name.encode("utf-8", "replace") for kw in self.keywords.all()]
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
InternalError: current transaction is aborted, commands ignored until end of transaction block
current transaction is aborted, commands ignored until end of transaction block
Traceback (most recent call last):
  File "./geonode/layers/views.py", line 211, in layer_upload
    metadata_upload_form=form.cleaned_data["metadata_upload_form"])
  File "./geonode/layers/utils.py", line 595, in file_upload
    defaults=defaults
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in get_or_create
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_or_create
    return self._create_object_from_params(lookup, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 500, in _create_object_from_params
    obj = self.create(**params)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in create
    return super(MultilingualQuerySet, self).create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 82, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 848, in save_base
    update_fields=update_fields, raw=raw, using=using,
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in send
    for receiver in self._live_receivers(sender)
  File "./geonode/decorators.py", line 44, in wrapper
    return func(*args, **kwargs)
  File "./geonode/geoserver/signals.py", line 92, in geoserver_post_save
    producer.geoserver_upload_layer(payload)
  File "<decorator-gen-3>", line 2, in geoserver_upload_layer
  File "./geonode/messaging/producer.py", line 71, in sync_if_local_memory
    worker.run(timeout=broker_socket_timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume
    conn.drain_events(timeout=safety_interval)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
    get(self._deliver, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 1001, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, in _poll
    return cycle.get(callback)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, in _get_and_deliver
    callback(message, queue)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver
    callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, in _callback
    return callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks]
  File "./geonode/messaging/consumer.py", line 104, in on_geoserver_messages
    geoserver_post_save_local(layer)
  File "./geonode/geoserver/signals.py", line 290, in geoserver_post_save_local
    if any(instance.keyword_list()):
  File "./geonode/base/models.py", line 915, in keyword_list
    return [kw.name.encode("utf-8", "replace") for kw in self.keywords.all()]
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
InternalError: current transaction is aborted, commands ignored until end of transaction block
Internal Server Error: /layers/upload
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "./geonode/layers/views.py", line 242, in layer_upload
    upload_session.error = pickle.dumps(error).decode("utf-8", "replace")
  File "/usr/local/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle traceback objects
[pid: 25|app: 0|req: 3/28] 10.255.0.2 () {52 vars in 1095 bytes} [Fri Feb 15 10:48:54 2019] POST /layers/upload => generated 11917 bytes in 3265 msecs (HTTP/1.1 500) 7 headers in 258 bytes (194 switches on core 0)
[pid: 26|app: 0|req: 12/29] 10.255.0.2 () {46 vars in 924 bytes} [Fri Feb 15 10:48:57 2019] GET /jsi18n/?_=1550224126273 => generated 11591 bytes in 10 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 25|app: 0|req: 4/30] 10.255.0.2 () {46 vars in 924 bytes} [Fri Feb 15 10:48:57 2019] GET /lang.js?_=1550224126274 => generated 13361 bytes in 5 msecs (HTTP/1.1 200) 7 headers in 230 bytes (1 switches on core 0)
[pid: 25|app: 0|req: 5/31] 127.0.0.1 () {28 vars in 294 bytes} [Fri Feb 15 10:49:31 2019] GET / => generated 29556 bytes in 104 msecs (HTTP/1.1 200) 8 headers in 393 bytes (1 switches on core 0)

[t-book]

duplicate of #4233 ?

[frafra]

No, different commits sadly, and setting ASYNC=False has no effect.

[frafra]

Cannot update metadata either.

[afabiani]

@frafra can you maybe send instructions on how to setup the environment e try to reproduce this?

[frafra]

Not easy to reproduce at the moment, but I will try to explain what is happening.

We have 2 almost identical setups, one for testing, one for production.
The main difference between the two is that the one in production is behind a web server for HTTPS.
As the error is related to OAuth2, I opened /en/admin/oauth2_provider/ and I have seen that the URL related to the application is HTTP and all the tokens are expired. As the web server has an HTTP to HTTPS redirection, I tried to set the URL to HTTPS, to see if that could change anything, but I got an SSL error:

[...]
  File "/usr/local/lib/python2.7/site-packages/geoserver/catalog.py", line 1015, in get_workspaces
    description = self.get_xml("%s/workspaces.xml" % self.service_url)
  File "/usr/local/lib/python2.7/site-packages/geoserver/catalog.py", line 213, in get_xml
    response, content = self.http.request(rest_url)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1452, in _request
    redirections=redirections - 1)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1399, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1319, in _conn_request
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1092, in connect
    raise SSLHandshakeError(e)
SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
Internal Server Error: /layers/upload

The certificate is valid: https://www.ssllabs.com/ssltest/analyze.html?d=geodata.nina.no&latest
I suppose that some libraries or ca-certs could be outdated.

[afabiani]

@frafra no matter if the certificate is valid or not, you need to add it to the java truststore in order to make GeoServer recognize it and avoid throwing that exception.

See if this link might be helpful somehow.

[frafra]

Thanks @afabiani for the reply.

The documentation says:

When using a custom Keystore or trying to access a non-trusted or self-signed SSL-protected OAuth2 Provider from a non-SSH connection,

I am not used a self-signed nor a non-trusted certificate, so it seems strange to me that I have to manually add it to a custom keystore.

The error comes from Django, so I do not understand why GeoServer truststore should be related.
I added the certificate nonetheless, but I get the same error.

[giohappy]

@frafra

1. are you still facing the AccessToken exception?
2. did adding the cert to the keystore solve the invalid cert error?

[frafra]

@giohappy yes, I still get that error and adding the certificate to the keystore (see #4245) didn't solve the issue. But I still don't understand why a SSLHandshakeError raised by httplib2 coming from the geonode container is related with the trustworthiness of the certificate by geoserver. It seems to me that the problem is the opposite: geonode is not able to connect to geoserver because of that SSL error.

[frafra]

$ docker container exec -it spcgeonode_django_1 python2
Python 2.7.14 (default, May  1 2018, 23:38:34) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> h = httplib2.Http(".cache")
>>> resp, content = h.request("https://geodata.nina.no/", "GET")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1399, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1319, in _conn_request
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1092, in connect
    raise SSLHandshakeError(e)
httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
>>> resp, content = h.request("https://google.com/", "GET")
>>>

Even if the browsers are able to connect to the website, it looks like the OS certificates are not enough.

[frafra]

SSL issue is now resolved, but it would be nice to have a mechanism similar to #4245 to import the certificate chain.

[frafra]

Still have the same old error: "DoesNotExist: AccessToken matching query does not exist."

[afabiani]

@frafra the issue here is not depending on the AccessToken. That is just a warning.

The real cause is an issue with the layer upload, causing some sort of error/exception and next to the impossibility to unpickle the StackTrace stored into the UploadSession object.

I remember facing a similar issue some time ago and tried to fix it like this

https://github.com/GeoNode/geonode/blob/master/geonode/layers/views.py#L245

Question is, are you using GeoNode master branch code? Or 2.8.x one?

If the latter, maybe it is just matter of backporting the fix, otherwise that means that the issue was not solved on master too.

You might envisage to put some log lines on the view (see the reference above) and see what GeoServer Importer is returning.

[frafra]

@afabiani Thank you for the reply. As I said in the first comment, I am running a quite recent version of GeoNode from master:

Running SPCGeonode, GeoNode commit 287708d (Jan 15 2019).

As git blame says that your change is older than 4 months, I suppose that the issue has not been entirely fixed on master.

[frafra]

Reverting commit 671c250 allows me to see the error: it looks like it is not able to follow redirects, so I replaced http with https for GEOSERVER_LOCATION.

Now I am back to #4241 :-)