Improvements to PyCSW Constraints and Local Mappings

docs/reference/index.txt

@@ -33,3 +33,4 @@ In this section, you will find information about every component of GeoNode, suc
     developers/settings
     browsers
     developers/geosites
+    worldmap

docs/reference/worldmap.txt

@@ -0,0 +1,204 @@
+.. _worldmap:
+
+========
+WorldMap
+========
+
+By using the WorldMap optional application, GeoNode is extended with the following additional features:
+
+* a customized GeoExplorer viewer
+    * the table of contents is hierarchical with layer categories. When a layer is added a new category containing the layer is added to the table of contents. If the category is already in the table of contents, then the layer is added to it. By default the category is the same as the layer's topic category, but that can be renamed by right clicking on it
+    * the "Add Layers" dialog comes with a "Search" tab which uses Hypermap Registry (Hypermap) as a catalogue of remote and local layers. Hypermap is a requirement when using the WorldMap contrib application
+* a gazetteer application: it is possible to add a given layer to a gazetteer. The gazetteer can be checked using the map client. When a layer is part of the gazetter it is possible to include it in a general gazetteer or in a specific project one. It is possible to search place names in the gazetteer by date range, in which case it is necessary to specify the layer attributes for the start and end depict dates
+
+Installation
+============
+
+Requirements
+------------
+
+We are assuming a Ubuntu 16.04.1 LTS development environment, but these instructions can be adapted to any recent Linux distributions::
+
+    # Install Ubuntu dependencies
+    sudo apt-get update
+    sudo apt-get install python-virtualenv python-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev libjpeg-dev libpq-dev libgdal-dev git default-jdk postgresql postgis
+
+    # Install Java 8 (needed by latest GeoServer 2.13)
+    sudo apt-add-repository ppa:webupd8team/java
+    sudo apt-get update
+    sudo apt-get install oracle-java8-installer
+
+Virtual environment creation and installation of Python packages
+----------------------------------------------------------------
+
+Create and activate the virtual environment::
+
+    cd ~
+    virtualenv --no-site-packages env
+    . env/bin/activate
+
+Now install GeoNode from source code::
+
+    git clone -b https://github.com/geonode/geonode.git
+    cd geonode
+    pip install -r requirements.txt
+    pip install pygdal==1.11.3.3
+    pip install -e .
+    paver setup
+    paver sync
+
+Set the following environment variables as needed (change SITE_NAME and SERVER_IP s needed. Also HYPERMAP_REGISTRY_URL and SOLR_URL may be different). Even better, create a file and source it::
+
+      export USE_WORLDMAP=True
+      export SITE_NAME=worldmap
+      export SERVER_IP=128.31.22.73
+      export PG_USERNAME=worldmap
+      export PG_PASSWORD=worldmap
+      export PG_WORLDMAP_DJANGO_DB=worldmap
+      export PG_WORLDMAP_UPLOADS_DB=wmdata
+      export OWNER=$PG_USERNAME
+      export ALLOWED_HOSTS="localhost, $SERVER_IP, "
+      export GEOSERVER_LOCATION=http://localhost:8080/geoserver/
+      export GEOSERVER_PUBLIC_LOCATION=http://$SERVER_IP/geoserver/
+      export SOLR_URL =http://localhost:8983/solr/hypermap/select/
+      export HYPERMAP_REGISTRY_URL =http://localhost:8001
+      export MAPPROXY_URL=http://localhost:8001
+
+
+You can install GeoNode WorldMap in two different ways:
+
+1) By installing GeoNode itself
+2) By using the recommended way of a geonode-project
+
+GeoNode/WorldMap without a geonode-project
+------------------------------------------
+
+Copy the included local_settings.py file and customize it to your needs::
+
+    cp local_settings.py.worldmap.sample local_settings.py
+
+GeoNode/WorldMap with a geonode-project
+---------------------------------------
+
+You will use a geonode-project in order to separate the customization of your instance from GeoNode.
+
+Create your geonode project by using the WorldMap geonode-project as a template  (https://github.com/cga-harvard/geonode-project). Rename it to something meaningful (for example your web site name)::
+
+    cd ~
+    django-admin startproject $SITE_NAME --template=https://github.com/cga-harvard/geonode-project/archive/master.zip -epy,rst
+    cd $SITE_NAME
+
+Create a local_settings.py by copying the included template::
+
+    cp $SITE_NAME/local_settings.py.sample $SITE_NAME/local_settings.py
+    make build
+    paver setup
+
+Start the Server
+----------------
+
+Start GeoNode with Worldmap using pavement::
+
+    python manage.py runserver 0.0.0.0:8000
+    paver start_geoserver
+
+To upload layers you can login with the default GeoNode administrative account:
+
+user: admin
+password: admin
+
+Configuring instance for production
+-----------------------------------
+
+Please follow best practices suggested by GeoNode documentation:
+
+http://docs.geonode.org/en/master/tutorials/advanced/geonode_production/
+
+Remember to add the ip of your server in ALLOWED_HOSTS in the local_settings.py file::
+
+    ALLOWED_HOSTS = ['localhost', '128.31.22.73', ]
+
+Hypermap Registry
+=================
+
+GeoNode with the WorldMap contribute module requires a Hypermap Registry (Hypermap) running instance.
+
+You can install Hypermap by following these instructions (use the "Manual Installation" section): https://github.com/cga-harvard/HHypermap/blob/master/_docs/developers.md
+
+Note that you can bypass Java 8 installation as it was installed previously. As a search engine you should install Solr, as we haven't tested Elasticsearch with WorldMap so far. Create a specific virtual environment for Hypermap in order not to interfere with the GeoNode/WorldMap virtual environment.
+
+After installing Hypermap, start it on a different port than 8000, for example::
+
+    python manage.py runserver 0.0.0.0:8001
+
+In another shell start the Celery process as well::
+
+    cd HHypermap
+    celery -A hypermap worker --beat --scheduler django -l info
+
+Test the stack
+==============
+
+Now that GeoNode/WorldMap and Hypermap are both running, test the stack by uploading a layer.
+
+Login in GeoNode (admin/admin) and upload a shapefile from this page: http://localhost:8000/layers/upload
+
+Make sure the shapefile is correctly displayed in GeoNode by going to the layer page.
+
+Now login in Hypermap (admin/admin) and go to the admin services page: http://localhost:8001/admin/aggregator/service/ Add a service like this:
+
+    * Title: My GeoNode WorldMap SDI
+    * Url: http://localhost:8000/
+    * Type: GeoNode WorldMap
+
+Go to the Hypermap service page and check it the service and the layer is there:
+http://localhost:8001/registry/
+
+In order to have layers in the search engine (Solr) there are two options:
+
+1) from task runner press the "Index cached layers" button
+2) schedule a task in celery
+
+We recommend the second option, which can be configured in the next section.
+
+Schedule Celery tasks
+=====================
+
+Go to the Periodic Task administrative interface: http://localhost:8001/admin/django_celery_beat/periodictask/
+
+Create the following two tasks:
+
+Index Cached Layer Task
+-----------------------
+
+This task will sync the layers from the cache to the search engine. Layers are sent in the cache every time they are saved:
+
+    * Name: Index Cached Layer
+    * Task (registered): hypermap.aggregator.tasks.index_cached_layers
+    * Interval: every 1 minute (or as needed)
+
+Check Worldmap Service
+----------------------
+
+This task will do a check of all of WorldMap service:
+
+    * Name: Check WorldMap Service
+    * Task (registered): hypermap.aggregator.tasks.check_service
+    * Interval: every 1 minute (or as needed)
+    * Arguments: [1] # 1 is the id of the service. Change it as is needed
+
+Now upload a new layer in GeoNode/WorldMap and check if it appears in Hypermap and in Solr (you may need to wait for the tasks to be executed)
+
+Update Last GeoNode WorldMap Layers
+-----------------------------------
+
+If your GeoNode/WorldMap instance has many layers, it is preferable to runt the check_service not so often, as it can be time consuming, and rather use the update_last_wm_layers.
+
+As a first thing, change the interval for the check_service task you created for GeoNode/WorldMap to a value such as "one day" or "one week".
+
+Then create the following periodic task:
+
+    * Name: Sync last layers in WorldMap Service
+    * Task (registered): hypermap.aggregator.update_last_wm_layers
+    * Interval: every 1 minute
+    * Arguments: [1] # 1 is the id of the service. Change it as is needed

geonode/api/resourcebase_api.py

@@ -413,15 +413,14 @@ def build_haystack_filters(self, parameters):
 
         # Filter by geographic bounding box
         if bbox:
-            print(bbox)
             left, bottom, right, top = bbox.split(',')
             sqs = (
                 SearchQuerySet() if sqs is None else sqs).exclude(
                 SQ(
-                    ll_bbox_top__lte=bottom) | SQ(
-                    ll_bbox_bottom__gte=top) | SQ(
-                    ll_bbox_left__gte=right) | SQ(
-                        ll_bbox_right__lte=left))
+                    bbox_top__lte=bottom) | SQ(
+                    bbox_bottom__gte=top) | SQ(
+                    bbox_left__gte=right) | SQ(
+                        bbox_right__lte=left))
 
         # Apply sort
         if sort.lower() == "-date":

geonode/contrib/worldmap/gazetteer/views.py

@@ -102,7 +102,7 @@ def set_none_if_empty(str):
         gazetteer_attributes.append(attribute)
     template = 'gazetteer/edit_layer_gazetteer.html'
 
-    return render(template, {
+    return render(request, template, {
         "layer": layer,
         "gazetteer_attributes": gazetteer_attributes,
         "status_message": status_message,

geonode/documents/models.py

@@ -121,10 +121,13 @@ def get_related_documents(resource):
 
 def get_related_resources(document):
     if document.links:
-        return [
-            link.content_type.get_object_for_this_type(id=link.object_id)
-            for link in document.links.all()
-        ]
+        try:
+            return [
+                link.content_type.get_object_for_this_type(id=link.object_id)
+                for link in document.links.all()
+            ]
+        except BaseException:
+            return []
     else:
         return []
 

geonode/documents/tasks.py

@@ -18,6 +18,10 @@
 #
 #########################################################################
 
+import os
+from os import access, R_OK
+from os.path import isfile
+
 from celery.app import shared_task
 from celery.utils.log import get_task_logger
 
@@ -55,10 +59,15 @@ def create_document_thumbnail(self, object_id):
             logger.debug("Could not convert document #{}: {}."
                          .format(object_id, e))
 
+    try:
+        assert isfile(image_path) and access(image_path, R_OK) and os.stat(image_path).st_size > 0
+    except AssertionError:
+        image_path = None
+
     if not image_path:
         image_path = document.find_placeholder()
 
-    if not image_path:
+    if not image_path or not os.path.exists(image_path):
         logger.debug("Could not find placeholder for document #{}"
                      .format(object_id))
         return

geonode/utils.py

@@ -199,7 +199,10 @@ def bbox_to_projection(native_bbox, target_srid=4326):
     box = native_bbox[:4]
     proj = native_bbox[-1]
     minx, maxx, miny, maxy = [float(a) for a in box]
-    source_srid = int(proj.split(":")[1])
+    try:
+        source_srid = int(proj.split(":")[1]) if proj and ':' in proj else int(proj)
+    except BaseException:
+        source_srid = target_srid
 
     def _v(coord, x, source_srid=4326, target_srid=3857):
         if source_srid == 4326 and target_srid != 4326:
@@ -927,7 +930,10 @@ def check_shp_columnnames(layer):
         inShapefile = unzip_file(inShapefile, '.shp', tempdir=tempdir)
 
     inDriver = ogr.GetDriverByName('ESRI Shapefile')
-    inDataSource = inDriver.Open(inShapefile, 1)
+    try:
+        inDataSource = inDriver.Open(inShapefile, 1)
+    except BaseException:
+        inDataSource = None
     if inDataSource is None:
         logger.warning('Could not open %s' % (inShapefile))
         return False, None, None

requirements.txt

@@ -107,7 +107,7 @@ django-utils<=0.0.2 # python-django-utils (0.0.2 in our ppa)
 django-basic-authentication-decorator==0.9 # python-django-basic-authentication-decorator (0.9 in our ppa)
 
 # GeoNode org maintained apps.
-django-geoexplorer>=4.0.0,<5.0 # python-django-geoexplorer (4.0.39 in our ppa)
+django-geoexplorer>=4.0.40,<5.0 # python-django-geoexplorer (4.0.39 in our ppa)
 geonode-user-messages==0.1.14 # python-geonode-user-messages (0.1.14 in our ppa)
 geonode-avatar==2.1.8 # python-geonode-avatar (2.1.7 in our ppa) FIXME
 geonode-announcements==1.0.13 # python-geonode-announcements (1.0.13 in our ppa)