Merge pull request #68 from NaturalHistoryMuseum/josh/ckan-upgrade-2.9

CKAN 2.9.x upgrade

.coveragerc

@@ -0,0 +1,2 @@
+[run]
+relative_files = True

.gitignore

@@ -2,4 +2,6 @@
 *.pyc
 .noseids
 
-\.coverage
+
+.coverage
+.idea

.travis.yml

@@ -1,19 +1,16 @@
-dist: trusty
-language: python
-
-python:
-  - "2.7"
+sudo: required
 
-install:
-  - sh ckanext/ldap/tests/bin/build.sh
+language: python
 
 services:
-  - redis-server
-  - postgresql
+  - docker
 
-addons:
-  postgresql: "9.4"
+# we need coveralls and this also prevents travis from running pip install -r requirements.txt
+install: pip install coveralls
 
-script: coverage run --source=ckanext.ldap setup.py nosetests --ckan --with-pylons=ckanext/ldap/tests/bin/test.ini --nologcapture --debug=ckantest,ckanext.ldap --rednose
+script:
+  - docker-compose build
+  - docker-compose run ckan
 
 after_success: coveralls
+

CHANGELOG.md

@@ -2,10 +2,23 @@
 
 (This file may not be historically complete, as it is a recent addition to the project).
 
+## [3.0.0] - 2021-01-25
+
+- Drop python2 support, add complete python3.6-3.8 support
+- Switch docker based tests to run on python3
+- Remove python2 specific code (_u_ string prefixes primarily), embrace some python3 code (use of
+  fstrings)
+- Remove dependency on six
+
+## [2.1.0] - 2021-01-25
+
+- Updated to work with CKAN 2.9.1.
+- Switched to docker based testing.
+- Add dependency on six
 
 ## [2.0.0-alpha] - 2019-07-23
 
 - Updated to work with CKAN 2.9.0a, e.g.:
     - uses toolkit wherever possible
     - references to Pylons removed
-- Standardised README, CHANGELOG, setup.py and .github files to match other Museum extensions
+- Standardised README, CHANGELOG, setup.py and .github files to match other Museum extensions

README.md

@@ -4,7 +4,8 @@
 
 [![Travis](https://img.shields.io/travis/NaturalHistoryMuseum/ckanext-ldap/master.svg?style=flat-square)](https://travis-ci.org/NaturalHistoryMuseum/ckanext-ldap)
 [![Coveralls](https://img.shields.io/coveralls/github/NaturalHistoryMuseum/ckanext-ldap/master.svg?style=flat-square)](https://coveralls.io/github/NaturalHistoryMuseum/ckanext-ldap)
-[![CKAN](https://img.shields.io/badge/ckan-2.9.0a-orange.svg?style=flat-square)](https://github.com/ckan/ckan)
+[![CKAN](https://img.shields.io/badge/ckan-2.9.1-orange.svg?style=flat-square)](https://github.com/ckan/ckan)
+[![Python](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8-blue.svg?style=flat-square)](https://www.python.org/)
 
 _A CKAN extension that provides LDAP authentication._
 
@@ -75,10 +76,10 @@ These are the options that can be specified in your .ini config file.
 Name|Description|Options
 --|--|--
 `ckanext.ldap.uri`|The URI of the LDAP server, of the form _ldap://example.com_. You can use the URI to specify TLS (use 'ldaps' protocol), and the port number (suffix ':port').|True/False
-`ckanext.ldap.base_dn`|The base dn in which to perform the search. Example: 'ou=USERS,dc=example,dc=com'.|  
-`ckanext.ldap.search.filter`|This is the search string that is sent to the LDAP server, in which '{login}' is replaced by the user name provided by the user. Example: 'sAMAccountName={login}'. The search performed here **must** return exactly 0 or 1 entry.|  
-`ckanext.ldap.username`|The LDAP attribute that will be used as the CKAN username. This **must** be unique.|  
-`ckanext.ldap.email`|The LDAP attribute to map to the user's email address. This **must** be unique.|  
+`ckanext.ldap.base_dn`|The base dn in which to perform the search. Example: 'ou=USERS,dc=example,dc=com'.|
+`ckanext.ldap.search.filter`|This is the search string that is sent to the LDAP server, in which '{login}' is replaced by the user name provided by the user. Example: 'sAMAccountName={login}'. The search performed here **must** return exactly 0 or 1 entry.|
+`ckanext.ldap.username`|The LDAP attribute that will be used as the CKAN username. This **must** be unique.|
+`ckanext.ldap.email`|The LDAP attribute to map to the user's email address. This **must** be unique.|
 
 ## Other options
 
@@ -109,7 +110,12 @@ Name|Description|Options|Default
 
 1. `setup-org`: create the organisation specified in `ckanext.ldap.organization.id`.
     ```bash
-    paster --plugin=ckanext-ldap ldap setup-org -c $CONFIG_FILE
+    ckan -c $CONFIG_FILE ldap setup-org
+    ```
+
+2. `initdb`: ensure the tables needed by this extension exist.
+    ```bash
+    ckan -c $CONFIG_FILE ldap initdb
     ```
 
 ## Templates
@@ -127,10 +133,24 @@ The helper function `h.is_ldap_user()` is also provided for templates.
 
 
 # Testing
-
 _Test coverage is currently extremely limited._
 
-To run the tests, use nosetests inside your virtualenv. The `--nocapture` flag will allow you to see the debug statements.
+To run the tests in this extension, there is a Docker compose configuration available in this
+repository to make it easy.
+
+To run the tests against ckan 2.9.x on Python3:
+
+1. Build the required images
 ```bash
-nosetests --ckan --with-pylons=$TEST_CONFIG_FILE --where=$INSTALL_FOLDER/src/ckanext-ldap --nologcapture --nocapture
+docker-compose build
 ```
+
+2. Then run the tests.
+   The root of the repository is mounted into the ckan container as a volume by the Docker compose
+   configuration, so you should only need to rebuild the ckan image if you change the extension's
+   dependencies.
+```bash
+docker-compose run ckan
+```
+
+The ckan image uses the Dockerfile in the `docker/` folder which is based on `openknowledge/ckan-dev:2.9`.

ckanext/ldap/cli.py

@@ -0,0 +1,60 @@
+import click
+
+from ckan.plugins import toolkit
+from ckanext.ldap.model.ldap_user import ldap_user_table
+
+
+def get_commands():
+    return [ldap]
+
+
+@click.group()
+def ldap():
+    '''
+    The LDAP CLI.
+    '''
+    pass
+
+
+@ldap.command(name='initdb')
+def init_db():
+    '''
+    Ensures the database tables we need exist in the database and creates them if they don't.
+    '''
+    if not ldap_user_table.exists():
+        ldap_user_table.create()
+        click.secho(f'Created {ldap_user_table.name} table', fg='green')
+    else:
+        click.secho(f'Table {ldap_user_table.name} already exists', fg='green')
+
+
+@ldap.command(name='setup-org')
+def setup_org():
+    '''
+    Sets up the default organisation which all ldap users will be automatically made members of.
+    '''
+    # get the organisation all users will be added to
+    organization_id = toolkit.config['ckanext.ldap.organization.id']
+
+    # set up context
+    user = toolkit.get_action('get_site_user')({
+        'ignore_auth': True
+    }, {})
+    context = {
+        'user': user['name']
+    }
+
+    try:
+        toolkit.get_action('organization_show')(context, {
+            'id': organization_id
+        })
+        click.secho(u"Organisation already exists, doing nothing", fg=u"green")
+    except toolkit.ObjectNotFound:
+        # see the following commit to understand why this line is here
+        # https://github.com/ckan/ckanext-harvest/commit/f315f41c86cbde4a49ef869b6993598f8cb11e2d
+        context.pop('__auth_audit', None)
+        toolkit.get_action('organization_create')(context, {
+            'id': organization_id,
+            'name': organization_id
+        })
+        click.secho(u"New organisation created", fg=u"green")

ckanext/ldap/lib/helpers.py

@@ -3,49 +3,38 @@
 #
 # This file is part of ckanext-ldap
 # Created by the Natural History Museum in London, UK
-
 from ckan.common import session
 from ckan.plugins import toolkit
 
-try:
-    # In case we are running Python3
-    from urllib.parse import urlparse, parse_qs
-except ImportError:
-    from urlparse import urlparse, parse_qs
+from urllib.parse import urlparse, parse_qs
 
 
 def is_ldap_user():
-    '''Helper function for determining if current user is LDAP user
+    '''
+    Helper function for determining if current user is LDAP user.
 
     :returns: boolean
-
     '''
-
-    return u'ckanext-ldap-user' in session
+    return 'ckanext-ldap-user' in session
 
 
 def get_login_action():
-    ''' Returns ldap login handler. Preserves parameter `came_from`
-    as stored in context object's login_handler.
-
+    '''
+    Returns ldap login handler. Preserves parameter `came_from` as stored in context object's
+    login_handler.
     '''
     if hasattr(toolkit.c, 'login_handler'):
-        camefrom = parse_qs(urlparse(toolkit.c.login_handler).query).get(u'came_from')
+        came_from = parse_qs(urlparse(toolkit.c.login_handler).query).get('came_from')
     else:
-        camefrom = None
-    if camefrom:
-        action = toolkit.url_for(u'ldap.login_handler', came_from=str(camefrom[0]))
+        came_from = None
+    if came_from:
+        action = toolkit.url_for('ldap.login_handler', came_from=str(came_from[0]))
     else:
-        action = toolkit.url_for(u'ldap.login_handler')
+        action = toolkit.url_for('ldap.login_handler')
     return action
 
 
-def decode_str(s, encoding=u'utf-8'):
-    try:
-        # this try throws NameError if this is python3
-        if isinstance(s, basestring) and isinstance(s, str):
-            return unicode(s, encoding)
-    except NameError:
-        if isinstance(s, bytes):
-            return s.decode(encoding)
+def decode_str(s, encoding='utf-8'):
+    if isinstance(s, bytes):
+        return s.decode(encoding)
     return s