Skip to content

Commit bcb11f3

Browse files
luccasmmgmpolidoristeveoni
authored
Sync dev staging mar 19 (#653)
* Harvest dynamic configs pt. 2 (#635) * Update ckanext-s3filestore branch * Fix typo * Update layer manager to use wri release * Fix Dockerfile * [ODP-235] Allow private and hidden teams (#537) * update teams create and edit form * update organization list api * update schema * update api * remove extras * fix type errors * update user_list_wri to support private teams * fix api call * remove chained action * add visibility * fix breaking changes * update env * update env plugin * fix test * update test * add visibility to org during dataset edit * add test for private teams * remove logs * update private team functionality * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * resolve conflict * undo changes * update organization patch and create * add private attribute to fetched orgs * update organization_list sql logic * improve team visbility * improve error message * update private team test * update failing test * update test * update settings * update test * update test * update test * update test * update test * update test * update test * update docker compose * add tooltip * update test * add padlock to private teams * add back auth plugin * Harvest config feb 03 2025 (#637) * Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app * Add whoami output to start_ckan.sh * [download_event.py] Add missing closing parenthesis * Set supervisor home by variable * Fix harvest config permissions * Use CKAN_IMAGE variable instead of local image name * Handle sed supervisor changes as root; Add start_ckan sub-script * Revert sub-script; Always use user ckan for supervisor * Fix sed commands * Use root user with ckan home for supervisord * Revert "Harvest config feb 03 2025 (#637)" This reverts commit e68155a. * Harvest dynamic configs pt. 3 (#638) * Add supercronic for harvest run command (#640) * Fix alignment * Fix build * Fix build * [ODP-414] add email notification when download-flow fails (#641) * add email notification when flow fails * update error notification * empty * remove pgclient version * update libpq --------- Co-authored-by: Michael Polidori <[email protected]> * Upgrade to CKAN 2.11 (#632) * Update setup.py * Update Members permission to team form + ODP-370 (#642) * members can not * update role: Editor can create sub-team and admin moving subteam from public to private should not throw error * disabled Public option if parent is private * Trigger CI --------- Co-authored-by: Luccas Mateus <[email protected]> * Change setup.py and setup.cfg * Fix ckanext-wri install (#643) * Improve resource location design (#646) * Trigger CI * [odp-426]: Add edit icon to dashboard entities (#648) * [odp-426]: Add edit icon to dashboard entities * update schema * Add bulk purge collaborator issue patch (#645) * Increase UWSGI timeouts; Extract URLs from markdown links in learn_more (#649) * Resource location search improvements (#650) * add authorized check to group/teams last node (#651) * Trigger CI * Rm global from DatafileLocation * Trigger CI * enable edit icon for collaborators (#652) * Fix typo * Remove dev helm config values and GH Actions file --------- Co-authored-by: Michael Polidori <[email protected]> Co-authored-by: Stephen Oni <[email protected]>
1 parent 6356c6b commit bcb11f3

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+909
-553
lines changed

‎.github/workflows/main.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -112,8 +112,6 @@ jobs:
112112
run: |
113113
docker compose -f docker-compose.test.yml --env-file .env.example up --build -d
114114
working-directory: ./ckan-backend-dev
115-
- name: Initialize the Extensions
116-
run: bash ./ckan-backend-dev/ckan/scripts/init-extensions.sh
117115
- name: Cypress Install and CKAN setup
118116
if: ${{ env.ENABLE_INT_TESTS == 'true' }}
119117
uses: cypress-io/github-action@v6
@@ -131,8 +129,6 @@ jobs:
131129
CKAN_IMAGE: '${{ steps.login-ecr.outputs.registry }}/${{ secrets.ECR_CKAN_REPO }}:staging-${{ github.sha }}'
132130
run: docker compose -f docker-compose.test.yml --env-file .env.example up --build -d frontend
133131
working-directory: ./ckan-backend-dev
134-
- name: Initialize the Extensions
135-
run: bash ./ckan-backend-dev/ckan/scripts/init-extensions.sh
136132
- name: Print Logs
137133
if: ${{ env.ENABLE_DEBUG_LOGS == 'true' }}
138134
env:
@@ -148,6 +144,8 @@ jobs:
148144
node-version: 18
149145
runTests: false
150146
working-directory: ./e2e-tests
147+
- name: Initialize the Extensions
148+
run: bash ./ckan-backend-dev/ckan/scripts/init-extensions.sh
151149
- name: Run Integration tests 🧪
152150
if: ${{ env.ENABLE_INT_TESTS == 'true' }}
153151
uses: cypress-io/github-action@v6
@@ -195,6 +193,8 @@ jobs:
195193
working-directory: ./ckan-backend-dev
196194
- name: Initialize the pending datasets table
197195
run: docker exec ckan-wri sh -c "ckan -c production.ini pendingdatasetsdb"
196+
- name: Initialize the issues table
197+
run: docker exec ckan-wri sh -c "ckan -c production.ini issuesdb"
198198
- name: Run Unit Tests 🧪
199199
env:
200200
CKAN_IMAGE: '${{ steps.login-ecr.outputs.registry }}/${{ secrets.ECR_CKAN_REPO }}:${{ github.sha }}'

‎ckan-backend-dev/.env.example

+3-3
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ TEST_CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore_test
4343
TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_test
4444

4545
# CKAN core
46-
CKAN_VERSION=2.10.0
46+
CKAN_VERSION=2.11.0
4747
CKAN_SITE_ID=default
4848
CKAN_SITE_URL=http://ckan-dev:5000
4949
CKAN_PORT=5000
@@ -84,8 +84,8 @@ DATAPUSHER_REWRITE_RESOURCES=True
8484
DATAPUSHER_REWRITE_URL=http://ckan-dev:5000
8585

8686
# Extensions
87-
CKAN__PLUGINS=image_view text_view webpage_view resource_proxy datatables_view datastore datapusher activity s3filestore scheming_datasets scheming_organizations scheming_groups harvest wri wri_harvester auth hierarchy_display hierarchy_form hierarchy_group_form issues envvars
88-
CKAN__VIEWS__DEFAULT_VIEWS=image_view text_view webpage_view datatables_view
87+
#CKAN__PLUGINS='text_view image_view webpage_view resource_proxy datatables_view datastore datapusher activity s3filestore scheming_datasets scheming_organizations scheming_groups harvest wri wri_harvester auth hierarchy_display hierarchy_form hierarchy_group_form envvars'
88+
#CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view webpage_view datatables_view"
8989
CKAN__HARVEST__MQ__TYPE=redis
9090
CKAN__HARVEST__MQ__HOSTNAME=redis
9191
CKAN__HARVEST__MQ__PORT=6379

‎ckan-backend-dev/ckan/Dockerfile.dev

+16-6
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM viderum/ckan:v2.10.3-alpine-3.18
1+
FROM viderum/ckan:v2.11.0
22

33
# Set up environment variables
44
ENV APP_DIR=/srv/app
@@ -22,16 +22,22 @@ ENV SRC_EXTENSIONS_DIR=${APP_DIR}/src_extensions
2222
# to get them mounted in this image at runtime
2323
USER root
2424
RUN pip3 install --upgrade setuptools==70.0.0
25-
# Echo the version of setuptools to check if it is installed correctly
26-
RUN pip3 install -e 'git+https://github.com/datopian/ckanext-scheming.git@0983735ca163f906693c46f5315ee5abf68d2742#egg=ckanext-scheming' && \
27-
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-scheming/ckan-2.10/test-requirements.txt' && \
25+
26+
RUN pip3 install -e 'git+https://github.com/ckan/ckanext-scheming.git@5ce30cf2856aee97fa7b49de46847f8119dad355#egg=ckanext-scheming' && \
27+
pip3 install -r 'https://raw.githubusercontent.com/ckan/ckanext-scheming/5ce30cf2856aee97fa7b49de46847f8119dad355/test-requirements.txt' && \
2828
pip3 install -e 'git+https://github.com/datopian/ckanext-s3filestore.git@wri/cost-splitting-orgs#egg=ckanext-s3filestore' && \
2929
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/requirements.txt' && \
30-
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/dev-requirements.txt' && \
30+
#pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/dev-requirements.txt' && \
31+
# Use latest versions for now due to build errors. TODO: Update to specific newer versions (hopefully these are compatible)
32+
pip3 install moto && \
33+
pip3 install ckanapi && \
34+
pip3 install httpretty && \
3135
pip3 install -e 'git+https://github.com/datopian/ckanext-auth.git@okta#egg=ckanext-auth' && \
3236
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-auth/okta/requirements.txt' && \
33-
pip3 install -e 'git+https://github.com/ckan/ckanext-hierarchy.git@master#egg=ckanext-hierarchy' && \
37+
pip3 install -e 'git+https://github.com/datopian/ckanext-hierarchy.git@wri#egg=ckanext-hierarchy' && \
3438
pip3 install -e 'git+https://github.com/datopian/[email protected]#egg=ckanext-issues' && \
39+
# Required by ckanext-issues, but it's no longer included in any of the requirements files or the subdependencies
40+
pip3 install mock && \
3541
pip3 install -e 'git+https://github.com/ckan/[email protected]#egg=ckanext-harvest' && \
3642
pip3 install -r 'https://raw.githubusercontent.com/ckan/ckanext-harvest/refs/tags/v1.6.0/requirements.txt'
3743

@@ -106,6 +112,9 @@ RUN chown ckan:ckan ${APP_DIR}/start_ckan_development.sh
106112

107113
RUN apk --no-cache add openssl
108114

115+
RUN cat ${APP_DIR}/src/ckan/requirements.txt
116+
RUN pip3 install -r /srv/app/src/ckan/requirements.txt
117+
109118
RUN openssl genpkey -algorithm RSA -out ${APP_DIR}/jwtRS256.key && \
110119
openssl rsa -in ${APP_DIR}/jwtRS256.key -pubout -outform PEM -out ${APP_DIR}/jwtRS256.key.pub && \
111120
chown ckan:ckan ${APP_DIR}/jwtRS256.key && \
@@ -120,6 +129,7 @@ COPY docker-entrypoint.d/* /docker-entrypoint.d/
120129

121130
RUN ckan config-tool ${CKAN_INI} "ckan.cors.origin_allow_all = True"
122131
RUN ckan config-tool ${CKAN_INI} "ckan.root_path = /private-admin/{{LANG}}"
132+
RUN ckan config-tool ${CKAN_INI} -s app:main "ckan.datapusher.api_token = xxxx-xxxx-xxxx-xxxx"
123133

124134
COPY setup/who.ini ${APP_DIR}/src/ckan/who.ini
125135

‎ckan-backend-dev/ckan/patches/ckan/02_tests_reset_db.patch

+8-10
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,16 @@
1-
diff --git a/ckan/ckan/model/__init__.py b/ckan/ckan/model/__init__.py
2-
--- ckan/ckan/model/__init__.py
3-
+++ ckan/ckan/model/__init__.py
4-
@@ -219,8 +219,13 @@
5-
with warnings.catch_warnings():
1+
diff --git a/ckan/model/__init__.py b/ckan/model/__init__.py
2+
index 595ae3876..a03e50c25 100644
3+
--- a/ckan/model/__init__.py
4+
+++ b/ckan/model/__init__.py
5+
@@ -237,6 +237,11 @@ class Repository():
66
warnings.filterwarnings('ignore', '.*(reflection|tsvector).*')
7-
meta.metadata.reflect()
7+
meta.metadata.reflect(engine)
88

99
+ tables = meta.metadata.sorted_tables
1010
+ for table in tables:
1111
+ if table.name == "spatial_ref_sys":
1212
+ meta.metadata.remove(table)
1313
+
14-
meta.metadata.drop_all()
15-
self.tables_created_and_initialised = False
16-
log.info('Database tables dropped')
14+
with engine.begin() as conn:
15+
meta.metadata.drop_all(conn)
1716

18-

‎ckan-backend-dev/ckan/patches/ckan/03_fix_dataset_purge_collaborators.patch

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
diff --git a/ckan/logic/action/delete.py b/ckan/logic/action/delete.py
2+
index dc3ce8ec2..1828effca 100644
3+
--- a/ckan/logic/action/delete.py
4+
+++ b/ckan/logic/action/delete.py
5+
@@ -152,6 +152,14 @@ def dataset_purge(context: Context, data_dict: DataDict) -> ActionResult.Dataset
6+
model.PackageRelationship.object_package_id == pkg.id)).all():
7+
r.purge()
8+
9+
+ dataset_collaborators = (
10+
+ model.Session.query(model.PackageMember)
11+
+ .filter(model.PackageMember.package_id == id)
12+
+ .all()
13+
+ )
14+
+ for collaborator in dataset_collaborators:
15+
+ collaborator.delete()
16+
+
17+
pkg = model.Package.get(id)
18+
assert pkg
19+
pkg.purge()

‎ckan-backend-dev/ckan/scripts/init-extensions.sh

-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
# Initialize Issues and Notification DB
32
docker exec ckan-wri sh -c "ckan -c production.ini issuesdb"
43
docker exec ckan-wri sh -c "ckan -c production.ini downloadeventdb"

‎ckan-backend-dev/ckan/setup/prerun.py.override

+10-6
Original file line numberDiff line numberDiff line change
@@ -93,20 +93,24 @@ def check_solr_connection(retry=None):
9393

9494

9595
def init_db():
96+
# Output the contents of the CKAN_INI file
97+
print("[prerun] CKAN_INI file contents:")
98+
with open(ckan_ini, "r") as f:
99+
print(f.read())
96100

97101
db_command = ["ckan", "-c", ckan_ini, "db", "init"]
98102
print("[prerun] Initializing or upgrading db - start")
99103
try:
100104
subprocess.check_output(db_command, stderr=subprocess.STDOUT)
101105
print("[prerun] Initializing or upgrading db - end")
102106
except subprocess.CalledProcessError as e:
103-
if "OperationalError" in e.output:
104-
print(e.output)
107+
if "OperationalError" in e.output.decode("utf-8"):
108+
print(e.output.decode("utf-8"))
105109
print("[prerun] Database not ready, waiting a bit before exit...")
106110
time.sleep(5)
107111
sys.exit(1)
108112
else:
109-
print(e.output)
113+
print(e.output.decode("utf-8"))
110114
raise e
111115

112116

@@ -144,13 +148,13 @@ def init_datastore_db():
144148
print(str(e))
145149

146150
except subprocess.CalledProcessError as e:
147-
if "OperationalError" in e.output:
148-
print(e.output)
151+
if "OperationalError" in e.output.decode("utf-8"):
152+
print(e.output.decode("utf-8"))
149153
print("[prerun] Database not ready, waiting a bit before exit...")
150154
time.sleep(5)
151155
sys.exit(1)
152156
else:
153-
print(e.output)
157+
print(e.output.decode("utf-8"))
154158
raise e
155159
finally:
156160
cursor.close()

‎ckan-backend-dev/ckan/setup/start_ckan_development.sh.override

+19-5
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
echo "Looking for local extensions to install..."
55
echo "Extension dir contents:"
66
ls -la $SRC_EXTENSIONS_DIR
7+
78
for i in $SRC_EXTENSIONS_DIR/*
89
do
910
if [ -d $i ];
@@ -46,7 +47,7 @@ echo "Enabling debug mode"
4647
ckan config-tool $CKAN_INI -s DEFAULT "debug = true"
4748

4849
# Add ckan.datapusher.api_token to the CKAN config file (updated with corrected value later)
49-
ckan config-tool $CKAN_INI ckan.datapusher.api_token=xxx
50+
#ckan config-tool $CKAN_INI ckan.datapusher.api_token=xxx
5051

5152
# Set up the Secret key used by Beaker and Flask
5253
# This can be overriden using a CKAN___BEAKER__SESSION__SECRET env var
@@ -60,10 +61,6 @@ then
6061
ckan config-tool $CKAN_INI "api_token.jwt.decode.secret=${JWT_SECRET}"
6162
fi
6263

63-
# Update the plugins setting in the ini file with the values defined in the env var
64-
echo "Loading the following plugins: $CKAN__PLUGINS"
65-
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
66-
6764
# Update test-core.ini DB, SOLR & Redis settings
6865
echo "Loading test settings into test-core.ini"
6966
ckan config-tool $SRC_DIR/ckan/test-core.ini \
@@ -73,13 +70,30 @@ ckan config-tool $SRC_DIR/ckan/test-core.ini \
7370
"solr_url = $TEST_CKAN_SOLR_URL" \
7471
"ckan.redis.url = $TEST_CKAN_REDIS_URL"
7572

73+
ckan config-tool $CKAN_INI "ckan.plugins = datastore envvars"
74+
ckan config-tool $CKAN_INI "ckan.views.default_views ="
75+
76+
DEFAULT_VIEWS=$CKAN__VIEWS__DEFAULT_VIEWS
77+
PLUGINS=$CKAN__PLUGINS
78+
79+
unset CKAN__PLUGINS
80+
unset CKAN__VIEWS__DEFAULT_VIEWS
81+
7682
# Run the prerun script to init CKAN and create the default admin user
7783
python3 prerun.py
7884

85+
export CKAN__VIEWS__DEFAULT_VIEWS="$DEFAULT_VIEWS"
86+
export CKAN__PLUGINS="$PLUGINS"
87+
7988
unset CKAN__DATAPUSHER__API_TOKEN
8089
echo "Set up ckan.datapusher.api_token in the CKAN config file"
8190
ckan config-tool $CKAN_INI "ckan.datapusher.api_token=$(ckan -c $CKAN_INI user token add ckan_admin datapusher | tail -n 1 | tr -d '\t')"
8291

92+
# Update the plugins setting in the ini file with the values defined in the env var
93+
echo "Loading the following plugins: $CKAN__PLUGINS"
94+
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
95+
ckan config-tool $CKAN_INI "ckan.views.default_views = $CKAN__VIEWS__DEFAULT_VIEWS"
96+
8397
# Run any startup scripts provided by images extending this one
8498
if [[ -d "/docker-entrypoint.d" ]]
8599
then

‎ckan-backend-dev/docker-compose.dev.yml

+11-11
Original file line numberDiff line numberDiff line change
@@ -96,17 +96,17 @@ services:
9696
prefect:
9797
condition: service_healthy
9898

99-
my-migration:
100-
container_name: my-migration
101-
build:
102-
context: ../migration
103-
restart: always
104-
environment:
105-
- FLOW_DEPLOYMENT_ENV=dev
106-
- PREFECT_API_URL=http://prefect:4200/api
107-
depends_on:
108-
prefect:
109-
condition: service_healthy
99+
#my-migration:
100+
# container_name: my-migration
101+
# build:
102+
# context: ../migration
103+
# restart: always
104+
# environment:
105+
# - FLOW_DEPLOYMENT_ENV=dev
106+
# - PREFECT_API_URL=http://prefect:4200/api
107+
# depends_on:
108+
# prefect:
109+
# condition: service_healthy
110110

111111
db:
112112
container_name: ${POSTGRESQL_CONTAINER_NAME}

‎ckan-backend-dev/solr/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ FROM solr:8
22

33
EXPOSE 8983
44

5-
ARG CKAN_BRANCH="dev-v2.10"
5+
ARG CKAN_BRANCH="2.11"
66

77
ENV SOLR_INSTALL="/opt/solr"
88
ENV SOLR_CONFIG_DIR="$SOLR_INSTALL/server/solr/configsets"

‎ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py

+6-7
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,7 @@
1313
from ckan.common import config, asbool, aslist
1414
from ckan.model import Package
1515
from sqlalchemy import text, engine
16-
from shapely import MultiPolygon, Polygon, wkb, wkt
17-
from shapely import make_valid
16+
from shapely import wkb, wkt
1817
import ckan.model as model
1918
from ckan.logic.action.get import (
2019
_unpick_search,
@@ -827,6 +826,7 @@ def group_activity_list_wri(context: Context, data_dict: DataDict):
827826
return results
828827

829828

829+
830830
@logic.side_effect_free
831831
def user_list_wri(context: Context, data_dict: DataDict):
832832
model = context["model"]
@@ -1274,11 +1274,6 @@ def resource_search(context: Context, data_dict: DataDict):
12741274
shape = get_shape_from_dataapi(spatial_address, point)
12751275
if shape:
12761276
shape = wkt.loads(shape)
1277-
bbox = Polygon([(-180, -90), (180, -90), (180, 90), (-180, 90)])
1278-
shape = make_valid(shape)
1279-
shape = shape.intersection(bbox)
1280-
polygons = [geom for geom in shape.geoms if isinstance(geom, (Polygon, MultiPolygon))]
1281-
shape = MultiPolygon(polygons)
12821277
spatial_geom = geoalchemy2.functions.ST_GeomFromText(shape.wkt)
12831278
location_queries.append(
12841279
geoalchemy2.functions.ST_Intersects(
@@ -1453,6 +1448,7 @@ def organization_list_for_user(context: Context,
14531448
return orgs_list
14541449

14551450

1451+
14561452
@logic.side_effect_free
14571453
def organization_list(context: Context,
14581454
data_dict: DataDict) -> ActionResult.OrganizationList:
@@ -1516,6 +1512,7 @@ def organization_list(context: Context,
15161512
return _group_or_org_list(context, data_dict, is_org=True)
15171513

15181514

1515+
15191516
def _group_or_org_list(
15201517
context: Context, data_dict: DataDict, is_org: bool = False):
15211518
model = context['model']
@@ -1844,6 +1841,8 @@ def validate_visibility(context, data_dict):
18441841
raise ValidationError({"message": _("Organization has private visibility and cannot create public datasets")})
18451842

18461843

1844+
1845+
18471846
@logic.side_effect_free
18481847
def organization_show(context, data_dict):
18491848
data_dict = old_organization_show(context, data_dict)

0 commit comments

Comments
 (0)