Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Permissions rework #5163

Merged
merged 272 commits into from
Dec 22, 2022
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
272 commits
Select commit Hold shift + click to select a range
754f6a9
Adding new backend for roles in server for user
Thingus Jun 14, 2022
410de5f
UI rework in progress
Thingus Jun 15, 2022
0dfef70
Progress on TokenBuilder
Thingus Jun 15, 2022
e9ee8a1
Checkbox in place
Thingus Jun 15, 2022
f776064
Checkall implemented
Thingus Jun 16, 2022
2c29c21
Token builder now returning tokens
Thingus Jun 20, 2022
b852fd2
Token builder in place and working
Thingus Jun 21, 2022
3738afd
Mid role rework
Thingus Jun 21, 2022
281a6c3
Functionising RoleDetails
Thingus Jun 21, 2022
696cb75
RoleDetails in progress
Thingus Jun 22, 2022
2be48ec
accidentally did work on master...
Thingus Jun 22, 2022
954d223
Fixing typo in backend
Thingus Jun 22, 2022
620af42
Mid changing Groups to Roles
Thingus Jun 22, 2022
cb90171
Trying to figure out why I can't edit Roles
Thingus Jun 22, 2022
606979c
Role endpoint work
Thingus Jun 23, 2022
9efd936
GET role endpoint now running
Thingus Jun 23, 2022
af9092d
Adding role edit endpoint (see note in source code)
Thingus Jun 23, 2022
c223c37
More endpoints, and useEffect doesn't pass anything on to its caller....
Thingus Jun 23, 2022
12914ac
Got to a RolePicker (sortof)
Thingus Jun 24, 2022
38a0eef
Trying to convince RolePicker to pass internal state
Thingus Jun 24, 2022
9bb1531
Continuing with RoleDetails
Thingus Jun 27, 2022
fae97fb
Misunderstood how useEffect worked
Thingus Jun 27, 2022
94ab263
Carrying on with RolePicker
Thingus Jun 28, 2022
59408a8
Three days of high blood pressure for want of a return statement
Thingus Jun 29, 2022
6b685ac
scope.scope now scope.name, also server picker working
Thingus Jun 30, 2022
72d8d28
Altered scopes endpoint working on role picker
Thingus Jun 30, 2022
53ec642
RoleScopePicker now picking scopes for roles
Thingus Jun 30, 2022
8f0f664
New role+delete role now in place
Thingus Jul 1, 2022
3dbadbe
Reworking user admin interface
Thingus Jul 1, 2022
3ab8be8
Reworking some API bits
Thingus Jul 4, 2022
27aa8f3
Deal with javascript object equivalence
greenape Jul 5, 2022
14614e0
Integrating Jono's fix; pickers now working
Thingus Jul 5, 2022
6e36320
Missed some server bits
Thingus Jul 5, 2022
2db7822
Implementing James' new scope format (untested)
Thingus Jul 6, 2022
7ed4706
Working on resolving references for schema traversal
Thingus Jul 7, 2022
d42b391
Progress on resolvingparser
Thingus Jul 11, 2022
ddee369
Working on resolving parser
Thingus Jul 11, 2022
a8af8b9
We probably don't need to flatten dicts anymore
Thingus Jul 11, 2022
8dff1d9
Why are my iterators not iterating?
Thingus Jul 12, 2022
1bf52d4
Refactoring to use a return list
Thingus Jul 12, 2022
60042f4
Scope triplets being produced at this point
Thingus Jul 12, 2022
5365984
Working on frontend scope parser
Thingus Jul 13, 2022
efd1643
Changing undefined to unset to avoid JS trouble
Thingus Jul 14, 2022
85d0f0c
ScopePicker now working with new API
Thingus Jul 14, 2022
721e57f
Server scopes now loading
Thingus Jul 15, 2022
1d16607
Changing demo data to use new scopes
Thingus Jul 15, 2022
6f714b0
Working on MultiCascader
Thingus Jul 18, 2022
7e5f7e4
MultiSelect now working with scopes in server
Thingus Jul 19, 2022
2717cbf
Scopes updating now working
Thingus Jul 21, 2022
47f11d5
Putting ToekBuilder back in for now
Thingus Jul 26, 2022
c494383
Working on TokenHistory
Thingus Jul 26, 2022
778e6fc
TokenHistory might be working
Thingus Jul 26, 2022
db4ef0e
Token builder in progress
Thingus Jul 27, 2022
f4cb63a
Token builder + token history now in place
Thingus Jul 28, 2022
61dc75f
Replacing identity with sub in jwt
Thingus Jul 28, 2022
35f1a00
Beginning e2e tests
Thingus Jul 28, 2022
4d579e3
Working on parsing scopes from token
Thingus Jul 29, 2022
719eafb
Token gen changed in flowauth backend
Thingus Aug 1, 2022
f1ede54
Frontend now produces tokens with roles
Thingus Aug 1, 2022
4a2e0c6
New roles-base flowAPI permission checks now in place
Thingus Aug 2, 2022
0c14060
Sorting out scope picker for role
Thingus Aug 4, 2022
e63fd18
ScopePicker for roles now working correctly
Thingus Aug 5, 2022
42235bf
Fixed one bug, but backing out still not working
Thingus Aug 8, 2022
bcd1fd7
'Roles' in UserAdminDetail is bugged
Thingus Aug 9, 2022
35a9235
User changes now saving correctly
Thingus Aug 9, 2022
c39bd71
Issue with RoleMembers is now backend
Thingus Aug 9, 2022
5ec5c92
RoleScopePicker needs some internal rewiring
Thingus Aug 9, 2022
c26b274
Yet more wrestling with RoleScopePicker
Thingus Aug 10, 2022
0527fc1
More RoleScopePicker doings
Thingus Aug 10, 2022
f5b2639
More working on RoleScopePicker
Thingus Aug 10, 2022
8e1db1b
Reworking the async bit of RoleScopePicker
Thingus Aug 12, 2022
1557276
I think that roleScopes initialisation is now working?
Thingus Aug 12, 2022
424941d
RoleDetails edit button now seems to be working
Thingus Aug 12, 2022
a542bf7
All save buttons now backing out correctly
Thingus Aug 12, 2022
c0044a2
get_aggregation_unit in flowapi in prog
Thingus Aug 17, 2022
11c91c2
Breaking out get_async in progress
Thingus Aug 23, 2022
83aa123
test_granular_run_access now green
Thingus Aug 24, 2022
83e972d
Working on poll access test
Thingus Aug 24, 2022
c0d5531
Granular poll test now passing
Thingus Sep 5, 2022
7b22ed1
test_access_control now passing
Thingus Sep 5, 2022
fd8d2b1
Geography test passing
Thingus Sep 5, 2022
4ba81d0
More flowapi tests green
Thingus Sep 6, 2022
325ae5c
FlowAPI unit test suite now green!
Thingus Sep 7, 2022
dbd24cb
Flowauth backend tests largely green or skipped prior to review
Thingus Sep 8, 2022
844aad7
Starting work on frontend tests
Thingus Sep 8, 2022
78427e1
No-role warning in token gen
Thingus Sep 12, 2022
23dcbf6
TokenBuilder tests now green
Thingus Sep 13, 2022
4e3d22b
More cypress + starting work on PR comments
Thingus Sep 14, 2022
50a169d
Tokens endpoint now confirms user has roles
Thingus Sep 15, 2022
94f61ee
CSRF error in tests
Thingus Sep 15, 2022
8a0c45a
Changing timeout for test_roles to work around CSRF timeout in tests
Thingus Sep 16, 2022
7a93d40
Removing 'expiry' from createToken frontend
Thingus Sep 16, 2022
28e5dad
Set of reformatting + spelling changes
Thingus Sep 16, 2022
aacb843
Role funcs now awaited, other small comments addressed
Thingus Sep 16, 2022
28c7587
Working on redesign of RoleList
Thingus Sep 20, 2022
d9df224
New style of RoleList needs buttons wiring up
Thingus Sep 20, 2022
897078a
Mid-wiring up new role form
Thingus Sep 20, 2022
7976bdf
New role layout in place
Thingus Sep 21, 2022
e6f71e3
Role form looking good - next, force refresh on role list
Thingus Sep 21, 2022
7f15f24
Don't bother with refresh, just remove from list on delete
Thingus Sep 21, 2022
5fe052e
Tidying up RoleAdmin layout
Thingus Sep 21, 2022
a7ade9a
Added wrong-server test
Thingus Sep 22, 2022
412eb0f
Added role access test
Thingus Sep 22, 2022
7ddab54
Removing server editing from roles endpoint
Thingus Sep 22, 2022
bfbdb3a
Role expiry validation + test implemented
Thingus Sep 22, 2022
5a8fcca
Removing unused /roles/server/<server_id> endpoint
Thingus Sep 22, 2022
5555cd5
Fixing some imports+breakpoints
Thingus Sep 22, 2022
2b660db
Refactoring has_access args
Thingus Sep 23, 2022
37eea69
Adding server names to token views
Thingus Sep 23, 2022
90b24db
errors -> error in UserAdminDetails
Thingus Sep 23, 2022
8e14245
RoleScopesPicker now only has branches in labels; this may have borke…
Thingus Sep 23, 2022
6acaf1f
test_token_generation now checks role permissions on servers
Thingus Oct 10, 2022
01d8e7f
Reworking demo data to use a demo scopes file
Thingus Oct 10, 2022
fccf59e
Get minimum of latest expiries
greenape Oct 11, 2022
cc37692
Demo data update in progress
Thingus Oct 11, 2022
7f4e61b
This now issues tokens - will check work
Thingus Oct 12, 2022
9469172
Demo data now loads scopes from .json
Thingus Oct 13, 2022
2331784
Fixing duplicate scopes in demo data
Thingus Oct 13, 2022
bc14b84
RoleScopePicker not displaying set scopes
Thingus Oct 13, 2022
3b9e36c
Fixing duplicate scopes in demo data
Thingus Oct 13, 2022
fd062a7
Migrating to yarn + upgrading cypress
Thingus Oct 17, 2022
5c973c8
First cypress mount test now green
Thingus Oct 17, 2022
b6b3e5e
Fixing crash on no single scope being present
Thingus Oct 17, 2022
fab4440
Component test for RoleScopePicker in progress
Thingus Oct 17, 2022
280aabb
RolScopePicker test not hitting getRoleScope test fixture
Thingus Oct 18, 2022
a79d34f
Missing leading / causing tests to fail
Thingus Oct 19, 2022
c3ece61
RoleScopePicker test is now failing for the right reason
Thingus Oct 19, 2022
ce4a7ff
compare_graphs is not comparing correctly
Thingus Oct 19, 2022
e26836a
Rewrite of HCR + other role scope traversal utils
Thingus Oct 20, 2022
38e222f
Rebase clobbered new cypress filepaths, redoing...
Thingus Oct 21, 2022
ab342f8
Fixing pipfile
Thingus Oct 21, 2022
ee8508a
Cypress tests now running
Thingus Oct 25, 2022
416789b
Renaming group_list to role_list
Thingus Nov 4, 2022
a196dbc
Merge branch 'master' into permissions_rework
Thingus Nov 7, 2022
e0c3df8
Working on Cannot set property message of which has only a getter
Thingus Nov 8, 2022
3c00681
Merge branch 'master' into permissions_rework
Thingus Nov 9, 2022
c65a4b3
Fixed version.js test
Thingus Nov 9, 2022
0be9a30
Small edit to the regex
Thingus Nov 9, 2022
280452c
Lowering default timeout for cypress
Thingus Nov 9, 2022
99b945b
Fixing login test
Thingus Nov 9, 2022
f0831fa
Disabling POST on servers/scopes
Thingus Nov 11, 2022
b6e7a52
Adding message to ServerAdmin about missing enable/disable
Thingus Nov 11, 2022
ac53f8a
Rebuild rolescopepicker (#5580)
Thingus Nov 11, 2022
e86e7e5
Test for checking patch role respects server
Thingus Nov 11, 2022
8073c11
Scope-server check on Role in progress
Thingus Nov 14, 2022
e1bd104
Removing breakpoints to see if CI is broken
Thingus Nov 15, 2022
16f74ee
And another breakpoint....
Thingus Nov 15, 2022
7040835
Removing scope validation from model
Thingus Nov 15, 2022
19b6df9
test_demo_data respectes new demo structure
Thingus Nov 15, 2022
24574cd
Scope server test now passing, but others failing
Thingus Nov 15, 2022
7eb3ea9
All tests now passing
Thingus Nov 15, 2022
4e4fc93
Fixes from tests in progress
Thingus Nov 16, 2022
f34e50f
Fixing server scope not returning 404
Thingus Nov 17, 2022
0db66a8
Updating lockfile
Thingus Nov 17, 2022
38031a2
Delay in test to mitigate race condition RoleAdmin
Thingus Nov 17, 2022
05b7d38
Setting cypress scroll to 'center'
Thingus Nov 17, 2022
cdadee1
All role tests now in place
Thingus Nov 17, 2022
bac22a9
Role scope check now in place
Thingus Nov 17, 2022
b480e77
Role validation fires on before_flush event
Thingus Nov 18, 2022
dd2d18e
Scope check in progress
Thingus Nov 18, 2022
4ed87c4
Catch for unique constraint violation + constraint for scope
Thingus Nov 21, 2022
ed9766f
Updating lockfiles
Thingus Nov 22, 2022
f53c6f9
Relocking after merge
Thingus Nov 22, 2022
1f60e95
Fixing test to actually test update statement
Thingus Nov 22, 2022
3fcf461
Updating approval tests
Thingus Nov 23, 2022
0fd4956
Running Prettier
Thingus Nov 23, 2022
ada7016
Rebuilding circle docker caches
Thingus Nov 24, 2022
a9ce38f
Approval tests
Thingus Nov 24, 2022
901a0b9
universal_access_token now returns a role
Thingus Nov 25, 2022
724dd98
Updating role generation
Thingus Nov 25, 2022
4933af7
Possible exception handling in get_agg_unit + related
Thingus Nov 28, 2022
1b85ff7
App context in schema_to_scopes
Thingus Nov 28, 2022
54c827e
Fixing test_permissions in progress
Thingus Nov 29, 2022
061bc81
Claims replaced with roles in jwt tests
Thingus Nov 29, 2022
1da767d
Call to logger causing test to fail
Thingus Nov 30, 2022
f480b19
Tweak to error handling in role.py + unique name test now passing
Thingus Nov 30, 2022
c3e33d9
Trying without cypress cache
Thingus Nov 30, 2022
ed3ea56
Revert "Trying without cypress cache"
Thingus Nov 30, 2022
d0911f0
Pinning Cypress image to 16.13.0
Thingus Dec 1, 2022
1b0fa1e
Pinning Cypress image to 16.13.0
Thingus Dec 1, 2022
7dc89de
Fixing issue with server scope editing + test
Thingus Dec 1, 2022
3ae9231
Fix for nonspatial query issue
Thingus Dec 1, 2022
b89fae0
Getting query failure test to pass
Thingus Dec 1, 2022
3ae88c1
String 'None' to value None
Thingus Dec 2, 2022
a3666dd
Increasing race timeout
Thingus Dec 2, 2022
2c0d74c
Merge branch 'permissions_rework' of https://github.com/Flowminder/Fl…
Thingus Dec 2, 2022
d7bc61e
Increasing Cypress timeouts
Thingus Dec 2, 2022
6f2a1cf
Running prettier
Thingus Dec 2, 2022
1a0e226
Adding guards in to role view
Thingus Dec 2, 2022
ab7630d
Cranking Cypress timeout even higher
Thingus Dec 2, 2022
283532a
Adding print for client 400s
Thingus Dec 2, 2022
fbeae02
More printing for the mystery 404
Thingus Dec 2, 2022
62e3dbd
adding local_ci_process to gitignore
Thingus Dec 6, 2022
ea80c26
Moving log to before poll check
Thingus Dec 8, 2022
5f02457
Adding compression back to jwt.py and tests
Thingus Dec 8, 2022
94d065e
Query added to json_log
Thingus Dec 8, 2022
bc025e7
Decompress claims added user_loader_callback
Thingus Dec 8, 2022
97f709e
Adding decompress to flowauth test
Thingus Dec 8, 2022
75f1e71
New users now keep roles
Thingus Dec 9, 2022
67eb12d
Adding server def check for RoleAdmin
Thingus Dec 13, 2022
c4354b9
Reworking server error catching
Thingus Dec 13, 2022
152c543
Getting rid of error
Thingus Dec 13, 2022
16fea7a
Replacing role.id with role_id
Thingus Dec 13, 2022
7f35d2f
Role duplication now rolls back then reraises
Thingus Dec 15, 2022
247c133
Left breakpoint in
Thingus Dec 15, 2022
da44db2
Adding first_or_404 to new_role.server
Thingus Dec 15, 2022
ca5edcb
Adding catch for no scopes in role
Thingus Dec 15, 2022
a02ad26
RolePicker now fetches scopes explicitly as part of load
Thingus Dec 15, 2022
724f259
Role test error message is now db agnostic. Ish.
Thingus Dec 15, 2022
3ddff5a
Role test error message is now db agnostic. Ish.
Thingus Dec 15, 2022
23f7f3c
Reverting string test to prev version
Thingus Dec 15, 2022
6a9a8ff
Removing unneeded scroll from cypress
Thingus Dec 15, 2022
0e505dc
Mering and relocking
Thingus Dec 15, 2022
262889b
Prettifying
Thingus Dec 15, 2022
57aa5d6
Bumping caches in circle config
Thingus Dec 15, 2022
8373a37
First draft of ADR in place
Thingus Dec 16, 2022
072a07a
Fixing notebook token
Thingus Dec 16, 2022
09ef797
Add named role
greenape Dec 16, 2022
f9a51dc
TOKEN is now a role_dict
Thingus Dec 16, 2022
32b0456
Merge branch 'permissions_rework' of https://github.com/Flowminder/Fl…
Thingus Dec 16, 2022
f1b05c8
Update docs/source/developer/adr/0012-claims-role-scope-rework.md
Thingus Dec 16, 2022
ede890c
Cleaning up handle_unique_error a little
Thingus Dec 19, 2022
5def539
Merge remote-tracking branch 'origin/permissions_rework' into permiss…
Thingus Dec 19, 2022
797875c
Adding James' edits to ADR
Thingus Dec 19, 2022
58646f0
Jono PR comments 1
Thingus Dec 19, 2022
915b9ce
Changes from Jono PR comments 2
Thingus Dec 19, 2022
2d8117a
Simplifying
greenape Dec 19, 2022
f22927a
Fixes to edit_role endpoint
Thingus Dec 20, 2022
e015a98
Adding json to access logger info
Thingus Dec 20, 2022
83a358f
Merge remote-tracking branch 'origin/simplify-generators' into permis…
Thingus Dec 20, 2022
99c4663
Invalid key now throws 400
Thingus Dec 20, 2022
d0e4753
Cleaning up various permissions.py
Thingus Dec 20, 2022
392a081
Further bugfix in role edit endpoint
Thingus Dec 20, 2022
2587cfd
Moving role_to_dict to Role.to_dict()
Thingus Dec 20, 2022
7c5388b
Removing unneeded catach in run_query
Thingus Dec 20, 2022
76a643e
Removing extra commit in set_scopes
Thingus Dec 20, 2022
d56b5da
Dropping flowdb timeout back down
Thingus Dec 20, 2022
c78b579
Cleaning up
Thingus Dec 20, 2022
4499e1e
set_scopes does not delete all scopes anymore
Thingus Dec 21, 2022
a97556d
Adding Cypress component testing to CI
Thingus Dec 21, 2022
a139b29
Naming component testing properly
Thingus Dec 21, 2022
8521d72
Update changelog
Thingus Dec 21, 2022
dc2c465
License headers
Thingus Dec 21, 2022
d6a39f6
Docstring for grab_on_key_list
Thingus Dec 21, 2022
71a3fb2
Removing surplus comprehension
Thingus Dec 21, 2022
bf77ec1
Jono comments 3
Thingus Dec 21, 2022
1cc5a2f
Merge branch 'permissions_rework' of https://github.com/Flowminder/Fl…
Thingus Dec 21, 2022
248e240
Cleaning up scopes in demo_data
Thingus Dec 21, 2022
2d4026e
Merge branch 'master' into permissions_rework
Thingus Dec 22, 2022
cf93b8d
Token warning in CHANGELOG
Thingus Dec 22, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 24 additions & 21 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ jobs:
- attach_workspace:
at: /home/circleci/
- restore_cache:
key: flowdb-deps-4-{{ checksum "flowdb/tests/Pipfile.lock"}}
key: flowdb-deps-6-{{ checksum "flowdb/tests/Pipfile.lock"}}
- run:
name: "Switch to Python v3.9.4"
command: |
Expand All @@ -191,7 +191,7 @@ jobs:
command: |
PIPENV_PIPFILE=flowdb/tests/Pipfile pipenv install --deploy --dev
- save_cache:
key: flowdb-deps-4-{{ checksum "flowdb/tests/Pipfile.lock" }}
key: flowdb-deps-6-{{ checksum "flowdb/tests/Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/
- run:
Expand Down Expand Up @@ -386,13 +386,13 @@ jobs:
steps:
- checkout
- restore_cache:
key: flowmachine-deps-5-{{ checksum "flowmachine/Pipfile.lock" }}
key: flowmachine-deps-6-{{ checksum "flowmachine/Pipfile.lock" }}
# Need to install graphviz and pygraphviz manually because it was removed from the Pipfile
# (see https://github.com/Flowminder/FlowKit/issues/952)
- run: cd flowmachine && pipenv install --dev --deploy && pipenv run pip install -e .
- run: cd flowmachine && sudo apt-get update && sudo apt-get install -y libgraphviz-dev graphviz && pipenv run pip install pygraphviz
- save_cache:
key: flowmachine-deps-5-{{ checksum "flowmachine/Pipfile.lock" }}
key: flowmachine-deps-6-{{ checksum "flowmachine/Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/flowmachine-caaCcVrN

Expand All @@ -402,7 +402,7 @@ jobs:
steps:
- checkout
- restore_cache:
key: flowmachine-deps-5-{{ checksum "flowmachine/Pipfile.lock" }}
key: flowmachine-deps-6-{{ checksum "flowmachine/Pipfile.lock" }}
- run:
name: Linting files with black
# Installed the version of Black from flowmachine's lockfile
Expand All @@ -417,12 +417,12 @@ jobs:
path: /home/circleci/project/
- restore_cache:
keys:
- flowauth-deps-4-{{ checksum "package.json" }}
- flowauth-deps-6-{{ checksum "package.json" }}
- run:
name: Install deps
command: npm ci
- save_cache:
key: flowauth-deps-4-{{ checksum "package.json" }}
key: flowauth-deps-6-{{ checksum "package.json" }}
# cache NPM modules and the folder with the Cypress binary
paths:
- ~/.npm
Expand All @@ -443,7 +443,7 @@ jobs:
- attach_workspace:
at: /home/circleci/
- restore_cache:
key: flowmachine-deps-5-{{ checksum "Pipfile.lock" }}
key: flowmachine-deps-6-{{ checksum "Pipfile.lock" }}
- run:
name: Install graphviz
command: sudo apt-get update && sudo apt-get install -y xvfb libgraphviz-dev graphviz
Expand Down Expand Up @@ -568,7 +568,7 @@ jobs:

run_flowauth_end_to_end_tests:
docker:
- image: cypress/base
- image: cypress/base:16.13.0
- image: flowminder/flowauth:$CIRCLE_SHA1
environment:
DB_URI: postgresql://flowauth:{}@localhost:5432/flowauth
Expand All @@ -591,12 +591,12 @@ jobs:
path: /home/circleci/project/
- restore_cache:
keys:
- flowauth-deps-4-{{ checksum "package.json" }}
- flowauth-deps-6-{{ checksum "package.json" }}
- run:
name: Cypress setup
command: npm ci
- save_cache:
key: flowauth-deps-4-{{ checksum "package.json" }}
key: flowauth-deps-6-{{ checksum "package.json" }}
# cache NPM modules and the folder with the Cypress binary
paths:
- ~/.npm
Expand All @@ -609,6 +609,9 @@ jobs:
- run:
name: Run Cypress end-to-end tests
command: DEBUG="cypress:*" $(npm bin)/cypress run --record --reporter junit --reporter-options "mochaFile=results/flowauth-frontend.[hash].xml"
- run:
name: Run Cypress component tests
command: DEBUG="cypress:*" $(npm bin)/cypress run --component --record --reporter junit --reporter-options "mochaFile=results/flowauth-component.[hash].xml"
- store_test_results:
path: results
- store_artifacts:
Expand Down Expand Up @@ -643,7 +646,7 @@ jobs:
path: /home/circleci/project/
- restore_cache:
keys:
- autoflow-deps-4-{{ checksum "Pipfile.lock" }}
- autoflow-deps-6-{{ checksum "Pipfile.lock" }}
- run: &install_autoflow_deps
name: Install non-python autoflow dependencies
command: |
Expand All @@ -656,7 +659,7 @@ jobs:
command: |
pipenv install --deploy --dev
- save_cache:
key: autoflow-deps-4-{{ checksum "Pipfile.lock" }}
key: autoflow-deps-6-{{ checksum "Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/
- run:
Expand Down Expand Up @@ -715,7 +718,7 @@ jobs:
- checkout:
path: /home/circleci/project/
- restore_cache:
key: flowetl-unit-deps-5-{{ checksum "Pipfile.lock"}}
key: flowetl-unit-deps-6-{{ checksum "Pipfile.lock"}}
- run:
name: Install pipenv
command: pip install --upgrade pipenv
Expand All @@ -724,7 +727,7 @@ jobs:
command: |
pipenv install --deploy --dev
- save_cache:
key: flowetl-unit-deps-5-{{ checksum "Pipfile.lock" }}
key: flowetl-unit-deps-6-{{ checksum "Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/
- run:
Expand All @@ -749,7 +752,7 @@ jobs:
- checkout:
path: /home/circleci/project/
- restore_cache:
key: flowetl-deps-5-{{ checksum "Pipfile.lock"}}
key: flowetl-deps-6-{{ checksum "Pipfile.lock"}}
- run:
name: Install psycopg2 build deps
command: |
Expand All @@ -772,7 +775,7 @@ jobs:
command: |
sudo apt-get update && sudo apt-get install -y postgresql
- save_cache:
key: flowetl-deps-5-{{ checksum "Pipfile.lock" }}
key: flowetl-deps-6-{{ checksum "Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/
- run:
Expand Down Expand Up @@ -803,15 +806,15 @@ jobs:
- attach_workspace:
at: /home/circleci
- restore_cache:
key: integration-test-deps-4-{{ checksum "Pipfile.lock" }}
key: integration-test-deps-6-{{ checksum "Pipfile.lock" }}
- when:
condition:
equal: ["not query_tests", << parameters.pytest_selector >>]
steps:
- run: *install_autoflow_deps
- run: pipenv install --deploy --dev
- save_cache:
key: integration-test-deps-4-{{ checksum "Pipfile.lock" }}
key: integration-test-deps-6-{{ checksum "Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/
- run: *wait_for_flowdb
Expand Down Expand Up @@ -882,7 +885,7 @@ jobs:
- attach_workspace:
at: /home/circleci/
- restore_cache:
key: flowkit-docs-deps-5-{{ checksum "Pipfile.lock" }}
key: flowkit-docs-deps-6-{{ checksum "Pipfile.lock" }}
- run:
name: Install pandoc
command: |
Expand Down Expand Up @@ -976,7 +979,7 @@ jobs:
path: /home/circleci/project/docs/pg_log.zip
destination: pg_log
- save_cache:
key: flowkit-docs-deps-5-{{ checksum "Pipfile.lock" }}
key: flowkit-docs-deps-6-{{ checksum "Pipfile.lock" }}
paths:
- /home/circleci/.local/share/virtualenvs/

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -132,3 +132,4 @@ docs/source/_static/openapi-redoc.json
*.received.txt

secrets_quickstart/flowdb_pgdata
local_ci_process.yml
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Added views `etl.ingested_state`, `etl.available_dates` and `etl.deduped_post_etl_queries` in FlowDB, for convenient extraction of relevant information from the ETL tables. [#5641](https://github.com/Flowminder/FlowKit/issues/5641)

### Changed
- Move from `groups` to `roles` in flowauth; see [here](https://github.com/Flowminder/FlowKit/pull/5163#issuecomment-1216480419) for full details. [#5613](https://github.com/Flowminder/FlowKit/pull/5163)
- Changed `AIRFLOW__CORE__SQL_ALCHEMY_CONN` env var to `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`
- RoleScopePicker component redesigned and reimplemented.
- Docs now recommend creating a separate bind mount for airflow scheduler logs, and include this in the secrets quickstart. [#3622](https://github.com/Flowminder/FlowKit/issues/3622)

### Fixed
Expand Down
4 changes: 3 additions & 1 deletion docs/notebook_preamble.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ def format_dict(x):
username="docsuser",
private_key=load_private_key(os.environ["PRIVATE_JWT_SIGNING_KEY"]),
lifetime=timedelta(days=1),
claims=get_all_claims_from_flowapi(flowapi_url="http://localhost:9090"),
roles=dict(
universal_role=get_all_claims_from_flowapi(flowapi_url="http://localhost:9090")
),
flowapi_identifier=os.environ["FLOWAPI_IDENTIFIER"],
)
74 changes: 74 additions & 0 deletions docs/source/developer/adr/0012-claims-role-scope-rework.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Rework of permissions system

2022/12/16

**Status**

Accepted

**Context**

When a user passes their query to a Flowmachine instance, they first sign into a Flowauth instance. Within a Flowauth instance, a user has a set of claims that define what combination of queries can be run.
The space for these scopes was generated by a full tree walk of every combination of a query, its geographic components, and all sub-queries (and their geographic components) - though normally derived through targetted traversal of the query tree, there were places where a full walk was required. This offered a very fine degree of control over what queries you could permit users to run, but as the available set of queries increased this space expanded exponentially, eventually becoming unwieldy.
This manifested in several places:
- When building the API spec inside flowAPI
- When displaying the spec on the Flowauth frontend for servers (leading to timeouts and crashes on some lower-capability machines)
- When passing large or specific scopes around as JWTs

**Decision**


The core of the change that has been implemented is to change the definition of what a user can access. Instead of a combination of queries, users are now assigned `roles`, which are composed of `scopes`. A scope can be one of two types;
- *Simple scopes*

These control access to a server-wide capability and are all-or-nothing. The currently implemented simple scopes are:
- `run`; permits a user that holds this role to run queries using the `run` flowAPI endpoint
- `get_result`; permits a user to get the results of a query using the `get` flowAPI endpoint
- `get_available_dates`; permits a user to get the list of dates available to run queries against on a server.

These scopes do *not* imply that a role provided the capabilites to run a query by themselves - they must be provided in combination with one or more complex scopes.

- *Complex scopes*

These control access to a combination of a geographic component, a top-level query and a descendent query.
- Top-level queries are the methods that are available to the users of Flowmachine directly.
- Each top-level query may require a set of sub-queries to run, wich in turn may require sub-queries of their own; these are the descendent queries.
- Finally, the geographic component is the spatial presentation that is available to the role. Queries without spatial aggregates (such as `historgram_aggregate`) have the geographic component `nonspatial`.

Complex scopes are of the form `geographic_component:top_level_query:sub_query` - this ordering has been built on assumed order of importance to users. For example it will be a more common use case that an administrator will want to create a role that restrics access only to admin levels 0 or 1 than to the `most_frequest_location` sub-query of the `spatial_aggregate` top-level query.

As a consequence of this, the new Flowauth db schema is shown below:

```mermaid
erDiagram
Servers ||--|{ Roles : contains
Roles }|--|{ Scopes : "provides permissions for"
Users }|--|{ Roles : "can work with"
Servers ||--|{ Scopes : "provides"
```

And the new auth flow is as follows:


```mermaid
sequenceDiagram
participant Flowauth
actor User
participant FlowAPI
note over User: Selects roles for token
User ->>+ Flowauth: roles
note over Flowauth: Checks user is permitted roles
Flowauth ->>+ User: signed token w/ roles + scopes
note over User: Builds query
User ->>+ FlowAPI: query, token
note over FlowAPI: Checks query params are covered by a single role
FlowAPI ->>+ User: query results
```

**Consequences**
- Much faster and more lightweight building of tokens
- Flowauth frontend can now be used on lower-capability machines
- Less fine-grained control over query permissions
- This does have the knock-on effect of reducing the decision space to a more managable size,perhaps leading users to make meaningful decisions about scopes within a role instead of defaulting to 'all'.
- Users can't be explicitly grouped anymore (roles supercede group functions)
- Scopes cannot be assigned directly to users anymore.
Loading