Learnings Table and API Endpoints

[ypyelab]

Is your feature request related to a problem? Please describe.
No. We want to have GO API endpoints for displaying operational learnings coming from DREF and Emergency Appeals documents.

Describe the solution you'd like
We would like to have:

1. GO API GET, PUT, POST, PATCH, DELETE endpoints for the learnings table (table structure including column names and types attached). We will set up an external system to extract and categorize learnings from appeals documents and would like to post the results (categorized learnings) to be stored in GO. Therefore, we would like these API endpoints so that we can get/add/modify/delete this data.
2. GO ADMIN learnings section. We would like to surface this table in the GO admin site, as we would like to have admin users manually validate the learnings and categorizations. As part of the validation, the users should be able to modify the learning and categories. These modified learnings/categories values should be stored in the table columns 'learning_validated', 'type_validated', 'sector_validated', and 'per_component_validated'.

Additional context
This is part of the ops learnings project, which tries to display operational learning data in different platforms/sections. The first step is to have the data easily retrievable (i.e., via GO API).

Where does the data come from and how is it maintained?
Externally generated from appeal documents and posted to GO via a POST endpoint.

Should there be any restrictions around viewing or updating information related to this feature

• GET endpoint: Each learning row is associated with an appeal (MDR code), the learning should have the same visibility as the appeal.
• The other endpoints (PUT, POST, PATCH, DELETE) should be restricted.

Links to documentation
Table structure:
go_ops_learning_endpoint.xlsx
Context of project:
https://docs.google.com/presentation/d/1HiulBBMZLoaazieX9lDuu13XRE6ZGG-RBkRoWBlafcY/edit?usp=drive_link

@tovari @AlexxxH

[szabozoltan69]

Permissions

• not logged-in + logged-in without any group membership: can get the opslearning API endpoint that displays validated learnings
• superusers + logged-in with OpsLearning Admin membership: can get/put/post/delete in the opslearning API endpoint that displays validated and not validated learnings

[szabozoltan69]

The fully functional opslearning API endpoint is deployed to Staging.
https://goadmin-stage.ifrc.org/api/v2/ops-learning/

[ypyelab]

Context:
We will need API filtering for several use cases: mainly the new ops learning dashboard in GO, but also for analytical products like the disaster brief and PER analysis, and for ad-hoc scripts (e.g. for the auto ingest of new learnings, for checking the quality of the extractor and classifier models...)

Requests for the API endpoint:
We would need to be available to do the next type of queries:

• search. We would need this for the operational learning dashboard e.g. https://goadmin.ifrc.org/api/v2/ops-learning/?search=cholera (and that this search in the 'learning','learning_validated','appeal.name','appeal.name_en','appeal.name_es','appeal.name_fr','appeal.name_ar','code' fields)
• id. We would need this for the script for auto-ingest new learnings e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?id=6916 (it would be great if it was possible to also pass a list of ids)
• appeal_document_id. We would need this for the script to check the quality of the extraction model e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_document_id=3005
• type_validated. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?type_validated=1 (it would be great if it was possible to also have an endpoint for the type field)
• is_validated. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?is_validated=True
• modified_at.We would need this for the script for auto-ingest new learnings e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?modified_at__gte=2024-02-01T00%3A00%3A00 (it would be great if we can have gte, lte)
• organization_validated. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?organization_validated=1 (it would be great if it was possible to also have an endpoint for the organization field) AND (it would be great if we could pass a list)
  -sector_validated. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?sector_validated=6 (it would be great if it was possible to also have an endpoint for the sector field) AND (it would be great if we could pass a list)
  -per_component_validated. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?per_component_validated=6 (it would be great if it was possible to also have an endpoint for the per component field) AND (it would be great if we could pass a list)
• appeal_code. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_code=MDRVU008 (it would be great if we could pass a list)
• appeal_num_beneficiaries. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_num_beneficiaries__gte=100000 (it would be great if we can have gte, lte)
• appeal_start_date. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_start_date__gte=2024-02-01T00%3A00%3A00 (it would be great if we can have gte, lte)
• appeal_dtype. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_dtype=13 (it would be great if we could pass a list)
• appeal_country. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_country=7 (it would be great if we could pass a list)
• appeal_region. We would need this for the operational learning dashboard++ e.g.https://goadmin.ifrc.org/api/v2/ops-learning/?appeal_region=2 (it would be great if we could pass a list)

We also would need this type of filtering for the CSV export. Thanks!

[szabozoltan69]

All the above filterings are working for CSV output also (due to this filtering happens before the format conversion).
Also /api/v2/learningtype is created for learning types, giving:

  {
    "key": 1,
    "label": "Lesson learned"
  },
  {
    "key": 2,
    "label": "Challenge"
  }
]

Also you can check the already existing /api/v2/primarysector and /api/v2/per-formcomponent/ endpoints to have sectors and per_components.

[szabozoltan69]

@ypyelab the change is ready and can be tested on Staging.
My test results:

Given records from all: 370 records
11  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?search=MDRBF015'                                                                                  
1   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?search=nationwide'                                                                                
1   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/13977/' #  or similar:                                                                             
1   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?id=13977' # (a 1-element list)                                                                    
3   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?id__in=13977,13976,13975'                                                                         
3   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?id__in=13977,13976,13975&is_validated=true'                                                       
0   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?is_validated=false'                                                                               
176 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?type_validated=1'                                                                                 
194 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?type_validated=2'                                                                                 
2   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_document_id=8992'                                                                          
12  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?sector_validated=3'                                                                               
32  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?sector_validated=4'                                                                               
44  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?sector_validated__in=3,4'                                                                         
21  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?per_component_validated=10'                                                                       
5   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?per_component_validated=44'                                                                       
25  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?per_component_validated__in=10,44'                                                                
11  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?organization_validated=1'                                                                         
362 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?organization_validated=2'                                                                         
370 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?organization_validated__in=1,2'                                                                   
13  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?modified_at__gte=2024-02-01T00:00:00'                                                             
11  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__code=MDRBF015'                                                                       
11  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code=MDRBF015&'                                                                            
24  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__in=MDRBF015,MDRBF014&'                                                               
16  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__num_beneficiaries=5000&'                                                             
172 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__num_beneficiaries__gte=5000&'                                                        
170 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__num_beneficiaries__gte=5000&appeal_code__num_beneficiaries__lte=50000'               
26  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__start_date__gte=2022-12-23T00:00:00&'                                                
26  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__start_date__gte=2022-12-23T00:00:00&appeal_code__start_date__lte=2023-12-23T00:00:00'
344 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__start_date__lte=2022-12-23T00:00:00&'                                                
112 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__dtype=12&'                                                                           
122 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__dtype__in=12,2&'                                                                     
6   'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country=1'                                                                           
49  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country__in=1,21'                                                                    
123 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country__iso=AR'                                                                     
129 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country__iso__in=AR,UA'                                                              
43  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country__iso3=ARM'                                                                   
49  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country__iso3__in=ARM,UKR'                                                           
43  'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__country__name=Armenia'                                                               
103 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__region=3'                                                                            
226 'https://goadmin-stage.ifrc.org/api/v2/ops-learning/?appeal_code__region__in=1,3'