[New Feature] Tracking and Configuration of Candidate's Diagnosis Evolution #7560

jesscall · 2021-08-31T20:15:35Z

Brief summary of changes

This PR adds a way for project administrators to define and configure diagnosis trajectories for their project. The aim of this new feature is multifaceted; it allows study coordinators to

Track the evolution of a patient's diagnosis over time
Determine (& query) the candidate's latest diagnosis is without having to manually compare the longitudinal data

Interest for this feature was expressed by multiple projects during the DCC meeting on July 7th.

What is included in this PR?

1. A configuration page for diagnosis evolution

2. A tab in candidate parameters to view details on the evolution of the diagnosis.

3. Logic to propagate the latest diagnosis data as instruments are updated.
4. A script that all projects should run when configuring Dx Trajectories and updating their instances.
5. Updates the demographic import to couchDB so the latest diagnosis can be queried.

Info

A Diagnosis Trajectory is defined by a source field, or group of source fields, that are found within the test battery.
A patient's Diagnosis Evolution is defined by the progression of changing diagnoses over time. The evolution is determined by the order of configured Diagnosis Trajectories.

There are two ways that the Latest Diagnosis of the candidate can be updated. First, the script in this PR can be run. Secondly, when ANY instrument part of a configured diagnosis track is saved, the latest diagnosis is determined and updated in the candidate table.

EX:

Have you updated related documentation?

Testing instructions (if applicable)

fetch jesscall && git checkout jesscall/2021_07_27_diagnosis_evolution
cat SQL/New_patches/2021-07-28_diagnosis_evolution.sql | mysql
make dev
Follow instructions in configuration & caniddate parameters TestPlan.md files included in this PR
After setting up the configuration for a diagnosis evolution please do the following:
php tools/data_integrity/update_candidate_latest_diagnosis.php
Ensure that updating data in a configured instrument updates the latest diagnosis in the candidate table (if it's the latest) and the candidate parameters tab.

Link(s) to related PR(s)

Blocked by [Utility] Refactor getVisitInstruments function #7514

zaliqarosli

this is really cool! setting it all up works well. i have a couple bugs, comments and a request.

some bugs:

if I click on the trajectory tab I just created, then back to 'New Diagnosis Trajectory' tab, the form doesn't re-render as a new form, it still shows the trajectory I just created

reset button doesn't work

request: is there a possibility to add a timestamp?

questions: is candidate_latest_diagnosis really just candidate_diagnosis_rel? i.e. do all the entries in the Diagnosis Evolution table on the front-end get stored in there?

zaliqarosli

1 more bug, the second field from the group of source fields is not getting saved in the LatestDiagnosis column

SQL/New_patches/2021-07-28_diagnosis_evolution.sql

modules/configuration/templates/form_configuration.tpl

tools/data_integrity/update_candidate_latest_diagnosis.php

modules/candidate_parameters/jsx/DiagnosisEvolution.js

jesscall · 2021-10-05T14:49:44Z

Hey @zaliqarosli
I'm working on reactifying this PR as well as adding a proper endpoint as per @driusan request in a private convo.

I'll try to get to your feedbacks afterwards and see what is still relevant.

As for the rendering of diagnosis for QPN, that's a tough one. I imagine they are dropdown options? That sort of information isn't captured by data dictionary / parameter type. I guess you could hard-code the mapping or use grep on the instrument linst file... But these are both pretty hacky solutions. Maybe @ridz1208 has a suggestion

jesscall · 2021-10-05T20:23:17Z

@zaliqarosli
I've updated the PR. It is now reactified, using a php endpoint rather than ajax and I've also addressed your comments :)
Feel free to re-review at your convenience

@driusan your requested changes have been implemented

zaliqarosli

should the Source Field be the database field name without the instrument prepended to it? i believe that was how it was implemented before reactification

modules/configuration/jsx/DiagnosisEvolution.js

modules/candidate_parameters/ajax/getData.php

jesscall · 2021-10-06T14:37:13Z

should the Source Field be the database field name without the instrument prepended to it? i believe that was how it was implemented before reactification

@zaliqarosli

So in switching from templates to React, the datalist is implemented differently with our predefined react components. Before, you could search by the field name or the table name because both values were displayed in the select option. With the react component you can only have one of the two. So either:

The field name alone - EX: weight_lbs (parameter_type -> SourceField)
The compounded table name and field name - EX: bmi_weight_lbs (parameter_type -> Name)

I opted for (2) to make searching by instrument an option for the user. If you want this changed, let me know.

jesscall · 2021-10-25T18:07:28Z

Latest Changes:

Added timestamp to track last update of latest diagnosis
reverted back to using the sourceField names that are not prepended with instrument name.

jesscall · 2021-11-01T20:37:44Z

Refactored this PR to track the diagnosis evolution as a history rather than the current latest diagnosis. Also implemented @ridz1208 suggestion for showing the confirmation status of a diagnosis (whether the diagnosis comes from an approved visit or not).

SQL/New_patches/2021-07-28_diagnosis_evolution.sql

php/libraries/NDB_BVL_Instrument.class.inc

driusan · 2022-06-06T16:37:52Z

@jesscall what is the status of this?

driusan · 2023-11-21T19:54:56Z

I'm not sure who's maintaining this PR now but the tests on it are failing and there's a few eslint warnings that need to be fixed (the error ones are the reason it's failing)

/home/runner/work/Loris/Loris/modules/configuration/jsx/DiagnosisEvolution.js
  125:1   error    The type 'int' is undefined                                           jsdoc/no-undefined-types
  303:1   error    The type 'bool' is undefined                                          jsdoc/no-undefined-types
  399:13  warning  Unexpected console statement                                          no-console
  494:13  warning  Unexpected console statement                                          no-console
  504:1   warning  @param "id" does not match an existing function parameter             jsdoc/check-param-names
  548:1   warning  @param "pendingValKey" does not match an existing function parameter  jsdoc/check-param-names

zaliqarosli · 2023-11-21T20:01:21Z

@driusan ya im working on it

driusan · 2023-12-06T18:07:24Z

I am getting 2 errors and a blank page on the configuration -> Diagnosis Evolution page:

Warning: ReactDOM.render is no longer supported in React 18. Use createRoot instead. Until you switch to the new API, your app will behave as if it's running React 17. Learn more: https://reactjs.org/link/switch-to-createroot [react-dom.development.js:10:1247](http://localhost:8000/vendor/js/react/react-dom.development.js)

Uncaught ReferenceError: FormElement is not defined
    renderDiagnosisForm DiagnosisEvolution.js:158
    render DiagnosisEvolution.js:291
    React 16
    componentDidMount DiagnosisEvolution.js:54
    promise callback*componentDidMount DiagnosisEvolution.js:54
    React 12
    <anonymous> DiagnosisEvolution.js:582
    EventListener.handleEvent* DiagnosisEvolution.js:581
    <anonymous> DiagnosisEvolution.js:6
    <anonymous> DiagnosisEvolution.js:6
[DiagnosisEvolution.js:158:20](webpack://lorisjs.configuration/modules/configuration/jsx/DiagnosisEvolution.js)
    renderDiagnosisForm DiagnosisEvolution.js:158
    render DiagnosisEvolution.js:291
    React 11
    performSyncWorkOnRoot self-hosted:1406
    React 5
    componentDidMount DiagnosisEvolution.js:54
    (Async: promise callback)
    componentDidMount DiagnosisEvolution.js:54
    React 7
    performSyncWorkOnRoot self-hosted:1406
    React 5
    <anonymous> DiagnosisEvolution.js:582
    (Async: EventListener.handleEvent)
    <anonymous> DiagnosisEvolution.js:581
    <anonymous> DiagnosisEvolution.js:6
    <anonymous> DiagnosisEvolution.js:6

The first seems to be because something is written using an old/deprecated style from react. The second is because import statements from jsx/Form are missing (I'm getting similar errors on the candidate_parameters page but I'll fix those on the other tabs and send a PR since I think that's broken on main.)

driusan · 2023-12-06T18:08:54Z

Actually it looks like the imports are already there on main. The branch needs to be rebased so that the candidate_parameters loads (and the new tab may need to be fixed.)

author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561499 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561457 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561433 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561346 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561321 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561301 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699561256 -0500 parent ad0460c author jesscall <[email protected]> 1627406855 -0400 committer zaliqarosli <[email protected]> 1699560890 -0500 [Diagnosis Evolution] First commit [Diagnosis Evolution] Validation & progress start cand param cand param tab more progress, getLatestDx, multiple sourceFields switching to propagate data - not done propagate data done, caveat script done schema test plans more documentation Added LatestDX to CouchDB Demographics import bug fix on candidate get latest source phpcs fixing errors added project to config and cand param, need to revamp sql structure progress on the sql structure and reworking logic multi-project support implemented Amend CouchDB Demographics Import for multiproject support of latest Dx phpcs progress on reactification progress reactification & php endpoint cleanup addressing zal's comments align buttons timestamp reverted back to sourcefield from parameter type phan starting refactor revamped react, backend refactored to dx history and confrimed logic completely reworked, front end finished. Need to update documentation and DQT imports add configured order to cand param couchDB demographics import updated phpcs propagate logic fix utility fxn to update confrimation status some lil fixes changes requested & changing confirmed = passed visit changing db initialization phan Apply zal's suggestions from code review Co-authored-by: Zaliqa <[email protected]> Apply zal's suggestions to schema amend param comment Update php/libraries/Utility.class.inc Update php/libraries/NDB_BVL_Instrument.class.inc Update modules/configuration/php/diagnosis.class.inc Update modules/candidate_parameters/ajax/getData.php Update SQL/New_patches/2021-07-28_diagnosis_evolution.sql Update SQL/0000-00-00-schema.sql rm links to other pages Co-authored-by: Zaliqa <[email protected]> Apply suggestions from code review Co-authored-by: Zaliqa <[email protected]> Update modules/configuration/php/diagnosis.class.inc travis update diagnosis if at least 1 field has data Co-authored-by: Zaliqa <[email protected]> zal's comment about pselect dont duplicate same diagnosis if confirmed Update tools/data_integrity/update_candidate_latest_diagnosis.php Co-authored-by: Zaliqa <[email protected]> lintphp fix null checks add approval status check cleanup and fix reset restrict saving diagnosis evolution for cands projects check name conflict as case insensitive Update tools/CouchDB_Import_Demographics.php To avoid error with projects that have a space in their name Co-authored-by: CamilleBeau <[email protected]> Update tools/CouchDB_Import_Demographics.php To avoid error with projects that have a space in their name Co-authored-by: CamilleBeau <[email protected]> Update modules/configuration/php/diagnosis.class.inc This caused a bug on CCNA when trying to set diagnosis evolution if the visits haven't yet been started, so I did a select from test_battery instead of session Co-authored-by: CamilleBeau <[email protected]> Update modules/configuration/php/diagnosis.class.inc Co-authored-by: CamilleBeau <[email protected]> cleanup declare DB cleanup repeat declaration add Delete button [Diagnosis Evolution] First commit [Diagnosis Evolution] Validation & progress cand param tab more progress, getLatestDx, multiple sourceFields schema more documentation Added LatestDX to CouchDB Demographics import phpcs added project to config and cand param, need to revamp sql structure multi-project support implemented Amend CouchDB Demographics Import for multiproject support of latest Dx phpcs progress on reactification progress reactification & php endpoint cleanup addressing zal's comments align buttons timestamp reverted back to sourcefield from parameter type starting refactor logic completely reworked, front end finished. Need to update documentation and DQT imports propagate logic fix some lil fixes changes requested & changing confirmed = passed visit Apply zal's suggestions to schema amend param comment Update php/libraries/Utility.class.inc Update php/libraries/NDB_BVL_Instrument.class.inc Update SQL/New_patches/2021-07-28_diagnosis_evolution.sql Update SQL/0000-00-00-schema.sql rm links to other pages Co-authored-by: Zaliqa <[email protected]> Update modules/configuration/php/diagnosis.class.inc travis zal's comment about pselect dont duplicate same diagnosis if confirmed Update tools/data_integrity/update_candidate_latest_diagnosis.php Co-authored-by: Zaliqa <[email protected]> lintphp fix null checks add approval status check restrict saving diagnosis evolution for cands projects check name conflict as case insensitive Update tools/CouchDB_Import_Demographics.php To avoid error with projects that have a space in their name Co-authored-by: CamilleBeau <[email protected]> Update modules/configuration/php/diagnosis.class.inc Co-authored-by: CamilleBeau <[email protected]> declare DB cleanup repeat declaration phpcs Test suite changes fix action declaration whitespace Update modules/configuration/php/diagnosis.class.inc Update modules/configuration/test/TestPlan.md Co-authored-by: Zaliqa <[email protected]> Update modules/candidate_parameters/test/TestPlan.md Co-authored-by: Zaliqa <[email protected]> Fix Delete & testplan Cleanup, required elenents error, clearing form phpcs typo fix project checking in script phpcs Fix resetting of old diagnosis trajectories Add SubmitURL SubmitUrl proptype

delete debuggin comment

zaliqarosli · 2023-12-07T02:52:47Z

@driusan fixed, and checks passed. could you have another look?
@CamilleBeau would you be able to do a final quick run through test? i had to make some extra changes after the rebase (see last 4 commits)

driusan

This is pretty cool, just some small comments.

I also find the "SearchableDropdown" on the configuration page confusing because it looks like a text field.

SQL/0000-00-00-schema.sql

driusan · 2023-12-08T14:27:39Z

modules/candidate_parameters/jsx/DiagnosisEvolution.js

+          <p>This diagnosis is <strong style={{color: 'green'}}>
+            confirmed</strong>.
+            A confirmed diagnosis is one that belongs to a visit
+            that has been sent to DCC for approval.


I don't understand this criteria. Why does sending to DCC for approval mean the diagnosis is confirmed?

that's the workflow that was agreed on during a loris meeting, and what was implemented by CCNA. sending a timepoint to DCC makes the diagnosis for that configured visit confirmed, before that the diagnosis isn't confirmed

modules/candidate_parameters/jsx/DiagnosisEvolution.js

modules/configuration/php/diagnosis.class.inc

zaliqarosli · 2023-12-14T20:15:06Z

@driusan the SearchableDropdown being used is just the one that is in the jsx form elements file so its UI can be targeted in a separate if necessary. I added the FK for test name, ready for your review/testing again

After We merge the diagnosis new feature into 26.0-release([New Feature] Tracking and Configuration of Candidate's Diagnosis Evolution (#7560)), it cause a bug. if diagnosis is empty. the whole candidate param page will be broken.

jesscall requested review from CamilleBeau, ridz1208 and LeighMac August 31, 2021 20:16

zaliqarosli self-requested a review September 14, 2021 21:49

zaliqarosli requested changes Sep 30, 2021

View reviewed changes

SQL/New_patches/2021-07-28_diagnosis_evolution.sql Outdated Show resolved Hide resolved

zaliqarosli reviewed Oct 2, 2021

View reviewed changes

modules/configuration/templates/form_configuration.tpl Outdated Show resolved Hide resolved

zaliqarosli reviewed Oct 5, 2021

View reviewed changes

tools/data_integrity/update_candidate_latest_diagnosis.php Outdated Show resolved Hide resolved

zaliqarosli reviewed Oct 5, 2021

View reviewed changes

modules/candidate_parameters/jsx/DiagnosisEvolution.js Outdated Show resolved Hide resolved

jesscall added the State: Needs work PR awaiting additional work by the author to proceed label Oct 5, 2021

jesscall force-pushed the 2021_07_27_diagnosis_evolution branch from e68b0e0 to 35fea1d Compare October 5, 2021 20:08

jesscall removed the State: Needs work PR awaiting additional work by the author to proceed label Oct 5, 2021

jesscall requested a review from zaliqarosli October 5, 2021 20:23

zaliqarosli reviewed Oct 5, 2021

View reviewed changes

modules/configuration/jsx/DiagnosisEvolution.js Outdated Show resolved Hide resolved

zaliqarosli reviewed Oct 5, 2021

View reviewed changes

modules/candidate_parameters/ajax/getData.php Show resolved Hide resolved

zaliqarosli requested changes Nov 8, 2021

View reviewed changes

SQL/New_patches/2021-07-28_diagnosis_evolution.sql Outdated Show resolved Hide resolved

SQL/New_patches/2021-07-28_diagnosis_evolution.sql Outdated Show resolved Hide resolved

zaliqarosli reviewed Nov 8, 2021

View reviewed changes

php/libraries/NDB_BVL_Instrument.class.inc Outdated Show resolved Hide resolved

ridz1208 removed their request for review January 10, 2022 18:12

zaliqarosli force-pushed the 2021_07_27_diagnosis_evolution branch from fb9b34e to b0b36b0 Compare November 21, 2023 20:25

driusan assigned driusan and unassigned zaliqarosli Dec 5, 2023

jesscall and others added 13 commits December 6, 2023 20:04

fix rebase

53e7108

fix rebase

1cf5ac0

replace getAllInstruments

c8a6dba

fix resetting added source fields

888e2e0

fix sourcefield error issue

cb42a56

lint

94be4e3

specify diagnosis is confirmed when sent to DCC

9b6f6df

Update tools/CouchDB_Import_Demographics.php

4f3585e

delete debuggin comment

fix tests

5450f7d

update React render + add import statements

d37e65f

trim fullname to prevent \n going to json

07126cf

fix jsx console warnings

e3b0cd2

zaliqarosli force-pushed the 2021_07_27_diagnosis_evolution branch from b0b36b0 to e3b0cd2 Compare December 7, 2023 02:31

lint

ec9bf33

driusan reviewed Dec 8, 2023

View reviewed changes

add FK for testname

4339563

driusan merged commit a0a8187 into aces:main Jan 11, 2024
19 checks passed

ridz1208 added this to the 26.0.0 milestone Apr 9, 2024

kongtiaowang mentioned this pull request Sep 26, 2024

[candidate parameters] diagnosis fix #9365

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[New Feature] Tracking and Configuration of Candidate's Diagnosis Evolution #7560

[New Feature] Tracking and Configuration of Candidate's Diagnosis Evolution #7560

jesscall commented Aug 31, 2021 •

edited

Loading

zaliqarosli left a comment

zaliqarosli left a comment

jesscall commented Oct 5, 2021

jesscall commented Oct 5, 2021 •

edited

Loading

zaliqarosli left a comment

jesscall commented Oct 6, 2021

jesscall commented Oct 25, 2021 •

edited

Loading

jesscall commented Nov 1, 2021

driusan commented Jun 6, 2022

driusan commented Nov 21, 2023

zaliqarosli commented Nov 21, 2023

driusan commented Dec 6, 2023

driusan commented Dec 6, 2023

zaliqarosli commented Dec 7, 2023

driusan left a comment

driusan Dec 8, 2023

zaliqarosli Dec 13, 2023

zaliqarosli commented Dec 14, 2023

[New Feature] Tracking and Configuration of Candidate's Diagnosis Evolution #7560

[New Feature] Tracking and Configuration of Candidate's Diagnosis Evolution #7560

Conversation

jesscall commented Aug 31, 2021 • edited Loading

Brief summary of changes

What is included in this PR?

Info

Testing instructions (if applicable)

Link(s) to related PR(s)

zaliqarosli left a comment

Choose a reason for hiding this comment

zaliqarosli left a comment

Choose a reason for hiding this comment

jesscall commented Oct 5, 2021

jesscall commented Oct 5, 2021 • edited Loading

zaliqarosli left a comment

Choose a reason for hiding this comment

jesscall commented Oct 6, 2021

jesscall commented Oct 25, 2021 • edited Loading

jesscall commented Nov 1, 2021

driusan commented Jun 6, 2022

driusan commented Nov 21, 2023

zaliqarosli commented Nov 21, 2023

driusan commented Dec 6, 2023

driusan commented Dec 6, 2023

zaliqarosli commented Dec 7, 2023

driusan left a comment

Choose a reason for hiding this comment

driusan Dec 8, 2023

Choose a reason for hiding this comment

zaliqarosli Dec 13, 2023

Choose a reason for hiding this comment

zaliqarosli commented Dec 14, 2023

jesscall commented Aug 31, 2021 •

edited

Loading

jesscall commented Oct 5, 2021 •

edited

Loading

jesscall commented Oct 25, 2021 •

edited

Loading