-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CU-86956du3q revisit regression #470
Conversation
Make sure to compare the correct annotation, not just hoping for any CUI annotated to match the one we are looking for. Output the specifics of the type of match that was found: - Identical - Bigger / smaller span - Random overlap - Parents / grandparetns, or children Add strictness options to summary (success / failure).
Remove 'Failure reason' and 'Failre descriptor' - now using Finding instead. Remove simplified success/failure metrics wherever relevant. Fix tests that relied on old logic and fix test-time replacement/cui location.
Task linked: CU-86956du3q Revisit regression |
…es for multiple placeholders
And more specifically, treat each one as their own sub-case
…sion test failure
f6498ae
to
f70386a
Compare
As a comment, I've now added a few new files and a new step to the main workflow. The files include:
The workflow addition does the following:
EDIT: |
…time to avoid conflicts with other python versions running in parallel
…ation) and add args to output upon regression checking
…ter iteration) and add args to output upon regression checking" This reverts commit 4bf3089.
…reation model for regression
medcat/utils/regression/utils.py
Outdated
string is returned. | ||
If it's longer, the first `keep_front` are kept, then the number of chars | ||
is included in brackets (e.g `" [123 chars] "`), and finally the last | ||
`keeo_rear` characters are included. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo keeo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
eeo
medcat/utils/regression/utils.py
Outdated
def add_doc_strings_to_enum(cls: Type[Enum]) -> None: | ||
"""Add doc strings to Enum as they are described in code right below each constant. | ||
|
||
The way python works means that the doc strins defined after an Enum constant do not |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo strins
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm - just some doc string typos
Doing some major changes to the regression tests.
The idea would be to be able to actually put this to use and allow easy sanity checking of model performance.
WIP changes:
%s
)Include useful Gold Standard dataset for regression[Will be addressed later]Some preliminary results from sample data
I've got an example (simple) dataset that I've run this through for the 2023 SNOMED model with the following results:And the 2024-06 model (with only MIMIC-IV training):
UPDATE with same model, but updated code (2024-06-07 at around 17.30):
[Summary] Final results
Result summary for the 2023 SNOMED model:
Result sumamry for the Snomed2024-06 model:
[FULL] Final results with the 2023 SNOMED model
[FULL] Final results with the SNOMED2024-06 model
The JSON output example (truncated)
A truncated example json output.
It shows the overall findings along with the per-phrase findings.
But examples are only (currently) kept for the per-phrase parts to avoid duplicating data.