CU-86956du3q revisit regression #470
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make sure to compare the correct annotation, not just hoping for any CUI annotated to match the one we are looking for. Output the specifics of the type of match that was found: - Identical - Bigger / smaller span - Random overlap - Parents / grandparetns, or children Add strictness options to summary (success / failure).
Remove 'Failure reason' and 'Failre descriptor' - now using Finding instead. Remove simplified success/failure metrics wherever relevant. Fix tests that relied on old logic and fix test-time replacement/cui location.
|
Task linked: CU-86956du3q Revisit regression |
…es for multiple placeholders
And more specifically, treat each one as their own sub-case
…sion test failure
mart-r
force-pushed
the
CU-86956du3q-revisit-regression2
branch
from
f6498ae
to
f70386a
Compare
|
As a comment, I've now added a few new files and a new step to the main workflow. The files include:
The workflow addition does the following:
EDIT: |
…time to avoid conflicts with other python versions running in parallel
…ation) and add args to output upon regression checking
…ter iteration) and add args to output upon regression checking" This reverts commit 4bf3089.
…reation model for regression
tomolopolis
reviewed
Aug 28, 2024
medcat/utils/regression/utils.py
Outdated
| string is returned. | ||
| If it's longer, the first `keep_front` are kept, then the number of chars | ||
| is included in brackets (e.g `" [123 chars] "`), and finally the last | ||
| `keeo_rear` characters are included. |
tomolopolis
reviewed
Aug 28, 2024
medcat/utils/regression/utils.py
Outdated
| def add_doc_strings_to_enum(cls: Type[Enum]) -> None: | ||
| """Add doc strings to Enum as they are described in code right below each constant. | ||
|
|
||
| The way python works means that the doc strins defined after an Enum constant do not |
tomolopolis
approved these changes
Aug 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Doing some major changes to the regression tests.
The idea would be to be able to actually put this to use and allow easy sanity checking of model performance.
WIP changes:
%s)Include useful Gold Standard dataset for regression[Will be addressed later]Some preliminary results from sample data
I've got an example (simple) dataset that I've run this through for the 2023 SNOMED model with the following results:And the 2024-06 model (with only MIMIC-IV training):
UPDATE with same model, but updated code (2024-06-07 at around 17.30):
[Summary] Final results
Result summary for the 2023 SNOMED model:
Result sumamry for the Snomed2024-06 model:
[FULL] Final results with the 2023 SNOMED model
[FULL] Final results with the SNOMED2024-06 model
The JSON output example (truncated)
A truncated example json output.
It shows the overall findings along with the per-phrase findings.
But examples are only (currently) kept for the per-phrase parts to avoid duplicating data.