Not All Observations Being Exported to CSV

[mspence-nhs]

What happened?

I have found in various experiments that not all expected observations seem to be recorded when exporting to csv. For example, in the minimal example module below all patients are prescribed a fake drug, then a test observation is made. However, when I run this module every patients is prescribed the drug as expected, but only ~10% have the "test obs" appearing in their records.

  "name": "fake module",
  "remarks": [
    "A blank module"
  ],
  "states": {
    "Initial": {
      "type": "Initial",
      "direct_transition": "start_encounter",
      "name": "Initial"
    },
    "Terminal": {
      "type": "Terminal",
      "name": "Terminal"
    },
    "start_encounter": {
      "type": "Encounter",
      "reason": "",
      "telemedicine_possibility": "possible",
      "direct_transition": "obs",
      "name": "start_encounter",
      "wellness": true
    },
    "obs": {
      "type": "Observation",
      "category": "vital-signs",
      "unit": "",
      "codes": [
        {
          "system": "LOINC",
          "code": 0,
          "display": "test observation",
          "value_set": ""
        }
      ],
      "distribution": {
        "kind": "EXACT",
        "round": false,
        "parameters": {
          "value": 1
        }
      },
      "direct_transition": "fake_medication",
      "name": "obs"
    },
    "fake_medication": {
      "type": "MedicationOrder",
      "codes": [
        {
          "system": "RxNorm",
          "code": "1234",
          "display": "fake drug",
          "value_set": ""
        }
      ],
      "name": "fake_medication",
      "direct_transition": "End Encounter"
    },
    "End Encounter": {
      "type": "EncounterEnd",
      "direct_transition": "Terminal",
      "name": "End Encounter"
    }
  },
  "gmf_version": 2
}

I initially noticed this behaviour when working with the hypertension module, and was finding fewer blood pressure observations than expected. Any idea why this could be happening?

Thanks!

Environment

- OS: Ubuntu 22.04.1 LTS"
- Java: openjdk version 17.0.10

Relevant log output

No response

[jawalonoski]

It may be worth looking at Examplitis and reading the Examplitis walkthrough.

At the initial state, all of your patients are newborn babies -- who then wait for their next wellness encounter, which is going to happen within the first month of their life. In addition to whatever the wellness encounter module does, your module adds an observation and a medication, ends the encounter, and never runs again for the rest of the patients life.

If you are using the default settings, you're only exporting 10 years of history. See exporter.years_of_history in the src/main/resources/synthea.properties file:

# number of years of history to keep in exported records, anything older than this may be filtered out
# set years_of_history = 0 to skip filtering altogether and keep the entire history
exporter.years_of_history = 10

So, any patients older than age 10 probably won't have your data in their records.

You can either change the exporter.years_of_history setting, change your module to have some type of Delay, Guard, or loop so that it doesn't run on newborn babies, or both.

[mspence-nhs]

Hey Jason, thanks so much for your quick reply!

I've tried adding an age guard and have increased years of recorded history. It looks like I'm still getting the same problem though.

For reproducibility I'm running this command, ./run_synthea -p 1000 --exporter.csv.export true --exporter.years_of_history 0, with the module below.

The resulting medications table shows all patients being prescribed the "fake drug", while only 126 have the "test obs" observation.

Please let me know what I could be missing here. Thanks so much!

{
  "name": "fake module",
  "remarks": [
    "A blank module"
  ],
  "states": {
    "Initial": {
      "type": "Initial",
      "direct_transition": "New_State",
      "name": "Initial"
    },
    "Terminal": {
      "type": "Terminal",
      "name": "Terminal"
    },
    "start_encounter": {
      "type": "Encounter",
      "reason": "",
      "telemedicine_possibility": "possible",
      "direct_transition": "obs",
      "name": "start_encounter",
      "wellness": true
    },
    "obs": {
      "type": "Observation",
      "category": "vital-signs",
      "unit": "",
      "codes": [
        {
          "system": "LOINC",
          "code": "1234",
          "display": "test obs",
          "value_set": ""
        }
      ],
      "distribution": {
        "kind": "EXACT",
        "round": false,
        "parameters": {
          "value": 1
        }
      },
      "direct_transition": "fake_medication",
      "name": "obs"
    },
    "fake_medication": {
      "type": "MedicationOrder",
      "codes": [
        {
          "system": "RxNorm",
          "code": "1234",
          "display": "fake drug",
          "value_set": ""
        }
      ],
      "name": "fake_medication",
      "direct_transition": "Terminal"
    },
    "New_State": {
      "type": "Guard",
      "allow": {
        "condition_type": "Age",
        "operator": ">",
        "quantity": 20,
        "unit": "years"
      },
      "direct_transition": "start_encounter",
      "name": "New_State"
    }
  },
  "gmf_version": 2
}

[dehall]

Looks like this is actually a bug - the issue is the command line --exporter.years_of_history setting isn't working. Thanks for reporting this, I can try to get this resolved in the next couple days.

In the meantime, if you change that config setting in src/main/resources/synthea.properties it should work as expected

[jawalonoski]

@dehall See d10d07e

[mspence-nhs]

Changing the properties file worked for me. Thanks both for quick responses!

[JustDoItSascha]

Hey, I have exactly the same problem and it was driving me crazy for two days now. I just the build .jar file from Friday and I've set the exporter.years_of_history = 20 inside my .properties file, but it's still the default of 10 years. I know that because I used the -a 30-30 config to make only patients with exact age of 30 and then I my module I used a guard to determine the age when my observation is happening.

If I set it to > 19 I see nothing, if I set it to > 20 I see my Observation. What am I doing wrong?

That is my synthea.properties:

generate.log_patients.detail = detailed
exporter.years_of_history = 20

Edit: Please forget it. I took the latest version from today and the problem is solved.