Skip to content

Flexporter

Jump to bottom Edit New page
Dylan Hall edited this page Nov 8, 2023 · 15 revisions

⚠️⚠️ WARNING: Here be dragons. This feature is a very rough work in progress. Features that you expect may be missing, and features that do exist may be broken. Proceed with caution. ⚠️⚠️

The Synthea Flexible Exporter a.k.a. "Flexporter" is a utility designed to make it easy to add new fields, values, and resource types to Synthea-generated FHIR, without having to modify the Synthea engine. Primary users of this feature are expected to be Implementation Guide authors and users, electronic clinical quality measure developers and testers, and anyone who needs just a slight tweak to the data exported by Synthea.

As of 2023-03-29, this feature is not yet merged to master, and only available on the flexporter branch: https://github.com/synthetichealth/synthea/tree/flexporter

Approach

The basic idea of the Flexporter is that users will define a series of transformations to apply to Synthea data. A predefined set of transformations, such as "set field X to value Y" or "create a new resource" is intended to cover the majority of use cases while still remaining easy to use.

Usage

In Synthea

./run_synthea -fm mapping_file [-ig ig_folder]

Standalone

The Flexporter can also run standalone to post-process FHIR Bundles.

./run_flexporter -fm mapping_file -s source_fhir [-ig ig_folder]

https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/RunFlexporter.java

Mapping File

The format of the mapping file is as follows:

---
# name is just a friendly name for this mapping
name: Flexporter Mapping

# applicability determines whether this mapping applies to a given file.
# for now the assumption is 1 file = 1 synthea patient bundle.
# this should be a FHIRPath which should return a "truthy" value if the mapping applies
applicability: true

# actions is a list of Action objects. see below for details
actions:
 - name: Apply Profiles
...

Actions

https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/Actions.java

Actions are driven by a key property name on the object. (technical note: this is because the YML library doesn't support discriminators like GSON does for JSON, think like how state types get mapped into a different class by type)

Apply Profiles

Field Name Type Required Description
🗝️ profiles List Yes List of profile/applicability pairs
profile URL Yes Profile URL to be added to resources
applicability FHIRPath Yes FHIRPath to select resources that the profile should be added to

Example:
This example will apply the qicore-patient profile to all Patient resources, and qicore-encounter to all Encounters.

 - name: Apply Profiles
   profiles:
   - profile: http://hl7.org/fhir/us/qicore/StructureDefinition/qicore-patient
     applicability: Patient
   - profile: http://hl7.org/fhir/us/qicore/StructureDefinition/qicore-encounter
     applicability: Encounter

Set Values

Field Name Type Required Description
🗝️ set_values List Yes List of profile/applicability pairs
applicability FHIRPath Yes FHIRPath to select resources to set fields on
fields List Yes List of location/value pairs to set
location FHIRPath Yes FHIRPath to the field to set the value on
value raw value, Value Function, object Yes The value to set on the target, or the function to fetch a value to set on the target
transform Value Transformation No Transformation to apply to the value before setting on the target

Example 1:
This example will read the value from Immunization.occurence[x] and set it on the Immunization.recorded field, for all Immunization resources.

 - name: testSetValues_getField
   set_values:
     - applicability: Immunization
       fields:
         - location: Immunization.recorded
           value: $getField([Immunization.occurrence])

Example 2:
This example will set a full coding, with code/system/display, to Patient.maritalStatus.coding.

 - name: testSetValues_object
   set_values:
     - applicability: Patient
       fields:
         - location: Patient.maritalStatus.coding
           value:
               system: http://snomedct.io
               code: "36629006"
               display: "Legally married (finding)"

Create Resource

Field Name Type Required Description
🗝️ create_resource List Yes List of resource definitions to create
resourceType FHIR resource type Yes Resource type to create (Patient, Encounter, etc)
based_on object No
based_on.resource FHIRPath No FHIRPath to select resources to base the new created resource off of. (Example use case, create an Appointment based on every Encounter)
based_on.module GMF Module Name No The name of a Synthea Module, where an instance of the current resource will be created for each instance of the patient passing through the state
based_on.state GMF Module State No The state to use as the basis for creating the current resource
fields List Yes List of location/value pairs to set on the created resource
writeback List (same options as fields) No List of location/value pairs to set on the selected based_on resource
fields.location FHIRPath Yes FHIRPath to the field to set the value on
fields.value raw value, Value Function, object Yes The value to set on the target, or the function to fetch a value to set on the target
fields.transform Value Transformation No Transformation to apply to the value before setting on the target

Example:
This example will create an instance of ServiceRequest for every instance of Procedure, with the following:

  • ServiceRequest.intent set to "plan"
  • ServiceRequest.encounter.reference set to the same as the source Procedure.encounter.reference
  • ServiceRequest.subject.reference set to a reference to the Patient resource
  • Procedure.basedOn.reference set to a reference to the newly created ServiceRequest
 - name: testCreateResources_createBasedOn
   create_resource: 
     - resourceType: ServiceRequest
       based_on:
         resource: Procedure
       fields:
         - location: ServiceRequest.intent
           value: plan
         - location: ServiceRequest.encounter.reference
           value: $getField([Procedure.encounter.reference])
         - location: ServiceRequest.subject.reference
           value: $findRef([Patient])
       writeback:
         - location: Procedure.basedOn.reference
           value: $setRef([ServiceRequest])

Keep Resources

Field Name Type Required Description
🗝️ keep_resources List<FHIRPath> Yes List of FHIRPath to select resources that should be retained in the bundle, anything not selected by a rule will be removed

Example:
This example will keep only the Patient, Encounter, and Condition resources. All other resources will be removed.

 - name: testKeepResources
   keep_resources:
     - Patient
     - Encounter
     - Condition

Delete Resources

Field Name Type Required Description
🗝️ delete_resources List<FHIRPath> Yes List of FHIRPath to select resources that should be deleted in the bundle, anything selected by a rule will be removed

Example:
This example deletes the Provenance resource:

 - name: testDeleteResources
   delete_resources:
     - Provenance

Execute Script

If the above options aren't sufficient, the Flexporter allows for executing arbitrary JavaScript code against the Bundle. (The ability to load arbitrary libraries is not [yet] supported)

Field Name Type Required Description
🗝️ execute_script List Yes List of script definitions to execute (in order)
function JavaScript code (string) Yes Javascript code including the function to be executed. More than one function may be defined in this field at a time
function_name string Yes Name of the function to be invoked
apply_to "resources" or "bundle" Yes Whether the function should be invoked on the "bundle" as a whole, or on the "resources" each individually

Example:
This example executes two scripts. The first adds a dummy profile to the first resource in the Bundle, the second sets the birthDate on any Patient resources in the Bundle to '2022-02-22'.

 - name: testExecuteScript
   execute_script:
     - apply_to: bundle
       function_name: apply
       function: |
          function apply(bundle) {
            bundle['entry'][0]['resource']['meta'] = {profile: ['http://example.com/dummy-profile']}
          }
     - apply_to: resource
       function_name: apply2
       function: |
         function apply2(resource, bundle) {
           if (resource.resourceType == 'Patient') {
             resource.birthDate = '2022-02-22';
           }
         }

Value Functions

Function Arguments Description
getField FHIRPath pointing to field Get value from a field from current or based-on resource
findValue FHIRPath pointing to resource/field Find a value from another resource within the entire Bundle
setRef Create a reference to the current resource
findRef FHIRPath pointing to resource Find a reference to another resource in the bundle
getAttribute Attribute key Read value from a Synthea attribute. Note this can only be used when running in Synthea, not standalone.
randomCode ValueSet URL Pick a random code from the target ValueSet

Basic naming scheme here is "set" and "get" refer to a particular resource, (ie, the current one, or the resource a new one is being based on) and "find" refers to searching the entire bundle.

Value Transforms

https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/ValueTransforms.java

Function Description
toInstant Convert the given date/time into an instant: YYYY-MM-DDThh:mm:ss.sss+zz:zz. If the given item doesn't have that level of precision, random values will populate the missing pieces.
toDate Convert the given date/time into a date: YYYY-MM-DD. If the given item doesn't have that level of precision, random values will populate the missing pieces.
toDateTime Convert the given date/time into a dateTime: YYYY-MM-DDThh:mm:ss+zz:zz. If the given item doesn't have that level of precision, random values will populate the missing pieces.
toTime Convert the given date/time into a time: hh:mm:ss. If the given item doesn't have that level of precision, random values will populate the missing pieces.

Full Examples

mCODE

actions:
 - name: Apply Profiles
   profiles:
   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-patient
     applicability: Patient

   - profile: http://example.com/bloodpressure
     applicability: Observation.code.coding.where($this.code = '85354-9')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-stage-group
     applicability: Observation.code.coding.where($this.code = '21908-9')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-primary-tumor-category
     applicability: Observation.code.coding.where($this.code = '21905-5')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-regional-nodes-category
     applicability: Observation.code.coding.where($this.code = '21906-3')

   - profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-distant-metastases-category
     applicability: Observation.code.coding.where($this.code = '21907-1')


 - name: Create TNM References
   set_values:
     - applicability: Observation.code.coding.where($this.code = '21908-9')
       fields:
         - location: Observation.hasMember.where(display='Tumor Category').reference
           value: $findRef([Observation.code.coding.where($this.code = '21905-5')])
         - location: Observation.hasMember.where(display='Nodes Category').reference
           value: $findRef([Observation.code.coding.where($this.code = '21906-3')])
         - location: Observation.hasMember.where(display='Metastases Category').reference
           value: $findRef([Observation.code.coding.where($this.code = '21907-1')])

Known Limitations

  • ValueSets can only be loaded from the IG when they are fully defined and expanded within the IG. The existing Synthea terminology support works, but you have to provide your own terminology server.
  • Resources removed by keep_resources or delete_resources will still be referenced by other resources. Need to define the best approach for what to do here. (Maybe options? "Cascade" = delete resources that reference deleted resources, "dangle" = leave dangling references, "prune field" = remove the reference field but leave the rest of the resource)
  • Copying objects from one field to another doesn't currently work, only primitives. The workaround for now is to list out all the individual sub-fields, but this doesn't work well when the fields that are populated might change. One hack I tried to implement was to deconstruct the provided object into its component fields, but that didn't work because HAPI doesn't provide a way to serialize generic FHIR objects, only Resources. (Ex, you can't serialize a CodeableConcept) See Actions.createFhirPathMapping https://github.com/synthetichealth/synthea/blob/d08c545d168610df7ba42326477a14e17daaa78c/src/main/java/org/mitre/synthea/export/flexporter/Actions.java#L176
  • Sometimes YAML needs to be told what an object is. For example things that look like dates will become java Date objects unless explicitly marked as strings:
 - name: testSetValues
   set_values:
     - applicability: Patient
       fields:
         - location: Patient.birthDate
           value: !!str "1987-06-05"

Technical Notes

At the core of the flexporter is the CustomFHIRPathResourceGeneratorR4 class - this is based on a community contribution within HAPI which we've added some additional functionality to. It does some magic using HAPI's internal reflection (on top of Java reflection), so it can be a bear to wrap your head around. Debugging and stepping through examples makes it easier. https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/CustomFHIRPathResourceGeneratorR4.java

Add a custom footer
Clone this wiki locally