Malformed date in payer_transition CSV data

[thondeboer]

What happened?

In some, but not all individuals in my test run, the payer trainsition END_YEAR is a malformed string.

292278994-08-17T07:12:55Z

start year was correctly formatted in this case, although CSV page indicates this as simple YYYY format, but this is clearly a TIMESTAMP field, so not sure if we should update wiki or it it is intended as a timestamp

Environment

- OS: Linux beast 5.19.0-50-generic #50-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 10 18:24:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
- Java: openjdk version "11.0.20" 2023-07-18

Relevant log output

Here is the complete set of malformed dates


c539a294-07c6-a206-fdc2-cf168d3441f2,ccaa28a9-62d9-4694-3072-3ab93530e08f,2022-09-06T07:00:32Z,292278994-08-17T07:12:55Z,734afbd6-4794-363b-9bc0-6a3981533ed5,,Self,Gabriela205 Niño185
7752c228-b9ff-e6c1-9024-4f3383ad7f8d,230142c5-2a5a-6947-2d56-3c69e53247a9,2022-08-28T10:28:09Z,292278994-08-17T07:12:55Z,df166300-5a78-3502-a46a-832842197811,,Self,Lorena247 Mendoza384
7c2de1a4-5847-fcc9-dad5-eb17f74e93ef,0c6334a8-a2c3-0440-8d82-5e198e03de14,2022-09-01T07:16:38Z,292278994-08-17T07:12:55Z,df166300-5a78-3502-a46a-832842197811,,Self,Dottie687 Dierdre747 Schuster709
dc1fc8f2-7c34-6dd8-d797-8ce8226b9a76,3cfa3c01-6d62-e4e8-ee51-14d979895d27,2022-09-02T04:53:02Z,292278994-08-17T07:12:55Z,26aab0cd-6aba-3e1b-ac5b-05c8867e762c,,Self,Jerold208 Nolan344
a25a7ab6-ef0b-1deb-aa76-6b9519a15332,168f5b88-d6d7-6f2f-36f7-27eb93b8a606,2022-09-02T20:49:14Z,292278994-08-17T07:12:55Z,a735bf55-83e9-331a-899d-a82a60b9f60c,b046940f-1664-3047-bca7-dfa76be352a4,Spouse,Mr. Berríos934
738a0f35-8c16-a8b6-7128-d9784613982f,b9eb0815-9f8f-b92f-a9c0-26a873b4bb59,2022-09-03T18:48:02Z,292278994-08-17T07:12:55Z,df166300-5a78-3502-a46a-832842197811,,Self,Pete38 Waters156
44f45b33-b59d-5694-3d6f-6263a00fa653,0433e304-a470-923d-d682-8003f5e3f9ec,2022-08-30T14:07:03Z,292278994-08-17T07:12:55Z,26aab0cd-6aba-3e1b-ac5b-05c8867e762c,,Spouse,Mr. Schultz619
4000c278-169b-7e54-6e1c-8d7858b5ad65,40b5e146-293b-5e0e-2dc8-1b4550fabdb2,2022-09-02T18:47:54Z,292278994-08-17T07:12:55Z,26aab0cd-6aba-3e1b-ac5b-05c8867e762c,,Self,Loyd638 Deckow585
a617597a-9805-4a67-70a3-1e982695a7e0,d19544c9-2a22-b6f7-5b7c-d18544bfd43e,2022-09-01T04:38:29Z,292278994-08-17T07:12:55Z,df166300-5a78-3502-a46a-832842197811,,Self,Victor265 Cummings51
f0bbc604-6b98-ecc2-d1cc-671ff63c784d,0feb45a1-3b4a-d42a-136a-23e956614f34,2022-08-31T04:58:07Z,292278994-08-17T07:12:55Z,b046940f-1664-3047-bca7-dfa76be352a4,,Guardian,Alysa236 Swift555
0422a0c0-6d13-ae76-1fab-5c8769ca3bf0,a8fcfad9-2c55-2a77-7e3d-a28521300866,2012-08-02T10:03:24Z,292278994-08-17T07:12:55Z,a735bf55-83e9-331a-899d-a82a60b9f60c,b046940f-1664-3047-bca7-dfa76be352a4,Self,Denisha680 Denise470 Hamill307
30bba614-1bc2-820a-2e32-6f529a33643f,5a0626d5-eedc-e132-8ad6-0ac086a976eb,2022-09-01T11:27:56Z,292278994-08-17T07:12:55Z,df166300-5a78-3502-a46a-832842197811,,Self,Benito209 Longoria249
f385ff1b-c3fa-a264-1ef8-be5843cb78b9,f01870c0-9f23-6bf1-5ac9-289a330af9ae,2022-09-03T20:14:47Z,292278994-08-17T07:12:55Z,df166300-5a78-3502-a46a-832842197811,,Self,Francesca650 Boyle917
bee4df1b-08b0-44f6-a1a6-3dcf8640175e,089411e6-a70c-6d5d-5871-6e7a2950c18e,2022-09-05T23:13:46Z,292278994-08-17T07:12:55Z,8fa6c185-e44e-3e34-8bd8-39be8694f4ce,,Spouse,Mr. Beier427
ebd7fe2d-c044-3b7e-d3a0-5b05b01b04f5,,2023-09-02T19:42:32Z,292278994-08-17T07:12:55Z,e03e23c9-4df1-3eb6-a62d-f70f02301496,,,
96565115-02e7-5e78-4482-f3f5b4c3d04a,30d655d1-0377-b251-dab0-574e3ba64246,2022-09-03T02:31:22Z,292278994-08-17T07:12:55Z,26aab0cd-6aba-3e1b-ac5b-05c8867e762c,,Self,Jeramy610 Kautzer186
f9a481e7-b337-29bf-2206-161ff3c2f234,40f463d7-b6ea-9c63-0e12-ec6837f76b02,2022-09-01T00:52:30Z,292278994-08-17T07:12:55Z,a735bf55-83e9-331a-899d-a82a60b9f60c,b046940f-1664-3047-bca7-dfa76be352a4,Self,Nikki942 Zaida719 Keebler762

[jawalonoski]

This is another bug, not a wiki error.

The Imaging study fields are OIDs, not UUIDs, so your wiki fix for that is fine.

But this one is an error.

[jawalonoski]

I managed to reproduce the error.

[arvindshmicrosoft]

I managed to reproduce the error.

Can you share the repro conditions? I'm really curious about this issue. Is it occurring for a specific state and area?

[jawalonoski]

Can you share the repro conditions? I'm really curious about this issue. Is it occurring for a specific state and area?

It has been a while since I looked at this, but I believe it can occur if you also have CCDA export enabled (exporter.ccda.export = true).

In the CCDA exporter, there is a hack to set patient insurance coverage to "No Insurance" if they do not have a coverage record at export time (to avoid an exception), and it sets the expiration of the coverage to Long.MAX_VALUE.

synthea/src/main/java/org/mitre/synthea/export/CCDAExporter.java

Line 78 in 6192ed7

person.coverage.setPlanToNoInsurance(Long.MAX_VALUE);

If you are running both the CCDA exporter and the CSV exporter... and the CCDA exporter runs before the CSV exporter (it always does, see code below), then you can get this error for some patients.

synthea/src/main/java/org/mitre/synthea/export/Exporter.java

Lines 275 to 293 in 6192ed7

	if (Config.getAsBoolean("exporter.ccda.export")) {
	String ccdaXml = CCDAExporter.export(person, stopTime);
	File outDirectory = getOutputFolder("ccda", person);
	Path outFilePath = outDirectory.toPath().resolve(filename(person, fileTag, "xml"));
	writeNewFile(outFilePath, ccdaXml);
	}
	if (Config.getAsBoolean("exporter.json.export")) {
	String json = JSONExporter.export(person);
	File outDirectory = getOutputFolder("json", person);
	Path outFilePath = outDirectory.toPath().resolve(filename(person, fileTag, "json"));
	writeNewFile(outFilePath, json);
	}
	if (Config.getAsBoolean("exporter.csv.export")) {
	try {
	CSVExporter.getInstance().export(person, stopTime);
	} catch (IOException e) {
	e.printStackTrace();
	}
	}

[thondeboer]

Indeed it seems if I only use CSV export, the dates are correct. So lesson is, do each export separate or at least do not nclude CCDA when exporting CSV.

[arvindshmicrosoft]

Many thanks for the clarification @jawalonoski.