Investigate why GCS data is being saved without meta info

[trent-codecov]

https://sentry.slack.com/archives/C04M5GE2R5M/p1689792185704099 context

[giovanni-guidini]

I checked several commits in several repos and they all had the "meta" info.
However in the logs we do see quite a few "meta info not found in encoded SessionArray" (see here)

That hasn't happened for the last 2 days tho.

[trent-codecov]

Marking as in review. Please check it again before the end of sprint and lets close pending more cases arising.

[giovanni-guidini]

The are still cases of this happening. It's like a silent error because it doesn't matter anymore since the fix. But I'll add some log lines to help us debug where the problem is.

[giovanni-guidini]

You can play around the logs with the following messages:

• "Saving files_array data without meta"
  indicates worker saved report_details data and the stringified data doesn't include the substring "meta"
• "Setting files_array from the API"
  indicates the api is saving report_details data (which should not happen)
• "meta info not found in encoded SessionArray"
  the original warning message that sparked this whole thing

From what I see in the logs we had 2 times data without meta (when creating the report details, the data was []. In the same period 0 times data being saved by the api. And a staggering 57K times the warning.

So I think this leads us to believe that:

1. Data is being saved correctly but never updated (by the worker)
2. We're doing some strange transformation to the data that is making it not have the "meta" info, some how
3. There's another edge case that I'm missing that is the actual cause of this issue, because point 2 does seem unlikely

UPDATE
After a few days the trend above seems to be confirmed. We will need to investigate further the process of retrieving data from API.

[giovanni-guidini]

I reproduced this locally. The thing that uses this is the sunburst chart in the UI (see image)

Now it starts to get interesting.

I changed the retrieve function to do this:

# Roughly https://github.com/codecov/codecov-api/blob/14dfd7c868e34a840387f9529b51194924c2a669/utils/model_utils.py#L87
@lru_cache(maxsize=1)
    def _get_value_from_archive(self, obj):
        repository = obj.get_repository()
        archive_service = ArchiveService(repository=repository)
        archive_field = getattr(obj, self.archive_field_name)
        if archive_field:
            try:
                file_str = archive_service.read_file(archive_field)
                if self.public_name == 'files_array':
                    rehydrated_data = self.rehydrate_fn(obj, json.loads(file_str))
                    for file in rehydrated_data:
                        if "session_totals" not in file or "meta" not in file['session_totals']:
                            log.warning("META NOT IN SESSION_TOTALS", extra=dict(data=file))
                return self.rehydrate_fn(obj, json.loads(file_str))
            except FileNotInStorageError:
                log.error(
                    "Archive enabled field not in storage",
                    extra=dict(
                        storage_path=archive_field,
                        object_id=obj.id,
                        commit=obj.get_commitid(),
                    ),
                )
...

so we would get the META NOT IN SESSION_TOTALS message if something went wrong there.
I saw NO LOGS of those.

At the same time I changed the place where we use this in a similar fashion

# Roughly https://github.com/codecov/codecov-api/blob/14dfd7c868e34a840387f9529b51194924c2a669/services/report.py#L206
files_array = report_details.files_array
    for file in files_array:
        if "session_totals" not in file or "meta" not in file['session_totals']:
            log.warning("really missing meta info", extra=dict(data=file))
    return {
        file["filename"]: ReportFileSummary(
            file_index=file["file_index"],
            file_totals=ReportTotals(*file["file_totals"]),
            session_totals=file["session_totals"],
            diff_totals=file["diff_totals"],
        )
        for file in files_array
    }

And... well...

WARNING: really missing meta info --- {"asctime": "2023-08-08 21:38:27,275", "name": "services.report", "lineno": 210, "pathname": "/app/services/report.py", "funcName": "build_files", "threadName": "MainThread", "data": {"filename": "services/fetch_data/tests/test_fetch_data.py", "file_index": 24, "file_totals": [0, 1, 1, 0, 0, "100", 0, 0, 0, 0, 0, 0, 0], "session_totals": {"0": [0, 1, 1, 0, 0, "100"]}, "diff_totals": null}}

EDIT
Disabling the lru_cache of the function solves the problem (here)