-
-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test(backup): Create backup version snapshot tests #58173
Conversation
2bfa4d3
to
83f36fe
Compare
def test_at_23_9_1(self): | ||
with tempfile.TemporaryDirectory() as tmp_dir: | ||
_, snapshot_refval = read_snapshot_file(self.get_snapshot_path("23.9.1")) | ||
snapshot_data = yaml.safe_load(snapshot_refval) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I'm assuming most of this code will be reused in the future, is it worth abstracting that to another function? I guess we will only support 2 minor releases so it's not a big deal
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll have two test methods (as described in my comment on the main thread), so it will be a bit cleaner if we have a standalone function. This is especially true when we add automation to replace these test cases via codegen.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the steps to creating this new snapshot for 23.9.1 would be:
- git checkout 23.9.1
- run pytest with SENTRY_SNAPSHOTS_WRITEBACK=new
- Add
test_at_23_9_1
intest_releases.py
83f36fe
to
e88bfbc
Compare
When I do the 23.10.0 release later today, I'll:
This is a bit of a special case release though, because we only have 1 version test at the moment, so we only need to add a version test case, not replace the oldest one. In the future (23.11.0 going forward), the process will be modified to:
|
Codecov Report
@@ Coverage Diff @@
## master #58173 +/- ##
==========================================
+ Coverage 78.38% 79.05% +0.67%
==========================================
Files 5135 5135
Lines 223460 223471 +11
Branches 37623 37625 +2
==========================================
+ Hits 175150 176674 +1524
+ Misses 42570 41139 -1431
+ Partials 5740 5658 -82
|
e88bfbc
to
c8ff623
Compare
c8ff623
to
55b7c5d
Compare
The goal for the export.json feature is to support two minor versions (since we're using CalVer, roughly equivalent to two months) of releases when importing. That is to say, if the latest release is 23.10.0, and the previous two releases were 23.9.0 and 23.8.0, then a self-hosted release running code from [email protected] or self-hosted@latest (aka nightly) should be able to successfully import JSON backup data generated by [email protected] and [email protected], but not necessarily [email protected]. To support this, we add a new file `test_releases.py`, that provides a snapshot test of an "exhaustive" (aka, has at least one of every exportable model) export. Every time we cut a new release, we will add a new commit directly after the release commit that renames the `test_at_head` snapshot to `test_at_<RELEASED_VERSION>`, and creates a test case for that version (see `test_at_23_9_1` in this PR for one such example). Test cases older than the 2 version support window will simultaneously be removed. In the future, we plan to automate this "release test rotation" process via GitHub actions, but for now the ospo team will take care to do this manually for every release. Some other notes of interest: - Exports cannot be compared for equality using simple string comparisons, since they have special comparators for cases like passwords, dates that may or may not have been updated, etc (see comparators.py for greater detail). To accommodate this, we extend the `insta_snapshot` fixture with an extra argument, `inequality_comparator`, that allows a custom comparison lambda to be passed in for such cases. - The `test_at_23_9_1` snapshot was generated by taking an early version of this PR, cherry-picking it onto the `23.9.1` release tag, generating a new snapshot there, then copying the result back into the main commit. Closes getsentry/team-ospo#197
55b7c5d
to
1d31ff7
Compare
The goal for the export.json feature is to support two minor versions (since we're using CalVer, roughly equivalent to two months) of releases when importing. That is to say, if the latest release is 23.10.0, and the previous two releases were 23.9.0 and 23.8.0, then a self-hosted release running code from [email protected] or self-hosted@latest (aka nightly) should be able to successfully import JSON backup data generated by [email protected] and [email protected], but not necessarily [email protected].
To support this, we add a new file
test_releases.py
that provides a snapshot test of an "exhaustive" (aka, has at least one of every exportable model) export. Every time we cut a new release, we will add a new commit directly after the release commit that renames thetest_at_head
snapshot totest_at_<RELEASED_VERSION>
, and creates a test case for that version (seetest_at_23_9_1
in this PR for one such example). Test cases older than the 2 version support window will simultaneously be removed.In the future, we plan to automate this "release test rotation" process via GitHub actions, but for now the ospo team will take care to do this manually for every release.
Some other notes of interest:
insta_snapshot
fixture with an extra argument,inequality_comparator
, that allows a custom comparison lambda to be passed in for such cases.test_at_23_9_1
snapshot was generated by taking an early version of this PR, cherry-picking it onto the23.9.1
release tag, generating a new snapshot there, then copying the result back into the main commit.Closes getsentry/team-ospo#197