Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent end-point for Interop pages #286

Closed
gsnedders opened this issue Jan 27, 2023 · 6 comments
Closed

Inconsistent end-point for Interop pages #286

gsnedders opened this issue Jan 27, 2023 · 6 comments

Comments

@gsnedders
Copy link
Member

https://wpt.fyi/interop-2022 shows a graph ending at the end of 2022, but the scores shown above are for the latest run (currently 2023). These should probably be consistent.

This difference is more stark if you look at https://wpt.fyi/interop-2021, where the scores are notably higher than the graph shows.

Per https://twitter.com/olwinfr/status/1618721146802950144, this does appear to be causing some actual confusion among people looking at the dashboard, so we should ensure it's clear what data is being presented.

cc @web-platform-tests/interop

@foolip
Copy link
Member

foolip commented Jan 27, 2023

When we've talked about this in the past I've been skeptical of stopping the graphs at the end of the year, because that would make any improvements after that invisible.

But now that we have "previous focus areas" it's not totally invisible, and I think it's reasonable that the graphs eventually stop updating.

To make https://wpt.fyi/interop-2021 consistent by lowering the scores now presented would be not-great I think, so I wonder if we could let that run to 2022 instead, in the graph as well?

cc @DanielRyanSmith who I've discussed this with in the past.

@DanielRyanSmith
Copy link
Contributor

After making the changes to the results-analysis scoring, it seems that we can pretty easily delimit the any year's scores by defining the end date in which we collect runs to score (the --to flag in the script). Currently, the to date is always today's date when not explicitly defined, so runs are aggregated and added to the overall scores long after the year is over.

Basically, we can define whenever we'd like to stop accumulating the score by setting this flag for specific years that we would like to freeze.

@foolip
Copy link
Member

foolip commented Feb 15, 2023

@gsnedders can you file an issue in the interop repo and agenda+ it so we can decide what we want to do?

@gsnedders gsnedders transferred this issue from web-platform-tests/wpt.fyi Feb 15, 2023
@foolip
Copy link
Member

foolip commented Feb 25, 2023

@DanielRyanSmith there is one more complication that specifying an end date doesn't deal with, and that's changes in the test lists that happen after the year has ended.

I think what we'll need to do is go back and see what the test lists were and the end of 2021 and 2022, and freeze those lists somewhere. Then we could keep running the scripts because they should always produce the same output, or just take a frozen copy of the output and put in a storage bucket.

@foolip
Copy link
Member

foolip commented Feb 15, 2024

We have the same problem for Interop 2023 as well now, where the graph ends in December but we kept updating the scores until Jan 31.

@foolip
Copy link
Member

foolip commented May 2, 2024

This is resolved now with how we're archiving the dashboard for each year:
https://wpt.fyi/interop-2021
https://wpt.fyi/interop-2022
https://wpt.fyi/interop-2023

@foolip foolip closed this as completed May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants