Default Value for RoborazziOptions.CompareOptions.changeThreshold Should Be 0

[takahirom]

During our testing with the Roborazzi library, we noticed that tests were passing even when there were noticeable differences between the images being compared. This behavior can lead to false positives in our test results.

#179

[takahirom]

If you have any opinions or concerns about this change, please let me know. As long as I'm managing the changes with a value of 0 in the CI, I believe there shouldn't be any issues.

[ParkJong-Hun]

Is there anyone who experiences the symptom of a diff being displayed in compare even though there appears to be no change in the screenshot and there is no actual change in the code?

[takahirom]

@ParkJong-Hun Thank you. Roborazzi utilizes color pixel data. Therefore, I believe you can compare the highlighted color of the pixel, where Roborazzi identifies it as a red pixel.

[hellohj]

I have the exact same issue: neither code change nor visual diff from screenshots. Could you please tell me how to compare highlighted color of pixel from *_compare.png that was generated? Is there a tool I can use? Also, this issue seems to occur in CI only. Do you have any ideas on why? Thanks.

[takahirom]

@hellohj Thanks.

I have the exact same issue: neither code change nor visual diff from screenshots. Could you please tell me how to compare highlighted color of pixel from *_compare.png that was generated? Is there a tool I can use?

I sometimes use Mac's Preview to cut out original images and paste them next to the new images for comparison.

Also, this issue seems to occur in CI only. Do you have any ideas on why? Thanks.

It depends on where the problem occurs. You can refer to this article for more information.
https://medium.com/@takahirom/how-to-solve-flaky-robolectric-and-roborazzi-tests-5731e55581cd

[hellohj]

Thanks for answers. Although I've spent a few hours today, I still can't figure out anything much.
The test only fails for screens in dark mode in CI (with @config(qualifiers = "+night")). OS in CI is linux. Verification in local is always successful on my mac. This is one file. Obviously, there are differences in hex values between snapshots. Sorry for asking again, but I am out of ideas.

[takahirom]

@hellohj
May I ask if the environment is the same when you are recording and verifying?

[hellohj]

That's a great question. Recording is done from a developer's laptop (macbook pro M1 max) and uploaded to our git main. Then, the verification happens in CI in each PR. CI machine is linux (Ubuntu 22.04.4).

[hellohj]

@takahirom Would you suspect our differences on OSs may cause this issue? But, why dark mode only? 🤔
We also have Paparazzi in CI and don't have any issues although a method of capturing is different. The issue is currently blocking us to enable Roborazzi in CI unfortunately.

[takahirom]

@hellohj
You can refer to this comment. I think this issue can currently be a tradeoff of Roborazzi.
android/nowinandroid#1242 (comment)

[hellohj]

Thank you so much for the link. It's unfortunate, but there is nothing much we can do at this point. My team will figure things out. Thank you for your support for a while.

[jeprubio]

@hellohj We had a similar problem and now we do the recording also in CI with a github action for updating the baseline. Perhaps it's something you could consider