Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression tests timeout from time to time #479

Closed
s3rvac opened this issue Jan 30, 2019 · 2 comments
Closed

Regression tests timeout from time to time #479

s3rvac opened this issue Jan 30, 2019 · 2 comments

Comments

@s3rvac
Copy link
Member

s3rvac commented Jan 30, 2019

Some of our regression tests timeout from time to time. For an example, see this macOS build. I have seen the timeouts on both macOS and Linux, but maybe this is also an issue on Windows. I strongly believe that these timeouts are caused by non-determinism in our tools (#209, #129). Instead of keeping disabling failing tests (#266), we should try to debug this.

Here is an example of two .ll files produced by bin2llvmir in two separate runs of the features.new-static-code-signatures.TestNewARM32Signatures regression test on my Linux machine. As you can see, they differ on quite a lot of places. @PeterMatula, do you have an idea what is causing these differences?

@s3rvac
Copy link
Member Author

s3rvac commented Feb 7, 2019

More timeouts on macOS: https://retdec-tc.avast.com/viewLog.html?buildId=1249&buildTypeId=Retdec_MacBuild&tab=buildLog&_focus=5200

@PeterMatula Can you please take a look at this issue?

PeterMatula added a commit that referenced this issue Feb 20, 2019
This is preventing the infinite loop by storing already seen values to
sets. It is working, but is not a very elegant solution.
PeterMatula added a commit to avast/retdec-regression-tests that referenced this issue Feb 21, 2019
@PeterMatula
Copy link
Collaborator

Fixed in 55d46b5, all previously disabled tests enabled in avast/retdec-regression-tests@87dd055.

PeterMatula added a commit to avast/retdec-regression-tests that referenced this issue Feb 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants