CI showing random pylint failures

[DickJC123]

Description

The 'sanity' CI runner is randomly failing with the following pylint error:

python/mxnet/numpy_op_signature.py:21:0: E0402: Attempted relative import beyond top-level package (relative-beyond-top-level)

Some recent examples from different PR's:

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-16397/2/pipeline

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-16391/2/pipeline

While there may be a direct fix to numpy_op_signature.py, the bigger question is why are some pipelines showing this error and others not? Is there an inconsistency in the pylint run by python3 -m pylint ...?

@larroy

Environment info (Required)

What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

Package used (Python/R/Scala/Julia):
(I'm using ...)

For Scala user, please provide:

1. Java version: (java -version)
2. Maven version: (mvn -version)
3. Scala runtime if applicable: (scala -version)

For R user, please provide R sessionInfo():

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)

Build config:
(Paste the content of config.mk, or the build command.)

Error Message:

(Paste the complete error message, including stack trace.)

Minimum reproducible example

(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)

Steps to reproduce

(Paste the commands you ran that produced the error.)

What have you tried to solve it?

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended label(s): Bug, CI

[larroy]

Thanks for reporting this, we will have a look.

[reminisce]

It's a pylint bug. I have deleted that line in the master branch. The flakiness should not show up again.

[DickJC123]

One of the failures I posted (the one from my PR) is built off of a fairly recent master. It's only missing your 'add boolean ndarray' PR from 10/8/2019, which does not touch the numpy_op_signature.py file. @reminisce, what commit introduced the fix your mention?

I assume that the pylint bug was specific to certain pylint versions (as opposed to being truly non-deterministic). My point to @larroy was that I think the CI runners should use a consistent version of pylint, so the results are the same throughout the many pipelines we launch as we improve our PRs.

[DickJC123]

After merging the most recent master into my PR, I still get a failure:

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-16391/6/pipeline

@reminisce Could you tell us more about the pylint bug as you understand it?

[reminisce]

@DickJC123 I am not an expert in root causing pylint errors, but I guess CI still reports this maybe due to some cached code in the system before you merged the master tip into your branch for two reasons:

1. It's the same message as you posted before I deleted from __future__ import absolute_import.
2. The line where the error is reported does not match what pylint describes. It matches the code before you updated your branch to the latest.

I am not sure whether this is because you merged the master branch into yours. Maybe you can try to rebase all of your commits on top of the master branch and see how sanity check goes.

[DickJC123]

To answer my own question from above, #16305 introduced the file ./python/mxnet/numpy_op_signature.py with initial lines:

from __future__ import absolute_import
import sys

#16370 attempted to correct the pylint issues, changing the lines to:

from __future__ import absolute_import # pylint: disable=reimported
from sys import version, version_info

And finally the most recent change to numpy_op_signature.py came from commit:

4940ec0e7  2019-10-06  reminisce              Fix random op signature

... which simplified the lines to:

import sys

So why have I seen the pylint problem with my PR #16391? I rebased the changes of my PR onto a master that already had the 4940ec0 commit. I don't understand the mechanism by which there could be "some cached code in the system". Thoughts on this @larroy ?

[reminisce]

@DickJC123 The following is just my guess.

I looked at your branch for PR 16391. Your commits are interleaved with the master branch commits, which suggests it's a merge, instead of rebase. It may require a rebase, which means putting your commits on top of the master branch tip, to refresh the CI cache (if there is any). But again, this is just my speculation. I have submitted three PRs today and have not seen the pylint error come back.

[DickJC123]

I can confirm that a rebase of my commits to the latest master has corrected this problem. Thanks @reminisce!

@larroy Should I close this issue, or do you want to use my experience to diagnose the root cause of this behavior? In my mind, there's a bug in the way MXNet creates the repo used to test against. Perhaps it's confused by commit time-stamps. Does MXNet have a home-grown script to create the repo or is it an approach used by many git projects?

[larroy]

I'm having this in my PR: #16430 (comment)

I just restarted it.

[larroy]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-16430/1/pipeline

[DickJC123]

It appears the output of pylint differs between the case of jobs==1 and the case jobs==8 (as set in the ./ci/other/pylintrc file). I can get the error python/mxnet/numpy_op_signature.py:21:0: E0402: Attempted relative import beyond top-level package (relative-beyond-top-level) to appear consistently when I set the jobs==1. Perhaps jobs==1 should be the setting for our CI, since it is masking errors and is otherwise (apparently) non-deterministically reporting errors.

[ddavydenko]

@mxnet-label-bot , add [Build]

[DickJC123]

Issue fixed by now-merged #16462.