Skip to content
This repository has been archived by the owner on Apr 20, 2023. It is now read-only.

Codecov broken tests #100

Closed
cpu82 opened this issue May 28, 2017 · 8 comments
Closed

Codecov broken tests #100

cpu82 opened this issue May 28, 2017 · 8 comments

Comments

@cpu82
Copy link

cpu82 commented May 28, 2017

Hi,

It seems that there are some test cases that fail to pass.

Tested on FreeBSD by invoking the following command:

/usr/local/bin/python2.7 -m pytest -v tests/test.py

http://sprunge.us/hIEC

Is it possible to skip the tests explicitly when invoking the test suite?
@cpu82
Copy link
Author

cpu82 commented May 29, 2017 •
edited
Loading

I noticed that if the .git directory is present all test cases run fine, except 3 tests skipped.

% python2.7 -m pytest -v tests/test.py
============================= test session starts ==============================
platform freebsd12 -- Python 2.7.13, pytest-3.1.0, py-1.4.33, pluggy-0.4.0 -- /usr/local/bin/python2.7
cachedir: .cache
rootdir: /usr/home/cpm/work/gh/codecov-python, inifile:
collected 68 items 

tests/test.py::TestUploader::test_bowerrc PASSED
tests/test.py::TestUploader::test_bowerrc_none PASSED
tests/test.py::TestUploader::test_ci_appveyor PASSED
tests/test.py::TestUploader::test_ci_buildkite PASSED
tests/test.py::TestUploader::test_ci_circleci PASSED
tests/test.py::TestUploader::test_ci_codeship PASSED
tests/test.py::TestUploader::test_ci_drone PASSED
tests/test.py::TestUploader::test_ci_gitlab PASSED
tests/test.py::TestUploader::test_ci_jenkins PASSED
tests/test.py::TestUploader::test_ci_jenkins_blue_ocean PASSED
tests/test.py::TestUploader::test_ci_jenkins_env PASSED
tests/test.py::TestUploader::test_ci_magnum PASSED
tests/test.py::TestUploader::test_ci_none PASSED
tests/test.py::TestUploader::test_ci_semaphore PASSED
tests/test.py::TestUploader::test_ci_shippable PASSED
tests/test.py::TestUploader::test_ci_snap PASSED
tests/test.py::TestUploader::test_ci_travis PASSED
tests/test.py::TestUploader::test_ci_wercker PASSED
tests/test.py::TestUploader::test_command PASSED
tests/test.py::TestUploader::test_disable_detect PASSED
tests/test.py::TestUploader::test_disable_gcov SKIPPED
tests/test.py::TestUploader::test_disable_search PASSED
tests/test.py::TestUploader::test_discovers PASSED
tests/test.py::TestUploader::test_exits_0 PASSED
tests/test.py::TestUploader::test_exits_1 PASSED
tests/test.py::TestUploader::test_gcov SKIPPED
tests/test.py::TestUploader::test_ignore_report_1__coverage_worker10 PASSED
tests/test.py::TestUploader::test_ignore_report_2_coverage_jade PASSED
tests/test.py::TestUploader::test_ignore_report_3_include_lst PASSED
tests/test.py::TestUploader::test_ignore_report_4_inputFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_5_createdFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_6_scoverage_measurements_blackandwhite_xml PASSED
tests/test.py::TestUploader::test_ignore_report_7_test_hello_coverage_txt PASSED
tests/test.py::TestUploader::test_ignore_report_8_conftest_blackwhite_c_gcov PASSED
tests/test.py::TestUploader::test_ignored_path_10__tox PASSED
tests/test.py::TestUploader::test_ignored_path_11_venv PASSED
tests/test.py::TestUploader::test_ignored_path_12__venv_python_2_7 PASSED
tests/test.py::TestUploader::test_ignored_path_1_vendor PASSED
tests/test.py::TestUploader::test_ignored_path_2_node_modules PASSED
tests/test.py::TestUploader::test_ignored_path_3_js_generated_coverage PASSED
tests/test.py::TestUploader::test_ignored_path_4___pycache__ PASSED
tests/test.py::TestUploader::test_ignored_path_5_coverage_instrumented PASSED
tests/test.py::TestUploader::test_ignored_path_6_build_lib PASSED
tests/test.py::TestUploader::test_ignored_path_7_htmlcov PASSED
tests/test.py::TestUploader::test_ignored_path_8__egg_info PASSED
tests/test.py::TestUploader::test_ignored_path_9__git PASSED
tests/test.py::TestUploader::test_include_env PASSED
tests/test.py::TestUploader::test_is_report_10_luacov_report_out PASSED
tests/test.py::TestUploader::test_is_report_11_gcov_info PASSED
tests/test.py::TestUploader::test_is_report_12_nosetests_xml PASSED
tests/test.py::TestUploader::test_is_report_1_coverage_xml PASSED
tests/test.py::TestUploader::test_is_report_2_jacoco_xml PASSED
tests/test.py::TestUploader::test_is_report_3_jacocoTestResults_xml PASSED
tests/test.py::TestUploader::test_is_report_4_coverage_txt PASSED
tests/test.py::TestUploader::test_is_report_5_gcov_lst PASSED
tests/test.py::TestUploader::test_is_report_6_cov_gcov PASSED
tests/test.py::TestUploader::test_is_report_7_info_lcov PASSED
tests/test.py::TestUploader::test_is_report_8_clover_xml PASSED
tests/test.py::TestUploader::test_is_report_9_cobertura_xml PASSED
tests/test.py::TestUploader::test_none_found PASSED
tests/test.py::TestUploader::test_not_jacoco PASSED
tests/test.py::TestUploader::test_read_token_file PASSED
tests/test.py::TestUploader::test_require_branch_1 PASSED
tests/test.py::TestUploader::test_returns_none PASSED
tests/test.py::TestUploader::test_run_coverage SKIPPED
tests/test.py::TestUploader::test_run_coverage_fails PASSED
tests/test.py::TestUploader::test_send PASSED
tests/test.py::TestUploader::test_send_error PASSED

===================== 65 passed, 3 skipped in 2.20 seconds =====================

@cpu82 cpu82 changed the title codecov broken tests Codecov broken tests May 29, 2017
@stevepeak
Copy link
Contributor

We welcome a pull request if you feel inclined to adjust the code base for this use case.
Speaking of which: why is the use case for your teste suite?

@cpu82
Copy link
Author

cpu82 commented May 30, 2017 •
edited
Loading

According to the FreeBSD python port policy [1]: python ports must include a do-test target if the package includes tests.

For now, I have removed the failed tests using a simple patch [2].

Therefore, it would be great that all test cases work properly.

[1] https://wiki.freebsd.org/Python/PortsPolicy#Tests
[2] https://reviews.freebsd.org/D10873#1609c58f

@stevepeak
Copy link
Contributor

@naglalakk can you help out here. Thanks

@stevepeak
Copy link
Contributor

I'm cleaning up issues. What is the status of this one?

@cpu82
Copy link
Author

cpu82 commented Jan 9, 2018

Hi,

Today I updated codecov to 2.0.13 but it still failing with a couple of AppVeyor CI tests.

As a workaround I skipped both tests at the moment.

--- tests/test.py.orig  2018-01-08 10:57:15 UTC
+++ tests/test.py
@@ -134,7 +134,7 @@ class TestUploader(unittest.TestCase):
                 self.assertEqual(codecov.main(), None)
                 assert post.called and put.called
 
-    @unittest.skipIf(os.getenv('CI') == "True" and os.getenv('APPVEYOR') == 'True', 'Skip AppVeyor CI test')
+    @unittest.skipIf(sys.platform.startswith('freebsd'),'Skip test')
     def test_send(self):
         with patch('requests.post') as post:
             with patch('requests.put') as put:
@@ -249,7 +249,7 @@ class TestUploader(unittest.TestCase):
         else:
             raise Exception("Did not raise AssertionError")
 
-    @unittest.skipIf(os.getenv('CI') == "True" and os.getenv('APPVEYOR') == 'True', 'Skip AppVeyor CI test')
+    @unittest.skipIf(sys.platform.startswith('freebsd'),'Skip test')
     def test_bowerrc_none(self):
         with open(self.bowerrc, 'w+') as f:
             f.write('{"other_key": "tests"}')

Here is the make test output

$ make test
===>  License APACHE20 accepted by the user
===>   py27-codecov-2.0.13 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by py27-codecov-2.0.13 for building
===>  Extracting for py27-codecov-2.0.13
=> SHA256 Checksum OK for codecov-codecov-python-v2.0.13_GH0.tar.gz.
===>  Patching for py27-codecov-2.0.13
===>  Applying FreeBSD patches for py27-codecov-2.0.13
===>   py27-codecov-2.0.13 depends on package: py27-setuptools>0 - found
===>   py27-codecov-2.0.13 depends on file: /usr/local/bin/python2.7 - found
===>  Configuring for py27-codecov-2.0.13
running config
===>  Building for py27-codecov-2.0.13
running build
running build_py
creating build
creating build/lib
creating build/lib/codecov
copying codecov/__init__.py -> build/lib/codecov
copying codecov/__main__.py -> build/lib/codecov
running egg_info
creating codecov.egg-info
writing requirements to codecov.egg-info/requires.txt
writing codecov.egg-info/PKG-INFO
writing top-level names to codecov.egg-info/top_level.txt
writing dependency_links to codecov.egg-info/dependency_links.txt
writing entry points to codecov.egg-info/entry_points.txt
writing manifest file 'codecov.egg-info/SOURCES.txt'
reading manifest file 'codecov.egg-info/SOURCES.txt'
writing manifest file 'codecov.egg-info/SOURCES.txt'
===>  Staging for py27-codecov-2.0.13
===>   py27-codecov-2.0.13 depends on package: py27-coverage>0 - found
===>   py27-codecov-2.0.13 depends on package: py27-requests>=2.7.9 - found
===>   py27-codecov-2.0.13 depends on package: py27-setuptools>0 - found
===>   py27-codecov-2.0.13 depends on file: /usr/local/bin/python2.7 - found
===>   Generating temporary packing list
running install
running build
running build_py
running egg_info
writing requirements to codecov.egg-info/requires.txt
writing codecov.egg-info/PKG-INFO
writing top-level names to codecov.egg-info/top_level.txt
writing dependency_links to codecov.egg-info/dependency_links.txt
writing entry points to codecov.egg-info/entry_points.txt
reading manifest file 'codecov.egg-info/SOURCES.txt'
writing manifest file 'codecov.egg-info/SOURCES.txt'
running install_lib
creating /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7
creating /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages
creating /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov
copying build/lib/codecov/__init__.py -> /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov
copying build/lib/codecov/__main__.py -> /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov
byte-compiling /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov/__init__.py to __init__.pyc
byte-compiling /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov/__main__.py to __main__.pyc
writing byte-compilation script '/tmp/tmptHoyRL.py'
/usr/local/bin/python2.7 -O /tmp/tmptHoyRL.py
removing /tmp/tmptHoyRL.py
running install_egg_info
Copying codecov.egg-info to /usr/ports/devel/py-codecov/work-py27/stage/usr/local/lib/python2.7/site-packages/codecov-2.0.13-py2.7.egg-info
running install_scripts
Installing codecov script to /usr/ports/devel/py-codecov/work-py27/stage/usr/local/bin
writing list of installed files to '/usr/ports/devel/py-codecov/work-py27/.PLIST.pymodtmp'
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/codecov --> bin/codecov-2.7
Link: @bin/codecov --> bin/codecov-2.7
====> Compressing man pages (compress-man)
====> Running Q/A tests (stage-qa)
===>  Testing for py27-codecov-2.0.13
===>   py27-codecov-2.0.13 depends on package: py27-ddt>0 - found
===>   py27-codecov-2.0.13 depends on package: py27-mock>0 - found
===>   py27-codecov-2.0.13 depends on package: py27-pytest>0 - found
===>   py27-codecov-2.0.13 depends on file: /usr/local/bin/python2.7 - found
============================= test session starts ==============================
platform freebsd10 -- Python 2.7.14, pytest-3.2.5, py-1.5.2, pluggy-0.4.0 -- /usr/local/bin/python2.7
cachedir: .cache
rootdir: /usr/ports/devel/py-codecov/work-py27/codecov-python-2.0.13, inifile:
plugins: xdist-1.13.1
collected 67 items                                                              

tests/test.py::TestUploader::test_bowerrc PASSED
tests/test.py::TestUploader::test_bowerrc_none <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_appveyor <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_buildkite <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_circleci <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_codeship <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_drone <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_gitlab <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_jenkins <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_jenkins_blue_ocean <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_jenkins_env <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_magnum <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_none <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_semaphore <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_shippable <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_travis <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_ci_wercker <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_command PASSED
tests/test.py::TestUploader::test_disable_detect PASSED
tests/test.py::TestUploader::test_disable_gcov SKIPPED
tests/test.py::TestUploader::test_disable_search PASSED
tests/test.py::TestUploader::test_discovers PASSED
tests/test.py::TestUploader::test_exits_0 PASSED
tests/test.py::TestUploader::test_exits_1 PASSED
tests/test.py::TestUploader::test_gcov SKIPPED
tests/test.py::TestUploader::test_ignore_report_1__coverage_worker10 PASSED
tests/test.py::TestUploader::test_ignore_report_2_coverage_jade PASSED
tests/test.py::TestUploader::test_ignore_report_3_include_lst PASSED
tests/test.py::TestUploader::test_ignore_report_4_inputFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_5_createdFiles_lst PASSED
tests/test.py::TestUploader::test_ignore_report_6_scoverage_measurements_blackandwhite_xml PASSED
tests/test.py::TestUploader::test_ignore_report_7_test_hello_coverage_txt PASSED
tests/test.py::TestUploader::test_ignore_report_8_conftest_blackwhite_c_gcov PASSED
tests/test.py::TestUploader::test_ignored_path_10__tox PASSED
tests/test.py::TestUploader::test_ignored_path_11_venv PASSED
tests/test.py::TestUploader::test_ignored_path_12__venv_python_2_7 PASSED
tests/test.py::TestUploader::test_ignored_path_1_vendor PASSED
tests/test.py::TestUploader::test_ignored_path_2_node_modules PASSED
tests/test.py::TestUploader::test_ignored_path_3_js_generated_coverage PASSED
tests/test.py::TestUploader::test_ignored_path_4___pycache__ PASSED
tests/test.py::TestUploader::test_ignored_path_5_coverage_instrumented PASSED
tests/test.py::TestUploader::test_ignored_path_6_build_lib PASSED
tests/test.py::TestUploader::test_ignored_path_7_htmlcov PASSED
tests/test.py::TestUploader::test_ignored_path_8__egg_info PASSED
tests/test.py::TestUploader::test_ignored_path_9__git PASSED
tests/test.py::TestUploader::test_include_env PASSED
tests/test.py::TestUploader::test_is_report_10_luacov_report_out PASSED
tests/test.py::TestUploader::test_is_report_11_gcov_info PASSED
tests/test.py::TestUploader::test_is_report_12_nosetests_xml PASSED
tests/test.py::TestUploader::test_is_report_1_coverage_xml PASSED
tests/test.py::TestUploader::test_is_report_2_jacoco_xml PASSED
tests/test.py::TestUploader::test_is_report_3_jacocoTestResults_xml PASSED
tests/test.py::TestUploader::test_is_report_4_coverage_txt PASSED
tests/test.py::TestUploader::test_is_report_5_gcov_lst PASSED
tests/test.py::TestUploader::test_is_report_6_cov_gcov PASSED
tests/test.py::TestUploader::test_is_report_7_info_lcov PASSED
tests/test.py::TestUploader::test_is_report_8_clover_xml PASSED
tests/test.py::TestUploader::test_is_report_9_cobertura_xml PASSED
tests/test.py::TestUploader::test_none_found PASSED
tests/test.py::TestUploader::test_not_jacoco PASSED
tests/test.py::TestUploader::test_read_token_file PASSED
tests/test.py::TestUploader::test_require_branch_1 PASSED
tests/test.py::TestUploader::test_returns_none PASSED
tests/test.py::TestUploader::test_run_coverage SKIPPED
tests/test.py::TestUploader::test_run_coverage_fails PASSED
tests/test.py::TestUploader::test_send <- ../../../../../local/lib/python2.7/site-packages/unittest2/case.py SKIPPED
tests/test.py::TestUploader::test_send_error PASSED

==================== 47 passed, 20 skipped in 1.72 seconds =====================

@thomasrockhu
Copy link
Contributor

Closing this out as tests should be passing now.

@yzgyyang
Copy link

Closing this out as tests should be passing now.

This is not true - OP was talking about that, by simply removing the .git directory, tests will begin to fail (I got "12 failed, 38 passed, 20 skipped in 3.19s"). This happens if one downloads a release (which doesn't include .git) and try to run the test suite.

@yzgyyang yzgyyang mentioned this issue Nov 24, 2020
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stevepeak @cpu82 @thomasrockhu @yzgyyang