KeyError: sub.py

[nedbat]

Originally reported by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

This reproduces when using the ubuntu 2.7 from precise forward.

ERROR: test_subprocess_with_pth_files (tests.test_process.ProcessStartupTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buck/trees/mine/coverage.py/tests/test_process.py", line 673, in test_subprocess_with_pth_files
    self.assertEqual(data.summary()['sub.py'], 2)
KeyError: 'sub.py'

You can reproduce the scenario with:

mkdir $VIRTUALENV/local
ln -s $VIRTUALENV/lib $VIRTUALENV/local/lib

The essential cause is that site.py is seeing and executing subcover.py twice because of the symlink, first for local/lib/ then for lib/.

MAKEPATH: ('/home/buck/venv/coverage.py/local/lib/python2.7/site-packages',)
ADDPACKAGE: subcover.pth
MAKEPATH: ('/home/buck/venv/coverage.py/lib/python2.7/site-packages',)
ADDPACKAGE: subcover.pth

I don't quite follow the part in between, but the result is that only the data from the first trace persists, meaning we don't get the data we want. Presumably the trace of sub.py is written first, then the earlier trace data written on top of it.

The essential problem is that if a single process calls process_startup() twice, data is lost.
Worse, data generated by the second call is clobbered by data from the first.

I believe the fix would be to make coverage process_startup() idempotent.
Failing that, ensuring a last-one-wins behavior would also produce desirable behavior in this case.

@nedbat: How would you go about solving this problem?

(Most of this information copied from an older ticket centered on another topic: issue #337)

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/340

[nedbat]

Original comment by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

I've said this elsewhere, but not noted it here:

This problem stems from a hackitty virtualenv workaround to a poorly-written ubuntu patchset.

• Ubuntu patch: http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/python2.7/precise-security/view/head:/debian/patches/distutils-install-layout.diff#L282
• venv workaround: pypa/virtualenv@285679c

The problematic bit of the ubuntu patch is gone in the newest ubuntu (trusty), but the venv workaround remains, causing this issue.

[nedbat]

Original comment by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

This causes the "unexpected third case" in the pending __main__-support branch.
Ugh.

[nedbat]

What's the latest on this? Do you think it needs to be solved?

[nedbat]

Original comment by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

I've repeatedly run into this on ubuntu systems when using the stock python. It's quite annoying and extremely confusing.
The workaround is always rm -rf $VIRTUAL_ENV/local.

The idempotency/last-wins fixes proposed above would work, and might be simple enough to implement. I haven't looked.
I think the best fix would be to change virtualenv to fix their bug 118 a different way, not creating this local/ symlink.

[nedbat]

@bukzor Is this still happening for you? BTW, issue #353 also causes the "unexpected third case" assert, but for a different reason, if you want to chime in on what should happen there.

[nedbat]

Original comment by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

Yep, my situation unchanged since last post.

[nedbat]

@bukzor Thanks for the analysis, it was spot on. This is fixed in 9fd110c4bba0 (bb).

[nedbat]

Original comment by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

Sweet! I'll check that it fixes my specific use case too. I expect so.

--phone is hard.