Wednesday 15 May 2013

Re: Mysterious Python pyc file corruption problems

On Wed, May 15, 2013 at 04:36:56PM -0400, Barry Warsaw wrote:
> Hello Developers,
>
> I am trying to debug and fix a particularly vexing problem in Python that
> manifests on Ubuntu in several different ways. I have a hypothesis about the
> problem, but there are still some mysteries and I don't know how to reproduce
> it. I think I can fix it, but I'm sending this message (and soon, another one
> to python-dev with more technical details) in the hopes that you might have
> other ideas about how it can happen, or have a reliable way to reproduce the
> bug.
>
> The problem can show up in any package, but Brian has started to collect a
> number of bugs that all seem to be related (and I think Steve is going to open
> a megabug to dupe them all to). The common way this manifests is a traceback
> on an import statement. The actual error can be a "ValueError: bad marshal
> data (unknown type code)" such as in LP: #1010077, or an "EOFError: EOF read
> where not expected" as in LP: #1060842. We have many more instances of both
> of these.
>
> Both of these exceptions come from Python's marshal code (marshal.c). marshal
> is the low-level serialization protocol used to cache Python byte code into
> .pyc files, so both of these exception imply corrupt .pyc files, and in fact,
> the workaround is always to essentially blow away the .pyc file and re-create
> it. (Various different techniques can be used, but they all boil down to the
> same thing.)
>
> Another commonality is that this bug -- so far -- has not been observed in any
> Python 3.3 code, only 3.2 and earlier, including 2.7 and 2.6.

I've found one bug where the interpreter is python3.3:

http://launchpad.net/bugs/1178599

This looks to be a fresh install of Raring so the apport data seems
trustworthy. (There are some apport issues with delayed reporting and
distribution upgrades.) I'll be doing a full search of Launchpad bugs
shortly.

--
Brian Murray