Wednesday, 15 May 2013

Re: Mysterious Python pyc file corruption problems

On Wed, May 15, 2013 at 03:15:05PM -0700, Brian Murray wrote:
> On Wed, May 15, 2013 at 04:36:56PM -0400, Barry Warsaw wrote:
> > Hello Developers,
> >
> > I am trying to debug and fix a particularly vexing problem in Python that
> > manifests on Ubuntu in several different ways. I have a hypothesis about the
> > problem, but there are still some mysteries and I don't know how to reproduce
> > it. I think I can fix it, but I'm sending this message (and soon, another one
> > to python-dev with more technical details) in the hopes that you might have
> > other ideas about how it can happen, or have a reliable way to reproduce the
> > bug.
> >
> > The problem can show up in any package, but Brian has started to collect a
> > number of bugs that all seem to be related (and I think Steve is going to open
> > a megabug to dupe them all to). The common way this manifests is a traceback
> > on an import statement. The actual error can be a "ValueError: bad marshal
> > data (unknown type code)" such as in LP: #1010077, or an "EOFError: EOF read
> > where not expected" as in LP: #1060842. We have many more instances of both
> > of these.
> >
> > Both of these exceptions come from Python's marshal code (marshal.c). marshal
> > is the low-level serialization protocol used to cache Python byte code into
> > .pyc files, so both of these exception imply corrupt .pyc files, and in fact,
> > the workaround is always to essentially blow away the .pyc file and re-create
> > it. (Various different techniques can be used, but they all boil down to the
> > same thing.)
> >
> > Another commonality is that this bug -- so far -- has not been observed in any
> > Python 3.3 code, only 3.2 and earlier, including 2.7 and 2.6.
> I've found one bug where the interpreter is python3.3:
> This looks to be a fresh install of Raring so the apport data seems
> trustworthy. (There are some apport issues with delayed reporting and
> distribution upgrades.) I'll be doing a full search of Launchpad bugs
> shortly.


That's all the bugs tagged apport-crash and raring with python3.3 in
InterpreterPath and EOFError or bad marshal data in the apport generated

Looking both of these up at using the bug parameter
(yeah!) I only found one example which appears to be the same report as
in bug 1178599.

Brian Murray