Monday, 22 July 2013

Re: autopkgtests that fail on stderr output

Martin,

Thanks for doing this - it'll be much easier for me to write and debug
my tests now.

On Mon, Jul 22, 2013 at 08:27:17AM +0200, Martin Pitt wrote:
> It's a bit unfortunate that Python's unittest defaults to outputting
> the test progress to stderr (i. e. something which is not an error at
> all).

I strongly disagree. stderr is not just for errors; it is for
diagnostics as well. There is plenty of precedent for this. Even "set
-x" does it. It's quite clear why: writing diagnostics/debugging to
stdout would screw up pipelines, thus changing the behaviour you're
trying to observe.

So it is *perfectly reasonable* for CLI interfaces to write diagnostics
to stderr that are not errors, and then return an exit status of zero if
there was no error. This is not a bug, nor is it unfortunate. The exit
status tells you whether a command succeeded or failed, not the presence
of stderr output. This is the way Unix is supposed to work.

Further, it is *even more reasonable* to do this when you've enabled
some kind of debugging flag (-d, -vvv, or whatever). And we should be
able to do this easily in tests, because if our tests aren't working as
expected, we need to debug them and get that stderr output right to the
developer, bypassing any output to pipelines.

We can redirect everything about of course, but it's a pain to do so,
and you risk changing the thing you're trying to debug.

If it has been useful to catch programs generating warning output to
stderr when they previously did not, then perhaps these programs should
take an option or environment variable setting to turn all warnings into
errors?

But anyway, for this reason I *really* appreciate the new allow-stderr
option. Thank you!

Robie

--
ubuntu-devel mailing list
[email protected]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel