> On Fri, Feb 03, 2023 at 11:38:13AM -0800, Brian Murray wrote:
> > I recently discovered that autopkgtests which are getting OOM killed
> > were behaving differently. Paride and I have tracked this down to a
> > change in systemd's behavior[1] which results in the testbed exiting
> > abnormally. I've temporarily[2] modified[3] our autopkgtest code in
> > production so that this will not be considered a "testbed" failure which
> > will prevent packages with autopkgtests that are OOM killed from running
> > repeatedly.
> >
> > I mention it because the log files for tests in this situation have
> > changed and have become less informative. Looking the log file[4] for
> > r-cran-rstanarm you'll see the following:
> >
> > autopkgtest [18:46:11]: test run-unit-test: -----------------------]
> > autopkgtest [18:46:12]: test run-unit-test: - - - - - - - - - - results
> > - - - - - - - - - -
> > run-unit-test FAIL non-zero exit status 255
> >
> > Which is less helpful than the previous "Killed signal terminated"
> > message. So when looking at proposed migration please keep this in mind.
>
> Does the backend have any visibility into detecting when OOM killer was
> triggered?
I'm not exactly certain what you mean but as I mentioned later there are
signs in the journal of the cloud-worker units now and previously those
signs were in the log of the autopkgtest results.
One thing somebody could do is query the autopkgtest results database
for packages which have always failed in Lunar then download the log
file and search it for processes being killed. Those packages could then
be tested locally with more memory and / or seeing if they pass in Debian.
Ideally we would modify the server side of things to rerun some packages
whose autopkgtests fail under certain criteria with more memory
automatically.
Cheers,
--
Brian Murray
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
