On Thu, 23 Jan 2025 10:04:10 +0100
Julian Andres Klode <julian.klode@canonical.com> wrote:
> On Tue, Jan 21, 2025 at 03:19:14PM +0000, Robie Basak wrote:
> > I think it's about time that Ubuntu Developer realtime conversation
> > moved from IRC to Matrix. What do you think? Can we reach a
> > consensus on this topic?
>
> 1.
>
> One thing to point out is that we need to be careful; I tried
> joinining #matrix:matrix.org yesterday and the server became
> unusable as the workers were busy fetching data from there
> that there were no resources left to do anything else.
>
> Some people tell me that's entirely normal and can't be
> avoided, other people tell me that our server is of low
> quality. I don't know what the truth is :D
(Matrix Council hat on)
It is entirely normal (even the best servers on the Matrix network have
severe problems with the #matrix:matrix.org at least to begin with),
but it *can* be avoided. I've asked the people with the needed admin
access about blocking the ability to join the room so that this sort of
thing doesn't happen in the future, and they're looking into blocking
rooms that have too high of a "complexity". If nothing else, this will
at least mitigate accidental denial-of-service attacks, and if it
blocks a room someone really needs access to, we can look into bumping
up server resources or whatever else needs to happen to make the room
tolerable.
Also, IS is planning on migrating our Synapse instance to a dedicated
cluster (no idea what the timeline is there) which I believe will
increase the server's resources. That should help it be able to deal
with heavy load better.
> The way I understand it there are worker processes and you
> simply need enough cores and RAM such that they can all run;
> one thing that was pointed out is that Synapse's PostgreSQL
> queries are huge; and you need to tweak PostgreSQL to allow
> these to be done in memory:
>
> ALTER DATABASE synapse SET work_mem TO '512 MB';
>
> (adjust 512 MB for size needed)
>
> As otherwise, PostgreSQL will start using temporary files
> to do the queries and slow to a crawl.
Could you share a link to where you saw this? I might be able to talk
someone into running that and that might help things out. (If it
happened in the chat on #matrix-ops:ubuntu.com, I accidentally missed
it.)
> 2.
>
> Internally we need to check that the system is independent
> of Canonical's internal mattermost. If it's in the same
> data center and down at the same time, this affects our
> ability to work :D
Good point, I think it probably is on the same system more or less
right now, but the dedicated cluster stuff may help with that. Stay
tuned :)
