Bug #18490
closed
Permission table bloat
Description
Postgres seems to be creating a lot of fragmentation in the materialized_permissions table in a context with a lot of users (~1000) and large groups. Reported the table size went from 200MB to 2GB to 20 GB in the span of a few days (? or even faster).
The group sync script is very slow in this context while there are other updates ongoing.
Normal usage (e.g. navigating a large project with lots of subprojects in a read-only arv-mount) is also very slow when the table is fragmented that way.
Vacuum full helps but can often not complete because normal usage patterns prevent it from completing.
This has been observed on Postgres 9.5 and 13.
Updated by
Updated by Peter Amstutz about 3 years ago
18490-redundant-updates @ 26aa25c76d3ea4285e724fe874c76aa9da03b4c9
Add a where clause to "insert on conflict update" to suppress redundant updates of rows that haven't changed.
Updated by Peter Amstutz about 3 years ago
- Assigned To set to Peter Amstutz
- Status changed from New to In Progress
- Category set to API
- Subject changed from Research postgres tuning for tables with lots of updates to Permission table bloat
Feedback from customer after emailing them and they applied a hotfix from #note-4:
"This seems to have solved the issue. We had over 100 new users added to a large group, and the table is currently sitting at 273MB. I'll continue to monitor it to check for any changes but it definitely looks promising."
Updated by Lucas Di Pentima about 3 years ago
My only observation is that it would be convenient to add a comment explaining the workaround, just for future reference. Apart from that, LGTM.
Updated by Peter Amstutz about 3 years ago
Lucas Di Pentima wrote:
My only observation is that it would be convenient to add a comment explaining the workaround, just for future reference. Apart from that, LGTM.
Yea, that's a good idea. Will do.
Updated by Peter Amstutz about 3 years ago
- Status changed from In Progress to Resolved