Permission table bloat
Postgres seems to be creating a lot of fragmentation in the materialized_permissions table in a context with a lot of users (~1000) and large groups. Reported the table size went from 200MB to 2GB to 20 GB in the span of a few days (? or even faster).
The group sync script is very slow in this context while there are other updates ongoing.
Normal usage (e.g. navigating a large project with lots of subprojects in a read-only arv-mount) is also very slow when the table is fragmented that way.
Vacuum full helps but can often not complete because normal usage patterns prevent it from completing.
This has been observed on Postgres 9.5 and 13.
#4 Updated by Peter Amstutz about 2 months ago
18490-redundant-updates @ 26aa25c76d3ea4285e724fe874c76aa9da03b4c9
Add a where clause to "insert on conflict update" to suppress redundant updates of rows that haven't changed.
#5 Updated by Peter Amstutz about 2 months ago
- Assigned To set to Peter Amstutz
- Status changed from New to In Progress
- Category set to API
- Subject changed from Research postgres tuning for tables with lots of updates to Permission table bloat
Feedback from customer after emailing them and they applied a hotfix from #note-4:
"This seems to have solved the issue. We had over 100 new users added to a large group, and the table is currently sitting at 273MB. I'll continue to monitor it to check for any changes but it definitely looks promising."