Arvados

I'm not sure adding an array of container UUIDs to the container_requests table would solve this problem. Often the most valuable troubleshooting information is in the log files, which would still be inaccessible.

It might be more useful to focus on preserving all relevant logs in the container request's log collection, even if they span multiple containers. Perhaps the API server could merge the logs: e.g., instead of replacing the CR's entire log collection when the container's log is updated, just copy the container's log files into a "container ${uuid}" subdir in the container request's log collection. This would disturb existing scripts/users who expect the log files to be at the top level, but it would be compatible with multiple concurrent containers (e.g., speculative retry without killing, and replication>1 service containers).

This also helps in the case where the container record itself is really what's wanted, since that is included in the container's log collection. (There are currently some exceptions -- e.g., a log collection isn't created at all when a container doesn't fit any instance type -- but those could be fixed.)

It's also worth addressing the permission issue, at least for admins (currently even the dispatcher isn't allowed to see that a container has state=Cancelled if all matching CRs have had different containers assigned!). If we need to do it for users, we should consider the performance implications of an array vs. a separate table to express the many-to-many relationship.

One more refinement: Put a copy of the latest container's logs in the root dir of the container request's log collection, in addition to a subdir named after the container UUID. This way, existing scripts continue to work on new logs.