Happy New Year from Curoverse!
Arvados engineering sprint updates
Happy New Year, everyone! We're poking our heads up through the snow here in New England to bring you our latest engineering updates!
One of the new features to come out of our most recent sprint is a revised Collection API (issue #4823) that more closely resembles the classic POSIX filesystem API. The Arvados storage system, Keep, is a content-addressed storage system that doesn't offer a POSIX interface for accessing data. A few sprints ago, in 7bf8f6c701, we released an interface that presents Arvados data streams as "file-like objects", but the overall collection API still just isn't very similar to the POSIX calls that Unix and Linux programmers are so familiar with.
Our new release brings us much closer to that goal, offering users a single API for both reading and writing collections, with familiar methods for addressing files like
remove() and so on. We anticipate that new users will find it much easier to get into the flow of using Arvados with these patterns, and existing users will find it more convenient to port pipelines to Arvados.
Crunch failure reporting¶
Another useful new tool is a Crunch job failure report (issue #4598). At present, while Crunch reports every job failure and success into Arvados, identifying the underlying causes of failed jobs can be a little tedious. Moreover, a breakdown report that gives visibility into why jobs have failed can be a huge boon to debugging.
With this tool, an administrator can now set up a nightly report breaking down job failures over the last 24 hours -- or get a report over any time period at all. We've already used this tool in development to help quickly debug knotty job failures, and expect that Arvados administrators everywhere will find it a great help.
New User Interfaces¶
We're also experimenting with newer ways of writing more responsive interfaces. Our primary interface to Arvados, Workbench, is written as a Rails application that uses the Arvados API instead of a local database. This makes it easy for Rails developers to work on, but also means that every request to Arvados goes through two software layers, which doubles the latency for every action a user takes.
Thanks to everyone for your ongoing input. Please give us a shout if you have any questions or idea for us!