Happy New Year from Curoverse!

Arvados engineering sprint updates
Added by Tim Pierce over 5 years ago

Happy New Year, everyone! We're poking our heads up through the snow here in New England to bring you our latest engineering updates!

Collection API

One of the new features to come out of our most recent sprint is a revised Collection API (issue #4823) that more closely resembles the classic POSIX filesystem API. The Arvados storage system, Keep, is a content-addressed storage system that doesn't offer a POSIX interface for accessing data. A few sprints ago, in 7bf8f6c701, we released an interface that presents Arvados data streams as "file-like objects", but the overall collection API still just isn't very similar to the POSIX calls that Unix and Linux programmers are so familiar with.

Our new release brings us much closer to that goal, offering users a single API for both reading and writing collections, with familiar methods for addressing files like open(), rename(), remove() and so on. We anticipate that new users will find it much easier to get into the flow of using Arvados with these patterns, and existing users will find it more convenient to port pipelines to Arvados.

Crunch failure reporting

Another useful new tool is a Crunch job failure report (issue #4598). At present, while Crunch reports every job failure and success into Arvados, identifying the underlying causes of failed jobs can be a little tedious. Moreover, a breakdown report that gives visibility into why jobs have failed can be a huge boon to debugging.

With this tool, an administrator can now set up a nightly report breaking down job failures over the last 24 hours -- or get a report over any time period at all. We've already used this tool in development to help quickly debug knotty job failures, and expect that Arvados administrators everywhere will find it a great help.

New User Interfaces

We're also experimenting with newer ways of writing more responsive interfaces. Our primary interface to Arvados, Workbench, is written as a Rails application that uses the Arvados API instead of a local database. This makes it easy for Rails developers to work on, but also means that every request to Arvados goes through two software layers, which doubles the latency for every action a user takes.

Since December we've been working with newer JavaScript frameworks like Mithril and Angular, to build a pure JavaScript interface to Arvados that eliminates the need for a Workbench middle layer entirely. Our recently-launched collection uploader tool is written in Angular, and we've used Mithril to build an experimental administrative interface we're calling "Backstage." The improved responsiveness we're getting from these tools is really showing a lot of promise, and we hope to have lots more to show you soon!

Thanks to everyone for your ongoing input. Please give us a shout if you have any questions or idea for us!

(Photo DNA of Bridge by Matthew Fang published CC-BY-SA on Flickr)

dna_bridge.jpg (23.8 KB) dna_bridge.jpg Tim Pierce, 01/12/2015 09:41 PM Delete