April development review: Workbench makes it easy
Over the last month, we've made it easier than ever to get started with Arvados. You'll literally see the results as soon as you log in to Workbench: now a guide pops up to walk you through each step of running your own pipeline. The material will be familiar if you've already been through our tutorials, but the integration will make things easier for first-timers.
But even if you've been with us for a while, the improvements don't end there. Workbench now also recognizes a wider range of pipeline input errors, like insufficient rights to read a collection, and will prompt you to fix them before you submit the pipeline to run. We all make mistakes now and then, and Workbench will help you find and fix them faster than ever.
There's another tool to help you verify and debug your pipelines: we built a Git repository browser into Workbench. You can see the commit that a job used, and browse your repository at that point. This makes it super simple to do quick code checks on a job, like seeing whether files or particular bug fixes are missing.
Git integration improvements don't end with Workbench, though. We also taught Crunch to fetch job code from remote repositories. Now if you find a Crunch script you'd like to run in a public repository, you can do that just by submitting a job or pipeline by writing a public repository URL instead of an Arvados repository name. Arvados will automatically fetch and store everything it needs.
Finally, we made Crunch's resource allocation smarter. In the previous sprint, we imposed memory limits on tasks to prevent one from interfering with others, allocating an equal amount of memory per compute node core. Crunch now recognizes when a job is running fewer tasks than cores available, and provides as much memory as it can to each task while preventing interference. This means Crunch provides better resource allocation to map-reduce and scatter-gather jobs.
We hope these new features are as useful to you as they are to us. You can still try out our open beta program if you haven't already. We'd be happy to hear your feedback on our IRC channel or mailing list. Get in touch!