May development review: Project promotion and Keep performance

Added by Brett Smith almost 2 years ago

We want Arvados to be the premier way for bioinformaticians and data scientists to collaborate. And as an open source project, we know that helping people understand your work is an important step to get others interested and joining you. In the last sprint, we added Open Graph support to public projects. Now when you share those projects on social media, people will see a useful preview with basic project information and a short introduction. We also improved the project description editing interface, so you'll have an easier time writing the information visitors need right on the project's front page.

Looking under Arvados' hood a bit, we also made several improvements to Keep performance, both on the server and client. The server's request handling has been reorganized to send each request through a wait queue fewer times, and it notifies clients when that queue is full to avoid overloading. Our Python Keep client library now makes its requests with PycURL library. In our experience, PycURL lets us more accurately detect when a server is unreachable. This means our code can still retry requests quickly in that scenario, but we have more reliability in other bad conditions like a strained network. In the I/O-heavy pipelines our users run, these little improvements add up quickly.

The performance improvements won't end there. We're also putting the speed of Arvados collections under a microscope. We've collected a variety of performance metrics about them across the system, and we have plans for improvement over the next few development cycles. Watch this space for the updates.

If you have questions or feedback about these or other updates, don't hesitate to drop us a line in our IRC channel or mailing list. Or if you just want to kick the tires, our open beta program is running the very latest Arvados code. Let us know what you think.