Arvados

{{>toc}}

_The first time you use this template, read through the entire document to understand how to use it._

_When designing a new system or component, copy this template and then replace the contents of each section with information about your project. Some sections may not be relevant to your project, in which case you should delete them._

_Each section contains italicized instructions explaining what belongs in that section. Many sections also contain example content in_ roman typeface.

_This introduction should be deleted (but not the table of contents markup above it)._

h1=. Design Doc Template

_Replace the previous line with the the title of your project or component and replace the following lines with your name(s), email and the date._

p>. *"Misha Zatsman":mailto:misha@curoverse.com

Last Updated: September 22, 2014*

h2. Overview

_Nothing goes here; all the content belongs in the subsections._

_You should complete these high-level sections first. If your design is elaborate, you may want to pause after these sections to wait for review comments before investing time into planning details that may get changed in the review._

h3. Objective

_In the objective section you should include a succinct 1-3 sentence statement of the objective of your project. It is also useful to state non-goals. Finally state who the intended audience for the document is._

A design document introduces a system or component to a fellow engineer. It should be written *before* implementing the system to aid in planning and to facilitate discussions about design decisions. After implementation, the design doc will serve as a reference for users, maintainers and anyone else interested in the system (and for that reason it is still useful to write design docs for systems that have already been written).

A design document is not a press release, a vision statement, a research report or a business plan.

The intended audience for this document is software engineers, but it should make sense to anyone familiar with software development.

h3. Background

_The background section should contain information the reader needs to know to understand the problem being solved. This can be a combination of text and links to other documents._

At Curoverse, we would like to do a better job documenting our software components. Our current documentation is located in:

* "our public wiki":https://arvados.org/projects/arvados/wiki/

* "our private wiki":https://arvados.org/projects/orvos-private/wiki/

* "other documents":http://doc.arvados.org/

Some Design Docs we've written are:

* [[Keep Design Doc]]

* [[Data Manger Design Doc]]

h3. Alternatives

_This section contains alternative solutions to the stated objective, as well as explanations for why they weren't used. In the planning stage, this section is useful for understanding the value added by the proposed solution. Once the system has been implemented, this section will inform readers of alternative solutions so they can find the best system to address their needs._

Some existing resources are available online:

* Guides to writing design docs:

** http://www.bitformation.com/art/writing_design_docs.html - This is a great guide which I think would complement this template well.

** http://blog.slickedit.com/2007/05/how-to-write-an-effective-design-document/ - Not quite as good (a little too focused on the author's template) but still makes some good points.

* Alternative doc templates:

** http://openacs.org/doc/filename.html - A great template, too specific to OpenACS for us, but still has some great ideas.

** http://www.bitformation.com/art/sample_sw_design_doc.html - Another good template, but more detailed than we're looking for.

* Examples of design docs:

** https://code.google.com/p/google-api-python-client/wiki/ClientDesignDocument

** http://www.chromium.org/developers/design-documents

h3. Tradeoffs

_What tradeoffs were made in this design and *why*. Types of tradeoffs can include: different approaches that could have been taken (e.g. storing data in memory/on disk/on network), or design decisions such as optimizing for latency vs throughput. The important part is to explain your reasoning for making the choice you did (or admitting if you felt the choice was arbitrary)._

Although a design doc takes time to write well, we believe the net result will save time for both the author and fellow engineers. Through writing a design doc, the author is forced to think through and receive feedback on design decisions, thus saving wasted implementation effort. Additionally, once the project is in use, the design doc serves as an overview to new users and a reference to all, reducing the time required for a user or maintainer to familiarize themselves with the system.

We have opted for a shorter format, with fewer sections than other templates, to reduce the pain to authors. In particular, to reduce the pain of redesigns, we have split our template into an [[Design_Doc_template#Overview|Overview]] section to reduce the amount of design required before an initial design review of larger systems.

h3. High Level Design

_A high-level description of the system. This is the most valuable section of the document and will probably receive the most attention. You should explain, at a high level, how your system will work. Don't get bogged down with details, those belong later in the document._

_A diagram showing how the major components communicate is very useful and a great way to start this section. If this system is intended to be a component in a larger system, a diagram showing how it fits in to the larger system will also be appreciated by your readers._

_Most diagrams will need to be updated over time as the design evolves, so please create your diagrams with a program that is easily (and freely) available and attach the diagram source to the document to make it easy for a future maintainer (who could be you) to update the diagrams along with the document._

h2. Specifics

_Nothing goes here; all the content belongs in the subsections._

h3. Detailed Design

_Designs that are too detailed for the above [[Design_Doc_Template#High-Level-Design|High Level Design]] section belong here. Anything that will require a day or more of work to implement, should be described here._

_This is a great place to put APIs, communication protocols, file formats and the like._

_It is important to include assumptions about what external systems will provide. For example if this system has a method that takes a user id as input, will your implementation assume that the user id is valid? Or if a method has a string parameter, does it assume that the parameter has been sanitized against injection attacks? Having such assumptions explicitly spelled out here before you start implementing increases the chances that misunderstandings will be caught by a reviewer before they lead to bugs or vulnerabilities. Please reference the external system's documentation justifying your assumption whenever possible (and if such documentation doesn't exist, ask the external system's author to document the behavior or at least confirm it in an email)._

_Here's an easy rule of thumb for deciding what to write here: Think of anything that would be a pain to change if you were requested to do so in a code review. If you put that implementation detail in here, you'll be less likely to be asked to change it once you've written all the code._

h3. Code Location

_The path of the source code in the repository._

h3. Testing Plan

_How you will verify the behavior of your system. Once the system is written, this section should be updated to reflect the current state of testing and future aspirations._

h3. Logging

_What your system will record and how._

h3. Debugging

_How users can debug interactions with your system. When designing a system it's important to think about what tools you can provide to make debugging problems easier. Sometimes it's unclear whether the problem is in your system at all, so a mechanism for isolating a particular interaction and examining it to see if your system behaved as expected is very valuable. Once a system is in use, this is a great place to put tips and recipes for debugging. If this section grows too large, the mechanisms can be summarized here and individual tips can be moved to another document._

h3. Caveats

_Gotchas, differences between the design and implementation, other potential stumbling blocks for users or maintainers, and their implications and workarounds. Unless something is known to be tricky ahead of time, this section will probably start out empty._

_Rather than deleting it, it's recommended that you keep this section with a simple place holder, since caveats will almost certainly appear down the road._

To be determined.

h3. Security Concerns

_This section should describe possible threats (denial of service, malicious requests, etc) and what, if anything, is being done to protect against them. Be sure to list concerns for which you don't have a solution or you believe don't need a solution. Security concerns that we don't need to worry about also belong here (e.g. we don't need to worry about denial of service attacks for this system because it only receives requests from the api server which already has DOS attack protections)._

h3. Open Questions and Risks

_This section should describe design questions that have not been decided yet, research that needs to be done and potential risks that could make make this system less effective or more difficult to implement._

_Some examples are: Should we communicate using TCP or UDP? How often do we expect our users to interrupt running jobs? This relies on an undocumented third-party API which may be turned off at any point._

_For each question you should include any relevant information you know. For risks you should include estimates of likelihood, cost if they occur and ideas for possible workarounds._

h3. Work Estimates

_Split the work into milestones that can be delivered, put them in the order that you think they should be done, and estimate roughly how much time you expect it each milestone to take. Ideally each milestone will take one week or less._

h3. Future Work

_Features you'd like to (or will need to) add but aren't required for the current release. This is a great place to speculate on potential features and performance improvements._

h3. Revision History

_The table below should record the major changes to this document. You don't need to add an entry for typo fixes, other small changes or changes before finishing the initial draft._

|_.Date            |_.Revisions Made |_.Author            |_.Reviewed By     |

| June 5, 2014 | Initial Draft         | Misha Zatsman |=. ----              |

| June 9, 2014 | Updated after comments         | Misha Zatsman | Tom Clegg, Tim Pierce              |