Arvados

h1. Go

{{toc}}

h2. Using Go with Arvados

h3. Install Go

You'll need *Go 1.6 or newer*. If you're installing Go on your linux machine you'll want to follow the "tarball instructions":http://golang.org/doc/install#tarball to get the latest stable version, because if you use <code>apt-get</code> you'll get an outdated version.

The first time you install Go, you'll need to set @GOPATH@ to an empty directory.  The Go toolchain will install Go packages and dependencies here. 

<pre>

export GOPATH=~/gocode

mkdir -p $GOPATH

</pre>

The rest of these instructions assume that you have a working Go installation on your system, and that your @GOPATH@ environment variable is set appropriately.

h3. Install Arvados source

Clone the Arvados git repository, if you have not already:

<pre>

cd

git clone git://git.curoverse.com/arvados.git

</pre>

*Note:* If you are an authorized committer, clone @git@git.curoverse.com:arvados.git@ instead so you may push directly to git.curoverse.com.

h3. Tell Go to use your local Arvados source

This step ensures that your development environment uses your locally-modified code, instead of fetching the master branch from git.curoverse.com:

<pre>

mkdir -p $GOPATH/src/git.curoverse.com

ln -s ~/arvados $GOPATH/src/git.curoverse.com/arvados.git

</pre>

Reason: The Keepstore and Keepproxy packages import other Go packages from the Arvados source tree. These packages have names like:

<pre>

git.curoverse.com/arvados.git/sdk/go/keepclient

git.curoverse.com/arvados.git/sdk/go/arvadosclient

</pre>

When the Go compiler needs to import one of these packages, it will look in @$GOPATH/src@ for the package source code. If it does not find the code locally, it will fetch the code from git.curoverse.com automatically.  This symlink ensures that Go will find your local source code under @$GOPATH/src/git.curoverse.com/arvados.git/...@

h3. Run some tests

e.g.

<pre>

go get -t git.curoverse.com/arvados.git/services/keepstore

cd ~/arvados/services/keepstore

go test

</pre>

The @go test@ command will print a few dozen lines of logging output.  If the tests succeeded, it will print PASS followed by a summary of the packages which passed testing, e.g.:

<pre>

PASS

ok  	_/home/you/arvados/services/keepstore	1.023s

</pre>

h2. Arvados Go SDK

h3. Documentation

For more information consult the "Arvados Go SDK documentation":http://doc.arvados.org/sdk/go/

h3. Recipes

This section contains code demonstrating aspects of the Arvados Go SDK. Additional examples can be found in the "Arvados Go SDK documentation":http://doc.arvados.org/sdk/go/

h4. Collections filtered and ordered

This example prints collections last modified after a given date, grouped by owner (owner UUID sorted DESCending) and then ordered by oldest modifications (i.e. ASCending modification date).

_Although not an issue in this example, it's worth noting that Arvados only stores the latest modification date for each collection. Therefore if you request collections modified before a given date, the SDK will not return collections who had also been modified after that date._

<pre>

package main

// *******************

// Import the modules.

import (

	"git.curoverse.com/arvados.git/sdk/go/arvadosclient"

	"log"

)

type SdkCollectionInfo struct {

	Uuid           string   `json:"uuid"`

	OwnerUuid      string   `json:"owner_uuid"`

	Redundancy     int      `json:"redundancy"`

	ModifiedAt     string   `json:"modified_at"`

	ManifestText   string   `json:"manifest_text"`

}

type SdkCollectionList struct {

	ItemsAvailable   int                   `json:"items_available"`

	Items            []SdkCollectionInfo   `json:"items"`

}

func main() {

	// ********************************

	// Set up an API client user agent.

	//

	arv, err := arvadosclient.MakeArvadosClient()

	if err != nil {

		log.Fatalf("Error setting up arvados client %s", err.Error())

	}

	GetCollections(arv)

}

func GetCollections(arv arvadosclient.ArvadosClient) () {

	fieldsWanted := []string{"manifest_text",

		"owner_uuid",

		"uuid",

		"redundancy",

		"modified_at"}

	sdkParams := arvadosclient.Dict{

		"select": fieldsWanted,

		"order": []string{"owner_uuid DESC", "modified_at ASC"},

		"filters": [][]string{[]string{"modified_at", ">=", "2014-01-03T06:53:08Z"}}}

	var collections SdkCollectionList

	err := arv.List("collections", sdkParams, &collections)

	if err != nil {

		log.Fatalf("error querying collections: %v", err)

	}

	for _, collection := range collections.Items {

		log.Printf("Seeing owner uuid, modification date: %s %s",

			collection.OwnerUuid,

			collection.ModifiedAt)

	}

	return

}

</pre>

h4. Writing a Log Entry

This example records a minimal log entry.

For more fields you can write, see "the Log Schema":http://doc.arvados.org/api/schema/Log.html

<pre>

package main

import (

	"git.curoverse.com/arvados.git/sdk/go/arvadosclient"

	"log"

)

func main() {

	arv, err := arvadosclient.MakeArvadosClient()

	if err != nil {

		log.Fatalf("Error setting up arvados client %v", err)

	}

	err = arv.Create("logs",

		arvadosclient.Dict{"log": arvadosclient.Dict{

			"event_type": "experimental-logger-testing",

			"properties": arvadosclient.Dict{

				"ninja": "Misha"}}},

		nil)

	if err != nil {

		log.Fatalf("Error writing log: %v", err)

	}

}

</pre>

h2. Learning Go

Getting good at Go, and concurrency/goroutines in particular, is an excellent use of time.

h3. Introductions

* "Go Tour":http://tour.golang.org/ is a great way to spend a few hours getting your feet wet.

* "Effective Go":http://golang.org/doc/effective_go.html The web site says that you should do both the tour and the language specification first before reading this, but honestly, I think it's ridiculous to try to read the language specification before you even start writing code in the language. I wouldn't try to read either the specification or "Effective Go" in depth before trying to write any code, but they're both at least worth skimming at this point.

* The slides for Rob Pike's talk on "Go Concurrency Patterns":http://talks.golang.org/2012/concurrency.slide#1. I haven't watched all of the talks yet; these slides are really nice because they show very elegant ways of using channels and goroutines to build rich and complex concurrent abstractions.

* "The package documentation.":http://golang.org/pkg/ Note that the package docs include links to web versions of the package source code, which help very much in learning idiomatic Go patterns.

h3. Resources

h4. Free online resources:

* http://golang.org

* http://golang.org/doc/code.html

* http://tour.golang.org/

* https://gobyexample.com/

* http://learnxinyminutes.com/docs/go/

h4. Books

* http://www.golang-book.com/ (Free!)

* "Programming in Go":http://www.amazon.com/Programming-Go-Creating-Applications-Developers/dp/0321774639/ref=pd_bxgy_b_img_y (a bit dated now)