Story #9053

Port API server to Go

Added by Tom Clegg almost 4 years ago. Updated 9 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
10/01/2020
Due date:
04/01/2021
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

Background

We want to replace our Rails server with a Go server. However:
  • Porting the entire code base to Go, and cutting over everything at once, seems unlikely to go smoothly.
  • In the time it takes to port, the Rails functionality will change. Keeping them in sync will be challenging.
  • Porting the functionality and the corresponding tests at once is likely to result in unnoticed errors and omissions.

Development approach

The first version of the Go service is just an HTTP proxy: it forwards each request to a configured upstream Rails API server. Nginx configs in the install docs (and existing deployments) are updated to send API requests to this new server. #13497

Next, port some functional/integration tests from the Rails server code base to the Go code base. These tests will pass because Go server is already proxying the relevant requests to the Rails server. (Mocking won't be portable, though!)

Each test that has been ported to Go can be skipped in the Rails code base.

When enough Rails tests are skipped to remove an API from Rails without making the remaining tests fail, that API can be ported to Go. The relevant tests are already in place in the Go code base, so they should be a reasonable way to confirm that the new Go port is correct.

Repeat until all tests are ported to Go, and they pass even with the proxy-to-Rails feature removed.

Caveats / TBD

Rails tests are fast (at least faster than Workbench tests, for example) because they use transaction rollback to reset database state. This option won't be available when tests run in a different (Go) process. Practically, this means the Go tests will need to use the "database reset" API, and they will run slowly.
  • Perhaps we can start by implementing "database reset" in Go? It will have direct access to the Postgres database, and it could take a snapshot of the database at startup instead of reading the Rails YAML fixtures.
  • Is there a better way around this?

Related issues

Related to Arvados - Story #9502: [API] Update permissions cache as needed after select writesResolved06/28/2016

Related to Arvados - Feature #13493: Federated record retrievalResolved06/28/2018

Related to Arvados - Story #13995: [Controller] Port "collection update" functional tests from Rails API to controller test suiteNew

Related to Arvados - Story #13997: [Controller] Port "collection update" API from RailsNew

Blocked by Arvados - Story #13497: [API] Initial "arvados-controller" server that proxies API endpoints to Rails serverResolved06/15/2018

History

#1 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)

#2 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)

#3 Updated by Tom Clegg almost 2 years ago

#4 Updated by Tom Clegg almost 2 years ago

  • Subject changed from [API] Deployable Go API server that proxies unimplemented APIs to Rails to [Epic] Port APIs to Go
  • Description updated (diff)

#5 Updated by Tom Clegg almost 2 years ago

  • Blocked by Story #13497: [API] Initial "arvados-controller" server that proxies API endpoints to Rails server added

#6 Updated by Tom Clegg over 1 year ago

  • Description updated (diff)

#7 Updated by Tom Clegg over 1 year ago

  • Related to Story #13995: [Controller] Port "collection update" functional tests from Rails API to controller test suite added

#8 Updated by Tom Clegg over 1 year ago

  • Related to Story #13997: [Controller] Port "collection update" API from Rails added

#9 Updated by Tom Morris over 1 year ago

  • Release deleted (12)

#10 Updated by Peter Amstutz 3 months ago

  • Subject changed from [Epic] Port APIs to Go to Port API server to Go
  • Project changed from Arvados to Arvados Epics

#11 Updated by Peter Amstutz 3 months ago

  • Start date set to 08/01/2020
  • Due date set to 02/01/2021

#12 Updated by Peter Amstutz 3 months ago

  • Release set to 28

#13 Updated by Peter Amstutz 9 days ago

  • Start date changed from 08/01/2020 to 09/01/2020
  • Due date changed from 02/01/2021 to 03/01/2021

#14 Updated by Peter Amstutz 9 days ago

  • Start date changed from 09/01/2020 to 10/01/2020
  • Due date changed from 03/01/2021 to 04/01/2021

Also available in: Atom PDF