Feature #16462

Expand arvados-controller to expose forecast features

Added by Nico César 5 months ago. Updated 3 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Nico César
Category:
API
Target version:
2020-11-18
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Arvados-controller forecast endpoints should be created based on the work done in git.arvados.org:arvados-forecaster.git

timeline.html (8.34 KB) timeline.html Nico César, 06/03/2020 11:01 PM
report.html (2.81 MB) report.html Nico César, 06/03/2020 11:01 PM
su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf (26.7 KB) su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf Nico César, 06/06/2020 01:43 PM
su92l-xvhdp-3gri0mi1vtakaf4_just_circles.dot (8.1 KB) su92l-xvhdp-3gri0mi1vtakaf4_just_circles.dot sourcecode for the pdf Nico César, 06/06/2020 01:43 PM
su92l-xvhdp-3gri0mi1vtakaf4.png (810 KB) su92l-xvhdp-3gri0mi1vtakaf4.png Nico César, 06/06/2020 01:43 PM

Subtasks

Task #16546: ReviewNewTom Clegg

Related issues

Related to Arvados Epics - Story #16517: Workflow runtime/cost visibility and forecastingIn Progress05/01/202010/30/2020

Related to Arvados - Story #16534: [controller] facilitate database access by localdb API methodsResolved06/26/2020

Blocked by Arvados - Feature #17014: Add Container Requests endpoint to controllerIn Progress

Associated revisions

Revision 3e8a3101 (diff)
Added by Nico Cesar 5 months ago

added type ContainerRequestList

refs #16462

Arvados-DCO-1.1-Signed-off-by: <>

Revision 96e01b6f (diff)
Added by Nico Cesar 2 months ago

initial go mod tidy

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision ad05fa2c (diff)
Added by Nico Cesar 2 months ago

Boilerplate for all the controller and federation needed for forecast

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 3d4c3231 (diff)
Added by Nico Cesar about 1 month ago

replaced arvados.ForecastDatapointsOptions for arvados.GetOptions

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 86d6ed00 (diff)
Added by Nico Cesar about 1 month ago

added generated code for CR, EndPoints for /a/v1/container_requests

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision fadbf1d8 (diff)
Added by Nico Cesar about 1 month ago

copy and paste leftover

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 5a2c3c48 (diff)
Added by Nico Cesar about 1 month ago

CR route were missing in controller/router/router.go

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision f716696e (diff)
Added by Nico Cesar about 1 month ago

Trying out sqlx options and StructScan()

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 549c1548 (diff)
Added by Nico Cesar about 1 month ago

Added all the ContainerRequest api call to APIStub

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0875fa95 (diff)
Added by Nico Cesar about 1 month ago

First working datapoint endpoints for CR

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision d8d31995 (diff)
Added by Nico Cesar about 1 month ago

Typos

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 5743f9d6
Added by Nico Cesar about 1 month ago

Merge branch 'master' into 16462-forecast-wip

Once more having go.sum in my branch has lots of issues.
adding the one in master for now. I'll make a better commit once this
is about to be merge to master

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 9cce9619 (diff)
Added by Nico Cesar about 1 month ago

test are not happy. maybe the transaction gets rolledback?

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 7f216ff3 (diff)
Added by Nico Cesar 26 days ago

adding fixtures for container and container_requests

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0c302f31 (diff)
Added by Nico Cesar 25 days ago

fixtures now are in rails app

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 5846dac4 (diff)
Added by Nico Cesar 25 days ago

circular dependencies, trying to get a real connection

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 9893aad5 (diff)
Added by Nico Cesar 11 days ago

owner of the fixtures should be the "active user" of the test

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0c0e656b (diff)
Added by Nico Cesar 11 days ago

Added context to use fixture user's credentials

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 790e0d29
Added by Nico Cesar 11 days ago

Merge branch 'master' into 16462-forecast-wip-ldp

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0dccb3d7
Added by Nico Cesar 10 days ago

Merge branch 'master' into 16462-forecast-wip

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision d81436f3 (diff)
Added by Nico Cesar 10 days ago

Good progress! first tests pass

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 74a8f47e (diff)
Added by Nico Cesar 9 days ago

added *golden to .licenseignore

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 71733fbe (diff)
Added by Nico Cesar 9 days ago

First batch of passing tests with golden files

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision f6ccc08c (diff)
Added by Nico Cesar 9 days ago

Added /arvados/v1/container_requests/UUID/datapoints endpoint

This is the first commit to try to implement Historical Forecasting
https://dev.arvados.org/projects/arvados/wiki/API_HistoricalForcasting_data_for_CR

This will enable to create Gantt diagrams that will help visualize the
times and dependencies of a run.

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

History

#1 Updated by Peter Amstutz 5 months ago

  • Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint

**

#2 Updated by Nico César 5 months ago

As a reference this is what nextflow has as a report

https://www.nextflow.io/docs/latest/tracing.html

#3 Updated by Nico César 5 months ago

In order to determine dependencies I added a "<workflow>-start" and "<workflow>-end" in the intermediate graph you can see this in: su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf

Output.Source and Input.Source have been a source of trouble to connect the dots to make dependencies work.

#4 Updated by Nico César 5 months ago

Longest running haplotypecaller: 1d21h49m:
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-iqy6soi7sz4runm (Notice that the command doesn't have the -L)

While most of the other ones are 2h to 4h range, as an example haplotypecaller_9:

https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-cpoakbi81qqsbke

As we discussed before, we can't take just the name of the step to do the bookkeeping of the metrics (running time in this case, but more to come),
Currently we use: "duration:<checkpoint>#<containerUUID>" as the key where checkpoint is "haplotypecaller" and anything after "#" is ignored on summarizing results.

Here the strategies are several:
  1. get the parent container request that has a workflow.json to compare to "the family of workflows" and use that as part of the key when storing the
  2. make some kind of signature of the command used for that Container, this will take into account the command line, but also get a little more creative and have input parsing and having some extra metrics comparing input sizes for example.
  3. a mix of 1 and 2, starting with one and try to get a sense of the clustering happens for all data

#5 Updated by Nico César 5 months ago

added branch 16462-go-sdk-patch 3e8a3101cb9191813f3c8ed557d6f189d3e42063

#7 Updated by Nico César 5 months ago

  • Related to Story #16517: Workflow runtime/cost visibility and forecasting added

#8 Updated by Nico César 4 months ago

  • Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint

After talking to Lucas and Tom this is the WIP: https://dev.arvados.org/projects/arvados/wiki/API_HistoricalForcasting_data_for_CR

(for later review)

#9 Updated by Tom Clegg 4 months ago

  • Related to Story #16534: [controller] facilitate database access by localdb API methods added

#10 Updated by Peter Amstutz 4 months ago

  • Target version changed from 2020-07-01 Sprint to 2020-07-15

#11 Updated by Nico César 3 months ago

  • Target version changed from 2020-07-15 to 2020-08-12 Sprint

#12 Updated by Nico César 3 months ago

  • Status changed from New to In Progress

#13 Updated by Nico César 2 months ago

  • Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint

#14 Updated by Nico César about 2 months ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint

#15 Updated by Nico César about 2 months ago

  • Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint

#16 Updated by Nico César about 1 month ago

  • Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint

#17 Updated by Nico César about 1 month ago

9cce9619f3438011a42e5d10ca6c8584f0788381 (branch 16462-forecast-wip ) has the initial version of database tests

#18 Updated by Nico César 26 days ago

Note to future self/reviewer : make sure GET https://&lt;API&gt;/arvados/v1/container_requests/ works as expected

#19 Updated by Peter Amstutz 18 days ago

  • Release set to 25

#20 Updated by Peter Amstutz 17 days ago

  • Release deleted (25)

#21 Updated by Peter Amstutz 17 days ago

  • Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint

#22 Updated by Nico César 9 days ago

16462-forecast-wip2 (f6ccc08c3f6b1ad42f2c827b19df0300f2c3c3db) can be reviewed now, This is unfinished work but might be useful to know if I'm heading in the right direction.

I see that some of my changes broke federation:
https://ci.arvados.org/job/developer-run-tests-remainder/2227/console

#23 Updated by Nico César 3 days ago

  • Blocked by Feature #17014: Add Container Requests endpoint to controller added

#24 Updated by Nico César 3 days ago

  • Target version changed from 2020-10-21 Sprint to 2020-11-18

Also available in: Atom PDF