Feature #16462

Expand arvados-controller to expose forecast features

Added by Nico César 8 months ago. Updated about 1 month ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Nico César
Category:
API
Target version:
2021-01-20 Sprint
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Arvados-controller forecast endpoints should be created based on the work done in git.arvados.org:arvados-forecaster.git

timeline.html (8.34 KB) timeline.html Nico César, 06/03/2020 11:01 PM
report.html (2.81 MB) report.html Nico César, 06/03/2020 11:01 PM
su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf (26.7 KB) su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf Nico César, 06/06/2020 01:43 PM
su92l-xvhdp-3gri0mi1vtakaf4_just_circles.dot (8.1 KB) su92l-xvhdp-3gri0mi1vtakaf4_just_circles.dot sourcecode for the pdf Nico César, 06/06/2020 01:43 PM
su92l-xvhdp-3gri0mi1vtakaf4.png (810 KB) su92l-xvhdp-3gri0mi1vtakaf4.png Nico César, 06/06/2020 01:43 PM

Subtasks

Task #16546: ReviewNewTom Clegg

Related issues

Related to Arvados Epics - Story #16517: Workflow runtime/cost visibility and forecastingIn Progress05/01/202006/30/2021

Related to Arvados - Story #16534: [controller] facilitate database access by localdb API methodsResolved06/26/2020

Related to Arvados - Feature #17054: Custom naming for scatter stepsNew

Blocked by Arvados - Feature #17014: Add Container Requests endpoint to controllerIn Progress11/10/2020

Associated revisions

Revision 3e8a3101 (diff)
Added by Nico Cesar 7 months ago

added type ContainerRequestList

refs #16462

Arvados-DCO-1.1-Signed-off-by: <>

Revision 96e01b6f (diff)
Added by Nico Cesar 5 months ago

initial go mod tidy

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision ad05fa2c (diff)
Added by Nico Cesar 5 months ago

Boilerplate for all the controller and federation needed for forecast

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 3d4c3231 (diff)
Added by Nico Cesar 4 months ago

replaced arvados.ForecastDatapointsOptions for arvados.GetOptions

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 86d6ed00 (diff)
Added by Nico Cesar 4 months ago

added generated code for CR, EndPoints for /a/v1/container_requests

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision fadbf1d8 (diff)
Added by Nico Cesar 4 months ago

copy and paste leftover

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 5a2c3c48 (diff)
Added by Nico Cesar 4 months ago

CR route were missing in controller/router/router.go

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision f716696e (diff)
Added by Nico Cesar 4 months ago

Trying out sqlx options and StructScan()

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 549c1548 (diff)
Added by Nico Cesar 4 months ago

Added all the ContainerRequest api call to APIStub

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0875fa95 (diff)
Added by Nico Cesar 4 months ago

First working datapoint endpoints for CR

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision d8d31995 (diff)
Added by Nico Cesar 4 months ago

Typos

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 5743f9d6
Added by Nico Cesar 4 months ago

Merge branch 'master' into 16462-forecast-wip

Once more having go.sum in my branch has lots of issues.
adding the one in master for now. I'll make a better commit once this
is about to be merge to master

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 9cce9619 (diff)
Added by Nico Cesar 4 months ago

test are not happy. maybe the transaction gets rolledback?

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 7f216ff3 (diff)
Added by Nico Cesar 4 months ago

adding fixtures for container and container_requests

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0c302f31 (diff)
Added by Nico Cesar 4 months ago

fixtures now are in rails app

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 5846dac4 (diff)
Added by Nico Cesar 4 months ago

circular dependencies, trying to get a real connection

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 9893aad5 (diff)
Added by Nico Cesar 3 months ago

owner of the fixtures should be the "active user" of the test

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0c0e656b (diff)
Added by Nico Cesar 3 months ago

Added context to use fixture user's credentials

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 790e0d29
Added by Nico Cesar 3 months ago

Merge branch 'master' into 16462-forecast-wip-ldp

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 0dccb3d7
Added by Nico Cesar 3 months ago

Merge branch 'master' into 16462-forecast-wip

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision d81436f3 (diff)
Added by Nico Cesar 3 months ago

Good progress! first tests pass

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 74a8f47e (diff)
Added by Nico Cesar 3 months ago

added *golden to .licenseignore

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision 71733fbe (diff)
Added by Nico Cesar 3 months ago

First batch of passing tests with golden files

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision f6ccc08c (diff)
Added by Nico Cesar 3 months ago

Added /arvados/v1/container_requests/UUID/datapoints endpoint

This is the first commit to try to implement Historical Forecasting
https://dev.arvados.org/projects/arvados/wiki/API_HistoricalForcasting_data_for_CR

This will enable to create Gantt diagrams that will help visualize the
times and dependencies of a run.

refs #16462

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

Revision c0916b95 (diff)
Added by Nico Cesar about 1 month ago

Added /a/v1/<cr_uuid>/datapoints endpoint to controller

This builds on top of the work we've done in 17014 this is
the inclusion of the forecast datapoints and the relevant tests

refs #16462
refs #17014

Arvados-DCO-1.1-Signed-off-by: Nico Cesar <>

History

#1 Updated by Peter Amstutz 8 months ago

  • Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint

**

#2 Updated by Nico César 8 months ago

As a reference this is what nextflow has as a report

https://www.nextflow.io/docs/latest/tracing.html

#3 Updated by Nico César 8 months ago

In order to determine dependencies I added a "<workflow>-start" and "<workflow>-end" in the intermediate graph you can see this in: su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf

Output.Source and Input.Source have been a source of trouble to connect the dots to make dependencies work.

#4 Updated by Nico César 7 months ago

Longest running haplotypecaller: 1d21h49m:
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-iqy6soi7sz4runm (Notice that the command doesn't have the -L)

While most of the other ones are 2h to 4h range, as an example haplotypecaller_9:

https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-cpoakbi81qqsbke

As we discussed before, we can't take just the name of the step to do the bookkeeping of the metrics (running time in this case, but more to come),
Currently we use: "duration:<checkpoint>#<containerUUID>" as the key where checkpoint is "haplotypecaller" and anything after "#" is ignored on summarizing results.

Here the strategies are several:
  1. get the parent container request that has a workflow.json to compare to "the family of workflows" and use that as part of the key when storing the
  2. make some kind of signature of the command used for that Container, this will take into account the command line, but also get a little more creative and have input parsing and having some extra metrics comparing input sizes for example.
  3. a mix of 1 and 2, starting with one and try to get a sense of the clustering happens for all data

#5 Updated by Nico César 7 months ago

added branch 16462-go-sdk-patch 3e8a3101cb9191813f3c8ed557d6f189d3e42063

#7 Updated by Nico César 7 months ago

  • Related to Story #16517: Workflow runtime/cost visibility and forecasting added

#8 Updated by Nico César 7 months ago

  • Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint

After talking to Lucas and Tom this is the WIP: https://dev.arvados.org/projects/arvados/wiki/API_HistoricalForcasting_data_for_CR

(for later review)

#9 Updated by Tom Clegg 7 months ago

  • Related to Story #16534: [controller] facilitate database access by localdb API methods added

#10 Updated by Peter Amstutz 7 months ago

  • Target version changed from 2020-07-01 Sprint to 2020-07-15

#11 Updated by Nico César 6 months ago

  • Target version changed from 2020-07-15 to 2020-08-12 Sprint

#12 Updated by Nico César 6 months ago

  • Status changed from New to In Progress

#13 Updated by Nico César 5 months ago

  • Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint

#14 Updated by Nico César 5 months ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint

#15 Updated by Nico César 4 months ago

  • Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint

#16 Updated by Nico César 4 months ago

  • Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint

#17 Updated by Nico César 4 months ago

9cce9619f3438011a42e5d10ca6c8584f0788381 (branch 16462-forecast-wip ) has the initial version of database tests

#18 Updated by Nico César 4 months ago

Note to future self/reviewer : make sure GET https://&lt;API&gt;/arvados/v1/container_requests/ works as expected

#19 Updated by Peter Amstutz 3 months ago

  • Release set to 25

#20 Updated by Peter Amstutz 3 months ago

  • Release deleted (25)

#21 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint

#22 Updated by Nico César 3 months ago

16462-forecast-wip2 (f6ccc08c3f6b1ad42f2c827b19df0300f2c3c3db) can be reviewed now, This is unfinished work but might be useful to know if I'm heading in the right direction.

I see that some of my changes broke federation:
https://ci.arvados.org/job/developer-run-tests-remainder/2227/console

#23 Updated by Nico César 3 months ago

  • Blocked by Feature #17014: Add Container Requests endpoint to controller added

#24 Updated by Nico César 3 months ago

  • Target version changed from 2020-10-21 Sprint to 2020-11-18

#25 Updated by Nico César 3 months ago

#26 Updated by Nico César 3 months ago

  • Target version changed from 2020-11-18 to 2020-12-02 Sprint

#27 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2020-12-02 Sprint to 2020-12-16 Sprint

#28 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2020-12-16 Sprint to 2021-01-06 Sprint

#30 Updated by Peter Amstutz about 1 month ago

  • Target version changed from 2021-01-06 Sprint to 2021-01-20 Sprint

Also available in: Atom PDF