Feature #16462
Expand arvados-controller to expose forecast features
0%
Description
Arvados-controller forecast endpoints should be created based on the work done in git.arvados.org:arvados-forecaster.git
Subtasks
Related issues
Associated revisions
Boilerplate for all the controller and federation needed for forecast
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
replaced arvados.ForecastDatapointsOptions for arvados.GetOptions
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
added generated code for CR, EndPoints for /a/v1/container_requests
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
CR route were missing in controller/router/router.go
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Trying out sqlx options and StructScan()
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Added all the ContainerRequest api call to APIStub
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
First working datapoint endpoints for CR
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Merge branch 'master' into 16462-forecast-wip
Once more having go.sum in my branch has lots of issues.
adding the one in master for now. I'll make a better commit once this
is about to be merge to master
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
test are not happy. maybe the transaction gets rolledback?
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
adding fixtures for container and container_requests
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
fixtures now are in rails app
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
circular dependencies, trying to get a real connection
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
owner of the fixtures should be the "active user" of the test
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Added context to use fixture user's credentials
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Merge branch 'master' into 16462-forecast-wip-ldp
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Merge branch 'master' into 16462-forecast-wip
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Good progress! first tests pass
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
added *golden to .licenseignore
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
First batch of passing tests with golden files
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Added /arvados/v1/container_requests/UUID/datapoints endpoint
This is the first commit to try to implement Historical Forecasting
https://dev.arvados.org/projects/arvados/wiki/API_HistoricalForcasting_data_for_CR
This will enable to create Gantt diagrams that will help visualize the
times and dependencies of a run.
refs #16462
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
Added /a/v1/<cr_uuid>/datapoints endpoint to controller
This builds on top of the work we've done in 17014 this is
the inclusion of the forecast datapoints and the relevant tests
Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>
History
#1
Updated by Peter Amstutz 11 months ago
- Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint
**
#2
Updated by Nico César 11 months ago
- File timeline.html timeline.html added
- File report.html report.html added
As a reference this is what nextflow has as a report
#3
Updated by Nico César 11 months ago
- File su92l-xvhdp-3gri0mi1vtakaf4.png su92l-xvhdp-3gri0mi1vtakaf4.png added
- File su92l-xvhdp-3gri0mi1vtakaf4_just_circles.dot su92l-xvhdp-3gri0mi1vtakaf4_just_circles.dot added
- File su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf added
In order to determine dependencies I added a "<workflow>-start" and "<workflow>-end" in the intermediate graph you can see this in: su92l-xvhdp-3gri0mi1vtakaf4_intermediate_graph.pdf
Output.Source and Input.Source have been a source of trouble to connect the dots to make dependencies work.
#4
Updated by Nico César 10 months ago
Longest running haplotypecaller: 1d21h49m:
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-iqy6soi7sz4runm (Notice that the command doesn't have the -L)
While most of the other ones are 2h to 4h range, as an example haplotypecaller_9:
https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-cpoakbi81qqsbke
As we discussed before, we can't take just the name of the step to do the bookkeeping of the metrics (running time in this case, but more to come),
Currently we use: "duration:<checkpoint>#<containerUUID>" as the key where checkpoint is "haplotypecaller" and anything after "#" is ignored on summarizing results.
- get the parent container request that has a workflow.json to compare to "the family of workflows" and use that as part of the key when storing the
- make some kind of signature of the command used for that Container, this will take into account the command line, but also get a little more creative and have input parsing and having some extra metrics comparing input sizes for example.
- a mix of 1 and 2, starting with one and try to get a sense of the clustering happens for all data
#5
Updated by Nico César 10 months ago
added branch 16462-go-sdk-patch 3e8a3101cb9191813f3c8ed557d6f189d3e42063
#6
Updated by Tom Clegg 10 months ago
3e8a3101cb9191813f3c8ed557d6f189d3e42063 LGTM, thanks
#7
Updated by Nico César 10 months ago
- Related to Story #16517: Workflow runtime/cost visibility and forecasting added
#8
Updated by Nico César 10 months ago
- Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint
After talking to Lucas and Tom this is the WIP: https://dev.arvados.org/projects/arvados/wiki/API_HistoricalForcasting_data_for_CR
(for later review)
#9
Updated by Tom Clegg 10 months ago
- Related to Story #16534: [controller] facilitate database access by localdb API methods added
#10
Updated by Peter Amstutz 10 months ago
- Target version changed from 2020-07-01 Sprint to 2020-07-15
#11
Updated by Nico César 9 months ago
- Target version changed from 2020-07-15 to 2020-08-12 Sprint
#12
Updated by Nico César 9 months ago
- Status changed from New to In Progress
#13
Updated by Nico César 8 months ago
- Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint
#14
Updated by Nico César 8 months ago
- Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint
#15
Updated by Nico César 7 months ago
- Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint
#16
Updated by Nico César 7 months ago
- Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint
#17
Updated by Nico César 7 months ago
9cce9619f3438011a42e5d10ca6c8584f0788381 (branch 16462-forecast-wip ) has the initial version of database tests
#18
Updated by Nico César 7 months ago
Note to future self/reviewer : make sure GET https://<API>/arvados/v1/container_requests/ works as expected
#19
Updated by Peter Amstutz 6 months ago
- Release set to 25
#20
Updated by Peter Amstutz 6 months ago
- Release deleted (
25)
#21
Updated by Peter Amstutz 6 months ago
- Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint
#22
Updated by Nico César 6 months ago
16462-forecast-wip2 (f6ccc08c3f6b1ad42f2c827b19df0300f2c3c3db) can be reviewed now, This is unfinished work but might be useful to know if I'm heading in the right direction.
I see that some of my changes broke federation:
https://ci.arvados.org/job/developer-run-tests-remainder/2227/console
#23
Updated by Nico César 6 months ago
- Blocked by Feature #17014: Add Container Requests endpoint to controller added
#24
Updated by Nico César 6 months ago
- Target version changed from 2020-10-21 Sprint to 2020-11-18
#25
Updated by Nico César 6 months ago
- Related to Feature #17054: Custom naming for scatter steps added
#26
Updated by Nico César 5 months ago
- Target version changed from 2020-11-18 to 2020-12-02 Sprint
#27
Updated by Peter Amstutz 5 months ago
- Target version changed from 2020-12-02 Sprint to 2020-12-16 Sprint
#28
Updated by Peter Amstutz 5 months ago
- Target version changed from 2020-12-16 Sprint to 2021-01-06 Sprint
#29
Updated by Nico César 4 months ago
c0916b956054b2e16ff61bd33b11bfe07e81787d branch 16462-CR-datapoints
https://ci.arvados.org/view/Developer/job/developer-run-tests/2227/
#30
Updated by Peter Amstutz 4 months ago
- Target version changed from 2021-01-06 Sprint to 2021-01-20 Sprint
#31
Updated by Nico César 3 months ago
- Target version changed from 2021-01-20 Sprint to To Be Groomed
#32
Updated by Nico César 17 days ago
- Status changed from In Progress to New
added type ContainerRequestList
refs #16462
Arvados-DCO-1.1-Signed-off-by: <nico@nicocesar.com>