Project

General

Profile

API HistoricalForcasting data for CR » History » Version 3

Nico César, 06/22/2020 02:43 PM

1 1 Nico César
h1. API Historical/Forcasting data for CR
2
3
Goal: create a pipeline forecaster and visualization for historical data. This should expose APIs that can be used in the ContainerRequest visualization and 
4
also could be use to provide extra information for the current running CR
5
6
Glossary:
7
8 3 Nico César
* Checkpoint: is a generic name that currently corresponds to a step name. This id together with "family" make a unique cluster to summarize results. This summarization for the unique cluster includes: a) several runs with similar parameters and b) scattered steps that have the pattern: name_2, name_3,..., name_229
9 1 Nico César
10 3 Nico César
* Family: A common name like "gatk" or "haplotypecaller" can be used as a step name. The family definition will help to separate the 2 populationsn terms of checkpoints. We think that implementing this based the parameters of CommandLineTool and parent workflow md5sum or a combination of both
11 1 Nico César
12 2 Nico César
* Datapoint: a concrete data that can be plotted as historical data. Currently we're bounding together the container request and the associated container to have a unified view of the times involved. This should not get confused with forecast data since can be used separately
13 1 Nico César
14
h2. API 
15
16
GET /container-request/aaaaa-xvhdp-123456789abc/checkpoints
17
18
Output:
19
20
<pre>
21
{
22
  "checkpoints": [
23
    {
24 3 Nico César
      "name": "merge-tilelib",
25
      "family": "family22",
26 1 Nico César
      "dependencies": [
27
        "createsglf"
28
      ],
29
      "time_average": 8254.534873,
30
      "time_count": 1,
31
      "time_min": 8254.534873,
32
      "time_min_comment": "duration:merge-tilelib#su92l-dz642-cc7799yfwi5jmd9",
33
      "time_max": 8254.534873,
34
      "time_max_comment": "duration:merge-tilelib#su92l-dz642-cc7799yfwi5jmd9"
35
    },
36
    {
37 3 Nico César
      "name": "createsglf",
38
      "family": "family9",
39 1 Nico César
      "dependencies": [],
40
      "time_average": 4741.290203,
41
      "time_count": 58,
42
      "time_min": 82.138309,
43
      "time_min_comment": "duration:createsglf_57#su92l-dz642-3u3g4bq1yh4pqje",
44
      "time_max": 5818.898387,
45
      "time_max_comment": "duration:createsglf_8#su92l-dz642-8d094xhqciin5m2"
46
    },
47
...
48
],
49
"time_average": <average time for the CR family>,
50
</pre>
51
52
53
GET /container-request/aaaaa-xvhdp-123456789abc/datapoints
54
55
Output:
56
57
<pre>
58
[
59
  {
60
    "step_name": "createsglf",
61
    "start_1": "2020-01-15 19:49:34.213 +0000",
62
    "end_1": "2020-01-15 21:19:39.001 +0000",
63
    "start_2": "2020-01-15 19:54:44.864 +0000",
64
    "end_2": "2020-01-15 21:19:39.001 +0000",
65
    "reuse": false,
66
    "status": "completed",
67
    "legend": "<p>createsglf</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-zfc3ffxk3slmkzv\">su92l-xvhdp-zfc3ffxk3slmkzv</a></p><p>Container duration: 1h24m54.137122s\n</p>"
68
  },
69
  {
70
    "step_name": "createsglf_2",
71
    "start_1": "2020-01-15 19:49:34.288 +0000",
72
    "end_1": "2020-01-15 21:29:11.399 +0000",
73
    "start_2": "2020-01-15 19:54:51.275 +0000",
74
    "end_2": "2020-01-15 21:29:11.399 +0000",
75
    "reuse": false,
76
    "status": "completed",
77
    "legend": "<p>createsglf_2</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-py99va9hnvuxzp5\">su92l-xvhdp-py99va9hnvuxzp5</a></p><p>Container duration: 1h34m20.123849s\n</p>"
78
  },
79
....
80
</pre>
81
82
GET /container-request/aaaaa-xvhdp-123456789abc/workflow-dot
83
84
Output:
85
86
<pre>
87
digraph cwlgraph {
88
rankdir=LR;
89
graph [compound=true];
90
91
subgraph cluster_0 {
92
label="#createcgf-wf.cwl";
93
node [style=filled];
94
shape=box
95
style="filled";
96
color="#dddddd";
97
"#createcgf-wf.cwl" [ label = "#createcgf-wf.cwl", style = invis ];
98
....
99
</pre>
100
101
102
h2. Frontend
103
104
Dot file can be rendered with  https://domparfitt.com/graphviz-react/ we already tested some big files 
105
106
h2. Schema and queries on the postgres DB 
107
108
TODO: Outline the transformation from the current local leveldb cache to some per-user caching table.  
109
TODO: list the queries to INSERT and SELECT the data for a particular checkpoint. 
110
111
112
h2. Permissions
113
114
One concern is permissions. we'll behave similar to everything else in Arvados: if it's a CR that the token doesn't have access to, then is a 404. This includes the idea of "sumarized data" as in the historical time and prices of the CRs 
115 3 Nico César
116
h2. Real World Example
117
118
Take the case of su92l-xvhdp-bs4tseq26te2bnz ( a hasher function that Ops usually use as smoke test)
119
120
h3. graph
121
122
!su92l-xvhdp-bs4tseq26te2bnz.png!
123
124
the dotty representation would be:
125
126
<pre>
127
digraph cwlgraph {
128
rankdir=LR;
129
graph [compound=true];
130
131
subgraph cluster_0 {
132
label="#main";
133
node [style=filled];
134
shape=box
135
style="filled";
136
color="#dddddd";
137
"#main" [ label = "#main", style = invis ];
138
139
140
"#main
141
inputfile" -> "step #main
142
hasher1";
143
"#main
144
hasher1_outputname" -> "step #main
145
hasher1";
146
"step #main
147
hasher1" -> "#main
148
hasher1
149
hasher_out";
150
"#main
151
hasher1
152
hasher_out" -> "step #main
153
hasher2";
154
"#main
155
hasher2_outputname" -> "step #main
156
hasher2";
157
"step #main
158
hasher2" -> "#main
159
hasher2
160
hasher_out";
161
"#main
162
hasher2
163
hasher_out" -> "step #main
164
hasher3";
165
"#main
166
hasher3_outputname" -> "step #main
167
hasher3";
168
"step #main
169
hasher3" -> "#main
170
hasher3
171
hasher_out";
172
}
173
174
175
176
"step #main
177
hasher1" [fillcolor="#FFD700", style="rounded,filled", shape=box];
178
"step #main
179
hasher2" [fillcolor="#FFD700", style="rounded,filled", shape=box];
180
"step #main
181
hasher3" [fillcolor="#FFD700", style="rounded,filled", shape=box];
182
"#hasher.cwl" [fillcolor="#FF9912", style="rounded,filled", shape=box];
183
184
185
"step #main
186
hasher1" -> "#hasher.cwl" [label="runs", style="dashed"];
187
"step #main
188
hasher2" -> "#hasher.cwl" [label="runs", style="dashed"];
189
"step #main
190
hasher3" -> "#hasher.cwl" [label="runs", style="dashed"];
191
}</pre>
192
193
h3. datapoints
194
<pre>
195
196
[
197
  {
198
    "checkpoint": "hasher1",
199
    "start_1": "2020-05-12 16:35:33.594 +0000",
200
    "end_1": "2020-05-12 16:37:30.597 +0000",
201
    "start_2": "2020-05-12 16:37:27.893 +0000",
202
    "end_2": "2020-05-12 16:37:30.597 +0000",
203
    "reuse": false,
204
    "legend": "<p>hasher1</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-pbpkli9qovdo4q8\">su92l-xvhdp-pbpkli9qovdo4q8</a></p><p>Container duration: 2.70491s\n</p>"
205
  },
206
  {
207
    "checkpoint": "hasher2",
208
    "start_1": "2020-05-12 16:37:33.673 +0000",
209
    "end_1": "2020-05-12 16:39:56.562 +0000",
210
    "start_2": "2020-05-12 16:39:51.455 +0000",
211
    "end_2": "2020-05-12 16:39:56.562 +0000",
212
    "reuse": false,
213
    "legend": "<p>hasher2</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-l8je8tws556fqcp\">su92l-xvhdp-l8je8tws556fqcp</a></p><p>Container duration: 5.10645s\n</p>"
214
  },
215
  {
216
    "checkpoint": "hasher3",
217
    "start_1": "2020-05-12 16:39:57.608 +0000",
218
    "end_1": "2020-05-12 16:42:17.628 +0000",
219
    "start_2": "2020-05-12 16:42:14.836 +0000",
220
    "end_2": "2020-05-12 16:42:17.628 +0000",
221
    "reuse": false,
222
    "legend": "<p>hasher3</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-jx5vk6lq26dsbba\">su92l-xvhdp-jx5vk6lq26dsbba</a></p><p>Container duration: 2.792018s\n</p>"
223
  }
224
]</pre>
225
226
227
h3. checkpoints
228
229
<pre>
230
{
231
  "checkpoints": [
232
    {
233
      "name": "hasher2",
234
      "family": "abde1234-9876543",
235
      "dependencies": [
236
        "hasher1"
237
      ],
238
      "time_average": 5.10645,
239
      "time_count": 1,
240
      "time_min": 5.10645,
241
      "time_min_comment": "duration:hasher2#su92l-dz642-eouma4xv1qpnhvc",
242
      "time_max": 5.10645,
243
      "time_max_comment": "duration:hasher2#su92l-dz642-eouma4xv1qpnhvc"
244
    },
245
    {
246
      "name": "hasher3",
247
      "family": "87654321-fedcba01",
248
      "dependencies": [
249
        "hasher2"
250
      ],
251
      "time_average": 2.792018,
252
      "time_count": 1,
253
      "time_min": 2.792018,
254
      "time_min_comment": "duration:hasher3#su92l-dz642-tn9t07438jd1zrt",
255
      "time_max": 2.792018,
256
      "time_max_comment": "duration:hasher3#su92l-dz642-tn9t07438jd1zrt"
257
    },
258
    {
259
      "name": "hasher1",
260
      "family": "deadbeef-deafbeef",
261
      "dependencies": [],
262
      "time_average": 2.70491,
263
      "time_count": 1,
264
      "time_min": 2.70491,
265
      "time_min_comment": "duration:hasher1#su92l-dz642-e6d8emz3ez54owu",
266
      "time_max": 2.70491,
267
      "time_max_comment": "duration:hasher1#su92l-dz642-e6d8emz3ez54owu"
268
    }
269
  ]
270
}
271
</pre>