API HistoricalForcasting data for CR » History » Revision 3
Revision 2 (Nico César, 06/18/2020 08:44 AM) → Revision 3/6 (Nico César, 06/22/2020 02:43 PM)
h1. API Historical/Forcasting data for CR Goal: create a pipeline forecaster and visualization for historical data. This should expose APIs that can be used in the ContainerRequest visualization and also could be use to provide extra information for the current running CR Glossary: * Checkpoint: is a generic name that currently corresponds to a step name. This id together with "family" make name followed by a unique "family". The reason behind having a family is to cluster to summarize results. This summarization for all the unique cluster includes: a) several runs with similar parameters and b) executions (including the scattered steps that have the pattern: name_2, name_3,..., name_229 name_229) for runs that the token has access to with similar characteristics. * Family: A common name like "gatk" or "haplotypecaller" can be used as a step name. The family definition will help to separate the In practice 2 populationsn executions would create 2 different populations (in terms of checkpoints. We think that implementing this based checkpoints) depending on the parameters of the CommandLineTool and parent workflow md5sum or a combination of both * Datapoint: a concrete data that can be plotted as historical data. Currently we're bounding together the container request and the associated container to have a unified view of the times involved. This should not get confused with forecast data since can be used separately h2. API GET /container-request/aaaaa-xvhdp-123456789abc/checkpoints Output: <pre> { "checkpoints": [ { "name": "merge-tilelib", "merge-tilelib@family22", "family": "family22", "dependencies": [ "createsglf" ], "time_average": 8254.534873, "time_count": 1, "time_min": 8254.534873, "time_min_comment": "duration:merge-tilelib#su92l-dz642-cc7799yfwi5jmd9", "time_max": 8254.534873, "time_max_comment": "duration:merge-tilelib#su92l-dz642-cc7799yfwi5jmd9" }, { "name": "createsglf", "createsglf@family22", "family": "family9", "dependencies": [], "time_average": 4741.290203, "time_count": 58, "time_min": 82.138309, "time_min_comment": "duration:createsglf_57#su92l-dz642-3u3g4bq1yh4pqje", "time_max": 5818.898387, "time_max_comment": "duration:createsglf_8#su92l-dz642-8d094xhqciin5m2" }, ... ], "time_average": <average time for the CR family>, </pre> GET /container-request/aaaaa-xvhdp-123456789abc/datapoints Output: <pre> [ { "step_name": "createsglf", "start_1": "2020-01-15 19:49:34.213 +0000", "end_1": "2020-01-15 21:19:39.001 +0000", "start_2": "2020-01-15 19:54:44.864 +0000", "end_2": "2020-01-15 21:19:39.001 +0000", "reuse": false, "status": "completed", "legend": "<p>createsglf</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-zfc3ffxk3slmkzv\">su92l-xvhdp-zfc3ffxk3slmkzv</a></p><p>Container duration: 1h24m54.137122s\n</p>" }, { "step_name": "createsglf_2", "start_1": "2020-01-15 19:49:34.288 +0000", "end_1": "2020-01-15 21:29:11.399 +0000", "start_2": "2020-01-15 19:54:51.275 +0000", "end_2": "2020-01-15 21:29:11.399 +0000", "reuse": false, "status": "completed", "legend": "<p>createsglf_2</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-py99va9hnvuxzp5\">su92l-xvhdp-py99va9hnvuxzp5</a></p><p>Container duration: 1h34m20.123849s\n</p>" }, .... </pre> GET /container-request/aaaaa-xvhdp-123456789abc/workflow-dot Output: <pre> digraph cwlgraph { rankdir=LR; graph [compound=true]; subgraph cluster_0 { label="#createcgf-wf.cwl"; node [style=filled]; shape=box style="filled"; color="#dddddd"; "#createcgf-wf.cwl" [ label = "#createcgf-wf.cwl", style = invis ]; .... </pre> h2. Frontend Dot file can be rendered with https://domparfitt.com/graphviz-react/ we already tested some big files h2. Schema and queries on the postgres DB TODO: Outline the transformation from the current local leveldb cache to some per-user caching table. TODO: list the queries to INSERT and SELECT the data for a particular checkpoint. h2. Permissions One concern is permissions. we'll behave similar to everything else in Arvados: if it's a CR that the token doesn't have access to, then is a 404. This includes the idea of "sumarized data" as in the historical time and prices of the CRs h2. Real World Example Take the case of su92l-xvhdp-bs4tseq26te2bnz ( a hasher function that Ops usually use as smoke test) h3. graph !su92l-xvhdp-bs4tseq26te2bnz.png! the dotty representation would be: <pre> digraph cwlgraph { rankdir=LR; graph [compound=true]; subgraph cluster_0 { label="#main"; node [style=filled]; shape=box style="filled"; color="#dddddd"; "#main" [ label = "#main", style = invis ]; "#main inputfile" -> "step #main hasher1"; "#main hasher1_outputname" -> "step #main hasher1"; "step #main hasher1" -> "#main hasher1 hasher_out"; "#main hasher1 hasher_out" -> "step #main hasher2"; "#main hasher2_outputname" -> "step #main hasher2"; "step #main hasher2" -> "#main hasher2 hasher_out"; "#main hasher2 hasher_out" -> "step #main hasher3"; "#main hasher3_outputname" -> "step #main hasher3"; "step #main hasher3" -> "#main hasher3 hasher_out"; } "step #main hasher1" [fillcolor="#FFD700", style="rounded,filled", shape=box]; "step #main hasher2" [fillcolor="#FFD700", style="rounded,filled", shape=box]; "step #main hasher3" [fillcolor="#FFD700", style="rounded,filled", shape=box]; "#hasher.cwl" [fillcolor="#FF9912", style="rounded,filled", shape=box]; "step #main hasher1" -> "#hasher.cwl" [label="runs", style="dashed"]; "step #main hasher2" -> "#hasher.cwl" [label="runs", style="dashed"]; "step #main hasher3" -> "#hasher.cwl" [label="runs", style="dashed"]; }</pre> h3. datapoints <pre> [ { "checkpoint": "hasher1", "start_1": "2020-05-12 16:35:33.594 +0000", "end_1": "2020-05-12 16:37:30.597 +0000", "start_2": "2020-05-12 16:37:27.893 +0000", "end_2": "2020-05-12 16:37:30.597 +0000", "reuse": false, "legend": "<p>hasher1</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-pbpkli9qovdo4q8\">su92l-xvhdp-pbpkli9qovdo4q8</a></p><p>Container duration: 2.70491s\n</p>" }, { "checkpoint": "hasher2", "start_1": "2020-05-12 16:37:33.673 +0000", "end_1": "2020-05-12 16:39:56.562 +0000", "start_2": "2020-05-12 16:39:51.455 +0000", "end_2": "2020-05-12 16:39:56.562 +0000", "reuse": false, "legend": "<p>hasher2</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-l8je8tws556fqcp\">su92l-xvhdp-l8je8tws556fqcp</a></p><p>Container duration: 5.10645s\n</p>" }, { "checkpoint": "hasher3", "start_1": "2020-05-12 16:39:57.608 +0000", "end_1": "2020-05-12 16:42:17.628 +0000", "start_2": "2020-05-12 16:42:14.836 +0000", "end_2": "2020-05-12 16:42:17.628 +0000", "reuse": false, "legend": "<p>hasher3</p><p>Container Request: <a href=\"https://workbench.su92l.arvadosapi.com/container_requests/su92l-xvhdp-jx5vk6lq26dsbba\">su92l-xvhdp-jx5vk6lq26dsbba</a></p><p>Container duration: 2.792018s\n</p>" } ]</pre> h3. checkpoints <pre> { "checkpoints": [ { "name": "hasher2", "family": "abde1234-9876543", "dependencies": [ "hasher1" ], "time_average": 5.10645, "time_count": 1, "time_min": 5.10645, "time_min_comment": "duration:hasher2#su92l-dz642-eouma4xv1qpnhvc", "time_max": 5.10645, "time_max_comment": "duration:hasher2#su92l-dz642-eouma4xv1qpnhvc" }, { "name": "hasher3", "family": "87654321-fedcba01", "dependencies": [ "hasher2" ], "time_average": 2.792018, "time_count": 1, "time_min": 2.792018, "time_min_comment": "duration:hasher3#su92l-dz642-tn9t07438jd1zrt", "time_max": 2.792018, "time_max_comment": "duration:hasher3#su92l-dz642-tn9t07438jd1zrt" }, { "name": "hasher1", "family": "deadbeef-deafbeef", "dependencies": [], "time_average": 2.70491, "time_count": 1, "time_min": 2.70491, "time_min_comment": "duration:hasher1#su92l-dz642-e6d8emz3ez54owu", "time_max": 2.70491, "time_max_comment": "duration:hasher1#su92l-dz642-e6d8emz3ez54owu" } ] } </pre>