Bug #18243

[controller] Error: json: cannot unmarshal number into Go struct field ContainerRequest.items.command of type string

Added by Ward Vandewege 2 months ago. Updated 2 months ago.

Status:
Closed
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

I am trying to retrieve the list of queued CR's on 9tee4 with the `arv` cli:

arv container_request list -e 200 --filters '[["state","=","Committed"]]'

This results in an error:

Error: json: cannot unmarshal number into Go struct field ContainerRequest.items.command of type string

The controller logs are:

{"PID":26289,"RequestID":"req-1epzlvli5ow7p1q59c5k","level":"info","msg":"request","remoteAddr":"[::1]:48158","reqBytes":0,"reqForwardedFor":"10.100.96.5","reqHost":"9tee4.arvadosapi.com","reqMethod":"GET","reqPath":"arvados/v1/container_requests","reqQuery":"_profile=true\u0026bypass_federation=false\u0026cluster_id=\u0026count=exact\u0026distinct=false\u0026filters=%5B%5B%22state%22%2C%22%3D%22%2C%22Committed%22%5D%5D\u0026filters_given=true\u0026help=false\u0026include_trash=false\u0026limit=100\u0026offset=200\u0026offset_given=true\u0026order=\u0026select=\u0026where=","time":"2021-10-01T14:31:48.378615311Z"}
{"PID":26289,"RequestID":"req-1epzlvli5ow7p1q59c5k","level":"info","msg":"response","remoteAddr":"[::1]:48158","reqBytes":0,"reqForwardedFor":"10.100.96.5","reqHost":"9tee4.arvadosapi.com","reqMethod":"GET","reqPath":"arvados/v1/container_requests","reqQuery":"_profile=true\u0026bypass_federation=false\u0026cluster_id=\u0026count=exact\u0026distinct=false\u0026filters=%5B%5B%22state%22%2C%22%3D%22%2C%22Committed%22%5D%5D\u0026filters_given=true\u0026help=false\u0026include_trash=false\u0026limit=100\u0026offset=200\u0026offset_given=true\u0026order=\u0026select=\u0026where=","respBody":"{\"errors\":[\"json: cannot unmarshal number into Go struct field ContainerRequest.items.command of type string\"]}\n","respBytes":112,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2021-10-01T14:31:49.563665389Z","timeToStatus":1.185014,"timeTotal":1.185040,"timeWriteBody":0.000026}

The rails log entry shows no error:

{"method":"GET","path":"/arvados/v1/container_requests","format":"html","controller":"Arvados::V1::ContainerRequestsController","action":"index","status":200,"duration":1169.2,"view":0.41,"db":1044.94,"request_id":"req-1epzlvli5ow7p1q59c5k","client_ipaddr":"127.0.0.1","client_auth":"9tee4-gj3su-sl3gnl6tj3vtvdv","params":{"cluster_id":"","count":"exact","filters":"[[\"state\",\"=\",\"Committed\"]]","forwarded_for":"9tee4-","include":"","limit":"100","offset":"200"},"@timestamp":"2021-10-01T14:31:49.558453209Z","@version":"1","message":"[200] GET /arvados/v1/container_requests (Arvados::V1::ContainerRequestsController#index)"}

I tracked the problem down to these uuids:

9tee4-xvhdp-lh8ky1vz1bakd3w
9tee4-xvhdp-xdvse40sqa70l7b
$ arv container_request get --uuid 9tee4-xvhdp-lh8ky1vz1bakd3w
Error: json: cannot unmarshal number into Go struct field ContainerRequest.command of type string
$ arv container_request get --uuid 9tee4-xvhdp-xdvse40sqa70l7b
Error: json: cannot unmarshal number into Go struct field ContainerRequest.command of type string

I looked up the offending value in the database:

uuid command
9tee4-xvhdp-lh8ky1vz1bakd3w ["sleep",60]
9tee4-xvhdp-xdvse40sqa70l7b ["sleep",60]

History

#1 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)
  • Subject changed from [controller] Error: json: cannot unmarshal number into Go struct field ContainerRequest.items.command of type string to [controller] Error: json: cannot unmarshal number into Go struct field ContainerRequest.items.command of type string

#2 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#3 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#4 Updated by Ward Vandewege 2 months ago

  • Release set to 42

#5 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#6 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#7 Updated by Tom Clegg 2 months ago

Whoops, it looks like we didn't have a validation for command being an array of strings. I think we could fix this in Rails by casting non-string elements to strings (x.to_str) when loading records, and adding a validation for new container requests.

#8 Updated by Peter Amstutz 2 months ago

  • Assigned To set to Peter Amstutz

#9 Updated by Ward Vandewege 2 months ago

uuid command created_at
9tee4-xvhdp-lh8ky1vz1bakd3w ["sleep",60] 2018-01-19 14:48:53.710259
9tee4-xvhdp-xdvse40sqa70l7b ["sleep",60] 2018-01-19 14:51:56.006449

#10 Updated by Peter Amstutz 2 months ago

Validation was added in #14082 (August 2018) and these records are older: 2018-01-19 14:51:56.006449

I propose we just delete the problematic records.

#11 Updated by Peter Amstutz 2 months ago

  • Release deleted (42)
  • Status changed from New to Closed

#12 Updated by Ward Vandewege 2 months ago

Peter Amstutz wrote:

Validation was added in #14082 (August 2018) and these records are older: 2018-01-19 14:51:56.006449

I propose we just delete the problematic records.

I've updated the database like so:

arvados_production=# update containers set command='["sleep","60"]' where command='["sleep",60]';
UPDATE 1
arvados_production=# update container_requests set command='["sleep","60"]' where command='["sleep",60]';
UPDATE 2

and that resolved the issue.

Also available in: Atom PDF