Bug #16263

Cannot list users on federated cluster with LoginCluster set

Added by Lucas Di Pentima 15 days ago. Updated 2 days ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
03/23/2020
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

When having logged in as an admin user on a federated cluster that uses another cluster as a LoginCluster, the following happens:

$ arv user list
Error: error updating local user records: request failed: http://localhost:8004/arvados/v1/users/batch_update: 422 Unprocessable Entity: #<ActiveRecord::RecordInvalid: Validation failed: Username must begin with a letter and contain only alphanumerics> (req-x4wnsmrfmqxx20u233ur)

Subtasks

Task #16268: Review 16263-logincluster-user-list-fixIn ProgressPeter Amstutz

Task #16272: Set up jenkins job to run federation migrateNew

History

#1 Updated by Peter Amstutz 15 days ago

Part of this story should be a user listing integration test that uses a similar arvados-boot setup to the new federation test.

#2 Updated by Lucas Di Pentima 15 days ago

  • Status changed from New to In Progress

#3 Updated by Peter Amstutz 14 days ago

A related error was reported:

error updating local user records: request failed: http://localhost:8000/arvados/v1/users/batch_update: 422 Unprocessable Entity: #<ActiveRecord::RecordInvalid: Validation failed: Modified by client uuid 'arlog-ozdt8-8bsvojxdycrr5qe' not found> (req-54bqf3xe8p5y1sppyu4u)

modified_by_client_uuid is a special field (I believe it is the client_uuid from token that last touched the record) and it doesn't really make sense to try and copy it to the local users table, because the client_uuid is cluster specific.

#4 Updated by Lucas Di Pentima 10 days ago

Updates at 8ba91a5558a9fbb743c726a7e38f7e7dcd7bf8ee - branch 16263-logincluster-user-list-fix
Test run: https://ci.arvados.org/job/developer-run-tests/1781/

  • Adds a hook to create and update calls so that every object can declare a list of fields that should be saved as nil when receiving an empty value.
  • Adds the same behavior to batch_update on users.
  • Adds tests.

#5 Updated by Lucas Di Pentima 10 days ago

Updates at 9fb4398f2
Test run: https://ci.arvados.org/job/developer-run-tests/1782/

  • Removes modified_by_*_uuid fields to the user's cache field whitelist.

#7 Updated by Peter Amstutz 9 days ago

Lucas Di Pentima wrote:

Updates at 9fb4398f2
Test run: https://ci.arvados.org/job/developer-run-tests/1782/

  • Removes modified_by_*_uuid fields to the user's cache field whitelist.

I noticed 'uuid' isn't in the whitelist. Does it get added in somewhere else?

#8 Updated by Peter Amstutz 9 days ago

Peter Amstutz wrote:

Lucas Di Pentima wrote:

Updates at 9fb4398f2
Test run: https://ci.arvados.org/job/developer-run-tests/1782/

  • Removes modified_by_*_uuid fields to the user's cache field whitelist.

I noticed 'uuid' isn't in the whitelist. Does it get added in somewhere else?

Actually I see now, the uuid is a dictionary key.

#9 Updated by Peter Amstutz 9 days ago

I'm running the federation-migrate test (which tests the LoginCluster feature). It's getting an API error:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.6:8000/arvados/v1/users?alt=json&limit=1&filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D returned "invalid filter operand %!q(bool=true)">

Here's the error from controller (I think it fails in controller, it doesn't make it to the API server):

@400000005e7a57190e6981ac.s:2020-03-24_18:50:52.75547 {"PID":40,"RequestID":"req-5otj2n35rtxdh6x5bir3","level":"info","msg":"response","remoteAddr":"127.0.0.1:47752","reqBytes":0,"reqForwardedFor":"172.17.0.9","reqHost":"172.17.0.6:8000","reqMethod":"GET","reqPath":"arvados/v1/users","reqQuery":"alt=json\u0026limit=1\u0026filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D","respBody":"{\"errors\":[\"invalid filter operand %!q(bool=true)\"]}\n","respBytes":53,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2020-03-24T18:50:52.755416243Z","timeToStatus":0.000604,"timeTotal":0.000614,"timeWriteBody":0.000010}

#10 Updated by Peter Amstutz 9 days ago

This is the API call from Python:

existing = api.users().list(filters=[["email", "=", "test@example.com"],
                                     ["is_active", "=", True]], limit=1).execute()

#11 Updated by Lucas Di Pentima 9 days ago

Peter Amstutz wrote:

I'm running the federation-migrate test (which tests the LoginCluster feature). It's getting an API error:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.6:8000/arvados/v1/users?alt=json&limit=1&filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D returned "invalid filter operand %!q(bool=true)">

Fixed at 5c941aa10b7b4bf9d0804f98eb93a8104bfb7658 (rebased branch)
Test run: https://ci.arvados.org/job/developer-run-tests/1785/

The problem was that the GoSDK wasn't supporting boolean operands on filters when unmarshalling from JSON. Added some tests.

#12 Updated by Peter Amstutz 9 days ago

Now I'm getting this:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.8:8000/arvados/v1/users?offset=0&limit=9223372036854775807&alt=json returned "request failed: https://172.17.0.6:8000/arvados/v1/users?cluster_id=&count=&limit=9223372036854776000&offset=0: 500 Internal Server Error: strconv.ParseInt: parsing "9223372036854776000": value out of range">

This is on this SDK call:

ul = arvados.util.list_all(arv.users().list)

#13 Updated by Peter Amstutz 8 days ago

  • Target version changed from 2020-03-25 Sprint to 2020-04-08 Sprint

#14 Updated by Lucas Di Pentima 7 days ago

The above problem is related to the fact that json.Unmarshal() uses the float64 type to decode JSON numbers, and when decoding really big numbers, it loses precision. See https://stackoverflow.com/questions/16946306/preserve-int64-values-when-parsing-json-in-go

Fix with tests at 12b534152
Test run: https://ci.arvados.org/job/developer-run-tests/1786/

I think changing controller's requestAndDecode() behavior is a bit overkill, I'm reading a bit about json's Unmarshaler interface to see if something more specific can be implemented: https://golang.org/pkg/encoding/json/#Unmarshaler

#15 Updated by Peter Amstutz 3 days ago

Got another error while trying run the federation migrate test. I'm looking into this one.

(case3@test) Error getting user info for x31je-tpzed-kjnjvyl0m4n1n2i from x31je: <HttpError 401 when requesting https://172.17.0.4:8000/arvados/v1/users/current?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=&remote=&uuid=: 401 Unauthorized: Not logged in (req-y6de37eqyvwzh16wbs37)">

#16 Updated by Peter Amstutz 3 days ago

(case3@test) No user listed with same email to migrate xsdok-tpzed-0w3rz8pyzrgryxr to x31je, will create new user with username 'case3'
(case3@test) Error getting user info for x31je-tpzed-o5zmv7q6i8cj5pq from x31je: <HttpError 401 when requesting https://172.17.0.4:8000/arvados/v1/users/current?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=&remote=&uuid=: 401 Unauthorized: Not logged in (req-mfgs9p4efyof1honpt26)">
(case3@test) Migrating xsdok-tpzed-0w3rz8pyzrgryxr to x31je-tpzed-o5zmv7q6i8cj5pq on xqcb8
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-0w3rz8pyzrgryxr&new_user_uuid=x31je-tpzed-o5zmv7q6i8cj5pq&new_owner_uuid=xqcb8-j7d0g-9m0sjeteyv2v88y&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 422 Unprocessable Entity: request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in new_user_uuid not found (req-ghtmi4tlfenkpp1u1ub4)">
(case3@test) Migrating xsdok-tpzed-0w3rz8pyzrgryxr to x31je-tpzed-o5zmv7q6i8cj5pq on xsdok
(case3@test) Updating username of x31je-tpzed-o5zmv7q6i8cj5pq to 'case3' on xsdok

2020-03-30_15:04:25.82187 {"PID":9470,"RequestID":"req-mfgs9p4efyof1honpt26","level":"info","msg":"request","remoteAddr":"127.0.0.1:35268","reqBytes":0,"reqForwardedFor":"172.17.0.1","reqHost":"172.17.0.4:8000","reqMethod":"GET","reqPath":"arvados/v1/users/current","reqQuery":"alt=json","time":"2020-03-30T15:04:25.821786163Z"}

2020-03-30_15:04:25.83428 {"PID":9470,"RequestID":"req-mfgs9p4efyof1honpt26","level":"info","msg":"response","remoteAddr":"127.0.0.1:35268","reqBytes":0,"reqForwardedFor":"172.17.0.1","reqHost":"172.17.0.4:8000","reqMethod":"GET","reqPath":"arvados/v1/users/current","reqQuery":"alt=json","respBody":"{\"errors\":[\"request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=\\u0026remote=\\u0026uuid=: 401 Unauthorized: Not logged in (req-mfgs9p4efyof1honpt26)\"]}\n","respBytes":177,"respStatus":"Unauthorized","respStatusCode":401,"time":"2020-03-30T15:04:25.834223741Z","timeToStatus":0.012422,"timeTotal":0.012436,"timeWriteBody":0.000013}

ESC[1mESC[36mApiClientAuthorization Load (0.4ms)ESC[0m ESC[1mESC[34mSELECT "api_client_authorizations".* FROM "api
_client_authorizations" WHERE (uuid='x31je-gj3su-9zgjsz5tpyw1ewx' and (expires_at is null or expires_at > CURRENT_TIME
STAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1ESC[0m "LIMIT", 1
ESC[1mESC[36mUser Load (0.6ms)ESC[0m ESC[1mESC[34mSELECT "users".* FROM "users" WHERE (redirect_to_user_uuid is nul
l) AND "users"."id" = 13ESC[0m
ESC[1mESC[36mApiClient Load (0.5ms)ESC[0m ESC[1mESC[34mSELECT "api_clients".* FROM "api_clients" WHERE "api_clients
"."id" = 1ESC[0m
[req-mfgs9p4efyof1honpt26] Error 1585580665+9a33fe5f: 401 {"method":"GET","path":"/arvados/v1/users/current","format":"html","controller":"Arvados::V1::UsersController","action
":"current","status":401,"duration":0.87,"view":0.29,"db":0.0,"request_id":"req-mfgs9p4efyof1honpt26","client_ipaddr":
"127.0.0.1","client_auth":null,"params":{"forwarded_for":"","remote":"","uuid":""},"@timestamp":"2020-03-30T15:04:25.8
33481839Z","@version":"1","message":"[401] GET /arvados/v1/users/current (Arvados::V1::UsersController#current)"}

#17 Updated by Peter Amstutz 3 days ago

Fixed previous error by adding omitempty to fields of GetOptions.

Next thing failing:

(case3@test) No user listed with same email to migrate xsdok-tpzed-5wtzbkn5iw66h8g to x31je, will create new user with username 'case3'
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on x31je
(case3@test) Error migrating user: <HttpError 403 when requesting https://172.17.0.4:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-5wtzbkn5iw66h8g&new_user_uuid=x31je-tpzed-2ss1g6pqjxhcg9d&new_owner_uuid=x31je-j7d0g-njks8t4fmc96e5r&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 403 Forbidden: request failed: http://localhost:8004/arvados/v1/users/merge: 403 Forbidden: cannot move objects into supplied new_owner_uuid: new user does not have write permission (req-qwbqtchw2ubsk3a57g41)">
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on xqcb8
(case3@test) Error migrating user: <HttpError 403 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-5wtzbkn5iw66h8g&new_user_uuid=x31je-tpzed-2ss1g6pqjxhcg9d&new_owner_uuid=xqcb8-j7d0g-wr33pehajc1edr8&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 403 Forbidden: request failed: http://localhost:8004/arvados/v1/users/merge: 403 Forbidden: cannot move objects into supplied new_owner_uuid: new user does not have write permission (req-mxz2gnky6zk8lswzw5em)">
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on xsdok
(case3@test) Updating username of x31je-tpzed-2ss1g6pqjxhcg9d to 'case3' on xsdok

#18 Updated by Peter Amstutz 3 days ago

  • Target version deleted (2020-04-08 Sprint)

Fix that error (it was incorrectly federating the user merge call) and now it's failing for a new reason!

(case3@test) No user listed with same email to migrate xsdok-tpzed-gotqha27ntc2769 to x31je, will create new user with username 'case3'
(case3@test) Migrating xsdok-tpzed-gotqha27ntc2769 to x31je-tpzed-u6nq17m7h5j5vfd on x31je
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.4:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-gotqha27ntc2769&new_user_uuid=x31je-tpzed-u6nq17m7h5j5vfd&new_owner_uuid=x31je-j7d0g-kld2rcfg7nqvjz3&redirect_to_new_user=true&alt=json returned "request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in old_user_uuid not found (req-orhh7a1czbrzv1752oq0)">
(case3@test) Migrating xsdok-tpzed-gotqha27ntc2769 to x31je-tpzed-u6nq17m7h5j5vfd on xqcb8
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-gotqha27ntc2769&new_user_uuid=x31je-tpzed-u6nq17m7h5j5vfd&new_owner_uuid=xqcb8-j7d0g-x21c9ws4bh9279p&redirect_to_new_user=true&alt=json returned "request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in old_user_uuid not found (req-g8nv9uhzj95c5i957zqu)">
(case3@test) Migrating xsdok-tpzed-gotqha27ntc2769 to x31je-tpzed-u6nq17m7h5j5vfd on xsdok
(case3@test) Updating username of x31je-tpzed-u6nq17m7h5j5vfd to 'case3' on xsdok

#19 Updated by Peter Amstutz 2 days ago

  • Target version set to 2020-04-08 Sprint

#20 Updated by Peter Amstutz 2 days ago

Fixed the error (it was trying to migrate users that didn't exist and didn't need to be migrated, fixed the existence check).

Next error:

(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je
(case7@test) Error updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je: <HttpError 422 when requesting https://172.17.0.4:8000/arvados/v1/users/x31je-tpzed-89a34lxw5j0v92d?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/x31je-tpzed-89a34lxw5j0v92d: 422 Unprocessable Entity: Username has already been taken (req-10dasoeab3iw7ifi1p9h)">
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on x31je
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on xqcb8
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on xqcb8
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on xsdok
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on xsdok

Also available in: Atom PDF