Bug #16263

Cannot list users on federated cluster with LoginCluster set

Added by Lucas Di Pentima 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
03/23/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

When having logged in as an admin user on a federated cluster that uses another cluster as a LoginCluster, the following happens:

$ arv user list
Error: error updating local user records: request failed: http://localhost:8004/arvados/v1/users/batch_update: 422 Unprocessable Entity: #<ActiveRecord::RecordInvalid: Validation failed: Username must begin with a letter and contain only alphanumerics> (req-x4wnsmrfmqxx20u233ur)

Subtasks

Task #16268: Review 16263-logincluster-user-list-fixResolvedPeter Amstutz

Task #16272: Set up jenkins job to run federation migrateResolvedPeter Amstutz

Associated revisions

Revision c9fe930b
Added by Peter Amstutz 3 months ago

Merge branch '16263-logincluster-user-list-fix' refs #16263

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision a7fa8488
Added by Peter Amstutz 2 months ago

Merge branch '16263-fed-test-on-jenkins' refs #16263

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz 4 months ago

Part of this story should be a user listing integration test that uses a similar arvados-boot setup to the new federation test.

#2 Updated by Lucas Di Pentima 4 months ago

  • Status changed from New to In Progress

#3 Updated by Peter Amstutz 4 months ago

A related error was reported:

error updating local user records: request failed: http://localhost:8000/arvados/v1/users/batch_update: 422 Unprocessable Entity: #<ActiveRecord::RecordInvalid: Validation failed: Modified by client uuid 'arlog-ozdt8-8bsvojxdycrr5qe' not found> (req-54bqf3xe8p5y1sppyu4u)

modified_by_client_uuid is a special field (I believe it is the client_uuid from token that last touched the record) and it doesn't really make sense to try and copy it to the local users table, because the client_uuid is cluster specific.

#4 Updated by Lucas Di Pentima 4 months ago

Updates at 8ba91a5558a9fbb743c726a7e38f7e7dcd7bf8ee - branch 16263-logincluster-user-list-fix
Test run: https://ci.arvados.org/job/developer-run-tests/1781/

  • Adds a hook to create and update calls so that every object can declare a list of fields that should be saved as nil when receiving an empty value.
  • Adds the same behavior to batch_update on users.
  • Adds tests.

#5 Updated by Lucas Di Pentima 4 months ago

Updates at 9fb4398f2
Test run: https://ci.arvados.org/job/developer-run-tests/1782/

  • Removes modified_by_*_uuid fields to the user's cache field whitelist.

#7 Updated by Peter Amstutz 4 months ago

Lucas Di Pentima wrote:

Updates at 9fb4398f2
Test run: https://ci.arvados.org/job/developer-run-tests/1782/

  • Removes modified_by_*_uuid fields to the user's cache field whitelist.

I noticed 'uuid' isn't in the whitelist. Does it get added in somewhere else?

#8 Updated by Peter Amstutz 4 months ago

Peter Amstutz wrote:

Lucas Di Pentima wrote:

Updates at 9fb4398f2
Test run: https://ci.arvados.org/job/developer-run-tests/1782/

  • Removes modified_by_*_uuid fields to the user's cache field whitelist.

I noticed 'uuid' isn't in the whitelist. Does it get added in somewhere else?

Actually I see now, the uuid is a dictionary key.

#9 Updated by Peter Amstutz 4 months ago

I'm running the federation-migrate test (which tests the LoginCluster feature). It's getting an API error:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.6:8000/arvados/v1/users?alt=json&limit=1&filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D returned "invalid filter operand %!q(bool=true)">

Here's the error from controller (I think it fails in controller, it doesn't make it to the API server):

@400000005e7a57190e6981ac.s:2020-03-24_18:50:52.75547 {"PID":40,"RequestID":"req-5otj2n35rtxdh6x5bir3","level":"info","msg":"response","remoteAddr":"127.0.0.1:47752","reqBytes":0,"reqForwardedFor":"172.17.0.9","reqHost":"172.17.0.6:8000","reqMethod":"GET","reqPath":"arvados/v1/users","reqQuery":"alt=json\u0026limit=1\u0026filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D","respBody":"{\"errors\":[\"invalid filter operand %!q(bool=true)\"]}\n","respBytes":53,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2020-03-24T18:50:52.755416243Z","timeToStatus":0.000604,"timeTotal":0.000614,"timeWriteBody":0.000010}

#10 Updated by Peter Amstutz 4 months ago

This is the API call from Python:

existing = api.users().list(filters=[["email", "=", "test@example.com"],
                                     ["is_active", "=", True]], limit=1).execute()

#11 Updated by Lucas Di Pentima 4 months ago

Peter Amstutz wrote:

I'm running the federation-migrate test (which tests the LoginCluster feature). It's getting an API error:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.6:8000/arvados/v1/users?alt=json&limit=1&filters=%5B%5B%22email%22%2C+%22%3D%22%2C+%22test%40example.com%22%5D%2C+%5B%22is_active%22%2C+%22%3D%22%2C+true%5D%5D returned "invalid filter operand %!q(bool=true)">

Fixed at 5c941aa10b7b4bf9d0804f98eb93a8104bfb7658 (rebased branch)
Test run: https://ci.arvados.org/job/developer-run-tests/1785/

The problem was that the GoSDK wasn't supporting boolean operands on filters when unmarshalling from JSON. Added some tests.

#12 Updated by Peter Amstutz 4 months ago

Now I'm getting this:

arvados.errors.ApiError: <HttpError 500 when requesting https://172.17.0.8:8000/arvados/v1/users?offset=0&limit=9223372036854775807&alt=json returned "request failed: https://172.17.0.6:8000/arvados/v1/users?cluster_id=&count=&limit=9223372036854776000&offset=0: 500 Internal Server Error: strconv.ParseInt: parsing "9223372036854776000": value out of range">

This is on this SDK call:

ul = arvados.util.list_all(arv.users().list)

#13 Updated by Peter Amstutz 4 months ago

  • Target version changed from 2020-03-25 Sprint to 2020-04-08 Sprint

#14 Updated by Lucas Di Pentima 4 months ago

The above problem is related to the fact that json.Unmarshal() uses the float64 type to decode JSON numbers, and when decoding really big numbers, it loses precision. See https://stackoverflow.com/questions/16946306/preserve-int64-values-when-parsing-json-in-go

Fix with tests at 12b534152
Test run: https://ci.arvados.org/job/developer-run-tests/1786/

I think changing controller's requestAndDecode() behavior is a bit overkill, I'm reading a bit about json's Unmarshaler interface to see if something more specific can be implemented: https://golang.org/pkg/encoding/json/#Unmarshaler

#15 Updated by Peter Amstutz 3 months ago

Got another error while trying run the federation migrate test. I'm looking into this one.

(case3@test) Error getting user info for x31je-tpzed-kjnjvyl0m4n1n2i from x31je: <HttpError 401 when requesting https://172.17.0.4:8000/arvados/v1/users/current?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=&remote=&uuid=: 401 Unauthorized: Not logged in (req-y6de37eqyvwzh16wbs37)">

#16 Updated by Peter Amstutz 3 months ago

(case3@test) No user listed with same email to migrate xsdok-tpzed-0w3rz8pyzrgryxr to x31je, will create new user with username 'case3'
(case3@test) Error getting user info for x31je-tpzed-o5zmv7q6i8cj5pq from x31je: <HttpError 401 when requesting https://172.17.0.4:8000/arvados/v1/users/current?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=&remote=&uuid=: 401 Unauthorized: Not logged in (req-mfgs9p4efyof1honpt26)">
(case3@test) Migrating xsdok-tpzed-0w3rz8pyzrgryxr to x31je-tpzed-o5zmv7q6i8cj5pq on xqcb8
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-0w3rz8pyzrgryxr&new_user_uuid=x31je-tpzed-o5zmv7q6i8cj5pq&new_owner_uuid=xqcb8-j7d0g-9m0sjeteyv2v88y&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 422 Unprocessable Entity: request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in new_user_uuid not found (req-ghtmi4tlfenkpp1u1ub4)">
(case3@test) Migrating xsdok-tpzed-0w3rz8pyzrgryxr to x31je-tpzed-o5zmv7q6i8cj5pq on xsdok
(case3@test) Updating username of x31je-tpzed-o5zmv7q6i8cj5pq to 'case3' on xsdok

2020-03-30_15:04:25.82187 {"PID":9470,"RequestID":"req-mfgs9p4efyof1honpt26","level":"info","msg":"request","remoteAddr":"127.0.0.1:35268","reqBytes":0,"reqForwardedFor":"172.17.0.1","reqHost":"172.17.0.4:8000","reqMethod":"GET","reqPath":"arvados/v1/users/current","reqQuery":"alt=json","time":"2020-03-30T15:04:25.821786163Z"}

2020-03-30_15:04:25.83428 {"PID":9470,"RequestID":"req-mfgs9p4efyof1honpt26","level":"info","msg":"response","remoteAddr":"127.0.0.1:35268","reqBytes":0,"reqForwardedFor":"172.17.0.1","reqHost":"172.17.0.4:8000","reqMethod":"GET","reqPath":"arvados/v1/users/current","reqQuery":"alt=json","respBody":"{\"errors\":[\"request failed: http://localhost:8004/arvados/v1/users/current?forwarded_for=\\u0026remote=\\u0026uuid=: 401 Unauthorized: Not logged in (req-mfgs9p4efyof1honpt26)\"]}\n","respBytes":177,"respStatus":"Unauthorized","respStatusCode":401,"time":"2020-03-30T15:04:25.834223741Z","timeToStatus":0.012422,"timeTotal":0.012436,"timeWriteBody":0.000013}

ESC[1mESC[36mApiClientAuthorization Load (0.4ms)ESC[0m ESC[1mESC[34mSELECT "api_client_authorizations".* FROM "api
_client_authorizations" WHERE (uuid='x31je-gj3su-9zgjsz5tpyw1ewx' and (expires_at is null or expires_at > CURRENT_TIME
STAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1ESC[0m "LIMIT", 1
ESC[1mESC[36mUser Load (0.6ms)ESC[0m ESC[1mESC[34mSELECT "users".* FROM "users" WHERE (redirect_to_user_uuid is nul
l) AND "users"."id" = 13ESC[0m
ESC[1mESC[36mApiClient Load (0.5ms)ESC[0m ESC[1mESC[34mSELECT "api_clients".* FROM "api_clients" WHERE "api_clients
"."id" = 1ESC[0m
[req-mfgs9p4efyof1honpt26] Error 1585580665+9a33fe5f: 401 {"method":"GET","path":"/arvados/v1/users/current","format":"html","controller":"Arvados::V1::UsersController","action
":"current","status":401,"duration":0.87,"view":0.29,"db":0.0,"request_id":"req-mfgs9p4efyof1honpt26","client_ipaddr":
"127.0.0.1","client_auth":null,"params":{"forwarded_for":"","remote":"","uuid":""},"@timestamp":"2020-03-30T15:04:25.8
33481839Z","@version":"1","message":"[401] GET /arvados/v1/users/current (Arvados::V1::UsersController#current)"}

#17 Updated by Peter Amstutz 3 months ago

Fixed previous error by adding omitempty to fields of GetOptions.

Next thing failing:

(case3@test) No user listed with same email to migrate xsdok-tpzed-5wtzbkn5iw66h8g to x31je, will create new user with username 'case3'
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on x31je
(case3@test) Error migrating user: <HttpError 403 when requesting https://172.17.0.4:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-5wtzbkn5iw66h8g&new_user_uuid=x31je-tpzed-2ss1g6pqjxhcg9d&new_owner_uuid=x31je-j7d0g-njks8t4fmc96e5r&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 403 Forbidden: request failed: http://localhost:8004/arvados/v1/users/merge: 403 Forbidden: cannot move objects into supplied new_owner_uuid: new user does not have write permission (req-qwbqtchw2ubsk3a57g41)">
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on xqcb8
(case3@test) Error migrating user: <HttpError 403 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-5wtzbkn5iw66h8g&new_user_uuid=x31je-tpzed-2ss1g6pqjxhcg9d&new_owner_uuid=xqcb8-j7d0g-wr33pehajc1edr8&redirect_to_new_user=true&alt=json returned "request failed: https://172.17.0.5:8000/arvados/v1/users/merge: 403 Forbidden: request failed: http://localhost:8004/arvados/v1/users/merge: 403 Forbidden: cannot move objects into supplied new_owner_uuid: new user does not have write permission (req-mxz2gnky6zk8lswzw5em)">
(case3@test) Migrating xsdok-tpzed-5wtzbkn5iw66h8g to x31je-tpzed-2ss1g6pqjxhcg9d on xsdok
(case3@test) Updating username of x31je-tpzed-2ss1g6pqjxhcg9d to 'case3' on xsdok

#18 Updated by Peter Amstutz 3 months ago

  • Target version deleted (2020-04-08 Sprint)

Fix that error (it was incorrectly federating the user merge call) and now it's failing for a new reason!

(case3@test) No user listed with same email to migrate xsdok-tpzed-gotqha27ntc2769 to x31je, will create new user with username 'case3'
(case3@test) Migrating xsdok-tpzed-gotqha27ntc2769 to x31je-tpzed-u6nq17m7h5j5vfd on x31je
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.4:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-gotqha27ntc2769&new_user_uuid=x31je-tpzed-u6nq17m7h5j5vfd&new_owner_uuid=x31je-j7d0g-kld2rcfg7nqvjz3&redirect_to_new_user=true&alt=json returned "request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in old_user_uuid not found (req-orhh7a1czbrzv1752oq0)">
(case3@test) Migrating xsdok-tpzed-gotqha27ntc2769 to x31je-tpzed-u6nq17m7h5j5vfd on xqcb8
(case3@test) Error migrating user: <HttpError 422 when requesting https://172.17.0.6:8000/arvados/v1/users/merge?old_user_uuid=xsdok-tpzed-gotqha27ntc2769&new_user_uuid=x31je-tpzed-u6nq17m7h5j5vfd&new_owner_uuid=xqcb8-j7d0g-x21c9ws4bh9279p&redirect_to_new_user=true&alt=json returned "request failed: http://localhost:8004/arvados/v1/users/merge: 422 Unprocessable Entity: User in old_user_uuid not found (req-g8nv9uhzj95c5i957zqu)">
(case3@test) Migrating xsdok-tpzed-gotqha27ntc2769 to x31je-tpzed-u6nq17m7h5j5vfd on xsdok
(case3@test) Updating username of x31je-tpzed-u6nq17m7h5j5vfd to 'case3' on xsdok

#19 Updated by Peter Amstutz 3 months ago

  • Target version set to 2020-04-08 Sprint

#20 Updated by Peter Amstutz 3 months ago

Fixed the error (it was trying to migrate users that didn't exist and didn't need to be migrated, fixed the existence check).

Next error:

(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je
(case7@test) Error updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je: <HttpError 422 when requesting https://172.17.0.4:8000/arvados/v1/users/x31je-tpzed-89a34lxw5j0v92d?alt=json returned "request failed: http://localhost:8004/arvados/v1/users/x31je-tpzed-89a34lxw5j0v92d: 422 Unprocessable Entity: Username has already been taken (req-10dasoeab3iw7ifi1p9h)">
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on x31je
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on x31je
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on xqcb8
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on xqcb8
(case7@test) Migrating xsdok-tpzed-os46kennicy3vp6 to x31je-tpzed-89a34lxw5j0v92d on xsdok
(case7@test) Updating username of x31je-tpzed-89a34lxw5j0v92d to 'case7' on xsdok

#21 Updated by Peter Amstutz 3 months ago

16263-logincluster-user-list-fix @ 5efed8f05e4616ed423124544e5b021e2c8d5a6f

https://ci.arvados.org/view/Developer/job/developer-run-tests/1795/

  • Adds bypass_federation flag, only usable by admins, to disable federation behavior and get responses only from the local cluster database
  • arv-federation-migrate uses bypass_federation flag.
  • passes federation migrate test and federated workflow tests. the federation migrate test also tests federated user listing.
  • Bumped API revision. arv-federation-migrate checks for new API revision.

#22 Updated by Lucas Di Pentima 3 months ago

After lots of struggling, was able to run the federation tests locally, but wasn't able to run the fed migration tests, I think because of OOM issues.

Code looks good, just one comment: bypass_federation param access test is OK, but it doesn’t test the param behavior. It's now possible to make a federation integration test to confirm that the flag works as intended. The federation migrate tests indirectly check this feature but I think a proper specific test would be needed.

Other than that, it LGTM.

#23 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-04-08 Sprint to 2020-04-22

#24 Updated by Lucas Di Pentima 3 months ago

  • Target version changed from 2020-04-22 to 2020-05-06 Sprint

#25 Updated by Lucas Di Pentima 3 months ago

  • Assigned To changed from Lucas Di Pentima to Peter Amstutz

#26 Updated by Peter Amstutz 3 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF