Project

General

Profile

Actions
Copy link

Bug #19192

closed

WebDAVCache not performing as expected for S3 requests

Added by Tom Clegg over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
Keep
Target version:
2022-07-06
Story points:
-
Release:
Arvados 2.4.2
Release relationship:
Auto

Files

Download all files
keep-web (11 MB) keep-web dev build at commit 042f0d37dacebddef9d2d7232fa2e5b9922a9451 Tom Clegg, 06/13/2022 06:15 PM
keep-web (11 MB) keep-web dev build at commit 65c1517193b5039769f288e84a11d1ebfddbf031 Tom Clegg, 06/20/2022 04:44 PM
keep-web (11 MB) keep-web dev build at commit b5db4b70878a3907db3691b71979e6e65511a12c Tom Clegg, 06/21/2022 03:22 AM

Subtasks 1 (0 open — 1 closed)

Task #19204: Review 19192-fix-deadlockResolvedLucas Di Pentima06/21/2022Actions

Related issues 2 (0 open — 2 closed)

Related to Arvados - Idea #19205: In Go services, monitor request times and record when they continue processing after client disconnects, or exceed a maximum request timeResolvedTom Clegg06/28/2022Actions
Related to Arvados - Bug #19368: [keep-web] [S3] slow requests caused by logUploadOrDownloadResolvedTom Clegg08/12/2022Actions
Actions
Copy link
 #1

Updated by Tom Clegg over 2 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Tom Clegg over 2 years ago

Actions
Copy link
 #3

Updated by Tom Clegg over 2 years ago

Debug logging indicates (*cache)pruneSessions() is not running, which causes expired sessions to get stuck in the cache forever. I'm guessing there's a situation where fs.MemorySize() deadlocks, and once that happens, pruneSessions can't continue/resume, and we effectively have no cache.

SIGABRT would produce a stack trace to confirm this and show where it's getting stuck.

Actions
Copy link
 #4

Updated by Tom Clegg over 2 years ago

Here's a debug version that will dump a stack trace and exit if cache pruning takes longer than 30s.

Actions
Copy link
 #5

Updated by Tom Clegg over 2 years ago

  • File keep-web added

19192-fix-deadlock @ dab0c5596e39dc455d88bba797717e829fe5caf5 -- developer-run-tests: #3177

Fixes a lock-ordering bug in lookupnode causing deadlock on concurrent open() and readdir() calls in the same lookupnode (project) directory.

Actions
Copy link
 #6

Updated by Tom Clegg over 2 years ago

  • File deleted (keep-web)
Actions
Copy link
 #7

Updated by Tom Clegg over 2 years ago

Here's a dev build with both the bugfix from note-5 and the watchdog / stack dump from note-4.

Actions
Copy link
 #8

Updated by Lucas Di Pentima over 2 years ago

19192-fix-deadlock LGTM, thanks!

Actions
Copy link
 #9

Updated by Peter Amstutz over 2 years ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions
Copy link
 #10

Updated by Tom Clegg over 2 years ago

  • Related to Idea #19205: In Go services, monitor request times and record when they continue processing after client disconnects, or exceed a maximum request time added
Actions
Copy link
 #11

Updated by Peter Amstutz over 2 years ago

  • Target version changed from 2022-07-06 to 2022-07-20
Actions
Copy link
 #12

Updated by Peter Amstutz over 2 years ago

  • Target version changed from 2022-07-20 to 2022-07-06
Actions
Copy link
 #13

Updated by Peter Amstutz over 2 years ago

  • Status changed from In Progress to Resolved
  • Category set to Keep
Actions
Copy link
 #14

Updated by Peter Amstutz over 2 years ago

  • Release set to 52
Actions
Copy link
 #15

Updated by Tom Clegg over 2 years ago

  • Related to Bug #19368: [keep-web] [S3] slow requests caused by logUploadOrDownload added
Actions
Copy link

Also available in: Atom PDF