crunch-job is failing to save Job outputs right now because it doesn't handle permission hints correctly. Example error message from the log: <pre> 2014-06-03_17:29:47 qr1hi-8i9sb-huk6pcv7w376ru0 4413 Failed to register output manifest: API call /collections failed: 422 Unprocessable Entity 2014-06-03_17:29:47 qr1hi-8i9sb-huk6pcv7w376ru0 4413 {"errors":["Uuid does not match checksum of manifest_text"]} at /usr/local/arvados/src/sdk/cli/bin/crunch-job line 813 </pre> This happens because it sends along a manifest+UUID pair with permission hints in it. However, the API server refuses to store manifests with permission hints. It strips them out, then compares the result against the provided UUID. After collating output, crunch-job should read back the data using arv-get; strip +A hints from it; compute hash of what's left; and use *that* hash as the uuid for collections.create(), and use whatever collections.create() returns as the job's output.

Ensure that when a progress bar is rendered in _job_progress.html.erb, at least one of {done, failed, running} is nonzero.

Review at commit:4b8a53c Ensure that: * All job progress bars reflect at least one task. If a job has no tasks at all (done, failed, finished, or todo) that should be considered a failed job, so render a progress bar as though one task ran and failed. * A failed job (j[:success] == false) always shows at least one failed task. If a job is marked as failed but has no failed tasks, then pretend there was one failed task so at least some red will appear in the progress bar.

Please review branch 2896-websocket-cli.

Changes @ commit:46460c9 * arvados.collection.CollectionWriter.finish() now sends a stripped manifest to Keep, storing a manifest without any variable locator hints. * New test suite test_cmdline.py tests the behavior of arv-put, confirming that @arvados.Keep.put@ stores a manifest in Keep without signatures, and that when apiserver is queried for the same manifest UUID, it sends a manifest with signatures.

Review master... commit:b12f667 This change implements: * A new configuration setting @permission_key@. * @Collections.create@ now verifies any signed locator found in a manifest. The signatures are removed from the manifest before storing it in the database or computing the collection UUID. * @Collections.show@ signs all locators in the manifest before returning it to the client.

Review @ commit:5214573 2755: add support for signed locators in the Python SDK. * @arvados.Keep.put()@ saves the response body (which may contain a signed locator) and returns it to the caller. * @arvados.Keep.get()@ passes the full signed locator to the remote Keep server. The bare MD5 hash is still used for caching and for shuffled_service_roots * @run_test_server.run_keep()@ takes arguments 'blob_signing_key' and 'enforce_permissions', for testing permission signatures in unit tests. * test_keep_client: new unit tests for permissions: ** with @--enforce-permissions=true@: *** GET with a signed locator works *** GET with an unsigned locator fails *** unauthenticated GET fails ** with @--enforce-permissions=false@: *** GET with a signed locator works *** GET with an unsigned locator works *** unauthenticated GET works Bug fixes to permission handling in the Keep server: * Locator hints may appear in any order; be flexible. Parse them in @GetBlockHandler@ rather than in the REST router. * Returned locators are terminated with newline (consistent with Warehouse, and more friendly for human debugging). * The locator returned from a PUT request always has a size hint. * The correct Authorization header keyword is "OAuth2", not "OAuth". D'oh. * Updated unit tests to accommodate newlines, size hints and OAuth2. Refs #2755.

Adding limits for crunch job log generation: * crunch_limit_log_events_per_job * crunch_limit_log_event_bytes_per_job ** Limit the number of events a job may log, and the total number of bytes a job may log. * crunch_log_bytes_per_event * crunch_log_seconds_between_events ** Set minimum logging thresholds: event logs will only be flushed to the database when at least crunch_log_bytes_per_event have been generated, or crunch_log_seconds_between_events have passed.

Todo: * -"View agreement" button doesn't work on "accept user agreement" page.- * -"Account is not active" condition is not detected on folders index page. Should redirect to user agreement etc., like previous dashboard.- * -Editable popup - make missing OK/cancel buttons come back- * -Editable popup - get rid of up/down scroll arrows- * -Pipeline inputs: "No further inputs required, but you can still edit stuff if you want"- * -Collection chooser - get rid of up/down scroll arrows- * -Collection page - when viewing by link uuid, show name instead of hash- * -Don't show "recent jobs/pipelines" that aren't in folders.- Add column for folder name.

Added several helper methods to workbench application_controller and used them in dashboard, metadata tab, and pipeline instances component display. No functional tests are added yet for the helper methods and I will take that up later. It will also help if we can comb through the code and identify pages where we can pre-load objects and reuse them. After adding the tests, I will make an attempt to identify some more of these areas.

To do this, put the desired information in appropriately-named files under @/var/tmp/ec2-data/user-data@. Exactly how we go about that TBD. Information we want: * total_ram_mb * total_scratch_mb * total_cpu_cores

The hardware runtime constraints are: * @min_ram_mb_per_node@ - The amount of RAM (MiB) available on each Node. * @min_scratch_mb_per_node@ - The amount of disk space (MiB) available for local caching on each Node. Corresponding information about each node will lives in its info hash, and will be updated each ping. Because we currently don't have a concept of Job priority, and it's not in this sprint, it seems best to stick pretty closely to Crunch's current FIFO strategy for working the Job queue. However, we need to take precautions to make sure that an unreasonably resource-large Job at the front of the queue doesn't prevent us from making progress on the rest of it. Plan: When the Job at the front of the queue can't be started because resource requirements aren't met, Crunch will wait for a few minutes to see if the Node Manager makes those resources available. If it does, great; proceed as normal. If not, continue through the queue and start the first job that can be run with available resources. Make sure that this wait only happens every so often, so lots of queue activity doesn't cause lots of waiting.

Very simple/kludgy mocking for Time in permission tests.

Absolute positioning on the provenance_graph div was resulting in shifting of the div. This is due to the container of the provenance_graph div had its own border. Removing the absolute positioning resolved the issue. We are now able to view the full graph for pipeline instances, jobs, and collections; all of which were broken earlier.

Both volume.Get and main.ReadAtMost use ioutil.ReadAll to read all available data. As Peter observed, ReadAll grows a buffer exponentially to read all available data, so in order to read a 64MB block it allocates a 128MB buffer. Some things we can do to relax memory usage: * Eliminate ioutil.ReadAll. Since we know the anticipated amount of data for both PUT (from the Content-Length header) and GET (from the file size on disk) we can allocate exactly the amount of space we need and use that buffer. * Possibly: remove the requirement that Keep return a 402 Corruption error on a corrupt block. If Keep is allowed to return the block anyway and simply log an alert if the block is found to be corrupt, GET can allocate a relatively small buffer (e.g. 1MB) for reads.

Review @ commit:70dd308 Eliminated ioutil.ReadAll, which was naively allocating buffers 2x as large as necessary. * Rewrote main.PutBlockHandler to allocate a buffer of exactly req.ContentLength bytes, and fill it with io.ReadFull. * UnixVolume.Read now uses ioutil.ReadFile to read the contents of a block file (which it should have been doing in the first place)

Removed the following" - jobs.log_tail_follow and associated references from code and docs - users.event_stream and associated references from code and docs - redis gem

See notes on #2752.