https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-01-27T17:11:57ZArvadosArvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=901462021-01-27T17:11:57ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/90146/diff?detail_id=86871">diff</a>)</li><li><strong>Subject</strong> changed from <i>Singularity MVP</i> to <i>Singularity proof of concept</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=901472021-01-27T17:15:57ZWard Vandewegeward@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/90147/diff?detail_id=86872">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=901512021-01-27T19:14:50ZWard Vandewegeward@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed" href="/issues/17241">Idea #17241</a>: Scoping/grooming Singularity support work</i> added</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=902642021-02-02T18:13:17ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/90264/diff?detail_id=86983">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=902652021-02-02T18:26:03ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Story points</strong> set to <i>5.0</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/90265/diff?detail_id=86984">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=902662021-02-02T18:29:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/90266/diff?detail_id=86986">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=902712021-02-02T18:39:18ZWard Vandewegeward@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=905002021-02-10T17:47:05ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-02-17 sprint</i> to <i>2021-03-03 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=907202021-02-18T17:04:04ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-03-03 sprint</i> to <i>2021-03-17 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=908492021-02-23T14:49:40ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Category</strong> set to <i>Crunch</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=910252021-03-03T18:55:00ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> changed from <i>Tom Clegg</i> to <i>Nico César</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=911162021-03-05T22:14:44ZNico César
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=912002021-03-10T16:02:44ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/16305">Idea #16305</a>: Singularity support</i> added</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=912632021-03-15T20:49:10ZNico César
<ul></ul><p>very first take trying to create an abstraction that matches 1 to 1 with docker for now,</p>
<p><a class="changeset" title="First step in a 1-to-1 abstraction with docker Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@c..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/c1d1f0502b8a0f049dba41da2f6b19a0d4b03d77">c1d1f0502b8a0f049dba41da2f6b19a0d4b03d77</a></p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2380/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2380/">developer-run-tests: #2380 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2380" alt="" /></a></a></p>
<p>TODO:</p>
<ul>
<li>[DONE] <a class="changeset" title="moving HostConfig into the ThinContainerExecRunner interface Arvados-DCO-1.1-Signed-off-by: Nico..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0a27815bdf3f1d1bc1eb3771bcee9294b6f4136f">0a27815bd</a> review ContainerConfig, HostConfig settings and add them to the ThinContainerExecRunner interface as Get/Set methods to abstract from the internal representation</li>
<li>review all the networking related options and see if can be simplified</li>
<li>make a run with crunch-run --container-runner singularity to see how it behaves</li>
<li>add tests related to singularity</li>
</ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=912842021-03-16T18:22:52ZNico César
<ul></ul><p>Commit:0a27815bd 17241-singularity-take1</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=913152021-03-17T15:57:08ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-03-17 sprint</i> to <i>2021-03-31 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=913362021-03-17T18:57:21ZNico César
<ul></ul><p>As I'm reading all the documentation available about singularity I want to write down some notes:</p>
<blockquote>
<p>It is also important to note that the philosophy of Singularity is Integration over Isolation. Most container run times strive to isolate your container from the host system and other containers as much as possible. Singularity, on the other hand, assumes that the user’s primary goals are portability, reproducibility, and ease of use and that isolation is often a tertiary concern.<br />Therefore, Singularity only isolates the mount namespace by default, and will bind mount several host directories such as $HOME and /tmp into the container at runtime. If needed, additional levels of isolation can be achieved by passing options causing Singularity to enter any or all of the other kernel namespaces and to prevent automatic bind mounting. These measures allow users to interact with the host system from within the container in sensible ways.</p>
</blockquote>
<p>(taken from <a class="external" href="https://sylabs.io/guides/3.7/user-guide/security.html">https://sylabs.io/guides/3.7/user-guide/security.html</a> )</p>
<p>I see a potential problem here, since singularity tries to incorporate the HOST files as part of the container in a transparent way, this could cause problems if crunch-run is running everything with the same user and maybe in a shared environment</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=913762021-03-19T18:20:18ZNico César
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/91376/diff?detail_id=88086">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=914522021-03-25T20:17:28ZNico César
<ul></ul><p><a class="changeset" title="Still failing tests but we'll be reviewing with tom Arvados-DCO-1.1-Signed-off-by: Nico Cesar <n..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/fccb8a3decaaa33fb9b356b028dad3e257c90fe0">fccb8a3de</a> 17241-singularity-take1</p>
<p>Tom let's review this branch before I continue down the wrong path</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=915352021-03-31T14:23:13ZNico César
<ul><li><strong>Target version</strong> changed from <i>2021-03-31 sprint</i> to <i>2021-04-14 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=917612021-04-14T13:18:05ZNico César
<ul></ul><p><a class="changeset" title="finilizyn interface Arvados-DCO-1.1-Signed-off-by: Nico Cesar <nico@curii.com>" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/43b39915bae3f3c24ab31cfbc7aefdce88f84dcb">43b39915bae3f3c24ab31cfbc7aefdce88f84dcb</a> branch 17296-singularity-take2</p>
<p>we had a conversation with tom and we agree that lib/crunchrun/container_exec.go looks good to start making lib/crunchrun/container_exec_test.go to add all the tests for a contarner exec.</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=917882021-04-14T15:48:44ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-04-14 sprint</i> to <i>2021-05-12 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=921502021-04-23T17:36:24ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Added singularity v3 to <strong>compute{0,1}.9tee4</strong>, removed it from <strong>api.9tee4</strong> (commit ecfae2b..4a694a1@saltstack)</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=922432021-04-28T13:48:28ZNico César
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/92243/diff?detail_id=88918">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=922892021-04-28T20:11:12ZNico César
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/92289/diff?detail_id=88968">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=922932021-04-29T13:46:58ZNico César
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/92293/diff?detail_id=88974">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=923042021-04-29T20:00:01ZNico César
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/92304/diff?detail_id=88983">diff</a>)</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=923052021-04-29T20:02:36ZNico César
<ul></ul><p>e587495bf 17296-singularity-take2</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=925272021-05-11T18:07:01ZTom Cleggtom@curii.com
<ul><li><strong>Assigned To</strong> changed from <i>Nico César</i> to <i>Tom Clegg</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=925642021-05-12T15:49:58ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-05-12 sprint</i> to <i>2021-05-26 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=927762021-05-17T20:24:35ZTom Cleggtom@curii.com
<ul></ul><p>17296-singularity @ <a class="changeset" title="17296: Add crunch-run integration test. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/8fd331405028ebdbb97de58560057564aa530105">8fd331405028ebdbb97de58560057564aa530105</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2477/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2477/">developer-run-tests: #2477 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2477" alt="" /></a></a></p>
<p>I think this is approximately functional enough to merge as alpha/experimental. More work is definitely needed to make it a fully supported feature.</p>
Limitations
<ul>
<li>Container gateway won't work with singularity. (Todo: make sure it at least fails gracefully, and mention in docs.)</li>
<li>Singularity doesn't even attempt to pass through VCPUs/memory. (I tried passing VCPUs but it just made the container fail to start.)</li>
<li>Singularity "build" step (convert docker image) fails if squashfs-tools is not installed. (Should we try to report this more gracefully?)</li>
<li>No cache for the docker-to-singularity image conversion.</li>
<li>crunchstat.txt tracks host stats instead of container stats. (Haven't looked into how to do cgroups with singularity.)</li>
</ul>
Things that changed unexpectedly
<ul>
<li>We now read the docker image from arv-mount instead of a ManifestFileReader. Might give us slightly better startup latency due to readahead?</li>
<li>Rewrote a lot of the testing code. Tests are still rather copy-pastey and could use more cleanup but I felt like I needed to stop short of rewriting Everything.</li>
</ul>
Other things
<ul>
<li>Config key "Containers.RuntimeEngine" make sense? Should it be Containers.CloudVMs.RuntimeEngine instead?</li>
<li>I noticed our "load docker image" code assumes the sha256 in the tarball filename matches the ID of the image and probably behaves strangely if it doesn't. (I didn't try to fix this.)</li>
<li>Very exciting, we finally have an integration test for crunch-run that uses the real apiserver, arv-mount, and docker/singularity.</li>
</ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=928422021-05-18T21:11:44ZTom Cleggtom@curii.com
<ul></ul><p>17296-singularity @ <a class="changeset" title="17296: Don't use /var/lib/arvados/tmp as a cache dir for tests. Arvados-DCO-1.1-Signed-off-by: T..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5a902f04bc1efdd30398195cb50df85ebab75d84">5a902f04bc1efdd30398195cb50df85ebab75d84</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2478/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2478/">developer-run-tests: #2478 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2478" alt="" /></a></a></p>
<p>addresses a test failure (/var/lib/arvados/tmp isn't necessarily writable by the non-root user that runs tests).</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=930642021-05-25T18:07:55ZWard Vandewegeward@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>17296-singularity @ <a class="changeset" title="17296: Don't use /var/lib/arvados/tmp as a cache dir for tests. Arvados-DCO-1.1-Signed-off-by: T..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5a902f04bc1efdd30398195cb50df85ebab75d84">5a902f04bc1efdd30398195cb50df85ebab75d84</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2478/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2478/">developer-run-tests: #2478 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2478" alt="" /></a></a></p>
<p>addresses a test failure (/var/lib/arvados/tmp isn't necessarily writable by the non-root user that runs tests).</p>
</blockquote>
<ul>
<li>slight behavior change in loading docker images: before, the .tar file of the image had to be the first file in the collection, and that was loaded. Any subsequent .tar files were ignored. Now, it is an error if there is more than one .tar file. This change should probably be mentioned in the upgrade docs (though it's unlikely someone would run into it).</li>
</ul>
<ul>
<li>since this branch does not add any documentation yet, I've created a story for Singularity documentation (<a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: [singularity] add documentation (Resolved)" href="https://dev.arvados.org/issues/17726">#17726</a>) to the epic and added a note there about the previous bullet.</li>
</ul>
<ul>
<li>I think the config key "Containers.RuntimeEngine" makes sense; the plan is for singularity support to be available in non-cloud dispatchers as well, right?</li>
</ul>
<ul>
<li>In RunCommand() (lib/crunchrun/crunchrun.go), line 1695 and later, would it make sense to print a warning for the first two conditions that cause the gateway service not to be started?</li>
</ul>
<ul>
<li>Looks like the new integration test is failing because it can't find the docker binary. That is then followed by a panic in TearDownSuite (eek). I see you test for the presence of Docker in crunchrun/docker_test.go, but not yet in crunchrun/integration_test.go (<a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests-remainder/2592/console"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests-remainder/2592/">developer-run-tests-remainder: #2592 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests-remainder&build=2592" alt="" /></a>/console</a>):</li>
</ul>
<pre>
14:24:59 ----------------------------------------------------------------------
14:24:59 FAIL: integration_test.go:38: integrationSuite.SetUpSuite
14:24:59
14:24:59
14:24:59 integration_test.go:43:
14:24:59 c.Assert(err, IsNil)
14:24:59 ... value *exec.Error = &exec.Error{Name:"docker", Err:(*errors.errorString)(0xc0000a83c0)} ("exec: \"docker\": executable file not found in $PATH")
14:24:59
14:24:59
14:24:59 ----------------------------------------------------------------------
14:24:59 PANIC: integration_test.go:76: integrationSuite.TearDownSuite
14:24:59
14:24:59 ... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x43A7D8)
14:24:59
14:24:59 /usr/src/go/src/runtime/panic.go:965
14:24:59 in gopanic
14:24:59 /usr/src/go/src/runtime/panic.go:212
14:24:59 in panicmem
14:24:59 /usr/src/go/src/runtime/signal_unix.go:734
14:24:59 in sigpanic
14:24:59 /tmp/workspace/developer-run-tests-remainder/sdk/go/arvados/client.go:321
14:24:59 in Client.RequestAndDecodeContext
14:24:59 /tmp/workspace/developer-run-tests-remainder/sdk/go/arvados/client.go:312
14:24:59 in Client.RequestAndDecode
14:24:59 integration_test.go:77
14:24:59 in integrationSuite.TearDownSuite
14:24:59 /usr/src/go/src/reflect/value.go:337
14:24:59 in Value.Call
14:24:59 /usr/src/go/src/runtime/asm_amd64.s:1371
14:24:59 in goexit
</pre>
<p>Other than that, LGTM.</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=930772021-05-25T21:18:02ZTom Cleggtom@curii.com
<ul></ul>17296-singularity @ <a class="changeset" title="17296: Fix panic in teardown of skipped test. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curi..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/fb5cab780281c6cfb834cbfae64f3fcac7918d19">fb5cab780281c6cfb834cbfae64f3fcac7918d19</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2495/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2495/">developer-run-tests: #2495 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2495" alt="" /></a></a>
<ul>
<li>added note about docker image collections (will need to adjust before merging so it goes in the forthcoming "upgrading from 2.2" section instead of 2.1)</li>
<li>added logs when not starting gateway service</li>
<li>added docker check so tests should pass on jenkins</li>
</ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=931042021-05-26T16:08:24ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-05-26 sprint</i> to <i>2021-06-09 sprint</i></li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=931852021-05-27T16:59:50ZWard Vandewegeward@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
17296-singularity @ <a class="changeset" title="17296: Fix panic in teardown of skipped test. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curi..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/fb5cab780281c6cfb834cbfae64f3fcac7918d19">fb5cab780281c6cfb834cbfae64f3fcac7918d19</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2495/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2495/">developer-run-tests: #2495 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2495" alt="" /></a></a>
<ul>
<li>added note about docker image collections (will need to adjust before merging so it goes in the forthcoming "upgrading from 2.2" section instead of 2.1)</li>
<li>added logs when not starting gateway service</li>
<li>added docker check so tests should pass on jenkins</li>
</ul>
</blockquote>
<p>LGTM, thanks!</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=932992021-06-02T17:59:37ZTom Cleggtom@curii.com
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '17296-singularity' closes #17296 Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@cu..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/bcab6c50f54f155f14d1dd635becd85a5e91289d">arvados|bcab6c50f54f155f14d1dd635becd85a5e91289d</a>.</p> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=936132021-06-11T15:28:52ZWard Vandewegeward@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed parent" href="/issues/17755">Idea #17755</a>: Test singularity support on a cloud cluster by running some real workflows</i> added</li></ul> Arvados - Idea #17296: Singularity proof of concepthttps://dev.arvados.org/issues/17296?journal_id=985442021-11-16T16:25:53ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>42</i></li></ul>