Bug #6173

[Workbench] Make public data crawlable

Added by Brett Smith about 4 years ago. Updated about 4 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Workbench
Target version:
Start date:
05/27/2015
Due date:
05/27/2015
% Done:

0%

Estimated time:
Story points:
-

Description

Right now our application layout sets the noindex and nofollow meta tags on every page, telling search engines to not list the page or follow its links.

Our general goal is that we want to allow search engines to browse and index public data (i.e., it's part of a shared project). We may need to still include nofollow for collections pages, to prevent search engines from downloading the large files inside them and using excess resources.

Implementation idea: only include the meta tags when an API token is set. Now that we have the public projects index, search engines should be able to enter the site and crawl all the public data from there. It's okay if real users with real API tokens get the meta tags even for public data—the tags aren't for them.


Related issues

Follows (1 day) Arvados - Story #6057: [Workbench] Provide an attractive index of public projectsResolved05/25/2015

Also available in: Atom PDF