Arvados

Right now our application layout sets the noindex and nofollow meta tags on every page, telling search engines to not list the page or follow its links.

Our general goal is that we want to allow search engines to browse and index public data (i.e., it's part of a shared project). We may need to still include nofollow for collections pages, to prevent search engines from downloading the large files inside them and using excess resources.

Implementation idea: only include the meta tags when an API token is set. Now that we have the public projects index, search engines should be able to enter the site and crawl all the public data from there. It's okay if real users with real API tokens get the meta tags even for public data—the tags aren't for them.