Generate sitemap

[jvwong]

• Module: sitemap exposing generateSitemap( docs ):
  • assembles a JSON version of the sitemap data
  • uses xml-js to generate the xml string
  • writes to static folder
  • Tests for sensible xml
• web service API: /document/sitemap?apiKey=xyz
  • filter for 'published' documents (see Update document status fields. #844)
  • uses generateSitemap
  • apiKey protected
  • Swagger docs

Sample out

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://biofactoid.org/document/7af64ad2-64a5-466e-b42d-acf259c3883d</loc>
    <lastmod>2020-09-28T20:19:27.063Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.5</priority>
  </url>
  <url>
    <loc>https://biofactoid.org/document/13a08fd5-0299-47c3-be94-0efc0f878a90</loc>
    <lastmod>2020-09-28T20:18:18.196Z</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.5</priority>
  </url>
</urlset>

Refs #842

• Notes:
  • sitemaps.org
  • Should sync status filter with Update document status fields. #844
  • Maybe create on boot?

[maxkfranz]

Would it be simpler for the sitemap to be generated dynamically? It's always up-to-date that way.

[jvwong]

Would it be simpler for the sitemap to be generated dynamically? It's always up-to-date that way.

I think I misread your 'attach it to a route' comment.

Maybe you mean when a document becomes PUBLIC, regenerate the sitemap?

[maxkfranz]

I meant that it's a dynamic route rather than a static file. So something like /sitemap.xml is just a dynamic route like /api/documents that returns all the current documents.

[jvwong]

I meant that it's a dynamic route rather than a static file. So something like /sitemap.xml is just a dynamic route like /api/documents that returns all the current documents.

Oh ok that's a better idea.

[maxkfranz]

The xml format may be OK, but you can also just use a simple text file with each url separated by newlines, e.g.

https://biofactoid.org/document/foo
https://biofactoid.org/document/bar
https://biofactoid.org/document/baz

Maybe the xml format has some other advantages, but the text format is really simple.

[maxkfranz]

The robots.txt should be included in this PR to activate the site map. See #842 (comment)

I think that this should be enough to start:

User-agent: *
Allow: /

Sitemap: https://biofactoid.org/api/sitemap.xml

[maxkfranz]

Note that the sitemap should also include the homepage: https://biofactoid.org/

[maxkfranz]

Maybe append the home page here?

[jvwong]

It's already appended in the sitemap.js code. Isn't that OK?