The existing sitemap plugin doesn't scale very well and is indexed by search engines once for every document in the site. It needs to be replaced by a partial sitemap with the appearance of a dynamic sitemap.
MKDoc currently has flo::plugin::Sitemap, a plugin that displays a nested sitemap of the whole site at locations like this:
http://www.example.com/path/to/.sitemap.html
This is ok but the content is exactly the same for all ‘parent documents’ and can get extremely large as sites get bigger.
It also gets crawled by robots repeatedly, which is why I tend to redirect all requests to a single sitemap at the site root. Ie. whichever sitemap you request, you end-up getting this one:
http://www.example.com/.sitemap.html
We have a plan to replace the current sitemap with a shorter one that just lists ‘nearby’ documents by default.
By nearby, I mean just: parents, parent_siblings, siblings and children documents of the current document.
Each document in the list that itself has children will have a little [+] link that links directly to the sitemap of that document.
The idea is that it will appear to fully expandable. If you expand and explore a collapsed branch in the tree, documents that are ‘further away’ will collapse so that the overall length of the sitemap page will stay the same – but in fact you are just visiting sitemaps of other documents.
Since it is going to be much shorter, we can afford to include document descriptions as well as titles.
Since there are only as many variations of the sitemap as there are documents, we can afford to let robots crawl it too.