Skip Navigation

Sign up

If you sign up for an account on this web site you can customise elements of this site and subscribe to an email newsletter.

If you have an account on this web site you may login.

If you have an account on this site but have forgotten your user name and / or your password then you can request an account reminder email.

Sitemap plugin doesn't scale

The existing sitemap plugin doesn't scale very well and is indexed by search engines once for every document in the site. It needs to be replaced by a partial sitemap with the appearance of a dynamic sitemap.

MKDoc currently has flo::plugin::Sitemap, a plugin that displays a nested sitemap of the whole site at locations like this:

http://www.example.com/path/to/.sitemap.html

This is ok but the content is exactly the same for all ‘parent documents’ and can get extremely large as sites get bigger.

It also gets crawled by robots repeatedly, which is why I tend to redirect all requests to a single sitemap at the site root. Ie. whichever sitemap you request, you end-up getting this one:

http://www.example.com/.sitemap.html

We have a plan to replace the current sitemap with a shorter one that just lists ‘nearby’ documents by default.

By nearby, I mean just: parents, parent_siblings, siblings and children documents of the current document.

Each document in the list that itself has children will have a little [+] link that links directly to the sitemap of that document.

The idea is that it will appear to fully expandable. If you expand and explore a collapsed branch in the tree, documents that are ‘further away’ will collapse so that the overall length of the sitemap page will stay the same – but in fact you are just visiting sitemaps of other documents.

Since it is going to be much shorter, we can afford to include document descriptions as well as titles.

Since there are only as many variations of the sitemap as there are documents, we can afford to let robots crawl it too.

Up

This document was last modified by Bruno Postle on 2005-01-12 03:57:43
MKDoc Ltd., 31 Psalter Lane, Sheffield, S11 8YL, UK.
Copyright © 2001-2005 MKDoc Ltd.