Wiki Sitemap

A wiki-sitemap is a site map that we use for federated wiki that uses JSON rather than XML to make it fast to load and utilise in the browser. It is used for things like Wiki Find, and displaying Recent Changes and Twins of the current page.

Here we maintain a Roster of federation directories. Directories will have a well groomed sitemap optimized for finding sites by keyword search.

We should get sitemaps quickly and keep up with their changes. This is especially true of our own.

We collect various counts while scraping sitemaps and report them as a text file. Here we plot the most recent counts available.

This is a rewrite of the first ruby scraper that saves text files useful for searching instead of whole sites in export format. It incrementally refetches pages that have changed based on dates in the sitemaps. github

The Ruby Sitemap Scrape provides the first full text search of the visible federation. We've learned a lot by building this ourselves from grep-like utilities. Here we list todos that have surfaced and been completed.

The Sitemaps Testing Tool allows you to examine different types of crawl errors that Google finds on your site. You can examine these here -

We make available the search index downloads available files in several formats including json objects designed for network graphing.

Here we look at the most sustainable way to integrate robust cross-federation search into wiki. This is currently experimental - see Future Search

In the future we look to experiment with speeding up search. With the Search Plugin (below) we can see the effect of manipulating the results of searching the search index downloads: