Sitemap, robots.txt and llms.txt

Sitemap

The sitemap lists published pages, news items and events in every language where that content is active and indexable. Drafts, language versions that are switched off, and items marked noindex are excluded.

You do not need to maintain it manually. It rebuilds after content changes and also refreshes overnight. SEO settings show when it was last generated, and Refresh lets you rebuild it immediately. The interface warns you if the sitemap has not refreshed for more than 24 hours.

Tip: submit https://your-domain.example/sitemap.xml in Google Search Console if you want to give Google a direct sitemap reference. Submission can help discovery, but it does not guarantee indexing or ranking.

To check which pages Google and Bing have discovered and which searches bring people to your website, see our Google Search Console and Bing Webmaster Tools guide.

robots.txt

robots.txt tells compliant crawlers which paths they should not crawl. Demona automatically blocks settings, sign-in and other internal areas, as well as the search-results page, which could otherwise create many low-value URLs. The sitemap address is also included at the end of the file.

Custom blocked paths: add any extra paths under Custom disallowed paths, one per line and starting with a slash, for example /private. For a single public page, use that page's indexing control instead.

AI crawlers

Allow AI crawlers is enabled by default. If you turn it off, robots.txt blocks the AI crawler user agents supported by Demona, including agents associated with OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon and Mistral. The exact effect depends on whether each service respects robots.txt and which crawler it uses.

You can also exclude individual pages from supported AI crawlers in Page settings. These controls affect crawler access signals; they do not guarantee that a page will or will not appear in any particular AI service.

llms.txt

llms.txt is a short, structured index of public pages, news items and events, intended to make the site's content easier for AI assistants and agents to discover and parse.

• On multilingual sites, each language has its own index, such as /en/llms.txt, with references between language versions.
• Each listed page is also available at the same URL with a .md suffix in a simplified Markdown representation for tools that prefer text-first content.
• Pages excluded from AI crawlers are omitted.
• The file rebuilds after content changes and overnight, and can also be refreshed manually with Refresh.

llms.txt is only exposed when the site is live and both site indexing and AI crawler access are enabled. SEO settings show which switch is missing when it is unavailable. Providing llms.txt can make the site easier for compatible tools to inspect, but it does not guarantee inclusion in AI-generated answers.