#llms.txt (SSG-MD) experimental

#What is SSG-MD?

Rspress provides experimental Static Site Generation to Markdown (SSG-MD) capability, which is a brand new feature. Similar to the Static Site Generation (SSG) process, SSG-MD renders your pages as Markdown files instead of HTML files and generates llms.txt and llms-full.txt, making it easier for large language models to understand and use your technical documentation.

To help understand SSG-MD, here is an analogy between SSG and SSG-MD:

#What is llms.txt?

llms.txt is an emerging standard file format placed in a website's root directory to help large language models better understand and use website content.

Since LLMs have limited context windows and cannot process entire websites' HTML content, and converting complex HTML (with navigation, ads, JavaScript) to plain text is both difficult and imprecise, llms.txt uses Markdown format to provide a structured index of the website, including page URLs and their content descriptions, allowing AI to quickly locate and understand key information.

In simple terms:

• sitemap.xml → "Site map" for search engines
• llms.txt → "Documentation index" for AI

Output structure example:

doc_build
├── llms.txt
├── llms-full.txt
├── guide
│   └── start
│       └── introduction.md
└── ...

llms.txt content example:

# Rspress

> Rspress is a static site generator based on Rspack.

## Docs

- [Introduction](/guide/start/introduction.md): Introduction to Rspress
- [Quick Start](/guide/start/quick-start.md): Quick Start

#Why SSG-MD?

In frontend frameworks based on React dynamic rendering, there is often a problem of difficulty in extracting static information. This also exists in MDX, where .mdx files contain both Markdown content and support embedding React components, enhancing the interactivity of documents. For Rspress, Rspress allows users to use MDX fragments, custom components, React Hooks, tsx files as routes, etc. to enhance the expressiveness of document content. However, these dynamic contents are difficult to convert to Markdown format, and even if the html generated during the SSG phase is converted to markdown, the results are often unsatisfactory.

Static Site Generation (SSG) can generate static HTML files for crawlers to crawl, improving SEO. SSG-MD also solves similar problems, improving GEO and the quality of static information for large language models. Compared to converting html to markdown, React's virtual DOM during rendering has a better source of information.

#How to implement SSG-MD?

1. Rspress internally implements a renderToMarkdownString method similar to renderToString in react-dom, which renders React components to Markdown strings:

import { renderToMarkdownString } from 'react-render-to-markdown';

// HTML elements are converted to corresponding Markdown syntax
renderToMarkdownString(
  <div>
    <strong>foo</strong>
    <span>bar</span>
  </div>,
);
// Output: '**foo**bar'

// Supports React components and Hooks
const Article = () => {
  return (
    <>
      <h1>Hello World</h1>
      <p>This is a paragraph.</p>
    </>
  );
};
renderToMarkdownString(<Article />);
// Output: '# Hello World\n\nThis is a paragraph.\n'

In principle, this API works for any site built with React; see react-render-to-markdown if you're interested.

2. Provides import.meta.env.SSG_MD environment variable, making it easy for users to distinguish between SSG-MD rendering and browser rendering in React components, thus achieving more flexible content customization:

export function Tab({ label }: { label: string }) {
  if (import.meta.env.SSG_MD) {
    return <>{`** Here is a Tab named ${label}**`}</>;
  }
  return <div>{label}</div>;
}

3. Rspress internal component library has been adapted for SSG-MD to ensure reasonable Markdown content is rendered during the SSG-MD phase. For example:

<PackageManagerTabs command="create rspress@latest" />

Will be rendered as:

```sh [npm]
npm create rspress@latest
```

```sh [yarn]
yarn create rspress
```

```sh [pnpm]
pnpm create rspress@latest
```

```sh [bun]
bun create rspress@latest
```

```sh [deno]
deno init --npm rspress@latest
```

#Features

• Renders each site page as a .md file, convenient for vectorization or providing to large language models. /guide/start/introduction.html can be accessed by replacing the .html suffix with .md.
• Generates llms.txt, displaying the title and description of each page in navigation and sidebar order.
• Generates llms-full.txt, containing the Markdown content of each page, convenient for batch import.
• Supports multilingual sites, outputting corresponding {lang}/llms.txt and {lang}/llms-full.txt for non-default languages.

#Output example

doc_build
├── llms.txt
├── llms-full.txt
├── guide
│   └── start
│       └── introduction.md
└── ...

The actual files are placed in the build directory (such as guide/start/introduction.md), and the url in llms-full.txt will carry the site prefix, such as /guide/start/introduction.md.

llms-full.txt example snippet:

---
url: /guide/start/introduction.md
---

# Introduction

...

#How to enable

Enable llms in rspress.config.ts to generate the above files during the build phase:

import { defineConfig } from '@rspress/core';

export default defineConfig({
  llms: true,
});

After executing rspress build, you can see llms.txt, llms-full.txt and the .md files corresponding to each route in the output directory (default doc_build).

#Custom MDX splitting (Optional)

When documents contain custom components, you can control which components to keep or convert to plain text when converting to Markdown through remarkSplitMdxOptions:

import { defineConfig } from '@rspress/core';

export default defineConfig({
  llms: {
    remarkSplitMdxOptions: {
      excludes: [[['Demo'], '@project/components']],
    },
  },
});

• excludes: Matched components will be converted to plain text, with the highest priority.
• includes: If set, only matched components are allowed to be retained, and the rest will be converted to plain text.
• When configured simultaneously, excludes will be applied first, then filtered by includes.