Sitemap index

[omerp-explorium]

Goals

When using the multiple sitmap approach, Nextjs should produce a root sitemap that will point to all sitemap urlsets e.g /sitemap.xml

Non-Goals

Background

As you can see in the image, we should also have this sitemap pointing to multiple sitemaps

Taken from Google documentation on large sitemaps

Proposal

I propose Nextjs will keep track of all the generated sitemaps, and generate the proposed sitemap of sitemaps xml.

What do you think?

[Fedeorlandau]

I made a PR that adds support for sitemap-index.ts which you can use to dynamically add your sitemaps to the sitemap index file.

#61391

[Fedeorlandau]

Hi! This is just for having a sitemap-index file where you can manually generate the index and put your other sitemaps files. You still need to generate sitemap.xml files for your site.

[tomcru]

Thanks for the PR @Fedeorlandau. Hope to see this merged soon!

Could you possible give any pointers on how you have handled sitemap index creation until now?

[Fedeorlandau]

@tomcru absolutely. The way I found it to work is by creating a /app/sitemap-index.xml/route.ts with a route GET handler and returning a xml like response.

[tomcru]

For anyone else coming across this, here's a quick example I used to test a sitemap index via /app/sitemap-index.xml/route.ts as Fedeorlandau suggested.

const PAGE_SIZE = 1000;

const generateSitemapLink = (url: string) =>
  `<sitemap><loc>${url}</loc></sitemap>`;
  
export async function GET() {
  const gamesResponse = await fetch(
    `${API.base}/products?${new URLSearchParams({
      pageSize: String(PAGE_SIZE),
      pageNumber: String(1),
      currency: Currency.usd,
      order: 'nameAsc',
    })}`,
  );

  const games = (await gamesResponse.json()) as GetProductsOkResponse;

  const sitemapIndexXML = `<?xml version="1.0" encoding="UTF-8"?>
    <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${generateSitemapLink(`${URLS.base}/sitemap.xml`)}

        ${Array.from({ length: games.data.pageCount }, (_, i) => i + 1)
          .map((id) =>
            generateSitemapLink(`${URLS.base}/games/sitemap/${id}.xml`),
          )
          .join('')} 
    </sitemapindex>`;

  return new Response(sitemapIndexXML, {
    headers: { 'Content-Type': 'text/xml' },
  });
}

Result:

[MartineBoulanger]

Thank you for this! I was looking for a solution since this sitemapIndex() is still a WIP, and your solution works perfectly!

[antoinerousseau]

I don't understand the utility of generateSitemaps without a sitemaps index, am I missing something?

[hermbo]

Agreed, seems like this should just be automatic. Then you only need to provide google with the index file.

[CaptainN]

Next.js has been filling up with half baked "features" - I'm not sure why...

[antoinerousseau]

Meanwhile, I went with my own custom index at src/app/sitemap.xml/route.ts:

import { env } from "~/env";
import { getLastMod } from "~/lib/sitemap";
import { generateSitemaps as getArtistSitemaps } from "../api/sitemaps/artists/sitemap";
import { generateSitemaps as getEventSitemaps } from "../api/sitemaps/events/sitemap";
import { generateSitemaps as getVenueSitemaps } from "../api/sitemaps/venues/sitemap";

export const revalidate = 3600; // cache for 1 hour

function getFileName(id?: number) {
  if (id == null) {
    return "sitemap.xml";
  }
  /**
   * https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap#generating-multiple-sitemaps
   * In production, your generated sitemaps will be available at /.../sitemap/[id].xml. For example, /product/sitemap/1.xml.
   * In development, you can view the generated sitemap on /.../sitemap.xml/[id]. For example, /product/sitemap.xml/1. This difference is temporary and will follow the production format.
   * See the generateSitemaps API reference for more information.
   */
  return env.VERCEL_ENV === "development"
    ? `sitemap.xml/${id}`
    : `sitemap/${id}.xml`;
}
function getLoc(path: string, id?: number) {
  return `${env.URL}/api/sitemaps/${path}/${getFileName(id)}`;
}
function getSitemap(path: string, id?: number) {
  return /* XML */ `<sitemap><loc>${getLoc(path, id)}</loc><lastmod>${getLastMod()}</lastmod></sitemap>`;
}
function getSitemaps(ids: { id: number }[], path: string) {
  return ids.map(({ id }) => getSitemap(path, id)).join("");
}

export async function GET() {
  const xml = /* XML */ `<?xml version="1.0" encoding="UTF-8"?>
    <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      ${getSitemap("cities")}
      ${getSitemaps(await getEventSitemaps(), "events")}
      ${getSitemaps(await getArtistSitemaps(), "artists")}
      ${getSitemaps(await getVenueSitemaps(), "venues")}
    </sitemapindex>
  `;

  return new Response(xml, {
    headers: {
      "Content-Type": "application/xml",
    },
  });
}

[SemihOzden]

I have implemented same logic in my next.js project and It worked. Thanks in advance.

[jonathanlal]

legend thank you - elegant solution

[Sammeeey]

agreed, we need proper sitemap generation (incl. sitemap-index)

[pondorasti]

For anyone looking for a alternatives, I would recommend the server side utils from https://github.com/iamvishnusankar/next-sitemap?tab=readme-ov-file#server-side-sitemap-getserversidesitemap. You can plug them in into a route handler and it probably has all the features you would ever need.

[rtorrente]

I don't really understand the goal of generateSitemaps if it's just to generate the available url without indexing them.
Paginated sitemap are growing indefinitly on a news site as post are created. It's just not possible to add new sitemap in google console because there is no alert like "New sitemap generated". Especially since, when using generateSitemap, the basic sitemap.xml file returns a 404, where indexes would be more appropriate. It seems more like a bug than a new feature at this level.

At least, MetadataRoute.Sitemap have to accept both url or sitemap link to allow dev to reference another sitemap file and generate an xml <url></url> or <sitemap></sitemap> accordingly.

[tomcru]

You create a Sitemap index that links to all your sitemaps - then you just add the index to google search console and it's able to find all of them. Like so: https://gamegator.net/sitemaps/index.xml (if that's what you mean)

[rtorrente]

Yes but it need to create a route and generate manually XML and we can't use he sitemap.ts at the root app/ directory

Because sitemap.ts only generate <url> XML tag and not <sitemap> XML tag

[heymartinadams]

Maybe this can help @rtorrente https://raddy.dev/blog/multiple-sitemaps-in-nextjs-app-router-sitemap-index/

[dsbrianwebster]

Sharing the approach we’ve been using:

We have a script that runs at the end of our build process. This script scans the app directory for all instances of sitemap.ts, resolves their URLs based on relative paths, and generates a single sitemap index at the root.

We like this approach because, once it’s in place, all we need to do is create sitemap.ts instances following Next.js file conventions, with no additional work required. It’s also copy n' paste reusable across projects. Additionally, it even handles sitemap.ts files that use generateSitemaps and adds all the generated sitemaps to the index.

Here’s how you might integrate it into your package.json:

  "scripts": {
    "build": "next build && pnpm sitemap:generate",
    "sitemap:generate": "dotenv -c -- npx tsx ./scripts/sitemap.ts",
  },

And here is the script...

import { promises as fs } from 'fs';
import path from 'path';

import { SITE_URL } from '~/app/_lib/constants';

/**
 * 🗺️ Generates the main sitemap index file.
 *
 * This function creates a sitemap index file by discovering and aggregating
 * sitemap files generated across the project. It is temporary until Next.js
 * offers native support for sitemap indexing.
 */

const ROOT_DIR = process.cwd();
const APP_DIR = path.join(ROOT_DIR, 'app');

async function main() {
  try {
    const sitemaps = await findSitemapFiles(APP_DIR);
    const SITEMAP_INDEX_PATH = path.join(ROOT_DIR, 'public', '/sitemap.xml');

    const createSitemap = (url: string) => /* XML */ `
      <sitemap>
        <loc>${url}</loc>
      </sitemap>`;

    const createSitemapIndex = (urls: string[]) => /* XML */ `
      <?xml version="1.0" encoding="UTF-8"?>
      <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${urls.map(createSitemap).join('')}
      </sitemapindex>`;

    const xml = createSitemapIndex(sitemaps);
    await fs.mkdir(path.dirname(SITEMAP_INDEX_PATH), { recursive: true });
    await fs.writeFile(SITEMAP_INDEX_PATH, xml.trim());

    console.log(`Sitemap index generated with ${sitemaps.length} entries.`);
  } catch (error) {
    console.error('Error generating sitemap:', error);
  }
}

/**
 * 🔎 Recursively finds 'sitemap.ts' files and extracts sitemap URLs.
 *
 * Traverses directories from the specified start point, looking for 'sitemap.ts' files.
 * If a file contains a `generateSitemaps` function, the function is invoked
 * to retrieve sitemap URLs. If absent, a default sitemap URL is constructed.
 *
 * Known Limitations:
 * - While this does consider Next.js conventions for folders like route groups, do not place
 *   'sitemap.ts' under folders like route groups untill the following seperate issue is resolved:
 *    https://github.com/vercel/next.js/issues/68403
 *
 * @param {string} dir - The base directory to begin searching for sitemap files.
 * @returns {Promise<string[]>} A promise that resolves to an array of sitemap URLs.
 */
async function findSitemapFiles(dir: string): Promise<string[]> {
  let sitemaps: string[] = [];
  try {
    const files = await fs.readdir(dir, { withFileTypes: true });
    for (const file of files) {
      const fullPath = path.join(dir, file.name);
      if (file.isDirectory()) {
        // ingore private folders since all its child segments are opted out of routing
        // https://nextjs.org/docs/app/getting-started/project-structure#route-groups-and-private-folders
        if (!file.name.startsWith('_')) {
          const nestedSitemaps = await findSitemapFiles(fullPath);
          sitemaps = sitemaps.concat(nestedSitemaps);
        }
      } else if (file.isFile() && file.name === 'sitemap.ts') {
        const relativePath = path
          .relative(APP_DIR, dir)
          .split(path.sep)
          .filter(
            (segment) =>
              // Exclude specific Next.js folder conventions that should not be part of the resolved path
              // https://nextjs.org/docs/app/getting-started/project-structure#route-groups-and-private-folders
              // https://nextjs.org/docs/app/getting-started/project-structure#parallel-and-intercepted-routes
              // https://nextjs.org/docs/app/getting-started/project-structure#dynamic-routes
              !/^[(\[@]/.test(segment),
          )
          .join('/');
        const sitemapFile = await import(fullPath);
        if (sitemapFile.generateSitemaps) {
          const result = await sitemapFile.generateSitemaps();
          result.forEach((item: { id: number }) => {
            sitemaps.push(`${SITE_URL}/${relativePath}/sitemap/${item.id}.xml`);
          });
        } else {
          sitemaps.push(`${SITE_URL}/${relativePath}/sitemap.xml`);
        }
      }
    }
  } catch (error) {
    console.error('Error finding sitemaps:', error);
  }
  return sitemaps;
}

main();