Ping search engines on every deploy with IndexNow

You ship a release. The new pages are live. Then you wait. Sometimes for days. Search engines eventually wander back and notice something changed.

That waiting is dumb, and there's a fix for it. IndexNow lets you tell search engines "this URL changed, come look now" with a single HTTP POST. We added it to both SeggWat and infra.page and tied it to our deploy pipeline, so every release pings the search engines automatically. It took an afternoon. This post is the whole thing.

What IndexNow actually is, and isn't

IndexNow is a tiny open protocol. You POST a list of URLs to one endpoint, and it fans the submission out to every participating search engine: Bing, Yandex, Seznam, Naver. One request, everyone hears about it.

Be honest with yourself about the "everyone," though: Google is not on that list. Google has its own opinions about crawl scheduling and ignores IndexNow. So this isn't a Google play. It's a "get indexed fast on the other half of search" play, which is still worth an afternoon if you're a small product trying to show up anywhere at all.

There are two pieces to it:

1. Proof you own the domain. You serve a key at https://yoursite.com/{key}.txt that returns the key as plaintext. That's how IndexNow knows you're allowed to submit URLs for that host.
2. The submission itself. A POST with the list of URLs you want recrawled.

Most tutorials show you doing both by hand, once. We wanted it to be automatic and zero-maintenance. Here's how each piece works.

Piece 1: serve the key file, don't commit it

The lazy way is to drop a static key.txt in your public/ folder and forget about it. Don't. That puts your key in the repo, ships it to every environment including local dev, and means the file exists whether or not the feature is actually configured.

Instead, serve it from the app, gated on a secret. The route only exists when the key is set, and the route path is the key. The handler just echoes it back:

// Serve the IndexNow key file at /{key}.txt.
// The route is only registered when the key is configured, and its path
// literally is the key, so any request that reaches this handler returns
// that exact key as plaintext, which is all IndexNow wants.
pub async fn indexnow_key() -> Response {
    let state = app_state();
    match state.indexnow_key.as_deref() {
        Some(key) => (
            [(header::CONTENT_TYPE, "text/plain; charset=utf-8")],
            key.to_string(),
        )
            .into_response(),
        None => StatusCode::NOT_FOUND.into_response(),
    }
}

Wire it up conditionally. No key, no route:

// Registered only when INDEXNOW_KEY is set. The whole feature stays inert
// on dev and on self-hosted deployments that never configured it.
let router = if let Some(key) = state.indexnow_key.clone() {
    router.route(&format!("/{key}.txt"), get(indexnow_key))
} else {
    router
};

The key itself is just openssl rand -hex 16, loaded from an env var. The payoff is simple: the secret never touches the repo, the file only exists in production, and everywhere else the route 404s. Nothing to clean up, nothing to leak.

Piece 2: submit on deploy, not on build

Now the submission. The instinct is to fire it during the build. Resist it. You have to submit after the new content is actually reachable. Otherwise you're inviting crawlers to URLs that may still 404 because the deploy hasn't swapped over yet.

So this runs as the last step of the deploy job, after the container is up. Here's the real step from our pipeline, with comments left in:

- name: Submit URLs to IndexNow
  env:
    INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
    SITE: ${{ vars.SITE_URL || 'https://infra.page' }}
  run: |
    if [ -z "$INDEXNOW_KEY" ]; then
      echo "INDEXNOW_KEY not set — skipping."
      exit 0
    fi

    # host with no scheme/path, for the payload
    HOST="${SITE#http://}"; HOST="${HOST#https://}"; HOST="${HOST%%/*}"

    # Wait until the NEW release is live and serving the key file.
    # This is the bit everyone forgets: during a rolling swap there's a
    # window where the old container still answers. Poll until the key
    # file comes back with the expected value before submitting anything.
    KEY_URL="$SITE/$INDEXNOW_KEY.txt"
    ready=
    for i in $(seq 1 30); do
      body=$(curl -fsS --max-time 10 "$KEY_URL" 2>/dev/null || true)
      if [ "$body" = "$INDEXNOW_KEY" ]; then ready=1; break; fi
      echo "Waiting for $KEY_URL (attempt $i/30)…"
      sleep 10
    done
    if [ -z "$ready" ]; then
      echo "::warning::Key file not reachable after 5min — skipping."
      exit 0
    fi

    # The URL list IS the sitemap. One source of truth, can't drift.
    urls=$(curl -fsS --max-time 30 "$SITE/sitemap.xml" | grep -oP '(?<=<loc>)[^<]+' || true)
    count=$(printf '%s\n' "$urls" | grep -c . || true)
    [ "$count" -eq 0 ] && { echo "::warning::Empty sitemap — skipping."; exit 0; }

    payload=$(printf '%s\n' "$urls" | jq -R . | jq -s \
      --arg host "$HOST" --arg key "$INDEXNOW_KEY" --arg loc "$KEY_URL" \
      '{host: $host, key: $key, keyLocation: $loc, urlList: .}')

    status=$(curl -s -o /tmp/resp.txt -w '%{http_code}' \
      -X POST 'https://api.indexnow.org/indexnow' \
      -H 'Content-Type: application/json; charset=utf-8' \
      --data "$payload" || echo "000")

    if [ "$status" -ge 200 ] && [ "$status" -lt 300 ]; then
      echo "✅ IndexNow accepted $count URLs (HTTP $status)."
    else
      echo "::warning::IndexNow returned HTTP $status. Deploy is unaffected."
    fi

A few details matter more than they look.

The poll loop is the big one. During a rolling deploy there's a window where the old container is still answering requests. If you submit in that window, you might be telling crawlers about content that isn't live yet, or the request goes to a box that's about to die. So we poll the key file until the new release serves it. The key file does double duty here: it proves ownership and gives us a cheap "is the new version up?" check.

The sitemap is the URL list. We don't hand-maintain a list of URLs to submit. We read sitemap.xml and pull every <loc>. Your sitemap already knows what's indexable, so reusing it means the submission list can't drift from reality. Add a page, it shows up in the sitemap, it gets submitted. No second list to forget about.

Best-effort, always. Every failure path is a ::warning:: and an exit 0. An SEO ping must never fail your deploy. By the time this step runs, your app is already live and serving users. If IndexNow is having a bad day, that's IndexNow's problem, not your release's. Letting a search-engine nicety break a production deploy is exactly backwards.

The tradeoffs we made on purpose

This is a deploy-time mechanism, so it has edges. Fine. Just know which ones you're accepting:

• It fires on deploy, not on content change. For mostly-static sites like a marketing site, docs, or a blog, that's perfect. Content changes are deploys. For infra.page, where users create public pages at runtime, those pages only get re-pinged on the next deploy. They're still in the sitemap so they do get submitted, just not instantly. If you need instant pings for runtime content, fire one from your app when the content changes. The deploy step and a live ping can live together.
• Each deploy re-submits the whole sitemap. We don't diff. IndexNow is fine with this, and "submit everything that's indexable" beats maintaining a changed-since-last-time list. If your sitemap grows into the thousands and this starts to feel rude, that's when you add a diff. Not before.
• It's bash in CI, not code in the app. On purpose. The trigger is deterministic, it's visible in the workflow log, it's re-runnable, and it submits only after the release is provably live. Doing it in-app means wiring it into your boot sequence and reasoning about which instance submits. For deploy-time submission, the CI step is the simpler, more honest place for it.

That last point is also why the same pattern dropped cleanly into two different products with one decision flipped per product: SeggWat scopes submission to our own first-party marketing and docs pages, while infra.page lets the sitemap include public user pages too. Same machinery, different sitemap.xml. The pattern doesn't care what's in the list.

Did it work?

Generate your key, set INDEXNOW_KEY as a CI secret (and in your runtime env so the app serves the file), deploy, and then check Bing Webmaster Tools → IndexNow. It shows submission counts and any rejected URLs. If the key file isn't reachable it'll tell you there, which is a much nicer feedback loop than guessing.

Worth it?

This is the kind of infra chore that pays rent quietly. It cost an afternoon, it costs nothing to run, and now every release tells half the search world to come look, without anyone remembering to do it. For a small team shipping a product nobody's heard of yet, "get found faster, for free, forever" is an easy yes.

The whole thing is maybe sixty lines split across a Rust handler and a CI step. If you've got a sitemap and a deploy pipeline, you already have the hard parts. Go connect them.