HTTP header inspector: parse response headers and spot the important ones

This tool parses raw HTTP response headers and flags the ones that change a crawler’s behaviour. Paste the headers from curl -I, DevTools, or a proxy log. Nothing is uploaded.

What it flags

• Missing Content-Type.
• Missing Cache-Control.
• Missing Strict-Transport-Security (informational).
• X-Robots-Tag containing noindex or none.
• Retry-After on 429/503.
• Rate-limit headers (RateLimit-*, X-RateLimit-*).
• Multiple Set-Cookie headers.

Why these headers matter

• Content-Type tells you how to parse the body. A JSON endpoint served as text/html is a warning sign.
• Cache-Control tells you whether a response can be reused.
• ETag and Last-Modified enable conditional requests; a 304 saves the body.
• Retry-After is the server’s instruction for a retry.
• X-Robots-Tag can noindex a response even when the HTML has no meta tag.
• Set-Cookie is state. Keep it only if the crawl needs a session.

How to use it

1. Run curl -sI https://example.com/ or copy the response headers from DevTools.
2. Paste them into the textarea.
3. Read the parsed table and the notes.
4. Fix the request or the retry policy accordingly.

Limits

• It parses the text you paste; it does not fetch the URL.
• It does not validate every header against its RFC grammar.
• It cannot see headers added or removed by a proxy after the origin.
• It does not execute JavaScript.

For the HTTP model, read HTTP for crawlers. For retries, read retries and idempotency.

Sources

• RFC 9110
• RFC 9111
• MDN — HTTP headers