Published on

Building Agent-Friendly Endpoints: RFC 9457 for the Agentic Web

Authors

Cloudflare just shipped something that should change how every web developer thinks about their API surface: RFC 9457-compliant structured error responses for AI agents. Their key insight is that when an AI agent hits your site, sending it 46KB of HTML with CSS and JavaScript is like handing someone a novel when they asked for directions.

The numbers are stark. A Cloudflare HTML error page: 46,645 bytes, 14,252 tokens. The same error as structured Markdown: 798 bytes, 221 tokens. That's a 98% reduction.

But Cloudflare only solved the error case. What about the happy path? When an agent successfully reaches your site, it still has to parse your full HTML, strip the navigation, sidebar, footer, and extract the content it actually needs. That's wasteful too.

So I built an agent-friendly digest endpoint for this blog.

The Problem

An AI agent wants to know what I've written about. Today, it has to:

  1. Fetch the homepage (HTML, React hydration, CSS)
  2. Parse the DOM to find blog post links
  3. Follow each link to get titles, dates, summaries
  4. Strip all the presentation markup to get the actual content

That's 4+ HTTP requests, tens of thousands of tokens in HTML parsing, and brittle selectors that break whenever I change my layout.

The Solution

One endpoint: /api/digest

It uses content negotiation. Send Accept: text/markdown and you get a token-optimized Markdown response with YAML frontmatter. Send Accept: application/json and you get structured JSON. Errors follow RFC 9457.

Markdown Response

GET /api/digest
Accept: text/markdown

Returns:

---
site: https://johnseong.ca
author: John Seong
description: Tech Lead | AI Systems
generated: 2026-03-13T22:00:00.000Z
post_count: 42
content_type: technical_blog
topics: [ai, agents, azure, development, devops, kubernetes, ...]
---

# John Seong - Site Digest

> Tech Lead | AI Systems

## Posts (42)

- [Building Agent-Friendly Endpoints](https://johnseong.ca/blog/AgentFriendlyEndpoints) | 2026-03-13 | 850 words | development, ai, agents, nextjs
  How I added an RFC 9457-compliant agent digest endpoint to my Next.js blog.
- [Building AI Debate Agents with Google A2A Protocol](https://johnseong.ca/blog/A2AProtocol) | 2026-01-25 | 1200 words | development, ai, agents, python
  Implementing Google's Agent2Agent protocol to make two local LLMs debate each other.
...

The YAML frontmatter gives machines stable keys to parse. The Markdown body gives LLMs readable context. Total: ~2-3KB vs ~50KB+ for crawling the HTML site.

JSON Response

GET /api/digest
Accept: application/json

Returns RFC 9457-shaped JSON:

{
  "type": "https://johnseong.ca/api/digest",
  "title": "John Seong - Site Digest",
  "status": 200,
  "detail": "Technical blog by John Seong. 42 posts.",
  "site": "https://johnseong.ca",
  "post_count": 42,
  "topics": ["ai", "agents", "azure", "development"],
  "posts": [
    {
      "title": "Building Agent-Friendly Endpoints",
      "url": "https://johnseong.ca/blog/AgentFriendlyEndpoints",
      "date": "2026-03-13",
      "tags": ["development", "ai", "agents", "nextjs"],
      "summary": "How I added an RFC 9457-compliant agent digest endpoint.",
      "word_count": 850
    }
  ]
}

Error Responses

Errors also follow RFC 9457. If an agent sends a POST instead of GET:

{
  "type": "https://johnseong.ca/api/digest",
  "title": "Method Not Allowed",
  "status": 405,
  "detail": "Only GET requests are supported.",
  "error_code": 1405,
  "error_name": "method_not_allowed",
  "error_category": "unsupported",
  "retryable": false,
  "owner_action_required": false
}

The agent immediately knows: don't retry, change the request method. No HTML parsing, no guessing.

Filtering

The endpoint supports query parameters for targeted retrieval:

GET /api/digest?tag=ai&limit=5           # Latest 5 AI posts
GET /api/digest?since=2026-01-01          # Posts from 2026
GET /api/digest?tag=devops&since=2025-06-01&limit=10

This lets an agent ask precise questions without downloading the entire catalog.

How It Works

Content negotiation via the Accept header. Agents that send Accept: text/markdown get Markdown with YAML frontmatter. Accept: application/json gets RFC 9457-shaped JSON. Browsers never see it — they keep getting the normal HTML site.

The Markdown format uses YAML frontmatter for machine-parseable keys (post_count, topics, content_type) and Markdown body for LLM-readable content. The JSON format follows RFC 9457's type/title/status/detail base members with extension fields for the blog-specific data.

Agent discovery happens at /.well-known/agent.json — a metadata file describing available endpoints, accepted formats, and query parameters. An agent hits the discovery URL, reads the contract, then calls /api/digest with the right headers. Two requests, full site knowledge, under 3KB of tokens.

Why This Matters

We're at an inflection point. AI agents are becoming a significant source of web traffic. Cloudflare is seeing billions of agent HTTP requests per day. These agents are building knowledge, researching topics, and executing multi-step workflows.

If your site only speaks HTML, you're forcing every agent to:

  • Waste tokens parsing your layout
  • Write brittle scrapers that break on redesigns
  • Make multiple requests to assemble what could be one response

The Cloudflare blog put it well: "Error pages stop being decoration and become execution instructions." The same applies to your entire API surface. Every endpoint should be able to answer the question an agent is actually asking, in the format that's cheapest for them to consume.

RFC 9457 gives us a standard shape for errors. YAML frontmatter in Markdown gives us a standard shape for content. Content negotiation via Accept headers gives us the routing layer. The pieces exist. We just need to assemble them.

The agentic web needs sites that can talk to machines as fluently as they talk to humans. This is a start.

The endpoint is live at johnseong.ca/api/digest. Agent discovery at /.well-known/agent.json.