DOC-2262: Add OAuth authentication to the docs MCP server (Redpanda Cloud IdP)

[JakeSCahill]

Jira: DOC-2262

Goal

Add authentication to the docs MCP server (docs.redpanda.com/mcp) so AI tools (ChatGPT, Claude, Cursor, VS Code) have users sign in with their Redpanda Cloud account, letting us capture verified work emails and attribute docs usage to organizations.

Architecture (decided with Cloud Identity)

The docs service runs its own OAuth 2.1 Authorization Server (AS), with the Cloud IdP (Auth0) as the upstream identity provider. AI tools register and authenticate against our AS; we federate the human login to Auth0 and issue our own tokens.

Why this shape:

• ChatGPT only supports spec OAuth (no static tokens), so OAuth is required.
• The Cloud Auth0 tenant has DCR and CIMD disabled (tested — see comments), so AI tools can't register with it directly. Brokering means the Cloud IdP only ever sees one client (ours), and client self-registration happens on our side where we control it.
• We get the user's verified email/org from Auth0 to capture + attribute.

Division of responsibilities

• Cloud / Identity: one Auth0 public client (client_id, Authorization Code + PKCE, no secret), our /callback redirect URIs allow-listed, ID token returns email/email_verified + org. One app covers MCP now and docs-site login later.
• Us: the AS — /authorize, /callback, /token (+ refresh w/ rotation), client registration (DCR + CIMD), JWKS, consent/login UI; federate login to Auth0; issue/validate our own tokens. State in Netlify.

Also in this PR: documentation feedback tool

Adds an MCP tool, submit_documentation_feedback, so AI clients can forward user feedback for bugs, documentation gaps, incorrect/missing info, feature requests straight to the docs/DX team. The tool description instructs the agent to ask the user before submitting and to include the relevant page/context.

• Lands in the existing api-feedback Netlify form (same store as our docs feedback); the hidden registration form gains category, source, and user-identity fields.
• When the user is signed in, their email + domain are attached so the team can follow up; anonymous otherwise. Logs only category/domain/authed — never the raw email or feedback text.
• Surfaced in the server card + server.json descriptions; server version bumped to 1.3.0.

Future (phase 2)

The same Auth0 federation core also powers human login to the docs site — it just sets a browser session instead of issuing tokens. One Auth0 app, two consumers; MCP ships first.

Testing

2026-06-18_19-50-10.mp4

The deploy preview is wired to the integration Auth0 tenant, so you can test the full flow there.

1. Add the preview server to Claude Code:

claude mcp add --scope local --transport http redpanda-preview https://deploy-preview-181--redpanda-documentation.netlify.app/mcp

2. Authenticate and use it:

• Run /mcp, select redpanda-preview, choose Authenticate
• A browser opens → "Continue with Redpanda Cloud" → sign in on the integration tenant → pick an org at the prompt → it connects
• Ask something that hits a tool, e.g. "Using redpanda-preview, list the Redpanda API reference pages."
• Try the feedback tool, e.g. "Send feedback to the Redpanda team that the quickstart is missing a step." → check Netlify → Forms → api-feedback
• Clean up with claude mcp remove redpanda-preview

Notes:

• You need an account on the integration tenant (integration-cloudv2.us.auth0.com), not prod.
• Unit tests: npm run test:mcp plus the tests/mcp-oauth-*.test.ts suites.

Kapa now has your conversation along with your email and org

History

Explored a pure OAuth resource server pointing at the Cloud IdP (blocked: DCR/CIMD disabled on that tenant), landing on the AS-broker design above after the call with Cloud.

[netlify]

👷 Deploy Preview for redpanda-documentation processing.

Name	Link
🔨 Latest commit	425d2c9
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-documentation/deploys/6a3919de1ec37200087160b7

[JakeSCahill]

⛔ Blocked: identity team — Dynamic Client Registration is disabled on the Cloud IdP

The OAuth resource server, discovery, and token validation are implemented and verified, but end-to-end auth cannot work yet because MCP clients can't register with the Redpanda Cloud IdP.

What works (verified against prd auth.prd.cloud.redpanda.com)

• Protected-resource metadata (/.well-known/oauth-protected-resource) + authorization-server discovery
• PKCE (S256), email / email_verified scopes
• Token validation via /userinfo
• Our server's 401 → WWW-Authenticate → metadata chain (confirmed on the preview)

The blocker

MCP clients (ChatGPT, Claude, Cursor, VS Code) have no pre-registered client_id — they self-register at runtime via Dynamic Client Registration. The Cloud IdP rejects this:

POST https://auth.prd.cloud.redpanda.com/oidc/register
→ 400 {"error":"Bad Request","message":"dynamic client registration is disabled"}

Confirmed with a valid registration body. Reproduced in Claude Code, which fails at exactly this step:

No client info found
SDK auth error: L7H
Error during auth completion: SDK auth failed

(Failure happens before the browser opens — i.e. at client registration, not at user login.)

Needed from the identity team (DOC-2262)

1. Enable Dynamic Client Registration for public clients (authorization_code + PKCE, token_endpoint_auth_method: none, client-supplied loopback/https redirect URIs) — and/or confirm CIMD (Client ID Metadata Documents) is actually enabled. The discovery doc advertises client_id_metadata_document_supported: true, but DCR was advertised too and is disabled, so advertised ≠ enabled. CIMD is what ChatGPT prefers.
2. (Hardening) Register an Auth0 API/resource with audience https://docs.redpanda.com/mcp and add email, email_verified, and org (org_id/org_name) as access-token claims → lets us validate audience-bound JWTs instead of /userinfo.
3. Confirm whether to enable on prd or a staging tenant first.

Status

• Code is complete and verified up to the registration step; keeping this PR in draft until the IdP is configured.
• Tracking the identity-team request in DOC-2262.
• Recommend leaving REQUIRE_AUTH in grace (unenforced) on deployed environments until DCR/CIMD is enabled, so the server isn't gated for clients that can't yet authenticate.

[JakeSCahill]

Update — call with Cloud Identity (Santi): implementation direction decided

Architecture: We unblock MCP auth by having the docs service run its own OAuth 2.1 Authorization Server (AS), with the Cloud IdP (Auth0) as the upstream identity provider. This sidesteps the earlier blocker (the Cloud Auth0 tenant has both DCR and CIMD disabled — see prior comments): the AI tools register and authenticate against our AS, not the Cloud IdP directly. The Cloud IdP only ever sees one client — ours.

Division of responsibilities

• Cloud / Identity (Santi): provide one Auth0 public client (client_id, Authorization Code + PKCE/S256, token_endpoint_auth_method=none, no secret), with our /callback redirect URIs allow-listed, and the ID token returning email, email_verified, and org (org_id/org_name). One app covers MCP now and docs-site login later.
• Docs (us): implement the OAuth 2.1 AS — /authorize, /callback, /token (+ refresh with rotation), client registration (DCR + CIMD) for the AI tools, JWKS, and the consent/login UI; federate the human login to Auth0; issue our own tokens to MCP clients. State in Netlify DB (Neon Postgres).

Flow: AI client → our AS (/authorize, PKCE) → redirect to Auth0 login → our /callback (exchange code, read verified email/org) → we mint our own token → client calls /mcp with it → we validate + attribute usage to the user/org.

Open asks to Santi (in flight):

• Confirm public client + exact /callback allow-listing + email/org claims.
• Which tenant for dev vs prod (auth.prd.cloud.redpanda.com is prod) + a test user or two.

Future (phase 2): the same Auth0 federation core also powers human login to the docs site — it just sets a browser session instead of issuing tokens. One Auth0 app, two consumers; MCP ships first.

Next steps:

1. Spike the AS slice (/authorize → Auth0 → /callback → issue + validate a JWT) to confirm Netlify DB + serverless fit and that the Auth0 client works end-to-end.
2. Phased build: AS core → client registration (DCR/CIMD) → refresh + hardening → swap the resource-server to validate our own tokens → docs + rollout (REQUIRE_AUTH grace → enforce).
3. Rescope PR DOC-2262: Add OAuth authentication to the docs MCP server (Redpanda Cloud IdP) #181 from "resource server pointing at the Cloud IdP" to "we run the AS, Auth0 upstream."

[JakeSCahill]

Milestone 1 landed — production AS scaffold (jose + storage), federating to Auth0

Pivoted the branch from the superseded resource-server-pointing-at-Cloud approach to the agreed broker architecture: our service is the OAuth 2.1 Authorization Server; it federates the human login upstream to Auth0 and issues/validates its own tokens. The validated spike is now ported to production shape.

In this milestone

• lib/oauth/keys.mjs — jose RS256 sign/verify + JWKS. Key from env (MCP_OAUTH_SIGNING_JWK) or dev-generated + persisted in Blobs (the spike proved an in-memory key breaks the flow).
• lib/oauth/store.mjs — auth-requests + auth-codes on Netlify Blobs; the interface is the seam for a Netlify DB / Neon backend when we want relational queries (swap behind the interface; needs DB provisioning).
• lib/oauth/{pkce,config,upstream}.mjs — PKCE (S256), config, and Auth0 federation (id_token validated against Auth0 JWKS) with a dev-mock fallback.
• mcp-oauth.mjs — AS endpoints: discovery (RFC 8414), JWKS, /authorize, /mcp/callback, /token (authorization_code + PKCE).
• mcp.mjs resource server now validates our own access tokens (jose) instead of calling the upstream /userinfo; protected-resource metadata + server card point authorization_servers at us; removed lib/idp.mjs.

Tests: 22 pass (PKCE incl. the RFC 7636 vector; JWT issue/verify; JWKS leaks no private key; wrong-audience/tampered rejected).

Deferred (clearly marked in code): DCR/CIMD client registration (M2), refresh-token grant + rotation (M3), consent UI, revocation.

Still gated on:

• Santi: the Auth0 client_id (public client + /mcp/callback allow-listed + email/org claims). Until then upstream defaults to a dev mock; flip with REDPANDA_OAUTH_CLIENT_ID + MCP_OAUTH_UPSTREAM=auth0.
• Netlify DB provisioning if/when we move auth state off Blobs.

The flow itself was already validated end-to-end on Netlify Functions in the spike branch (spike/mcp-oauth-as).

[JakeSCahill]

After thinking through the storage options for the OAuth layer, I’m leaning toward using Neon Postgres (via Netlify’s integration) as the system of record for auth state, rather than Netlify Blobs.
Even though this MCP is currently lightweight (public docs + agent access), the direction we’re heading in includes a shared authentication system for both the docs site and a chatbot with saved conversations. That introduces more durable identity state (sessions, refresh tokens, conversation links), which benefits from strong consistency and transactional guarantees.

Postgres gives us:

• safe single-use semantics for authorization codes (transactional “check + consume”)
• clean modeling for users, sessions, orgs, and conversations
• a natural path to unify docs + chatbot identity under one system

Netlify Blobs feels fine for simple metadata or low-risk storage, but it doesn’t provide strong enough guarantees for OAuth flows where race conditions or replay could become an issue.
Given our current scale (~1k users/week), Neon is more than sufficient and keeps the system simple while leaving room for future expansion into persistent user identity and chat history.