REST API reference

JSON over HTTPS. Authenticate with Authorization: Bearer cb_live_… (or x-api-key). Limit: 300 requests/min per key. Submission endpoints only validate and enqueue, so they respond in milliseconds — every response carries an x-response-time header.

Quick start

curl -X POST http://localhost:3000/api/v1/submit \
  -H "Authorization: Bearer $CRAWLBEAM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/new-post"],"strategy":"fresh"}'

// Node 18+ / Next.js route handler — notify Crawlbeam when a post is published
await fetch("http://localhost:3000/api/v1/submit", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.CRAWLBEAM_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ urls: [post.url], strategy: "fresh" }),
});

Strategies

autoPer-URL: owned → fresh rules, external → guest rules.

freshAll channels now; checks at 10m, 1h, 6h, 24h, 72h; one re-push at 48h.

agedAll channels now; re-push at 24h and 72h; checks to day 7.

guestDiscovery feed + WebSub + pings; re-push at 12h and 48h.

Endpoints

POST/api/v1/submit

Enqueue up to 10,000 URLs. Returns 202 immediately with a batch id.

{
  "urls": ["https://example.com/a", "https://example.com/b"],  // or one newline-separated string
  "strategy": "auto",          // auto | fresh | aged | guest
  "project_id": "p_…",         // optional — inferred from the URL host
  "name": "Product reviews"    // optional
}

{ "data": { "batch_id": "b_…", "project_id": "p_…", "strategy": "auto", "accepted": 2,
  "status_url": "http://localhost:3000/api/v1/batches/b_…", "dashboard_url": "…" } }

GET/api/v1/batches

List recent batches with submitted / crawled / indexed counts. ?limit=25

GET/api/v1/batches/:id

Full batch: every URL, its status, first bot visit and per-channel results.

GET/api/v1/url-status?url=…

Latest tracked state for a single URL.

POST/api/v1/check

Indexability check for up to 50 URLs (+ Google URL Inspection for connected properties).

{ "urls": ["https://example.com/a"] }

GET/api/v1/stats

Totals, avg. bot visit time, channel acceptance and a 14-day series. ?project_id=

GET/api/v1/projects

List projects with IndexNow key location and credential status.

POST/api/v1/projects

Create a project.

{ "domain": "example.com", "name": "Example" }

POST/api/v1/crawl-events

Crawl beacon ingest. Auth with the x-crawlbeam-beacon header (per-project secret), not an API key.

{ "url": "https://example.com/a", "user_agent": "Mozilla/5.0 (compatible; Googlebot/2.1…)", "ip": "66.249.66.1" }

Errors

{ "error": { "code": "no_valid_urls", "message": "No valid http(s) URLs found" } }

400 invalid_body · 401 unauthorized · 404 not_found / no_project
413 too_many_urls · 429 rate_limited · 500 internal_error