# GET /crawl: $0.03 per call

Turn a small website, or one section of it, into clean Markdown: starts from your URL, finds pages from the sitemap or by following links on the same site, obeys robots.txt, and returns up to 10 pages (title, Markdown, address) in one call. Great for docs, blogs and company sites. ?url=https://example.com/docs/&maxPages=10&path=/docs/

- **Price:** $0.03 in USDC, the same on Base, Solana, Polygon, Arbitrum. Failed calls are never charged.
- **Free trial:** yes, 20 free calls a day from Claude, Cursor or any MCP client ([set-up](/mcp/setup)).
- **Tier:** live
- **Answers cached for:** 1 hour(s)
- **Data sources:** The site's own pages, robots.txt and sitemaps
- **Lane:** Web search and crawling ([OpenAPI](/openapi/web-search.json))
- **Live health:** [status page](/status)

## Free sample

See an answer for the demo input first, free (no payment, 10 a minute): [https://aayatai.com/sample/crawl](/sample/crawl). It is a stored real answer when we have one, otherwise an example marked `"kind": "illustrative"`.

```bash
curl "https://aayatai.com/sample/crawl"
```

## 1. See the price (free)

Call it without paying: you get `402 Payment Required` and a `PAYMENT-REQUIRED` header with the exact price and where to pay.

```bash
curl -i "https://aayatai.com/crawl?url=https%3A%2F%2Fexample.org&maxPages=2"
```

## 2. Pay and call (TypeScript)

```bash
npm install @x402/fetch @x402/evm viem
```

```ts
import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";

// A wallet used only by your agent, holding a little USDC on Base.
const account = privateKeyToAccount(process.env.WALLET_PRIVATE_KEY as `0x${string}`);
const pay = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }],
});

const res = await pay("https://aayatai.com/crawl?url=https%3A%2F%2Fexample.org&maxPages=2");
console.log(await res.json());
```

## 3. Or as an MCP tool

```ts
// MCP server: https://aayatai.com/mcp (Streamable HTTP). With the x402 MCP client (see /start):
const result = await client.callTool("crawl", {"url":"https://example.org","maxPages":2});
```

## Inputs

- `url` **(required)** (string): Start page, e.g. https://example.com/docs/.
- `maxPages` (integer; default `5`): Most pages to return (1-10).
- `path` (string): Only crawl pages under this path (default: the start page's folder).

Bad inputs are rejected with HTTP 400 before any payment is asked for.

## Example answer

```json
{
  "start": "https://example.com/",
  "pages": [
    {
      "url": "https://example.com/",
      "title": "Example Domain",
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples.",
      "chars": 64,
      "truncated": false
    }
  ],
  "count": 1,
  "discoveredFrom": "links",
  "skipped": [
    {
      "url": "https://example.com/admin",
      "reason": "robots.txt"
    }
  ],
  "robots": "ok",
  "complete": true,
  "crawledAt": "2026-09-28T12:00:00.000Z"
}
```

## Related

- [GET /search/open](/services/search-open) ($0.002): Cheapest search: Wikipedia articles, Hacker News stories, DuckDuckGo instant answers and Stack Overflow questions in one call, merged, de-duplicated and ranked,
- [GET /search](/services/search) ($0.003): Live search, open-sources tier: fresh results from Wikipedia, Hacker News, DuckDuckGo instant answers and Stack Overflow, merged, de-duplicated and ranked (titl
- [GET /search/answer](/services/search-answer) ($0.03): Search, read and summarise in one call (open-sources tier): finds pages via Wikipedia, Hacker News, DuckDuckGo and Stack Overflow, reads the top ones (robots.tx
- [GET /metadata](/services/metadata) ($0.002): A web page's metadata in one cheap call: title, description, canonical URL, language, site name, preview image, OpenGraph and Twitter card tags, author and date
- [GET /feed](/services/feed) ($0.003): Latest posts from any site or feed as clean JSON: pass a feed URL or just the site's address and we find its RSS, Atom or JSON Feed (from the page's links or co
- [POST /page/changes](/services/page-changes) ($0.002): Watch any public page for changes: we fetch it now (robots.txt respected), fingerprint its text and return a token.
- [POST /page/changes/summary](/services/page-changes-summary) ($0.008): Page change detection with an AI summary: like /page/changes (send back the token from your last check), plus a two-sentence summary of what changed and whether
- [GET /product](/services/product) ($0.003): Product and price from any public product page, as clean JSON: name, brand, price, currency, availability, condition, SKU/GTIN, seller, rating and image, read f

New here? [Getting started in 60 seconds](/start). All services: [Aayat AI](/).