> ## Documentation Index > Fetch the complete documentation index at: https://deepline.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Context.dev: Contextdev Post Web Crawl > Crawl Website & Scrape Markdown. Includes SDK V2 guidance, input constraints, response fields, and Deepline credit cost. Crawl Website & Scrape Markdown. > [!NOTE] > > Tool ID: `contextdev_post_web_crawl` ## Run this action Use the TypeScript SDK for one call. Use `ctx.tools.execute(...)` inside a Play for durable calls, scheduled calls, or CSV processing. ```ts theme={"languages":{"custom":["/languages/deepline-prompt.json"]}} import { Deepline } from 'deepline'; const deepline = await Deepline.connect(); const result = await deepline.tools.execute( 'contextdev_post_web_crawl', { "url": "https://example.com/resource" }, ); console.log(result.toolResponse.raw); ``` ### CLI ```bash theme={"languages":{"custom":["/languages/deepline-prompt.json"]}} deepline tools execute contextdev_post_web_crawl --input '{ "url": "https://example.com/resource" }' --json ``` ## Example response The SDK returns this shape at `result.toolResponse.raw`. The example values illustrate the response. ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}} { "data": { "results": [ { "markdown": "example", "metadata": { "sourceUrl": "https://example.com", "finalUrl": "https://example.com", "title": "VP of Engineering", "url": "https://example.com", "crawlDepth": 123, "statusCode": 123, "success": true } } ], "metadata": { "numUrls": 123, "maxCrawlDepth": 123, "numSucceeded": 123, "numFailed": 123, "numSkipped": 123 } } } ``` Use `deepline describe contextdev_post_web_crawl --json` for the latest machine-readable contract. ## Input reference Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages. | Name | Type | Required | Default | Details | | - | - | - | - | - | | `payload.url` | `string` | Yes | — | The starting URL for the crawl (must include http\:// or https\:// protocol) Format: `uri`. | | `payload.maxPages` | `integer` | No | `100` | Maximum number of pages to crawl. Hard cap: 500. Minimum: 1. Maximum: 500. | | `payload.maxDepth` | `integer` | No | — | Maximum link depth from the starting URL (0 = only the starting page) Minimum: 0. | | `payload.urlRegex` | `string` | No | — | Regex pattern. Only URLs matching this pattern will be followed and scraped. | | `payload.includeLinks` | `boolean` | No | `true` | Preserve hyperlinks in the Markdown output | | `payload.includeImages` | `boolean` | No | `false` | Include image references in the Markdown output | | `payload.shortenBase64Images` | `boolean` | No | `true` | Truncate base64-encoded image data in the Markdown output | | `payload.useMainContentOnly` | `boolean` | No | `false` | Extract only the main content, stripping headers, footers, sidebars, and navigation | | `payload.followSubdomains` | `boolean` | No | `false` | When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent. | | `payload.pdf` | `object` | No | `{"shouldParse":true,"ocr":false}` | PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range. | | `payload.includeFrames` | `boolean` | No | `false` | When true, the contents of iframes are rendered to Markdown for each crawled page. | | `payload.includeSelectors` | `array` | No | — | CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each crawled page is converted to Markdown. When omitted, the entire document is kept. Examples: "article.main", "#content", "\[role=main]". Maximum items: 50. | | `payload.excludeSelectors` | `array` | No | — | CSS selectors to remove before each crawled page is converted to Markdown. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: "nav", "footer", ".ad-banner", "\[aria-hidden=true]". Maximum items: 50. | | `payload.maxAgeMs` | `integer` | No | `86400000` | Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh. Minimum: 0. Maximum: 2592000000. | | `payload.waitForMs` | `integer` | No | — | Optional browser wait time in milliseconds after initial page load for each crawled page. Min: 0. Max: 30000 (30 seconds). Minimum: 0. Maximum: 30000. | | `payload.settleAnimations` | `boolean` | No | `false` | When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages. | | `payload.stopAfterMs` | `integer` | No | `80000` | Soft time budget for the crawl in milliseconds. After each scrape, the crawler checks the elapsed time and, if exceeded, returns the pages collected so far instead of continuing. Min: 10000 (10s). Max: 110000 (110s). Default: 80000 (80s). Minimum: 10000. Maximum: 110000. | | `payload.country` | `"ad" \| "ae" \| "af" \| "ag" \| "ai" \| "al" \| "am" \| "ao" \| "ar" \| "at" \| "au" \| "aw" \| "az" \| "ba" \| "bb" \| "bd" \| "be" \| "bf" \| "bg" \| "bh" \| "bi" \| "bj" \| "bm" \| "bn" \| "bo" \| "bq" \| "br" \| "bs" \| "bw" \| "by" \| "bz" \| "ca" \| "cd" \| "cf" \| "cg" \| "ch" \| "ci" \| "cl" \| "cm" \| "cn" \| "co" \| "cr" \| "cv" \| "cw" \| "cy" \| "cz" \| "de" \| "dj" \| "dk" \| "dm" \| "do" \| "dz" \| "ec" \| "ee" \| "eg" \| "es" \| "et" \| "fi" \| "fj" \| "fr" \| "ga" \| "gb" \| "gd" \| "ge" \| "gf" \| "gg" \| "gh" \| "gm" \| "gn" \| "gp" \| "gq" \| "gr" \| "gt" \| "gu" \| "gw" \| "gy" \| "hk" \| "hn" \| "hr" \| "ht" \| "hu" \| "id" \| "ie" \| "il" \| "im" \| "in" \| "iq" \| "ir" \| "is" \| "it" \| "je" \| "jm" \| "jo" \| "jp" \| "ke" \| "kg" \| "kh" \| "kn" \| "kr" \| "kw" \| "ky" \| "kz" \| "la" \| "lb" \| "lc" \| "lk" \| "lr" \| "ls" \| "lt" \| "lu" \| "lv" \| "ly" \| "ma" \| "mc" \| "md" \| "me" \| "mf" \| "mg" \| "mk" \| "ml" \| "mm" \| "mn" \| "mo" \| "mq" \| "mr" \| "mt" \| "mu" \| "mv" \| "mw" \| "mx" \| "my" \| "mz" \| "na" \| "nc" \| "ne" \| "ng" \| "ni" \| "nl" \| "no" \| "np" \| "nz" \| "om" \| "pa" \| "pe" \| "pf" \| "pg" \| "ph" \| "pk" \| "pl" \| "pr" \| "ps" \| "pt" \| "py" \| "qa" \| "re" \| "ro" \| "rs" \| "ru" \| "rw" \| "sa" \| "sc" \| "sd" \| "se" \| "sg" \| "si" \| "sk" \| "sl" \| "sm" \| "sn" \| "so" \| "sr" \| "ss" \| "st" \| "sv" \| "sx" \| "sy" \| "sz" \| "tc" \| "td" \| "tg" \| "th" \| "tj" \| "tl" \| "tm" \| "tn" \| "tr" \| "tt" \| "tw" \| "tz" \| "ua" \| "ug" \| "us" \| "uy" \| "uz" \| "vc" \| "ve" \| "vg" \| "vi" \| "vn" \| "ye" \| "yt" \| "za" \| "zm" \| "zw"` | No | — | Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2). Allowed: `ad`, `ae`, `af`, `ag`, `ai`, `al`, `am`, `ao`, `ar`, `at`, `au`, `aw`, `az`, `ba`, `bb`, `bd`, `be`, `bf`, `bg`, `bh`, `bi`, `bj`, `bm`, `bn`, `bo`, `bq`, `br`, `bs`, `bw`, `by`, `bz`, `ca`, `cd`, `cf`, `cg`, `ch`, `ci`, `cl`, `cm`, `cn`, `co`, `cr`, `cv`, `cw`, `cy`, `cz`, `de`, `dj`, `dk`, `dm`, `do`, `dz`, `ec`, `ee`, `eg`, `es`, `et`, `fi`, `fj`, `fr`, `ga`, `gb`, `gd`, `ge`, `gf`, `gg`, `gh`, See the live schema for the complete constraint. | | `payload.timeoutMS` | `integer` | No | — | Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes). Minimum: 1000. Maximum: 300000. | | `payload.zdr` | `"enabled" \| "disabled"` | No | `"disabled"` | Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact [support@context.dev](mailto:support@context.dev)), otherwise the request fails with ZDR\_NOT\_ENABLED. Successful ZDR responses include X-Context-ZDR: true. Allowed: `enabled`, `disabled`. | | `payload.tags` | `array` | No | — | Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters. Maximum items: 20. |
Show raw input schema ### Input JSON Schema ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}} { "type": "object", "description": "Performs a crawl starting from a given URL, extracts page content as Markdown, and returns results for all crawled pages.", "properties": { "url": { "type": "string", "description": "The starting URL for the crawl (must include http:// or https:// protocol)", "format": "uri" }, "maxPages": { "type": "integer", "description": "Maximum number of pages to crawl. Hard cap: 500.", "default": 100, "minimum": 1, "maximum": 500 }, "maxDepth": { "type": "integer", "description": "Maximum link depth from the starting URL (0 = only the starting page)", "minimum": 0 }, "urlRegex": { "type": "string", "description": "Regex pattern. Only URLs matching this pattern will be followed and scraped." }, "includeLinks": { "type": "boolean", "description": "Preserve hyperlinks in the Markdown output", "default": true }, "includeImages": { "type": "boolean", "description": "Include image references in the Markdown output", "default": false }, "shortenBase64Images": { "type": "boolean", "description": "Truncate base64-encoded image data in the Markdown output", "default": true }, "useMainContentOnly": { "type": "boolean", "description": "Extract only the main content, stripping headers, footers, sidebars, and navigation", "default": false }, "followSubdomains": { "type": "boolean", "description": "When true, follow links on subdomains of the starting URL's domain (e.g. docs.example.com when starting from example.com). www and apex are always treated as equivalent.", "default": false }, "pdf": { "type": "object", "description": "PDF parsing controls. Use start/end to limit text extraction and embedded-image detection/OCR to an inclusive 1-based page range.", "default": { "shouldParse": true, "ocr": false }, "properties": { "shouldParse": { "type": "boolean", "description": "When true, PDF pages are fetched and parsed. When false, PDF pages are skipped entirely (not included in results and not counted as failures).", "default": true }, "ocr": { "type": "boolean", "description": "When true, OCR the selected PDF pages that have no usable text layer (scans), replacing each recovered page's text with the OCR result while pages with a real text layer keep it.", "default": false }, "start": { "type": "integer", "description": "First 1-based PDF page to parse. When omitted, parsing starts at the first page.", "minimum": 1 }, "end": { "type": "integer", "description": "Last 1-based PDF page to parse. When omitted, parsing ends at the last page. Must be greater than or equal to start when both are provided.", "minimum": 1 } }, "additionalProperties": false }, "includeFrames": { "type": "boolean", "description": "When true, the contents of iframes are rendered to Markdown for each crawled page.", "default": false }, "includeSelectors": { "type": "array", "description": "CSS selectors. When provided, only matching HTML subtrees (and their descendants) are kept before each crawled page is converted to Markdown. When omitted, the entire document is kept. Examples: \"article.main\", \"#content\", \"[role=main]\".", "maxItems": 50, "items": { "type": "string", "maxLength": 2048 } }, "excludeSelectors": { "type": "array", "description": "CSS selectors to remove before each crawled page is converted to Markdown. Applied after includeSelectors. Exclusion takes precedence: an element matching both is removed. Examples: \"nav\", \"footer\", \".ad-banner\", \"[aria-hidden=true]\".", "maxItems": 50, "items": { "type": "string", "maxLength": 2048 } }, "maxAgeMs": { "type": "integer", "description": "Return a cached result if a prior scrape for the same parameters exists and is younger than this many milliseconds. Defaults to 1 day (86400000 ms) when omitted. Max is 30 days (2592000000 ms). Set to 0 to always scrape fresh.", "default": 86400000, "minimum": 0, "maximum": 2592000000 }, "waitForMs": { "type": "integer", "description": "Optional browser wait time in milliseconds after initial page load for each crawled page. Min: 0. Max: 30000 (30 seconds).", "minimum": 0, "maximum": 30000 }, "settleAnimations": { "type": "boolean", "description": "When true, waits briefly for CSS and transition animations to settle before extracting each crawled page. Defaults to false. This adds a bit of latency in exchange for more stable output on animated pages.", "default": false }, "stopAfterMs": { "type": "integer", "description": "Soft time budget for the crawl in milliseconds. After each scrape, the crawler checks the elapsed time and, if exceeded, returns the pages collected so far instead of continuing. Min: 10000 (10s). Max: 110000 (110s). Default: 80000 (80s).", "default": 80000, "minimum": 10000, "maximum": 110000 }, "country": { "type": "string", "description": "Fetch the target page through a residential proxy in this country (ISO 3166-1 alpha-2).", "enum": [ "ad", "ae", "af", "ag", "ai", "al", "am", "ao", "ar", "at", "au", "aw", "az", "ba", "bb", "bd", "be", "bf", "bg", "bh", "bi", "bj", "bm", "bn", "bo", "bq", "br", "bs", "bw", "by", "bz", "ca", "cd", "cf", "cg", "ch", "ci", "cl", "cm", "cn", "co", "cr", "cv", "cw", "cy", "cz", "de", "dj", "dk", "dm", "do", "dz", "ec", "ee", "eg", "es", "et", "fi", "fj", "fr", "ga", "gb", "gd", "ge", "gf", "gg", "gh", "gm", "gn", "gp", "gq", "gr", "gt", "gu", "gw", "gy", "hk", "hn", "hr", "ht", "hu", "id", "ie", "il", "im", "in", "iq", "ir", "is", "it", "je", "jm", "jo", "jp", "ke", "kg", "kh", "kn", "kr", "kw", "ky", "kz", "la", "lb", "lc", "lk", "lr", "ls", "lt", "lu", "lv", "ly", "ma", "mc", "md", "me", "mf", "mg", "mk", "ml", "mm", "mn", "mo", "mq", "mr", "mt", "mu", "mv", "mw", "mx", "my", "mz", "na", "nc", "ne", "ng", "ni", "nl", "no", "np", "nz", "om", "pa", "pe", "pf", "pg", "ph", "pk", "pl", "pr", "ps", "pt", "py", "qa", "re", "ro", "rs", "ru", "rw", "sa", "sc", "sd", "se", "sg", "si", "sk", "sl", "sm", "sn", "so", "sr", "ss", "st", "sv", "sx", "sy", "sz", "tc", "td", "tg", "th", "tj", "tl", "tm", "tn", "tr", "tt", "tw", "tz", "ua", "ug", "us", "uy", "uz", "vc", "ve", "vg", "vi", "vn", "ye", "yt", "za", "zm", "zw" ] }, "timeoutMS": { "type": "integer", "description": "Optional timeout in milliseconds for the request. If the request takes longer than this value, it will be aborted with a 408 status code. Maximum allowed value is 300000ms (5 minutes).", "minimum": 1000, "maximum": 300000 }, "zdr": { "type": "string", "description": "Set to enabled to bypass shared caches and omit request and response content from retained usage logs. Requires zero data retention to be enabled for your organization (contact support@context.dev), otherwise the request fails with ZDR_NOT_ENABLED. Successful ZDR responses include X-Context-ZDR: true.", "default": "disabled", "enum": [ "enabled", "disabled" ] }, "tags": { "type": "array", "description": "Optional tags for tracking usage. Up to 20 tags, each 1 to 50 characters.", "maxItems": 20, "items": { "type": "string", "minLength": 1, "maxLength": 50 } } }, "required": [ "url" ], "additionalProperties": false } ```
## Output reference | Name | Type | Required | Default | Details | | - | - | - | - | - | | `result.data` | `object` | Yes | — | — | | `result.data.results` | `array` | Yes | — | — | | `result.data.results[].markdown` | `string` | Yes | — | — | | `result.data.results[].metadata` | `object` | Yes | — | — | | `result.data.results[].metadata.sourceUrl` | `string` | Yes | — | — | | `result.data.results[].metadata.finalUrl` | `string` | Yes | — | — | | `result.data.results[].metadata.title` | `string` | Yes | — | — | | `result.data.results[].metadata.description` | `string` | No | — | — | | `result.data.results[].metadata.language` | `string` | No | — | — | | `result.data.results[].metadata.keywords` | `array` | No | — | — | | `result.data.results[].metadata.canonicalUrl` | `string` | No | — | — | | `result.data.results[].metadata.author` | `string` | No | — | — | | `result.data.results[].metadata.siteName` | `string` | No | — | — | | `result.data.results[].metadata.image` | `string` | No | — | — | | `result.data.results[].metadata.favicon` | `string` | No | — | — | | `result.data.results[].metadata.publishedTime` | `string` | No | — | — | | `result.data.results[].metadata.modifiedTime` | `string` | No | — | — | | `result.data.results[].metadata.robots` | `string` | No | — | — | | `result.data.results[].metadata.openGraph` | `record` | No | — | — | | `result.data.results[].metadata.twitter` | `record` | No | — | — | | `result.data.results[].metadata.alternates` | `array` | No | — | — | | `result.data.results[].metadata.alternates[].href` | `string` | Yes | — | — | | `result.data.results[].metadata.alternates[].hreflang` | `string` | No | — | — | | `result.data.results[].metadata.alternates[].type` | `string` | No | — | — | | `result.data.results[].metadata.alternates[].title` | `string` | No | — | — | | `result.data.results[].metadata.headings` | `array` | No | — | — | | `result.data.results[].metadata.headings[].level` | `integer` | Yes | — | Minimum: 1. Maximum: 6. | | `result.data.results[].metadata.headings[].text` | `string` | Yes | — | — | | `result.data.results[].metadata.jsonLd` | `array` | No | — | — | | `result.data.results[].metadata.additionalMeta` | `record` | No | — | — | | `result.data.results[].metadata.url` | `string` | Yes | — | — | | `result.data.results[].metadata.crawlDepth` | `integer` | Yes | — | — | | `result.data.results[].metadata.statusCode` | `integer` | Yes | — | — | | `result.data.results[].metadata.success` | `boolean` | Yes | — | — | | `result.data.metadata` | `object` | Yes | — | — | | `result.data.metadata.numUrls` | `integer` | Yes | — | — | | `result.data.metadata.maxCrawlDepth` | `integer` | Yes | — | — | | `result.data.metadata.numSucceeded` | `integer` | Yes | — | — | | `result.data.metadata.numFailed` | `integer` | Yes | — | — | | `result.data.metadata.numSkipped` | `integer` | Yes | — | — | | `result.data.key_metadata` | `object` | No | — | — | | `result.meta` | `object` | No | — | — |
Show raw output schema ### Output JSON Schema ```json theme={"languages":{"custom":["/languages/deepline-prompt.json"]}} { "type": "object", "properties": { "data": { "type": "object", "properties": { "results": { "type": "array", "items": { "type": "object", "properties": { "markdown": { "type": "string" }, "metadata": { "type": "object", "properties": { "sourceUrl": { "type": "string" }, "finalUrl": { "type": "string" }, "title": { "type": "string" }, "description": { "type": "string" }, "language": { "type": "string" }, "keywords": { "type": "array", "items": { "type": "string" } }, "canonicalUrl": { "type": "string" }, "author": { "type": "string" }, "siteName": { "type": "string" }, "image": { "type": "string" }, "favicon": { "type": "string" }, "publishedTime": { "type": "string" }, "modifiedTime": { "type": "string" }, "robots": { "type": "string" }, "openGraph": { "type": "object", "additionalProperties": { "type": "object", "additionalProperties": true } }, "twitter": { "type": "object", "additionalProperties": { "type": "object", "additionalProperties": true } }, "alternates": { "type": "array", "items": { "type": "object", "properties": { "href": { "type": "string" }, "hreflang": { "type": "string" }, "type": { "type": "string" }, "title": { "type": "string" } }, "required": [ "href" ], "additionalProperties": false } }, "headings": { "type": "array", "items": { "type": "object", "properties": { "level": { "type": "integer", "minimum": 1, "maximum": 6 }, "text": { "type": "string" } }, "required": [ "level", "text" ], "additionalProperties": false } }, "jsonLd": { "type": "array", "items": { "type": "object", "additionalProperties": true } }, "additionalMeta": { "type": "object", "additionalProperties": { "type": "object", "additionalProperties": true } }, "url": { "type": "string" }, "crawlDepth": { "type": "integer" }, "statusCode": { "type": "integer" }, "success": { "type": "boolean" } }, "required": [ "sourceUrl", "finalUrl", "url", "title", "crawlDepth", "statusCode", "success" ], "additionalProperties": false } }, "required": [ "markdown", "metadata" ], "additionalProperties": false } }, "metadata": { "type": "object", "properties": { "numUrls": { "type": "integer" }, "maxCrawlDepth": { "type": "integer" }, "numSucceeded": { "type": "integer" }, "numFailed": { "type": "integer" }, "numSkipped": { "type": "integer" } }, "required": [ "numUrls", "maxCrawlDepth", "numSucceeded", "numFailed", "numSkipped" ], "additionalProperties": false }, "key_metadata": { "type": "object", "properties": {}, "required": [], "additionalProperties": false } }, "required": [ "results", "metadata" ], "additionalProperties": false }, "meta": { "type": "object", "additionalProperties": true } }, "required": [ "data" ], "additionalProperties": false } ```
## Deepline cost * Pricing model: `provider_usage` (provider usage). * Estimated Deepline credits: `0.01` per pricing unit. ## Related documentation * [Context.dev provider guide](https://deepline.com/docs/providers/contextdev/guide.md) * [SDK V2 quickstart](https://deepline.com/docs/sdk-v2/quickstart.md) * [SDK reference](https://deepline.com/docs/sdk-v2/sdk-reference.md) * [Run tools across a CSV](https://deepline.com/docs/sdk-v2/batch-csv.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.