browser-rendering

1 article
sort: new top best
clear filter
0 2/10

Cloudflare announced a new /crawl endpoint for Browser Rendering that allows users to crawl entire websites with a single API call, automatically discovering and rendering pages in multiple formats (HTML, Markdown, JSON) while respecting robots.txt and AI Crawl Control directives.

Cloudflare Browser Rendering Workers AI robots.txt AI Crawl Control
developers.cloudflare.com · jeffpalmer · 3 days ago · details · hn