Screenshot

Scrapfly's screenshot feature allows capturing a screenshot of the scraped web page. These screenshots can be full page or focused on specific HTML elements via CSS selectors.

Screenshots feature requires Javascript Rendering enabled

Captured screenshots are stored on Scrapfly's servers and the URLs are available in the API response under result.screenshots key as well as monitoring logs Screenshot tab:

overview page of web interface for scraped screenshot information — screenshot details on the monitoring logs page

The screenshots feature is a great way to understand what is happening during a scrape. It can be used to debug a scraper or to monitor a website for visual changes.

Note that Scrapfly headless browsers are optimized for web scraping and do not render media files like images or videos, so they will not appear in the screenshots. If you need to render images in screenshot, you must enable the flag screenshot_flags=load_images, see flags section for more information.

The screenshot feature works beside the scraper and does not affect or alter the scraped result.

A maximum number of 10 different screenshots can be taken per single scrape.

Usage

To use the screenshots feature the screenshots parameter must be set with a screenshot name and the desired capture area:

fullpage is a reserved value for capturing the entire web page.
CSS selector value can be used to target specific HTML elements.

For example, to capture the full page and the reviews section of this mock product page http://web-scraping.dev/product/1 we'd use two screenshot parameters:

screenshots[all]=fullpage to capture the whole page under all name. The fullpage is a reserved value for capturing all page content.
screenshots[reviews]=.reviews to capture CSS selector targeting the reviews section under reviews name

curl -G \
--request "GET" \
--url "http://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "screenshots[all]=fullpage" \
--data-urlencode "screenshots[reviews]=#reviews" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=http://web-scraping.dev/product/1"

http://api.scrapfly.io/scrape?render_js=true&screenshots%5Ball%5D=fullpage&screenshots%5Breviews%5D=%2523reviews&key=&url=http%253A%252F%252Fweb-scraping.dev%252Fproduct%252F1

Example Of Response

{
    ...
    "result": {
        ...
        "screenshots": {
            "all": {
                "extension": "png",
                "format": "fullpage",
                "css_selector": null,
                "size": 2867974,
                "url": "http://api.scrapfly.io/4d1b8e8f-3803-4aa6-88fa-39d5aa81b6b3/scrape/screenshot/db475202-a7c0-4d7b-9179-98089901fce3/main"
            },
            "reviews": {
                "extension": "png",
                "format": "element",
                "css_selector": ".similarities-widget",
                "size": 109817,
                "url": "http://api.scrapfly.io/4d1b8e8f-3803-4aa6-88fa-39d5aa81b6b3/scrape/screenshot/db475202-a7c0-4d7b-9179-98089901fce3/similarities"
            }
        }
        ...
    }
    ...
}

Download Programmatically

The URL to download the screenshot is located in the response result.screenshots.${name}.url key. However, this URL requires authentication and to view it the key parameter must be added with your Scrapfly key. For example using curl:

curl "http://api.scrapfly.io/4d1b8e8f-3803-4aa6-88fa-39d5aa81b6b3/scrape/screenshot/db475202-a7c0-4d7b-9179-98089901fce3/main?key=" > screenshot.jpg

Related Errors

All related errors are listed below. You can see the full description and examples of error responses in Errors documentation page.

422 - ERR::SCRAPE::DOM_SELECTOR_INVISIBLE
422 - ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT

Options / Flags

You can enable additional options for the screenshot feature by adding the screenshot_flags parameter to the scrape request. You can set multiple flags by separating them with a comma. Here is the list of supported flags:

load_images Load images. +3 API Credit is billed per 100kb of media downloaded
dark_mode Enable dark mode display
block_banners Block cookies banners and overlay that cover the screen
high_quality No compression on the output image
print_media_format Render the page in the print mode

Example

curl -G \
--request "GET" \
--url "http://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "screenshots[all]=fullpage" \
--data-urlencode "screenshots[reviews]=#reviews" \
--data-urlencode "screenshot_flags=load_images,block_banners,high_quality" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=http://web-scraping.dev/product/1"

http://api.scrapfly.io/scrape?render_js=true&screenshots%5Ball%5D=fullpage&screenshots%5Breviews%5D=%2523reviews&screenshot_flags=load_images%252Cblock_banners%252Chigh_quality&key=&url=http%253A%252F%252Fweb-scraping.dev%252Fproduct%252F1

Limitations

The screenshots feature requires javascript rendering to be enabled. The screenshot capture can fail or be ignored in these situations:

Non-GET Requests - only GET request are eligible for screenshots
Javascript rendering difficulties - for unusually javascript-heavy pages screenshots can fail as javascript execution can interfere with the screenshot capture mechanism.
Cache use - if the cache feature is used and cache is being HIT the screenshots will be ignored.

Frequently Asked Questions

Question: How long are screenshot download links valid for?
Answer: Screenshot availability duration is determined by your log retention policy (starting at 1 week).

Question: Why do some screenshots look different compared to a real browser?
Answer: Scrapfly web browsers are optimized for web scraping, which can make visuals appear slightly different compared to real web browsers.

Question: Can I limit the API Credit consumption when flag load_images is set with cost_budget?
Answer: Bandwidth usage is not heuristic, we can't predict the consumption of a complete page - the budget limit will not have effect.

Integration

Use Screenshot Feature with Python SDK