scrapfly

package module
v0.0.0-...-c6547da Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 24, 2025 License: BSD-2-Clause Imports: 21 Imported by: 0

README

Scrapfly Go SDK

Go Reference

Go SDK for Scrapfly.io web scraping API.

This SDK allows you to easily:

  • Scrape the web without being blocked.
  • Use headless browsers to access Javascript-powered page data.
  • Take screenshots of websites.
  • Extract structured data using AI.

For web scraping guides see our blog and #scrapeguide tag for how to scrape specific targets.

Installation

go get github.com/scrapfly/go-scrapfly

Quick Intro

  1. Register a Scrapfly account for free
  2. Get your API Key on scrapfly.io/dashboard
  3. Start scraping: 🚀
package main

import (
	"fmt"
	"log"

	"github.com/scrapfly/go-scrapfly"
)

func main() {
	key := "YOUR_SCRAPFLY_KEY"

	client, err := scrapfly.New(key)
	if err != nil {
		log.Fatalf("failed to create client: %v", err)
	}

	// Create a scrape configuration
	scrapeConfig := &scrapfly.ScrapeConfig{
		URL:        "https://web-scraping.dev/product/1",
		RenderJS:   true,
		Country:    "us",
		ASP:        true,
		ProxyPool:  scrapfly.PublicResidentialPool,
	}

	// Perform the scrape
	apiResponse, err := client.Scrape(scrapeConfig)
	if err != nil {
		log.Fatalf("scrape failed: %v", err)
	}

	// HTML content is in apiResponse.Result.Content
	// fmt.Println(apiResponse.Result.Content)

	// Use the built-in HTML parser (go-query)
	selector, err := apiResponse.Selector()
	if err != nil {
		log.Fatalf("failed to get selector: %v", err)
	}
	
	fmt.Println("Product Title:", selector.Find("h3").First().Text())
}

Full Documentation

Documentation

Overview

Package scrapfly provides a Go SDK for the Scrapfly.io web scraping API.

Scrapfly is a comprehensive web scraping API that handles proxies, browser rendering, anti-bot protection, and more. This SDK provides a simple and idiomatic Go interface to interact with the Scrapfly API for various web scraping tasks.

Features

  • Web scraping with automatic proxy rotation
  • JavaScript rendering with headless browsers
  • Anti-bot protection (ASP) bypass
  • Screenshot capture with multiple formats
  • AI-powered structured data extraction
  • Session management for persistent browsing
  • Concurrent scraping with rate limiting
  • Caching and webhook support

Installation 

go get github.com/scrapfly/go-scrapfly

Quick Start

Create a client and perform a simple scrape: 

package main

import (
    "fmt"
    "log"
    "github.com/scrapfly/go-scrapfly"
)

func main() {
    // Create a new client
    client, err := scrapfly.New("YOUR_API_KEY")
    if err != nil {
        log.Fatal(err)
    }

    // Configure the scrape request
    config := &scrapfly.ScrapeConfig{
        URL:      "https://example.com",
        RenderJS: true,
        Country:  "us",
    }

    // Perform the scrape
    result, err := client.Scrape(config)
    if err != nil {
        log.Fatal(err)
    }

    // Access the content
    fmt.Println(result.Result.Content)

    // Or use the built-in HTML parser
    doc, err := result.Selector()
    if err != nil {
        log.Fatal(err)
    }
    title := doc.Find("title").First().Text()
    fmt.Println("Title:", title)
}

Advanced Features

JavaScript Rendering: 

config := &scrapfly.ScrapeConfig{
    URL:             "https://example.com",
    RenderJS:        true,
    WaitForSelector: ".content",
    RenderingWait:   2000,
    AutoScroll:      true,
}

Taking Screenshots: 

config := &scrapfly.ScreenshotConfig{
    URL:        "https://example.com",
    Format:     scrapfly.FormatPNG,
    Capture:    "fullpage",
    Resolution: "1920x1080",
}
result, err := client.Screenshot(config)
if err != nil {
    log.Fatal(err)
}
filePath, err := client.SaveScreenshot(result, "screenshot")

Concurrent Scraping: 

configs := []*scrapfly.ScrapeConfig{
    {URL: "https://example.com/page1"},
    {URL: "https://example.com/page2"},
    {URL: "https://example.com/page3"},
}
resultsChan := client.ConcurrentScrape(configs, 3)
for result := range resultsChan {
    if result.error != nil {
        log.Printf("Error: %v", result.error)
        continue
    }
    // Process result
}

AI Data Extraction: 

config := &scrapfly.ExtractionConfig{
    Body:             []byte("<html>...</html>"),
    ContentType:      "text/html",
    ExtractionPrompt: "Extract product name, price, and description",
}
result, err := client.Extract(config)

Error Handling

The SDK uses sentinel errors that can be checked with errors.Is(): 

result, err := client.Scrape(config)
if err != nil {
    if errors.Is(err, scrapfly.ErrUpstreamClient) {
        // Target website returned 4xx error
    } else if errors.Is(err, scrapfly.ErrProxyFailed) {
        // Proxy connection failed
    } else if apiErr, ok := err.(*scrapfly.APIError); ok {
        // Get detailed API error information
        fmt.Printf("Status: %d, Message: %s\n", apiErr.HTTPStatusCode, apiErr.Message)
    }
}

Debugging

Enable debug logging to see detailed request information: 

scrapfly.DefaultLogger.SetLevel(scrapfly.LevelDebug)

Enable debug mode in the API to access debug information in the dashboard: 

config := &scrapfly.ScrapeConfig{
    URL:   "https://example.com",
    Debug: true,
}

Documentation

For more information, visit:

Example (BasicGet)

basicGet demonstrates basic scraping with cache and ASP 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL: "https://httpbin.dev/html",
	// Anti Scraping Protection bypass - enable this when scraping protected targets
	ASP: true,
	// server side cache - great for repeated requests
	Cache:    true,
	CacheTTL: 3600, // in seconds
	// CacheClear: true,  // you can always clear the cache explicitly!
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}

// the scrape_result.Result contains all result details
fmt.Println("web log url:") // you can check web UI for request details:
fmt.Println(scrapeResult.Result.LogURL)

fmt.Println("\npage content:")
fmt.Println(scrapeResult.Result.Content)

fmt.Println("\nresponse headers:")
headersJSON, _ := json.MarshalIndent(scrapeResult.Result.ResponseHeaders, "", "  ")
fmt.Println(string(headersJSON))

fmt.Println("\nresponse cookies:")
cookiesJSON, _ := json.MarshalIndent(scrapeResult.Result.Cookies, "", "  ")
fmt.Println(string(cookiesJSON))

Output:
scrapfly: 2025/10/26 05:39:16 [DEBUG] scraping url https://httpbin.dev/html
scrapfly: 2025/10/26 05:39:19 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/01K8FGECZ0FDFSQ3W11V0M1MMG
web log url:
https://scrapfly.io/dashboard/monitoring/log/01K8FGECZ0FDFSQ3W11V0M1MMG
page content:
<!DOCTYPE html>
<html>
<head>
</head>
<body>
    <h1>Herman Melville - Moby-Dick</h1>
    <div>
        <p>
            Availing himself of the mild, summer-cool weather that now reigned in these latitudes, and in preparation
            for the peculiarly active pursuits shortly to be anticipated, Perth, the begrimed, blistered old blacksmith,
            had not removed his portable forge to the hold again, after concluding his contributory work for Ahab's leg,
            but still retained it on deck, fast lashed to ringbolts by the foremast; being now almost incessantly
            invoked by the headsmen, and harpooneers, and bowsmen to do some little job for them; altering, or
            repairing, or new shaping their various weapons and boat furniture. Often he would be surrounded by an eager
            circle, all waiting to be served; holding boat-spades, pike-heads, harpoons, and lances, and jealously
            watching his every sooty movement, as he toiled. Nevertheless, this old man's was a patient hammer wielded
            by a patient arm. No murmur, no impatience, no petulance did come from him. Silent, slow, and solemn; bowing
            over still further his chronically broken back, he toiled away, as if toil were life itself, and the heavy
            beating of his hammer the heavy beating of his heart. And so it was.—Most miserable! A peculiar walk in this
            old man, a certain slight but painful appearing yawing in his gait, had at an early period of the voyage
            excited the curiosity of the mariners. And to the importunity of their persisted questionings he had finally
            given in; and so it came to pass that every one now knew the shameful story of his wretched fate. Belated,
            and not innocently, one bitter winter's midnight, on the road running between two country towns, the
            blacksmith half-stupidly felt the deadly numbness stealing over him, and sought refuge in a leaning,
            dilapidated barn. The issue was, the loss of the extremities of both feet. Out of this revelation, part by
            part, at last came out the four acts of the gladness, and the one long, and as yet uncatastrophied fifth act
            of the grief of his life's drama. He was an old man, who, at the age of nearly sixty, had postponedly
            encountered that thing in sorrow's technicals called ruin. He had been an artisan of famed excellence, and
            with plenty to do; owned a house and garden; embraced a youthful, daughter-like, loving wife, and three
            blithe, ruddy children; every Sunday went to a cheerful-looking church, planted in a grove. But one night,
            under cover of darkness, and further concealed in a most cunning disguisement, a desperate burglar slid into
            his happy home, and robbed them all of everything. And darker yet to tell, the blacksmith himself did
            ignorantly conduct this burglar into his family's heart. It was the Bottle Conjuror! Upon the opening of
            that fatal cork, forth flew the fiend, and shrivelled up his home. Now, for prudent, most wise, and economic
            reasons, the blacksmith's shop was in the basement of his dwelling, but with a separate entrance to it; so
            that always had the young and loving healthy wife listened with no unhappy nervousness, but with vigorous
            pleasure, to the stout ringing of her young-armed old husband's hammer; whose reverberations, muffled by
            passing through the floors and walls, came up to her, not unsweetly, in her nursery; and so, to stout
            Labor's iron lullaby, the blacksmith's infants were rocked to slumber. Oh, woe on woe! Oh, Death, why canst
            thou not sometimes be timely? Hadst thou taken this old blacksmith to thyself ere his full ruin came upon
            him, then had the young widow had a delicious grief, and her orphans a truly venerable, legendary sire to
            dream of in their after years; and all of them a care-killing competency.
        </p>
    </div>
    <hr>
    <h2>Heading Level 2</h2>
    <h3>Heading Level 3</h3>
    <p>This is a paragraph with <strong>bold</strong> and <em>italic</em> text.</p>
    <blockquote>
        <p>This is a blockquote for markdown testing.</p>
    </blockquote>
    <pre><code>// This is a code block
function test() {
  return true;
}
</code></pre>
    <ul>
        <li>Unordered list item 1
            <ul>
                <li>Nested unordered item A</li>
                <li>Nested unordered item B</li>
            </ul>
        </li>
        <li>Unordered list item 2</li>
    </ul>
    <ol>
        <li>Ordered list item 1
            <ol>
                <li>Nested ordered item A</li>
                <li>Nested ordered item B</li>
            </ol>
        </li>
        <li>Ordered list item 2</li>
    </ol>
    <p>Link: <a href="https://httpbin.dev">httpbin.dev</a></p>
    <p>Image: <img src="https://httpbin.dev/image/png" alt="Test Image" width="100"></p>
    <h3>Vertical Table</h3>
    <table>
        <tr>
            <th>Header 1</th>
            <th>Header 2</th>
        </tr>
        <tr>
            <td>Cell 1</td>
            <td>Cell 2</td>
        </tr>xwx
        <tr>
            <td>Cell 3</td>
            <td>Cell 4</td>
        </tr>
    </table>
    <h3>Horizontal Table</h3>
    <table border="1">
        <tr>
            <th>Name</th>
            <th>Age</th>
            <th>Country</th>
        </tr>
        <tr>
            <td>Alice</td>
            <td>30</td>
            <td>USA</td>
        </tr>
        <tr>
            <td>Bob</td>
            <td>25</td>
            <td>UK</td>
        </tr>
        <tr>
            <td>Charlie</td>
            <td>35</td>
            <td>Canada</td>
        </tr>
    </table>
    <h3>Vertical Table</h3>
    <table border="1">
        <tr>
            <th>Field</th>
            <th>Value</th>
        </tr>
        <tr>
            <td>Name</td>
            <td>Alice</td>
        </tr>
        <tr>
            <td>Age</td>
            <td>30</td>
        </tr>
        <tr>
            <td>Country</td>
            <td>USA</td>
        </tr>
    </table>
</body>

</html>

response headers:
{
  "access-control-allow-credentials": "true",
  "access-control-allow-origin": "*",
  "alt-svc": "h3=\":443\"; ma=2592000",
  "content-security-policy": "frame-ancestors 'self' *.httpbin.dev; font-src 'self' *.httpbin.dev; default-src 'self' *.httpbin.dev; img-src 'self' *.httpbin.dev https://cdn.scrapfly.io; media-src 'self' *.httpbin.dev; object-src 'self' *.httpbin.dev https://web-scraping.dev; script-src 'self' 'unsafe-inline' 'unsafe-eval' *.httpbin.dev; style-src 'self' 'unsafe-inline' *.httpbin.dev https://unpkg.com; frame-src 'self' *.httpbin.dev https://web-scraping.dev; worker-src 'self' *.httpbin.dev; connect-src 'self' *.httpbin.dev",
  "content-type": "text/html; charset=utf-8",
  "date": "Sun, 26 Oct 2025 05:39:19 GMT",
  "permissions-policy": "fullscreen=(self), autoplay=*, geolocation=(), camera=()",
  "referrer-policy": "strict-origin-when-cross-origin",
  "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
  "x-content-type-options": "nosniff",
  "x-xss-protection": "1; mode=block"
}

response cookies:
[]
Example (DownloadFile) 
apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

// Build a JavaScript scenario using the scenario builder
scenario, err := js_scenario.New().
	Click("button[type='submit']").
	Wait(2000).
	Build()
if err != nil {
	log.Fatalf("failed to build scenario: %v", err)
}

scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL: "https://web-scraping.dev/file-download",
	// enable browsers:
	RenderJS: true,
	// this enables more options
	// you can wait for some element to appear:
	WaitForSelector: "#download-btn",
	// you can wait explicitly for N seconds
	RenderingWait: 3000, // 3 seconds
	// you can control the browser through scenarios:
	// https://scrapfly.io/docs/scrape-api/javascript-scenario
	JSScenario: scenario,
	// or even run any custom JS code!
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}

fmt.Println("attachments:")
attachmentsJSON, _ := json.MarshalIndent(scrapeResult.Result.BrowserData.Attachments, "", "  ")
fmt.Println(string(attachmentsJSON))

// use the shortcut to save attachments to file:
paths, err := scrapeResult.SaveAttachments("./tests_output")
if err != nil {
	log.Fatalf("failed to save attachments: %v", err)
}
for _, path := range paths {
	fmt.Printf("Attachment saved to: %s\n", path)
}

Output:
scrapfly: 2025/11/06 18:30:41 [DEBUG] scraping url https://web-scraping.dev/file-download
scrapfly: 2025/11/06 18:30:55 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
attachments:
[
  {
    "content": "https://api.scrapfly.io/scrape/attachment/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/7024607f-f336-4a79-9841-5fc2bb06c41e",
    "content_type": "application/pdf",
    "filename": "download-sample.pdf",
    "id": "7024607f-f336-4a79-9841-5fc2bb06c41e",
    "size": 10360,
    "state": "completed",
    "suggested_filename": "download-sample.pdf",
    "url": "https://web-scraping.dev/api/download-file"
  }
]
Attachment saved to: tests_output/download-sample.pdf
Example (ExtractionAutoExtract)

extractionAutoExtract demonstrates automatic extraction using predefined models 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

// First, get HTML either from Web Scraping API or your own storage
scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL: "https://web-scraping.dev/product/1",
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}
html := scrapeResult.Result.Content

// Auto Extract - extract content using predefined models
productResult, err := client.Extract(&scrapfly.ExtractionConfig{
	// identify content type like text/html or text/markdown etc.
	ContentType: "text/html",
	Body:        []byte(html),
	// define model type: product, article etc.
	// see https://scrapfly.io/docs/extraction-api/automatic-ai#models
	ExtractionModel: "product",
})
if err != nil {
	log.Fatalf("extraction failed: %v", err)
}

fmt.Println("product auto extract:")
productResultJSON, _ := json.MarshalIndent(productResult, "", "  ")
fmt.Println(string(productResultJSON))

Output:
scrapfly: 2025/11/17 05:22:31 [DEBUG] scraping url https://web-scraping.dev/product/1
scrapfly: 2025/11/17 05:22:35 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/XXXXXX
product auto extract:
{
  "data": {
    "aggregate_rating": {
      "best_rating": 5,
      "rating_value": 4.7,
      "review_count": 10
    },
    "bidding": null,
    "brand": "ChocoDelight",
    "breadcrumbs": [
      {
        "link": "/",
        "name": "Home"
      },
      {
        "link": "/products",
        "name": "Products"
      },
      {
        "link": null,
        "name": "Box of Chocolate Candy"
      }
    ],
    "canonical_url": null,
    "color": null,
    "delivery": "1 Day shipping",
    "description": "Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an assortment of rich, flavorful chocolates with a smooth, creamy filling. Choose from a variety of flavors including zesty orange and sweet cherry. Whether you're looking for the perfect gift or just want to treat yourself, our Box of Chocolate Candy is sure to satisfy.",
    "description_markdown": "Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an assortment of rich, flavorful chocolates with a smooth, creamy filling. Choose from a variety of flavors including zesty orange and sweet cherry. Whether you're looking for the perfect gift or just want to treat yourself, our Box of Chocolate Candy is sure to satisfy.",
    "identifiers": {
      "ean13": null,
      "gtin14": null,
      "gtin8": null,
      "isbn10": null,
      "isbn13": null,
      "ismn": null,
      "issn": null,
      "mpn": null,
      "sku": "1",
      "upc": null
    },
    "images": [
      {
        "url": "https://www.web-scraping.dev/assets/products/orange-chocolate-box-small-1.webp"
      },
      {
        "url": "https://www.web-scraping.dev/assets/products/orange-chocolate-box-small-2.webp"
      },
      {
        "url": "https://www.web-scraping.dev/assets/products/orange-chocolate-box-small-3.webp"
      },
      {
        "url": "https://www.web-scraping.dev/assets/products/orange-chocolate-box-small-4.webp"
      }
    ],
    "main_category": "Products",
    "main_image": "https://www.web-scraping.dev/assets/products/orange-chocolate-box-small-1.webp",
    "name": "Box of Chocolate Candy",
    "offers": [
      {
        "availability": "available",
        "currency": "USD",
        "price": 9.99,
        "regular_price": 12.99
      }
    ],
    "related_products": [
      {
        "availability": "available",
        "description": null,
        "images": [
          {
            "url": "https://www.web-scraping.dev/assets/products/blue-potion.webp"
          }
        ],
        "link": "https://web-scraping.dev/product/17",
        "name": "Blue Energy Potion",
        "price": {
          "amount": 4.99,
          "currency": "USD",
          "raw": "4.99"
        }
      },
      {
        "availability": "available",
        "description": null,
        "images": [
          {
            "url": "https://www.web-scraping.dev/assets/products/teal-potion.webp"
          }
        ],
        "link": "https://web-scraping.dev/product/27",
        "name": "Teal Energy Potion",
        "price": {
          "amount": 4.99,
          "currency": "USD",
          "raw": "4.99"
        }
      },
      {
        "availability": "available",
        "description": null,
        "images": [
          {
            "url": "https://www.web-scraping.dev/assets/products/orange-chocolate-box-medium-1.webp"
          }
        ],
        "link": "https://web-scraping.dev/product/25",
        "name": "Box of Chocolate Candy",
        "price": {
          "amount": 24.99,
          "currency": "USD",
          "raw": "24.99"
        }
      },
      {
        "availability": "available",
        "description": null,
        "images": [
          {
            "url": "https://www.web-scraping.dev/assets/products/hiking-boots-1.webp"
          }
        ],
        "link": "https://web-scraping.dev/product/7",
        "name": "Hiking Boots for Outdoor Adventures",
        "price": {
          "amount": 89.99,
          "currency": "USD",
          "raw": "89.99"
        }
      }
    ],
    "secondary_category": null,
    "size": null,
    "sizes": null,
    "specifications": [
      {
        "name": "material",
        "value": "Premium quality chocolate"
      },
      {
        "name": "flavors",
        "value": "Available in Orange and Cherry flavors"
      },
      {
        "name": "sizes",
        "value": "Available in small, medium, and large boxes"
      },
      {
        "name": "brand",
        "value": "ChocoDelight"
      },
      {
        "name": "care instructions",
        "value": "Store in a cool, dry place"
      },
      {
        "name": "purpose",
        "value": "Ideal for gifting or self-indulgence"
      }
    ],
    "style": null,
    "url": "https://web-scraping.dev/product/1",
    "variants": [
      {
        "color": "orange",
        "offers": [],
        "sku": null,
        "url": "https://web-scraping.dev/product/1?variant=orange-small"
      },
      {
        "color": "orange",
        "offers": [],
        "sku": null,
        "url": "https://web-scraping.dev/product/1?variant=orange-medium"
      },
      {
        "color": "orange",
        "offers": [],
        "sku": null,
        "url": "https://web-scraping.dev/product/1?variant=orange-large"
      },
      {
        "color": "cherry",
        "offers": [],
        "sku": null,
        "url": "https://web-scraping.dev/product/1?variant=cherry-small"
      },
      {
        "color": "cherry",
        "offers": [],
        "sku": null,
        "url": "https://web-scraping.dev/product/1?variant=cherry-medium"
      },
      {
        "color": "cherry",
        "offers": [],
        "sku": null,
        "url": "https://web-scraping.dev/product/1?variant=cherry-large"
      }
    ]
  },
  "content_type": "application/json",
  "data_quality": {
    "errors": [
      "breadcrumbs[2].link: Input should be a valid string",
      "breadcrumbs[2].link: Input should be a valid string"
    ],
    "fulfilled": false,
    "fulfillment_percent": 69.57
  }
}
Example (ExtractionLLM)

extractionLLM demonstrates using the extraction API with LLM prompts 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

// First, get HTML either from Web Scraping API or your own storage
scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL: "https://web-scraping.dev/product/1",
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}
html := scrapeResult.Result.Content

// LLM Parsing - extract content using LLM queries
llmResult, err := client.Extract(&scrapfly.ExtractionConfig{
	// identify content type like text/html or text/markdown etc.
	ContentType: "text/html",
	Body:        []byte(html),
	// use any prompt
	ExtractionPrompt: "get product price only",
})
if err != nil {
	log.Fatalf("extraction failed: %v", err)
}

fmt.Println("llm extraction:")
llmResultJSON, _ := json.MarshalIndent(llmResult, "", "  ")
fmt.Println(string(llmResultJSON))

// You can also request LLM to output specific formats like JSON or CSV
llmFormatResult, err := client.Extract(&scrapfly.ExtractionConfig{
	ContentType: "text/html",
	Body:        []byte(html),
	// directly request format
	ExtractionPrompt: "get product price and currency in JSON",
})
if err != nil {
	log.Fatalf("extraction failed: %v", err)
}

fmt.Println("\nllm extraction in JSON:")
llmFormatResultJSON, _ := json.MarshalIndent(llmFormatResult, "", "  ")
fmt.Println(string(llmFormatResultJSON))

Output:
scrapfly: 2025/11/07 01:01:58 [DEBUG] scraping url https://web-scraping.dev/product/1
scrapfly: 2025/11/07 01:02:00 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/XXXXXXXXXXXXX
llm extraction:
{
  "data": "The price of the \"Box of Chocolate Candy\" is $9.99 from $12.99. The lowPrice is \"9.99\" and the highPrice is \"19.99\" according to the JSON-LD metadata.",
  "content_type": "text/plain"
}

llm extraction in JSON:

	{
	  "data": {
	    "product": "Box of Chocolate Candy",
	    "similar_products": [
	      {
	        "currency": "USD",
	        "price": "4.99",
	        "product": "Dragon Energy Potion"
	      },
	      {
	        "currency": "USD",
	        "price": "89.99",
	        "product": "Hiking Boots for Outdoor Adventures"
	      },
	      {
	        "currency": "USD",
	        "price": "4.99",
	        "product": "Teal Energy Potion"
	      },
	      {
	        "currency": "USD",
	        "price": "14.99",
	        "product": "Cat-Ear Beanie"
	      }
	    ],
	    "variants": [
	      {
	        "currency": "USD",
	        "price": "9.99",
	        "variant": "orange, small"
	      },
	      {
	        "currency": "USD",
	        "price": "9.99",
	        "variant": "orange, medium"
	      },
	      {
	        "currency": "USD",
	        "price": "9.99",
	        "variant": "orange, large"
	      },
	      {
	        "currency": "USD",
	        "price": "9.99",
	        "variant": "cherry, small"
	      },
	      {
	        "currency": "USD",
	        "price": "9.99",
	        "variant": "cherry, medium"
	      },
	      {
	        "currency": "USD",
	        "price": "9.99",
	        "variant": "cherry, large"
	      }
	    ]
	  },
	  "content_type": "application/json"
	}
Example (ExtractionTemplates)

extractionTemplates demonstrates using extraction templates 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

// First, get HTML either from Web Scraping API or your own storage
scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL:             "https://web-scraping.dev/reviews",
	RenderJS:        true,
	WaitForSelector: ".review",
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}
html := scrapeResult.Result.Content

// Define your template, see https://scrapfly.io/docs/extraction-api/rules-and-template
template := map[string]interface{}{
	"source": "html",
	"selectors": []map[string]interface{}{
		{
			"name":     "date_posted",
			"type":     "css",
			"query":    "[data-testid='review-date']::text",
			"multiple": true,
			"formatters": []map[string]interface{}{
				{
					"name": "datetime",
					"args": map[string]interface{}{
						"format": "%Y, %b %d — %A",
					},
				},
			},
		},
	},
}

templateResult, err := client.Extract(&scrapfly.ExtractionConfig{
	Body:        []byte(html),
	ContentType: "text/html",
	// provide template:
	ExtractionEphemeralTemplate: template,
})
if err != nil {
	log.Fatalf("extraction failed: %v", err)
}

fmt.Println("template extract:")
templateResultJSON, _ := json.MarshalIndent(templateResult, "", "  ")
fmt.Println(string(templateResultJSON))

Output:
scrapfly: 2025/10/26 05:40:28 [DEBUG] scraping url https://web-scraping.dev/reviews
scrapfly: 2025/10/26 05:40:35 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/01K8FGGJX0AY63MHW2ACEY4ZFZ
template extract:
{
  "data": {
    "date_posted": [
      "2023, May 18 — Thursday",
      "2023, May 17 — Wednesday",
      "2023, May 16 — Tuesday",
      "2023, May 15 — Monday",
      "2023, May 15 — Monday",
      "2023, May 12 — Friday",
      "2023, May 10 — Wednesday",
      "2023, May 01 — Monday",
      "2023, May 01 — Monday",
      "2023, Apr 25 — Tuesday",
      "2023, Apr 25 — Tuesday",
      "2023, Apr 18 — Tuesday",
      "2023, Apr 12 — Wednesday",
      "2023, Apr 11 — Tuesday",
      "2023, Apr 10 — Monday",
      "2023, Apr 10 — Monday",
      "2023, Apr 09 — Sunday",
      "2023, Apr 07 — Friday",
      "2023, Apr 07 — Friday",
      "2023, Apr 05 — Wednesday"
    ]
  },
  "content_type": "application/json"
}
Example (GetAccount)

getAccount demonstrates fetching account information 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

account, err := client.Account()
if err != nil {
	log.Fatalf("failed to get account: %v", err)
}

fmt.Println("Account:")
accountJSON, _ := json.MarshalIndent(account, "", "  ")
fmt.Println(string(accountJSON))

Output:
Account:
{
	"account": {
	  "account_id": "XX-XXX-4c01-a9X97-XXXX",
	  "currency": "USD",
	  "timezone": "Europe/Paris",
	  "suspended": false,
	  "suspension_reason": ""
	},
	"project": {
	  "allow_extra_usage": true,
	  "allowed_networks": [],
	  "budget_limit": null,
	  "budget_spent": null,
	  "concurrency_limit": null,
	  "name": "default",
	  "quota_reached": false,
	  "scrape_request_count": 307,
	  "scrape_request_limit": null,
	  "tags": []
	},
	"subscription": {
	  "billing": {
		"current_extra_scrape_request_price": {
		  "currency": "USD",
		  "amount": 0
		},
		"extra_scrape_request_price_per_10k": {
		  "currency": "USD",
		  "amount": 0
		},
		"ongoing_payment": {
		  "currency": "USD",
		  "amount": 0
		},
		"plan_price": {
		  "currency": "USD",
		  "amount": 0
		}
	  },
	  "extra_scrape_allowed": false,
	  "max_concurrency": 5,
	  "period": {
		"start": "2025-08-01 00:06:34",
		"end": "2025-09-01 00:06:34"
	  },
	  "plan_name": "FREE",
	  "usage": {
		"spider": {
		  "current": 0,
		  "limit": 1
		},
		"schedule": {
		  "current": 0,
		  "limit": 1
		},
		"scrape": {
		  "concurrent_limit": 5,
		  "concurrent_remaining": 5,
		  "concurrent_usage": 0,
		  "current": 307,
		  "extra": 0,
		  "limit": 1000,
		  "remaining": 693
		}
	  }
	}
  }
Example (JsRender)

jsRender demonstrates JavaScript rendering with scenarios 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

// Build a JavaScript scenario using the scenario builder
scenario, err := js_scenario.New().
	Click("#load-more-reviews").
	Wait(2000).
	Build()
if err != nil {
	log.Fatalf("failed to build scenario: %v", err)
}

scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL: "https://web-scraping.dev/product/1",
	// enable browsers:
	RenderJS: true,
	// this enables more options
	// you can wait for some element to appear:
	WaitForSelector: ".review",
	// you can wait explicitly for N seconds
	RenderingWait: 3000, // 3 seconds
	// you can control the browser through scenarios:
	// https://scrapfly.io/docs/scrape-api/javascript-scenario
	JSScenario: scenario,
	// or even run any custom JS code!
	JS: `return document.querySelector(".review").innerText`,
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}

// the scrape_result.Result contains all result details:
fmt.Println("web log url:") // you can check web UI for request details:
fmt.Println(scrapeResult.Result.LogURL)

fmt.Println("\npage content (first 1000 chars):")
content := scrapeResult.Result.Content
if len(content) > 1000 {
	content = content[:1000] + "..."
}
fmt.Println(content)

fmt.Println("\nbrowser data capture:")
browserDataJSON, _ := json.MarshalIndent(scrapeResult.Result.BrowserData, "", "  ")
fmt.Println(string(browserDataJSON))

Output:
scrapfly: 2025/10/26 05:39:19 [DEBUG] scraping url https://web-scraping.dev/product/1
scrapfly: 2025/10/26 05:39:34 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/01K8FGEG1NB3RJ183CJ27YD25N
web log url:
https://scrapfly.io/dashboard/monitoring/log/01K8FGEG1NB3RJ183CJ27YD25N

page content (first 1000 chars):
<html lang="en"><head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <link rel="stylesheet" href="https://web-scraping.dev/assets/css/main.css">
    <link rel="stylesheet" href="https://web-scraping.dev/assets/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://web-scraping.dev/assets/css/bootstrap-icons.css">
    <link rel="stylesheet" href="https://web-scraping.dev/assets/css/highlight-nord.css">
    <link rel="icon" href="https://web-scraping.dev/assets/media/icon.png" type="image/png">
    <script src="https://web-scraping.dev/assets/js/cash.min.js"></script>
    <script src="https://web-scraping.dev/assets/js/main.js"></script>
    <script src="https://web-scraping.dev/assets/js/bootstrap.js"></script>
<title>web-scraping.dev product Box of Chocolate Candy</title>
<meta name="description" content="Mock product Box of Chocolate Candy page for web scraper testing">
<meta property="og:sit...
browser data capture:

	{
	  "javascript_evaluation_result": "2022-07-22     \n\nAbsolutely delicious! The orange flavor is my favorite.",
	  "js_scenario": {
	    "duration": 3.0700000000000003,
	    "executed": 2,
	    "response": null,
	    "steps": [
	      {
	        "action": "click",
	        "config": {
	          "ignore_if_not_visible": false,
	          "multiple": false,
	          "selector": "#load-more-reviews",
	          "timeout": 3500
	        },
	        "duration": 1.07,
	        "executed": true,
	        "result": null,
	        "success": true
	      },
	      {
	        "action": "wait",
	        "config": 2000,
	        "duration": 2,
	        "executed": true,
	        "result": null,
	        "success": true
	      }
	    ]
	  },
	  "local_storage_data": {},
	  "session_storage_data": {},
	  "websockets": [],
	  "xhr_call": [
	    {
	      "body": null,
	      "headers": {
	        "Accept": "* /*",
	        "Referer": "https://web-scraping.dev/product/1",
	        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
	        "sec-ch-ua": "\"Google Chrome\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\"",
	        "sec-ch-ua-mobile": "?0",
	        "sec-ch-ua-platform": "\"macOS\"",
	        "x-csrf-token": "secret-csrf-token-123"
	      },
	      "method": "GET",
	      "response": {
	        "body": "{\"order\":\"asc\",\"category\":null,\"total_results\":10,\"next_url\":null,\"results\":[{\"id\":\"chocolate-candy-box-6\",\"text\":\"Bought the large box, and it's lasted quite a while. Great for when you need a sweet treat.\",\"rating\":5,\"date\":\"2022-12-18\"},{\"id\":\"chocolate-candy-box-7\",\"text\":\"These chocolates are so tasty! Love the variety of flavors.\",\"rating\":5,\"date\":\"2023-01-24\"},{\"id\":\"chocolate-candy-box-8\",\"text\":\"The box is nicely packaged, making it a great gift option.\",\"rating\":5,\"date\":\"2023-02-15\"},{\"id\":\"chocolate-candy-box-9\",\"text\":\"The orange flavor wasn't my favorite, but the cherry ones are great.\",\"rating\":4,\"date\":\"2023-03-20\"},{\"id\":\"chocolate-candy-box-10\",\"text\":\"Delicious chocolates, and the box is pretty substantial. It'd make a nice gift.\",\"rating\":5,\"date\":\"2023-04-18\"}],\"page_number\":2,\"page_size\":5,\"page_total\":2}",
	        "content_encoding": null,
	        "content_type": "application/json",
	        "duration": 0,
	        "format": "text",
	        "headers": {
	          "alt-svc": "h3=\":443\"; ma=2592000",
	          "content-length": "839",
	          "content-type": "application/json",
	          "date": "Sun, 26 Oct 2025 05:39:31 GMT",
	          "permissions-policy": "fullscreen=(self), autoplay=*, geolocation=(), camera=()",
	          "referrer-policy": "strict-origin-when-cross-origin",
	          "server": "uvicorn",
	          "strict-transport-security": "max-age=31536000; includeSubDomains; preload",
	          "x-content-type-options": "nosniff",
	          "x-xss-protection": "1; mode=block"
	        },
	        "status": 200
	      },
	      "type": "fetch",
	      "url": "https://web-scraping.dev/api/reviews?product_id=1\u0026page=2"
	    }
	  ]
	}
Example (ScrapeExtraction)

scrapeExtraction demonstrates scraping with inline extraction using LLM prompts 

apiKey := getApiKey()
client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

scrapeResult, err := client.Scrape(&scrapfly.ScrapeConfig{
	URL: "https://web-scraping.dev/product/1",
	// enable browsers:
	RenderJS: true,
	// use LLM prompt for auto parsing
	ExtractionPrompt: "Extract the product specification in json format",
})
if err != nil {
	log.Fatalf("scrape failed: %v", err)
}

// access the extraction result
fmt.Println("extraction result:")
extractedDataJSON, _ := json.MarshalIndent(scrapeResult.Result.ExtractedData, "", "  ")
fmt.Println(string(extractedDataJSON))

Output:
scrapfly: 2025/11/07 01:01:46 [DEBUG] scraping url https://web-scraping.dev/product/1
scrapfly: 2025/11/07 01:01:58 [DEBUG] scrape log url: https://scrapfly.io/dashboard/monitoring/log/XXXXXXXXXXXXX
extraction result:
{
  "data": {
    "product": {
      "aggregateRating": {
        "ratingValue": "4.7",
        "reviewCount": "10"
      },
      "brand": "ChocoDelight",
      "description": "Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an assortment of rich, flavorful chocolates with a smooth, creamy filling. Choose from a variety of flavors including zesty orange and sweet cherry. Whether you're looking for the perfect gift or just want to treat yourself, our Box of Chocolate Candy is sure to satisfy.",
      "features": {
        "care instructions": "Store in a cool, dry place",
        "flavors": "Available in Orange and Cherry flavors",
        "material": "Premium quality chocolate",
        "purpose": "Ideal for gifting or self-indulgence",
        "sizes": "Available in small, medium, and large boxes"
      },
      "image": "https://web-scraping.dev/assets/products/orange-chocolate-box-medium-1.webp",
      "name": "Box of Chocolate Candy",
      "offers": {
        "availability": "InStock",
        "highPrice": "19.99",
        "lowPrice": "9.99",
        "priceCurrency": "USD"
      },
      "packs": [
        {
          "Delivery Type": "1 Day shipping",
          "Package Dimension": "100x230 cm",
          "Package Weight": "1,00 kg",
          "Variants": "6 available",
          "Version": "Pack 1"
        },
        {
          "Delivery Type": "1 Day shipping",
          "Package Dimension": "200x460 cm",
          "Package Weight": "2,11 kg",
          "Variants": "6 available",
          "Version": "Pack 2"
        },
        {
          "Delivery Type": "1 Day shipping",
          "Package Dimension": "300x690 cm",
          "Package Weight": "3,22 kg",
          "Variants": "6 available",
          "Version": "Pack 3"
        },
        {
          "Delivery Type": "1 Day shipping",
          "Package Dimension": "400x920 cm",
          "Package Weight": "4,33 kg",
          "Variants": "6 available",
          "Version": "Pack 4"
        },
        {
          "Delivery Type": "1 Day shipping",
          "Package Dimension": "500x1150 cm",
          "Package Weight": "5,44 kg",
          "Variants": "6 available",
          "Version": "Pack 5"
        }
      ],
      "reviews": [
        {
          "datePublished": "2022-07-22",
          "ratingValue": "5",
          "reviewBody": "Absolutely delicious! The orange flavor is my favorite."
        },
        {
          "datePublished": "2022-08-16",
          "ratingValue": "4",
          "reviewBody": "I bought these as a gift, and they were well received. Will definitely purchase again."
        },
        {
          "datePublished": "2022-09-10",
          "ratingValue": "5",
          "reviewBody": "Nice variety of flavors. The chocolate is rich and smooth."
        },
        {
          "datePublished": "2022-10-02",
          "ratingValue": "5",
          "reviewBody": "The cherry flavor is amazing. Will be buying more."
        },
        {
          "datePublished": "2022-11-05",
          "ratingValue": "4",
          "reviewBody": "A bit pricey, but the quality of the chocolate is worth it."
        }
      ],
      "variants": [
        "orange, small",
        "orange, medium",
        "orange, large",
        "cherry, small",
        "cherry, medium",
        "cherry, large"
      ]
    }
  },
  "content_type": "application/json"
}
Example (Screenshot)

screenshot demonstrates capturing screenshots 

apiKey := getApiKey()

client, err := scrapfly.New(apiKey)
if err != nil {
	log.Fatalf("failed to create client: %v", err)
}

screenshotResult, err := client.Screenshot(&scrapfly.ScreenshotConfig{
	URL: "https://web-scraping.dev/product/1",
	// by default 1920x1080 will be captured but resolution can take any value
	Resolution: "540x1200", // for example - tall smartphone viewport
	// to capture all visible parts use capture with full page
	Capture: "fullpage",

	// you can also capture specific elements with css or xpath
	// WaitForSelector: "#reviews", // wait for review to load
	// Capture: "#reviews",  // capture only reviews element

	// for pages that require scrolling to load elements (like endless paging) use
	AutoScroll: true,

	// Simulate vision deficiency (new)
	VisionDeficiencyType: scrapfly.VisionDeficiencyTypeBlurredVision,
})
if err != nil {
	log.Fatalf("screenshot failed: %v", err)
}

fmt.Println("captured screenshot:")
fmt.Printf("Format: %s, Size: %d bytes\n", screenshotResult.Metadata.ExtensionName, len(screenshotResult.Image))

// use the shortcut to save screenshots to file:
filePath, err := screenshotResult.Save("screenshot")
if err != nil {
	log.Fatalf("failed to save screenshot: %v", err)
}
fmt.Printf("saved screenshot to %s\n", filePath)

Output:
captured screenshot:
Format: jpeg, Size: 586399 bytes
saved screenshot to screenshot.jpeg

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrBadAPIKey indicates an invalid or empty API key was provided.
	ErrBadAPIKey = errors.New("invalid key, must be a non-empty string")

	// ErrScrapeConfig indicates invalid scrape configuration.
	ErrScrapeConfig = errors.New("invalid scrape config")

	// ErrScreenshotConfig indicates invalid screenshot configuration.
	ErrScreenshotConfig = errors.New("invalid screenshot config")

	// ErrExtractionConfig indicates invalid extraction configuration.
	ErrExtractionConfig = errors.New("invalid extraction config")

	// ErrContentType indicates an invalid content type for the requested operation.
	ErrContentType = errors.New("invalid content type for this operation")

	// ErrTooManyRequests indicates rate limiting is in effect.
	ErrTooManyRequests = errors.New("too many requests")

	// ErrQuotaLimitReached indicates the account quota has been exceeded.
	ErrQuotaLimitReached = errors.New("quota limit reached")

	// ErrScreenshotAPIFailed indicates a screenshot API error occurred.
	ErrScreenshotAPIFailed = errors.New("screenshot API error")

	// ErrExtractionAPIFailed indicates an extraction API error occurred.
	ErrExtractionAPIFailed = errors.New("extraction API error")

	// ErrUpstreamClient indicates a 4xx error from the target website.
	ErrUpstreamClient = errors.New("upstream http client error")

	// ErrUpstreamServer indicates a 5xx error from the target website.
	ErrUpstreamServer = errors.New("upstream http server error")

	// ErrAPIClient indicates a 4xx error from the Scrapfly API.
	ErrAPIClient = errors.New("API http client error")

	// ErrAPIServer indicates a 5xx error from the Scrapfly API.
	ErrAPIServer = errors.New("API http server error")

	// ErrScrapeFailed indicates the scraping operation failed.
	ErrScrapeFailed = errors.New("scrape failed")

	// ErrProxyFailed indicates a proxy connection error.
	ErrProxyFailed = errors.New("proxy error")

	// ErrASPBypassFailed indicates Anti-Scraping Protection bypass failed.
	ErrASPBypassFailed = errors.New("ASP bypass error")

	// ErrScheduleFailed indicates a scheduled job error.
	ErrScheduleFailed = errors.New("schedule error")

	// ErrWebhookFailed indicates a webhook delivery error.
	ErrWebhookFailed = errors.New("webhook error")

	// ErrSessionFailed indicates a browser session error.
	ErrSessionFailed = errors.New("session error")

	// ErrUnhandledAPIResponse indicates an unexpected API error response.
	ErrUnhandledAPIResponse = errors.New("unhandled API error response")
)

Sentinel errors for the Scrapfly client. These errors can be used with errors.Is() for error checking.

View Source 
var DefaultLogger = NewLogger("scrapfly")

DefaultLogger is the default logger used by the Scrapfly SDK.

You can configure the log level to control verbosity: 

scrapfly.DefaultLogger.SetLevel(scrapfly.LevelDebug)

Functions

func GetAnyEnumFor 

func GetAnyEnumFor[V Enumerable[T], T fmt.Stringer]() []any

func GetEnumFor 

func GetEnumFor[V Enumerable[T], T fmt.Stringer]() []T

func IsValidEnumType 

func IsValidEnumType[T fmt.Stringer](f T) bool

func ValidateEnums 

func ValidateEnums(s interface{}) error

ValidateEnums checks fields tagged with `validate:"enum"`. It calls the IsValid() bool method on the field if it's a single value, or on each element if it's a slice.

func ValidateExclusiveFields 

func ValidateExclusiveFields(s interface{}) error

ValidateExclusiveFields checks a struct for fields marked with the "exclusive" tag and ensures that only one field per exclusive group is set.

func ValidateRequiredFields 

func ValidateRequiredFields(s interface{}) error

ValidateRequiredFields checks a struct for fields with the `required:"true"` tag and returns an error if any of them are zero-valued.

Types

type APIError 

type APIError struct {
	// Message is the human-readable error message.
	Message string
	// Code is the error code identifier.
	Code string
	// HTTPStatusCode is the HTTP status code of the response.
	HTTPStatusCode int
	// DocumentationURL provides a link to relevant documentation.
	DocumentationURL string
	// APIResponse contains the full API response (if available).
	APIResponse *ScrapeResult
	// RetryAfterMs indicates how long to wait before retrying (for rate limits).
	RetryAfterMs int
	// Hint provides additional context or suggestions for resolving the error.
	Hint string
}

APIError represents a detailed error returned by the Scrapfly API.

It contains information about the error including status codes, error messages, documentation links, and retry information.

func (*APIError) Error 

func (e *APIError) Error() string

Error implements the error interface.

type APIErrorDetails 

type APIErrorDetails struct {
	Code      string            `json:"code"`
	HTTPCode  int               `json:"http_code"`
	Links     map[string]string `json:"links"`
	Message   string            `json:"message"`
	Retryable bool              `json:"retryable"`
	DocURL    string            `json:"doc_url"`
}

APIErrorDetails contains detailed error information from the API.

type Account 

type Account struct {
	AccountID        string `json:"account_id"`
	Currency         string `json:"currency"`
	Timezone         string `json:"timezone"`
	Suspended        bool   `json:"suspended"`
	SuspensionReason string `json:"suspension_reason"`
}

--- Account Data Structures ---

type AccountData 

type AccountData struct {
	Account      Account      `json:"account"`
	Project      Project      `json:"project"`
	Subscription Subscription `json:"subscription"`
}

AccountData represents detailed account information from Scrapfly.

This includes subscription details, usage statistics, billing information, and account limits.

type Attachment 

type Attachment struct {
	Content           string `json:"content"`
	ContentType       string `json:"content_type"`
	Filename          string `json:"filename"`
	ID                string `json:"id"`
	Size              int    `json:"size"`
	State             string `json:"state"`
	SuggestedFilename string `json:"suggested_filename"`
	URL               string `json:"url"`
	// contains filtered or unexported fields
}

func (*Attachment) Data 

func (a *Attachment) Data() ([]byte, error)

Data returns the attachment data as a byte slice.

func (*Attachment) Save 

func (a *Attachment) Save(savePath ...string) (string, error)

Save saves a scraped attachment result to disk.

Parameters:

  • savePath: Optional directory path where to save the file (defaults to current directory) (if savePath does not exists, it will be created in a best effort basis)

it is named as the filename of the attachment Returns the full path to the saved file.

Example: 

filePath, err := a.Save("./attachments")
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Attachment %s saved to: %s\n", a.Filename, filePath)

type BrowserData 

type BrowserData struct {
	JSEvaluationResult *string                `json:"javascript_evaluation_result"`
	JSScenario         interface{}            `json:"js_scenario"`
	LocalStorageData   map[string]interface{} `json:"local_storage_data"`
	SessionStorageData map[string]interface{} `json:"session_storage_data"`
	Websockets         []interface{}          `json:"websockets"`
	XHRCall            []interface{}          `json:"xhr_call"`
	Attachments        []Attachment           `json:"attachments"`
}

BrowserData contains data collected from the browser during JavaScript rendering.

type CacheContext 

type CacheContext struct {
	State string      `json:"state"`
	Entry interface{} `json:"entry"`
}

CacheContext contains information about cache usage for the request.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the main client for interacting with the Scrapfly API. It handles authentication, request execution, and response parsing.

func New 

func New(key string) (*Client, error)

New creates a new Scrapfly client with the provided API key. The API key can be obtained from https://scrapfly.io/dashboard.

Example: 

client, err := scrapfly.New("YOUR_API_KEY")
if err != nil {
    log.Fatal(err)
}

func NewWithHost 

func NewWithHost(key, host string, verifySSL bool) (*Client, error)

NewWithHost creates a new Scrapfly client with a custom API host. This is useful for enterprise deployments or testing against a custom endpoint.

Parameters:

  • key: Your Scrapfly API key
  • host: Custom API host URL (e.g., "https://custom-api.example.com")
  • verifySSL: Whether to verify SSL certificates (set to false only for testing)

Example: 

client, err := scrapfly.NewWithHost("YOUR_API_KEY", "https://custom-api.example.com", true)

func (*Client) APIKey 

func (c *Client) APIKey() string

APIKey returns the currently configured API key.

func (*Client) Account 

func (c *Client) Account() (*AccountData, error)

Account retrieves information about the current Scrapfly account.

Returns account details including:

  • Subscription plan and limits
  • API usage statistics
  • Billing information
  • Concurrency limits

Example: 

account, err := client.Account()
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Plan: %s\n", account.Subscription.PlanName)
fmt.Printf("Remaining requests: %d\n", account.Subscription.Usage.Scrape.Remaining)

func (*Client) ConcurrentScrape 

func (c *Client) ConcurrentScrape(configs []*ScrapeConfig, concurrencyLimit int) <-chan struct {
	*ScrapeResult
	error
}

ConcurrentScrape performs multiple scraping requests concurrently with controlled concurrency. This is useful for scraping multiple pages efficiently while respecting rate limits.

Parameters:

  • configs: A slice of ScrapeConfig objects to scrape
  • concurrencyLimit: Maximum number of concurrent requests. If <= 0, uses account's concurrent limit

Returns a channel that emits results as they complete. Each result contains either a successful ScrapeResult or an error.

Example: 

configs := []*scrapfly.ScrapeConfig{
    {URL: "https://example.com/page1"},
    {URL: "https://example.com/page2"},
    {URL: "https://example.com/page3"},
}
resultsChan := client.ConcurrentScrape(configs, 3)
for result := range resultsChan {
    if result.error != nil {
        log.Printf("Error: %v", result.error)
        continue
    }
    fmt.Println(result.ScrapeResult.Result.Content)
}

func (*Client) Extract 

func (c *Client) Extract(config *ExtractionConfig) (*ExtractionResult, error)

Extract performs AI-powered structured data extraction from HTML content.

This method uses Scrapfly's AI extraction capabilities to parse HTML and extract structured data based on templates or prompts.

Example: 

config := &scrapfly.ExtractionConfig{
    Body:               []byte("<html>...</html>"),
    ContentType:        "text/html",
    ExtractionTemplate: "product",
}
result, err := client.Extract(config)
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Extracted data: %+v\n", result.Data)

func (*Client) Scrape 

func (c *Client) Scrape(config *ScrapeConfig) (*ScrapeResult, error)

Scrape performs a web scraping request using the provided configuration. This is the main method for scraping web pages with Scrapfly.

The method supports various features including:

  • JavaScript rendering (RenderJS)
  • Proxy rotation and geo-targeting
  • Anti-bot protection (ASP)
  • Custom headers and cookies
  • Screenshot capture
  • Data extraction

Example: 

config := &scrapfly.ScrapeConfig{
    URL:      "https://example.com",
    RenderJS: true,
    Country:  "us",
    ASP:      true,
}
result, err := client.Scrape(config)
if err != nil {
    log.Fatal(err)
}
fmt.Println(result.Result.Content)

func (*Client) Screenshot 

func (c *Client) Screenshot(config *ScreenshotConfig) (*ScreenshotResult, error)

Screenshot captures a screenshot of a web page using the provided configuration.

Supports various features including:

  • Multiple image formats (JPG, PNG, WEBP, GIF)
  • Full page or element-specific capture
  • Custom resolution
  • Dark mode
  • Banner blocking

Example: 

config := &scrapfly.ScreenshotConfig{
    URL:        "https://example.com",
    Format:     scrapfly.FormatPNG,
    Capture:    "fullpage",
    Resolution: "1920x1080",
}
result, err := client.Screenshot(config)
if err != nil {
    log.Fatal(err)
}
// result.Image contains the screenshot bytes

func (*Client) SetAPIKey 

func (c *Client) SetAPIKey(key string)

SetAPIKey updates the API key for the client. This is useful for switching between different API keys at runtime.

func (*Client) VerifyAPIKey 

func (c *Client) VerifyAPIKey() (*VerifyAPIKeyResult, error)

VerifyAPIKey checks if the configured API key is valid. Returns a VerifyAPIKeyResult indicating whether the key is valid.

Example: 

result, err := client.VerifyAPIKey()
if err != nil {
    log.Fatal(err)
}
if result.Valid {
    fmt.Println("API key is valid")
}

type CompressionFormat 

type CompressionFormat string

CompressionFormat specifies the compression format for document body. 

const (
	// GZIP uses gzip compression (widely supported, good compression ratio).
	GZIP CompressionFormat = "gzip"
	// ZSTD uses Zstandard compression (better compression and speed than gzip).
	ZSTD CompressionFormat = "zstd"
	// DEFLATE uses DEFLATE compression (older format, similar to gzip).
	DEFLATE CompressionFormat = "deflate"
)

Available compression formats for document body compression.

type ConfigData 

type ConfigData struct {
	URL                string            `json:"url"`
	Method             string            `json:"method"`
	Country            *string           `json:"country"`
	RenderJS           bool              `json:"render_js"`
	Cache              bool              `json:"cache"`
	CacheClear         bool              `json:"cache_clear"`
	CacheTTL           int               `json:"cache_ttl"`
	SSL                bool              `json:"ssl"`
	DNS                bool              `json:"dns"`
	ASP                bool              `json:"asp"`
	Debug              bool              `json:"debug"`
	ProxyPool          string            `json:"proxy_pool"`
	Session            *string           `json:"session"`
	SessionStickyProxy bool              `json:"session_sticky_proxy"`
	Tags               []string          `json:"tags"`
	CorrelationID      *string           `json:"correlation_id"`
	Body               *string           `json:"body"`
	Headers            map[string]string `json:"headers"`
	JS                 *string           `json:"js"`
	RenderingWait      int               `json:"rendering_wait"`
	WaitForSelector    *string           `json:"wait_for_selector"`
	Screenshots        map[string]string `json:"screenshots"`
	WebhookName        *string           `json:"webhook_name"`
	Timeout            int               `json:"timeout"`
	JSScenario         interface{}       `json:"js_scenario"`
	Extract            interface{}       `json:"extract"`
	Lang               []string          `json:"lang"`
	OS                 *string           `json:"os"`
	AutoScroll         bool              `json:"auto_scroll"`
	Env                string            `json:"env"`
	Origin             string            `json:"origin"`
	Project            string            `json:"project"`
	UserUUID           string            `json:"user_uuid"`
	UUID               string            `json:"uuid"`
}

ConfigData contains the configuration that was used for a scrape request. This mirrors the 'config' object in the API response.

type ContextData 

type ContextData struct {
	ASP               interface{}  `json:"asp"`
	BandwidthConsumed int          `json:"bandwidth_consumed"`
	Cache             CacheContext `json:"cache"`
	Cookies           []Cookie     `json:"cookies"`
	Cost              CostContext  `json:"cost"`
	CreatedAt         string       `json:"created_at"`
	Debug             DebugContext `json:"debug"`
	Env               string       `json:"env"`
	//Fingerprint       string            `json:"fingerprint"`
	Fingerprint      interface{}       `json:"fingerprint"`
	Headers          map[string]string `json:"headers"`
	IsXMLHTTPRequest bool              `json:"is_xml_http_request"`
	Job              interface{}       `json:"job"`
	Lang             interface{}       `json:"lang"` // []string or string
	OS               OSContext         `json:"os"`
	Project          string            `json:"project"`
	Proxy            ProxyContext      `json:"proxy"`
	Redirects        interface{}       `json:"redirects"` // []string or string
	Retry            int               `json:"retry"`
	Schedule         interface{}       `json:"schedule"`
	Session          interface{}       `json:"session"`
	Spider           interface{}       `json:"spider"`
	Throttler        interface{}       `json:"throttler"`
	URI              URIContext        `json:"uri"`
	URL              string            `json:"url"`
	Webhook          interface{}       `json:"webhook"`
}

ContextData contains metadata about the scrape request execution. This includes proxy information, costs, cache status, and more. 

type Cookie struct {
	Name     string `json:"name"`
	Value    string `json:"value"`
	Expires  string `json:"expires"`
	Path     string `json:"path"`
	Comment  string `json:"comment"`
	Domain   string `json:"domain"`
	MaxAge   int    `json:"max_age"`
	Secure   bool   `json:"secure"`
	HTTPOnly bool   `json:"http_only"`
	Version  string `json:"version"`
	Size     int    `json:"size"`
}

Cookie represents an HTTP cookie.

type CostContext 

type CostContext struct {
	Details []CostDetail `json:"details"`
	Total   int          `json:"total"`
}

CostContext contains the cost breakdown for a scrape request.

type CostDetail 

type CostDetail struct {
	Amount      int    `json:"amount"`
	Code        string `json:"code"`
	Description string `json:"description"`
}

CostDetail represents a single cost item for a scrape request.

type DebugContext 

type DebugContext struct {
	ResponseURL   string      `json:"response_url"`
	ScreenshotURL interface{} `json:"screenshot_url"`
}

DebugContext contains URLs for debugging the request.

type Enumerable 

type Enumerable[T fmt.Stringer] interface {
	Enum() []T
	AnyEnum() []any
}

type ExtractionConfig 

type ExtractionConfig struct {
	// Body is the document content to extract data from (required).
	Body []byte `required:"true"`
	// ContentType specifies the document content type, e.g., "text/html" (required).
	ContentType string `required:"true"`
	// URL is the original URL of the document (optional, helps with context).
	URL string
	// Charset specifies the character encoding of the document.
	Charset string
	// ExtractionTemplate is the name of a saved extraction template.
	ExtractionTemplate string `exclusive:"extraction"`
	// ExtractionEphemeralTemplate is an inline extraction template definition.
	ExtractionEphemeralTemplate map[string]interface{} `exclusive:"extraction"`
	// ExtractionPrompt is an AI prompt describing what data to extract.
	ExtractionPrompt string `exclusive:"extraction"`
	// ExtractionModel specifies which AI model to use for extraction.
	ExtractionModel ExtractionModel `exclusive:"extraction" validate:"enum"`
	// IsDocumentCompressed indicates if the Body is compressed.
	IsDocumentCompressed bool
	// DocumentCompressionFormat specifies the compression format if IsDocumentCompressed is true.
	DocumentCompressionFormat CompressionFormat
	// Webhook is the name of a webhook to call after extraction completes.
	Webhook string
}

ExtractionConfig configures an AI-powered data extraction request to the Scrapfly API.

This struct contains all available options for extracting structured data from HTML or other document formats using AI models or predefined templates.

Example with template: 

config := &scrapfly.ExtractionConfig{
    Body:               []byte("<html>...</html>"),
    ContentType:        "text/html",
    ExtractionTemplate: "product",
}

Example with AI prompt: 

config := &scrapfly.ExtractionConfig{
    Body:             []byte("<html>...</html>"),
    ContentType:      "text/html",
    ExtractionPrompt: "Extract product name, price, and description",
}

type ExtractionModel 

type ExtractionModel string

ExtractionModel defines the type of extraction model to use for extraction. see https://scrapfly.io/docs/extraction-api/automatic-ai#models 

const (
	ExtractionModelArticle                   ExtractionModel = "article"
	ExtractionModelEvent                     ExtractionModel = "event"
	ExtractionModelFoodRecipe                ExtractionModel = "food_recipe"
	ExtractionModelHotel                     ExtractionModel = "hotel"
	ExtractionModelHotelListing              ExtractionModel = "hotel_listing"
	ExtractionModelJobListing                ExtractionModel = "job_listing"
	ExtractionModelJobPosting                ExtractionModel = "job_posting"
	ExtractionModelOrganization              ExtractionModel = "organization"
	ExtractionModelProduct                   ExtractionModel = "product"
	ExtractionModelProductListing            ExtractionModel = "product_listing"
	ExtractionModelRealEstateProperty        ExtractionModel = "real_estate_property"
	ExtractionModelRealEstatePropertyListing ExtractionModel = "real_estate_property_listing"
	ExtractionModelReviewList                ExtractionModel = "review_list"
	ExtractionModelSearchEngineResults       ExtractionModel = "search_engine_results"
	ExtractionModelSocialMediaPost           ExtractionModel = "social_media_post"
	ExtractionModelSoftware                  ExtractionModel = "software"
	ExtractionModelStock                     ExtractionModel = "stock"
	ExtractionModelVehicleAd                 ExtractionModel = "vehicle_ad"
	ExtractionModelVehicleAdListing          ExtractionModel = "vehicle_ad_listing"
	ExtractionModelNone                      ExtractionModel = ""
)

func (ExtractionModel) AnyEnum 

func (f ExtractionModel) AnyEnum() []any

func (ExtractionModel) Enum 

func (f ExtractionModel) Enum() []ExtractionModel

func (ExtractionModel) IsValid 

func (f ExtractionModel) IsValid() bool

func (ExtractionModel) String 

func (f ExtractionModel) String() string

type ExtractionResult 

type ExtractionResult struct {
	// Data contains the extracted structured data.
	Data interface{} `json:"data"`
	// ContentType is the content type of the extracted data.
	ContentType string `json:"content_type"`
	// DataQuality indicates the quality/confidence of the extraction (if available).
	DataQuality interface{} `json:"data_quality,omitempty"`
}

ExtractionResult represents the result of a data extraction request.

type Format 

type Format string

Format defines the format for the scraped content response. 

const (
	// FormatJSON returns content structured as JSON.
	FormatJSON Format = "json"
	// FormatText returns plain text content with HTML tags stripped.
	FormatText Format = "text"
	// FormatMarkdown converts HTML to Markdown format.
	FormatMarkdown Format = "markdown"
	// FormatCleanHTML returns cleaned and normalized HTML.
	FormatCleanHTML Format = "clean_html"
	// FormatRaw returns the raw HTML content without any processing.
	FormatRaw Format = "raw"
)

Available content formats for scrape responses.

func (Format) AnyEnum 

func (f Format) AnyEnum() []any

func (Format) Enum 

func (f Format) Enum() []Format

func (Format) IsValid 

func (f Format) IsValid() bool

func (Format) String 

func (f Format) String() string

type FormatOption 

type FormatOption string

FormatOption defines additional options for content formatting. 

const (
	// NoLinks removes all links from the formatted content.
	NoLinks FormatOption = "no_links"
	// NoImages removes all images from the formatted content.
	NoImages FormatOption = "no_images"
	// OnlyContent extracts only the main content, removing headers, footers, and navigation.
	OnlyContent FormatOption = "only_content"
)

Available format options that can be combined with Format settings.

func (FormatOption) AnyEnum 

func (f FormatOption) AnyEnum() []any

func (FormatOption) Enum 

func (f FormatOption) Enum() []FormatOption

func (FormatOption) IsValid 

func (f FormatOption) IsValid() bool

func (FormatOption) String 

func (f FormatOption) String() string

type HttpMethod 

type HttpMethod string
const (
	HttpMethodGet     HttpMethod = http.MethodGet
	HttpMethodPost    HttpMethod = http.MethodPost
	HttpMethodPut     HttpMethod = http.MethodPut
	HttpMethodPatch   HttpMethod = http.MethodPatch
	HttpMethodOptions HttpMethod = http.MethodOptions
)

func (HttpMethod) AnyEnum 

func (f HttpMethod) AnyEnum() []any

func (HttpMethod) Enum 

func (f HttpMethod) Enum() []HttpMethod

func (HttpMethod) IsValid 

func (f HttpMethod) IsValid() bool

func (HttpMethod) String 

func (f HttpMethod) String() string

type IFrame 

type IFrame struct {
	URL     string     `json:"url"`
	URI     URIContext `json:"uri"`
	Content string     `json:"content"`
}

IFrame represents an iframe found in the page.

type LogLevel 

type LogLevel int

LogLevel defines the severity level for log messages. 

const (
	// LevelDebug logs all messages including detailed debug information.
	LevelDebug LogLevel = iota
	// LevelInfo logs informational messages and above.
	LevelInfo
	// LevelWarn logs warnings and errors.
	LevelWarn
	// LevelError logs only error messages.
	LevelError
)

Available log levels, from most to least verbose.

type Logger 

type Logger struct {
	// contains filtered or unexported fields
}

Logger provides simple leveled logging for the Scrapfly SDK.

func NewLogger 

func NewLogger(name string) *Logger

NewLogger creates a new Logger instance with the given name prefix.

Example: 

logger := scrapfly.NewLogger("my-scraper")
logger.SetLevel(scrapfly.LevelDebug)
logger.Info("Starting scraper...")

func (*Logger) Debug 

func (l *Logger) Debug(v ...interface{})

Debug logs a debug-level message. These messages are only logged when the level is set to LevelDebug.

func (*Logger) Error 

func (l *Logger) Error(v ...interface{})

Error logs an error message. These messages are always logged regardless of the level setting.

func (*Logger) Info 

func (l *Logger) Info(v ...interface{})

Info logs an informational message. These messages are logged when the level is LevelInfo or lower.

func (*Logger) SetLevel 

func (l *Logger) SetLevel(level LogLevel)

SetLevel sets the minimum logging level. Only messages at this level or higher will be logged.

func (*Logger) Warn 

func (l *Logger) Warn(v ...interface{})

Warn logs a warning message. These messages are logged when the level is LevelWarn or lower.

type OSContext 

type OSContext struct {
	Distribution string `json:"distribution"`
	Name         string `json:"name"`
	Type         string `json:"type"`
	Version      string `json:"version"`
}

OSContext contains information about the operating system used for the request.

type Project 

type Project struct {
	AllowExtraUsage    bool     `json:"allow_extra_usage"`
	AllowedNetworks    []string `json:"allowed_networks"`
	BudgetLimit        *float32 `json:"budget_limit"`
	BudgetSpent        *float32 `json:"budget_spent"`
	ConcurrencyLimit   *int     `json:"concurrency_limit"`
	Name               string   `json:"name"`
	QuotaReached       bool     `json:"quota_reached"`
	ScrapeRequestCount int      `json:"scrape_request_count"`
	ScrapeRequestLimit *int     `json:"scrape_request_limit"`
	Tags               []string `json:"tags"`
}

type ProxyContext 

type ProxyContext struct {
	Country  string `json:"country"`
	Identity string `json:"identity"`
	Network  string `json:"network"`
	Pool     string `json:"pool"`
}

ProxyContext contains information about the proxy used for the request.

type ProxyPool 

type ProxyPool string

ProxyPool represents the type of proxy pool to use for scraping. 

const (
	// PublicDataCenterPool uses datacenter proxies. Fast and reliable but easier to detect.
	PublicDataCenterPool ProxyPool = "public_datacenter_pool"
	// PublicResidentialPool uses residential proxies. More expensive but harder to detect.
	PublicResidentialPool ProxyPool = "public_residential_pool"
)

Available proxy pool types for Scrapfly API requests.

func (ProxyPool) AnyEnum 

func (f ProxyPool) AnyEnum() []any

func (ProxyPool) Enum 

func (f ProxyPool) Enum() []ProxyPool

func (*ProxyPool) IsValid 

func (f *ProxyPool) IsValid() bool

func (ProxyPool) String 

func (f ProxyPool) String() string

type ResultData 

type ResultData struct {
	BrowserData     BrowserData            `json:"browser_data"`
	Content         string                 `json:"content"`
	ContentEncoding string                 `json:"content_encoding"`
	ContentType     string                 `json:"content_type"`
	Cookies         []Cookie               `json:"cookies"`
	Data            interface{}            `json:"data"`
	DNS             interface{}            `json:"dns"`
	Duration        float64                `json:"duration"`
	Error           *APIErrorDetails       `json:"error"`
	Format          string                 `json:"format"`
	IFrames         []IFrame               `json:"iframes"`
	LogURL          string                 `json:"log_url"`
	Reason          string                 `json:"reason"`
	RequestHeaders  map[string]string      `json:"request_headers"`
	ResponseHeaders map[string]interface{} `json:"response_headers"` // Can be string or []string
	Screenshots     map[string]Screenshot  `json:"screenshots"`
	Size            int                    `json:"size"`
	SSL             interface{}            `json:"ssl"`
	Status          string                 `json:"status"`
	StatusCode      int                    `json:"status_code"`
	Success         bool                   `json:"success"`
	URL             string                 `json:"url"`
	ExtractedData   *ExtractionResult      `json:"extracted_data"`
}

ResultData contains the scraped content and response information. This is the main data from the scrape request including HTML content, status codes, headers, cookies, and more.

type ScheduleUsage 

type ScheduleUsage struct {
	Current int `json:"current"`
	Limit   int `json:"limit"`
}

type ScrapeConfig 

type ScrapeConfig struct {
	// URL is the target URL to scrape (required).
	URL string `required:"true"`
	// Method is the HTTP method to use (GET, POST, PUT, PATCH). Defaults to GET.
	Method HttpMethod
	// Body is the raw request body for POST/PUT/PATCH requests.
	Body string
	// Data is a map that will be encoded as request body based on Content-Type.
	// Cannot be used together with Body.
	Data map[string]interface{}
	// Headers are custom HTTP headers to send with the request.
	Headers map[string]string
	// Cookies are cookies to include in the request.
	Cookies map[string]string
	// Country specifies the proxy country code (e.g., "us", "uk", "de").
	// Supports ISO 3166-1 alpha-2 country codes.
	Country string
	// ProxyPool specifies which proxy pool to use.
	ProxyPool ProxyPool
	// RenderJS enables JavaScript rendering using a headless browser.
	RenderJS bool
	// ASP enables Anti-Scraping Protection bypass.
	ASP bool
	// Cache enables response caching.
	Cache bool
	// CacheTTL sets the cache time-to-live in seconds.
	CacheTTL int
	// CacheClear forces cache refresh for this request.
	CacheClear bool
	// Timeout sets the maximum time in milliseconds to wait for the request.
	Timeout int
	// Retry enables automatic retries on failure (enabled by default).
	Retry bool
	// Session maintains a persistent browser session across requests.
	Session string
	// SessionStickyProxy keeps the same proxy for all requests in a session.
	SessionStickyProxy bool
	// Tags are custom tags for organizing and filtering requests.
	Tags []string
	// Webhook is the name of a webhook to call after the request completes.
	Webhook string
	// Debug enables debug mode for viewing request details in the dashboard.
	Debug bool
	// SSL enables SSL certificate verification details capture.
	SSL bool
	// DNS enables DNS resolution details capture.
	DNS bool
	// CorrelationID is a custom ID for tracking requests across systems.
	CorrelationID string
	// Format specifies the output format for the scraped content.
	Format Format `validate:"enum"`
	// FormatOptions are additional options for the content format.
	FormatOptions []FormatOption `validate:"enum"`
	// ExtractionTemplate is the name of a saved extraction template.
	// it is exclusve with other extraction options
	ExtractionTemplate string `exclusive:"extraction"`
	// ExtractionEphemeralTemplate is an inline extraction template definition.
	// it is exclusve with other extraction options
	ExtractionEphemeralTemplate map[string]interface{} `exclusive:"extraction"`
	// ExtractionPrompt is an AI prompt for extracting structured data.
	// it is exclusve with other extraction options
	ExtractionPrompt string `exclusive:"extraction"`
	// ExtractionModel specifies which AI model to use for extraction.
	// it is exclusve with other extraction options
	ExtractionModel ExtractionModel `exclusive:"extraction" validate:"enum"`
	// WaitForSelector waits for a CSS selector to appear before capturing (requires RenderJS).
	WaitForSelector string
	// RenderingWait is additional wait time in milliseconds after page load (requires RenderJS).
	RenderingWait int
	// AutoScroll automatically scrolls the page to load lazy content (requires RenderJS).
	AutoScroll bool
	// Screenshots is a map of screenshot names to CSS selectors (requires RenderJS).
	Screenshots map[string]string
	// ScreenshotFlags are options for screenshot capture.
	ScreenshotFlags []ScreenshotFlag `validate:"enum"`
	// JS is custom JavaScript code to execute in the browser (requires RenderJS).
	JS string
	// JSScenario is a sequence of browser actions to perform (requires RenderJS).
	JSScenario []js_scenario.JSScenarioStep
	// OS spoofs the operating system in the User-Agent.
	OS string
	// Lang sets the Accept-Language header values.
	Lang []string
}

ScrapeConfig configures a web scraping request to the Scrapfly API.

This struct contains all available options for customizing scraping behavior, including proxy settings, JavaScript rendering, data extraction, and more.

Example: 

config := &scrapfly.ScrapeConfig{
    URL:      "https://example.com",
    RenderJS: true,
    Country:  "us",
    ASP:      true,
    Cache:    true,
    Format:   scrapfly.FormatMarkdown,
}

type ScrapeResult 

type ScrapeResult struct {
	// Config contains the configuration used for this scrape request.
	Config ConfigData `json:"config"`
	// Context contains metadata about the request execution.
	Context ContextData `json:"context"`
	// Result contains the scraped content and response data.
	Result ResultData `json:"result"`
	// UUID is the unique identifier for this scrape request.
	UUID string `json:"uuid"`
	// contains filtered or unexported fields
}

ScrapeResult represents the complete response from a scrape request.

It contains the scraped content, metadata, configuration, and context information about the request. The result includes details about the upstream response, proxy used, cost, and more.

func (*ScrapeResult) SaveAttachments 

func (r *ScrapeResult) SaveAttachments(savePath ...string) ([]string, error)

SaveAttachments is a shortcut to save all attachments to disk

Parameters:

  • savePath: Optional directory path where to save the files (defaults to current directory) (if savePath does not exists, it will be created in a best effort basis)

Returns the full paths to the saved files.

Example: 

paths, err := r.SaveAttachments("./attachments")
if err != nil {
    log.Fatal(err)
}
for _, path := range paths {
	fmt.Printf("Attachment saved to: %s\n", path)
}

func (*ScrapeResult) SaveScreenshots 

func (r *ScrapeResult) SaveScreenshots(savePath ...string) ([]string, error)

SaveScreenshots is a shortcut to save all screenshots to disk

Parameters:

  • savePath: Optional directory path where to save the files (defaults to current directory) (if savePath does not exists, it will be created in a best effort basis)

Returns the full paths to the saved files.

Example: 

paths, err := r.SaveScreenshots("./screenshots")
if err != nil {
    log.Fatal(err)
}
for _, path := range paths {
	fmt.Printf("Screenshot saved to: %s\n", path)
}

func (*ScrapeResult) Selector 

func (r *ScrapeResult) Selector() (*goquery.Document, error)

Selector provides a goquery document for parsing HTML content.

The selector is lazy-loaded and cached, making it efficient to call multiple times. It can only be used with HTML content.

Example: 

result, err := client.Scrape(config)
if err != nil {
    log.Fatal(err)
}

doc, err := result.Selector()
if err != nil {
    log.Fatal(err)
}

title := doc.Find("title").First().Text()
fmt.Println(title)

type ScrapeUsage 

type ScrapeUsage struct {
	ConcurrentLimit     int `json:"concurrent_limit"`
	ConcurrentRemaining int `json:"concurrent_remaining"`
	ConcurrentUsage     int `json:"concurrent_usage"`
	Current             int `json:"current"`
	Extra               int `json:"extra"`
	Limit               int `json:"limit"`
	Remaining           int `json:"remaining"`
}

type Screenshot 

type Screenshot struct {
	// CSSSelector is the CSS selector of the element to capture. If Format == fullpage, this will be nil
	CSSSelector *string `json:"css_selector"`
	// Extension is the file extension (jpg, png, webp, gif)
	Extension string `json:"extension"` // Always jpg when request from scraping api.
	// Format is the format of the screenshot (fullpage, element)
	Format string `json:"format"`
	// Size is the size of the screenshot in bytes
	Size int `json:"size"`
	// URL is the URL to retrieve the screenshot from
	URL string `json:"url"`

	// Name is the name of the screenshot retrieved from the API response
	Name string `json:"-"`
	// contains filtered or unexported fields
}

Screenshot represents a screenshot captured during rendering.

func (*Screenshot) Image 

func (s *Screenshot) Image() ([]byte, error)

Image returns the screenshot data as a byte slice.

func (*Screenshot) Save 

func (s *Screenshot) Save(savePath ...string) (string, error)

Save saves a scraped screenshot result to disk.

Parameters:

  • savePath: Optional directory path where to save the file (defaults to current directory) (if savePath does not exists, it will be created in a best effort basis)

it is named as the name of the screenshot as set in config and retrieved from the API response Returns the full path to the saved file.

Example: 

filePath, err := s.Save("./screenshots")
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Screenshot saved to: %s\n", filePath)

type ScreenshotConfig 

type ScreenshotConfig struct {
	// URL is the target URL to capture (required).
	URL string
	// Format specifies the image format (jpg, png, webp, gif).
	Format ScreenshotFormat
	// Capture defines what to capture: "fullpage" for entire page, or a CSS selector for specific element.
	Capture string
	// Resolution sets the viewport size (e.g., "1920x1080").
	Resolution string
	// Country specifies the proxy country code (e.g., "us", "uk", "de").
	Country string
	// Timeout sets the maximum time in milliseconds to wait for the request.
	Timeout int
	// RenderingWait is additional wait time in milliseconds after page load.
	RenderingWait int
	// WaitForSelector waits for a CSS selector to appear before capturing.
	WaitForSelector string
	// Options are additional screenshot options (dark mode, block banners, etc.).
	Options []ScreenshotOption
	// AutoScroll automatically scrolls the page to load lazy content.
	AutoScroll bool
	// JS is custom JavaScript code to execute before capturing.
	JS string
	// Cache enables response caching.
	Cache bool
	// CacheTTL sets the cache time-to-live in seconds.
	CacheTTL int
	// CacheClear forces cache refresh for this request.
	CacheClear bool
	// Webhook is the name of a webhook to call after the request completes.
	Webhook string
	// VisionDeficiencyType specifies the type of vision deficiency to simulate.
	// see https://scrapfly.io/docs/screenshot-api/accessibility#vision_deficiency
	VisionDeficiencyType VisionDeficiencyType
}

ScreenshotConfig configures a screenshot capture request to the Scrapfly API.

This struct contains all available options for customizing screenshot behavior, including format, resolution, capture area, and rendering options.

Example: 

config := &scrapfly.ScreenshotConfig{
    URL:        "https://example.com",
    Format:     scrapfly.FormatPNG,
    Capture:    "fullpage",
    Resolution: "1920x1080",
    Options:    []scrapfly.ScreenshotOption{scrapfly.OptionBlockBanners},
}

type ScreenshotFlag 

type ScreenshotFlag string

ScreenshotFlag defines options for screenshot behavior when using Screenshots parameter. 

const (
	// LoadImages enables image loading for screenshots (disabled by default for performance).
	LoadImages ScreenshotFlag = "load_images"
	// DarkMode enables dark mode rendering.
	DarkMode ScreenshotFlag = "dark_mode"
	// BlockBanners blocks cookie banners and similar overlays.
	BlockBanners ScreenshotFlag = "block_banners"
	// PrintMediaFormat uses print media CSS for rendering.
	PrintMediaFormat ScreenshotFlag = "print_media_format"
	// HighQuality captures screenshots at higher quality settings.
	HighQuality ScreenshotFlag = "high_quality"
)

Available screenshot flags for customizing screenshot capture.

func (ScreenshotFlag) AnyEnum 

func (f ScreenshotFlag) AnyEnum() []any

func (ScreenshotFlag) Enum 

func (f ScreenshotFlag) Enum() []ScreenshotFlag

func (ScreenshotFlag) IsValid 

func (f ScreenshotFlag) IsValid() bool

func (ScreenshotFlag) String 

func (f ScreenshotFlag) String() string

type ScreenshotFormat 

type ScreenshotFormat string

ScreenshotFormat defines the image format for screenshots. 

const (
	// FormatJPG captures screenshots in JPEG format (smaller file size, lossy compression).
	FormatJPG ScreenshotFormat = "jpg"
	// FormatPNG captures screenshots in PNG format (larger file size, lossless compression).
	FormatPNG ScreenshotFormat = "png"
	// FormatWEBP captures screenshots in WebP format (modern format with good compression).
	FormatWEBP ScreenshotFormat = "webp"
	// FormatGIF captures screenshots in GIF format (animated screenshots support).
	FormatGIF ScreenshotFormat = "gif"
)

Available image formats for screenshot capture.

type ScreenshotMetadata 

type ScreenshotMetadata struct {
	// ExtensionName is the file extension (jpg, png, webp, gif).
	ExtensionName string
	// UpstreamStatusCode is the HTTP status code from the target website.
	UpstreamStatusCode int
	// UpstreamURL is the final URL after any redirects.
	UpstreamURL string
}

ScreenshotMetadata contains metadata about a captured screenshot.

type ScreenshotOption 

type ScreenshotOption string

ScreenshotOption defines options to customize screenshot capture behavior. 

const (
	// OptionLoadImages enables image loading (disabled by default for performance).
	OptionLoadImages ScreenshotOption = "load_images"
	// OptionDarkMode enables dark mode rendering.
	OptionDarkMode ScreenshotOption = "dark_mode"
	// OptionBlockBanners blocks cookie banners and similar overlays.
	OptionBlockBanners ScreenshotOption = "block_banners"
	// OptionPrintMediaFormat uses print media CSS for rendering.
	OptionPrintMediaFormat ScreenshotOption = "print_media_format"
)

Available options for customizing screenshot capture.

type ScreenshotResult 

type ScreenshotResult struct {
	// Image contains the raw screenshot image bytes.
	Image []byte
	// Metadata contains information about the screenshot.
	Metadata ScreenshotMetadata
}

ScreenshotResult represents a screenshot captured by the API.

func (*ScreenshotResult) Save 

func (s *ScreenshotResult) Save(name string, savePath ...string) (string, error)

Save saves a screenshot result to disk.

Parameters:

  • name: The base name for the file (without extension)
  • savePath: Optional directory path where to save the file (defaults to current directory)

Returns the full path to the saved file.

Example: 

filePath, err := s.Save("example", "./screenshots")
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Screenshot saved to: %s\n", filePath)

type SpiderUsage 

type SpiderUsage struct {
	Current int `json:"current"`
	Limit   int `json:"limit"`
}

type Subscription 

type Subscription struct {
	Billing            SubscriptionBilling `json:"billing"`
	ExtraScrapeAllowed bool                `json:"extra_scrape_allowed"`
	MaxConcurrency     int                 `json:"max_concurrency"`
	Period             SubscriptionPeriod  `json:"period"`
	PlanName           string              `json:"plan_name"`
	Usage              SubscriptionUsage   `json:"usage"`
}

type SubscriptionBilling 

type SubscriptionBilling struct {
	CurrentExtraScrapeRequestPrice SubscriptionIntegerPrice `json:"current_extra_scrape_request_price"`
	ExtraScrapeRequestPricePer10k  SubscriptionFloatPrice   `json:"extra_scrape_request_price_per_10k"`
	OngoingPayment                 SubscriptionFloatPrice   `json:"ongoing_payment"`
	PlanPrice                      SubscriptionIntegerPrice `json:"plan_price"`
}

type SubscriptionFloatPrice 

type SubscriptionFloatPrice struct {
	Currency string  `json:"currency"`
	Amount   float32 `json:"amount"`
}

type SubscriptionIntegerPrice 

type SubscriptionIntegerPrice struct {
	Currency string `json:"currency"`
	Amount   int    `json:"amount"`
}

type SubscriptionPeriod 

type SubscriptionPeriod struct {
	Start string `json:"start"`
	End   string `json:"end"`
}

type SubscriptionUsage 

type SubscriptionUsage struct {
	Spider   SpiderUsage   `json:"spider"`
	Schedule ScheduleUsage `json:"schedule"`
	Scrape   ScrapeUsage   `json:"scrape"`
}

type URIContext 

type URIContext struct {
	BaseURL    string      `json:"base_url"`
	Fragment   interface{} `json:"fragment"`
	Host       string      `json:"host"`
	Params     interface{} `json:"params"`
	Port       int         `json:"port"`
	Query      interface{} `json:"query"`
	RootDomain string      `json:"root_domain"`
	Scheme     string      `json:"scheme"`
}

URIContext contains parsed URI information about the requested URL.

type VerifyAPIKeyResult 

type VerifyAPIKeyResult struct {
	// Valid indicates whether the API key is valid.
	Valid bool `json:"valid"`
}

VerifyAPIKeyResult represents the result of an API key verification.

type VisionDeficiencyType 

type VisionDeficiencyType string
const (
	VisionDeficiencyTypeNone            VisionDeficiencyType = "none"
	VisionDeficiencyTypeDeuteranopia    VisionDeficiencyType = "deuteranopia"
	VisionDeficiencyTypeProtanopia      VisionDeficiencyType = "protanopia"
	VisionDeficiencyTypeTritanopia      VisionDeficiencyType = "tritanopia"
	VisionDeficiencyTypeAchromatopsia   VisionDeficiencyType = "achromatopsia"
	VisionDeficiencyTypeBlurredVision   VisionDeficiencyType = "blurredVision"
	VisionDeficiencyTypeReducedContrast VisionDeficiencyType = "reducedContrast"
)

func (VisionDeficiencyType) AnyEnum 

func (f VisionDeficiencyType) AnyEnum() []any

func (VisionDeficiencyType) Enum 

func (f VisionDeficiencyType) Enum() []VisionDeficiencyType

func (VisionDeficiencyType) IsValid 

func (f VisionDeficiencyType) IsValid() bool

func (VisionDeficiencyType) String 

func (f VisionDeficiencyType) String() string

Source Files

View all Source files

Directories

Path Synopsis
Package scenario provides a fluent builder for creating Scrapfly API JS Scenarios.
 Package scenario provides a fluent builder for creating Scrapfly API JS Scenarios.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.