Documentation
¶
Index ¶
- Constants
- Variables
- func Banner()
- func CreateGridSeedJobs(langCode string, r io.Reader, maxDepth int, email bool, bbox grid.BoundingBox, ...) ([]scrapemate.IJob, error)
- func CreateSeedJobs(fastmode bool, langCode string, r io.Reader, maxDepth int, email bool, ...) (jobs []scrapemate.IJob, err error)
- func LoadCustomWriter(pluginDir, pluginName string) (scrapemate.ResultWriter, error)
- func Telemetry() tlmt.Telemetry
- type Config
- type Runner
- type S3Uploader
Constants ¶
View Source
const ( RunModeFile = iota + 1 RunModeDatabase RunModeDatabaseProduce RunModeInstallPlaywright RunModeWeb RunModeAwsLambda RunModeAwsLambdaInvoker )
Variables ¶
View Source
var (
ErrInvalidRunMode = errors.New("invalid run mode")
)
Functions ¶
func CreateGridSeedJobs ¶ added in v1.11.0
func CreateGridSeedJobs( langCode string, r io.Reader, maxDepth int, email bool, bbox grid.BoundingBox, cellSizeKm float64, zoom int, dedup deduper.Deduper, exitMonitor exiter.Exiter, extraReviews bool, ) ([]scrapemate.IJob, error)
CreateGridSeedJobs reads search queries from r and produces one GmapJob per (query, grid-cell) pair. Each cell covers approximately cellSizeKm × cellSizeKm on the ground. The zoom level controls how much of the map Google Maps renders per cell (use 14-16 for most cases).
Deduplication across cells is handled automatically by the shared deduper.
func CreateSeedJobs ¶
func LoadCustomWriter ¶
func LoadCustomWriter(pluginDir, pluginName string) (scrapemate.ResultWriter, error)
Types ¶
type Config ¶
type Config struct {
Concurrency int
CacheDir string
MaxDepth int
InputFile string
ResultsFile string
JSON bool
LangCode string
Debug bool
Dsn string
ProduceOnly bool
ExitOnInactivityDuration time.Duration
Email bool
CustomWriter string
GeoCoordinates string
Zoom int
RunMode int
DisableTelemetry bool
WebRunner bool
AwsLamdbaRunner bool
DataFolder string
Proxies []string
AwsAccessKey string
AwsSecretKey string
AwsRegion string
S3Uploader S3Uploader
S3Bucket string
AwsLambdaInvoker bool
FunctionName string
AwsLambdaChunkSize int
FastMode bool
Radius float64
Addr string
DisablePageReuse bool
ExtraReviews bool
LeadsDBAPIKey string
// Grid scraping — divide a bounding box into cells to bypass the ~120
// results-per-search limit imposed by Google Maps.
GridBBox string // "minLat,minLon,maxLat,maxLon"
GridCellKm float64 // size of each grid cell in km (default: 1.0)
}
func ParseConfig ¶
func ParseConfig() *Config
Click to show internal directories.
Click to hide internal directories.