utils

Utils

A collection of small Go helpers that can be shared between projects. The repository is organised by package so you can import only the utilities you need.

Browser Transport

Reusable proxy-aware browser and HTTP transport helpers for scraping-heavy projects.

• Browser profiles - Model direct, HTTP proxy auth, and SOCKS forwarder browser transport modes.
• Session - Reuse one browser per transport and open short-lived render tabs on demand.
• RenderPage / RenderPages - One-shot convenience helpers for JS-rendered pages.
• NewHTTPClient - Build an HTTP client bound to the same transport profile model; direct profiles bypass ambient HTTP_PROXY/HTTPS_PROXY environment settings while explicit HTTP and SOCKS profiles stay profile-bound.

Crawler

Reusable crawler primitives for proxy-aware scraping workloads.

• ProxyLeaseSelector - Select provider/user-aware proxy leases, keep successful leases sticky, reuse the least-reserved healthy lease under concurrency saturation, release neutral terminal responses without poisoning proxy health, clear health from stale in-flight successes without rewinding provider cursors, and rotate providers immediately after reported failures or rotation-only retry decisions.
• RetryDecision.ProxyFailureSeverity - Let platform hooks distinguish normal rotate-proxy retries that only rotate leases from critical proxy failures that should immediately cooldown a candidate.
• RetryDecision.ProxyFailureKind - Attach structured proxy diagnostics such as challenge, status, transport, provider auth, and provider account reasons so shared selector pools can rotate content challenges without poisoning proxy health and can explain exhausted candidate pools.
• Provider credential failures - Status 402, status 407, Payment Required, and Proxy Authentication Required errors quarantine the affected lease and retry only alternate proxy candidates instead of burning the normal retry budget.
• ProxyLeaseAttemptScope - Track failed leases for one scrape or request batch so callers can skip candidates that already failed during that operation and stop with a typed exhausted-candidates error.

Configfile

Strict YAML configuration loading for applications.

• LoadYAML(path string, target any) error - Read a YAML config file, expand environment variables only inside YAML scalar values, reject missing environment variables and trailing YAML documents, and decode with known-field validation.
• LoadYAMLWithOptions(path string, target any, options EnvironmentOptions) error - Load a YAML config with an explicit environment registry so deployment preflights can require critical shell-sourced values before decoding.
• LoadYAMLBytes(configPayload []byte, target any) error - Apply the same contract to already-read YAML bytes.
• InterpolateYAML(configPayload []byte) ([]byte, error) - Expand YAML scalar environment references before application-specific decoding.
• EnvContract / EnvRegistry - Declare required and optional environment parameters, attach value schemas when needed, expose the mandatory registry, and validate shell-expanded config references without logging secret values.
• EnvValueSchemaForKind - Reuse built-in value schemas for booleans, URLs, JSON, base64/hex 32-byte secrets, host:port addresses, durations, positive integers, and email addresses.
• cmd/configenvcheck - Validate a YAML config plus dotenv inputs from deployment preflights, including optional variables and built-in value schemas for booleans, URLs, JSON, base64/hex keys, host:port values, durations, positive integers, and email addresses.

Runtimeconfig

Application runtime config loading built on top of configfile.

• Contract[T] / NewLoader[T] - Declare the application config shape with a typed Go target, optional edge validation, optional scalar value mappings, and an optional interpolation lookup. The loader resolves --config-style paths, reads one YAML file, expands ${NAME} scalar references exactly once, decodes with known-field validation, and runs application validation at the edge.
• Loaded[T] - Returns the typed config, expanded YAML, effective settings map, and selected scalar value map for legacy resolver-style code.
• LoadSection - Decode one required YAML section with the same strict contract, useful for split service binaries that share one runtime config file.
• ConfigValues - Expose mapped effective values through Lookup, Resolve, Map, and Resolver without requiring callers to know whether a value was literal YAML or populated through interpolation.

JSEval

Compatibility wrapper around browsertransport for existing callers that only need one-shot page rendering.

File

Utilities that simplify common file system operations.

• RemoveAll(dir string) - Recursively delete a directory while ignoring errors.

  file.RemoveAll("/tmp/cache")
  

• RemoveFile(path string) - Delete a single file and log any failures.

  file.RemoveFile("/tmp/out.log")
  

• CloseFile(c io.Closer) - Safely close a file descriptor and log errors.

  f, _ := os.Open("data.txt")
  file.CloseFile(f)
  

• ReadLines(filename string) ([]string, error) - Read a text file into a slice of lines.

  lines, err := file.ReadLines("notes.txt")
  if err != nil {
      log.Fatal(err)
  }
  

• SaveFile(dir, name string, data []byte) error - Write a .html file to a directory, creating it if necessary.

  err := file.SaveFile("public", "index", []byte("<h1>Hello</h1>"))
  if err != nil {
      log.Fatal(err)
  }
  

• *ReadFile(path string) (bytes.Reader, error) - Load file contents into a bytes.Reader.

  r, err := file.ReadFile("public/index.html")
  if err != nil {
      log.Fatal(err)
  }
  

Math

Helpers for basic numeric calculations and probability checks.

• Min(a, b int) int and Max(a, b int) int - Return the smaller or larger of two integers.

  m := math.Min(3, 5) // 3
  M := math.Max(3, 5) // 5
  _ = m
  _ = M
  

• *FormatNumber(f float64) string - Convert a floating number to a human-friendly string without trailing zeros.

  v := pointers.FromFloat(12.3400)
  s := math.FormatNumber(v) // "12.34"
  _ = s
  

• ChanceOf(p float64) bool - Return true with the given probability using cryptographic randomness.

  if math.ChanceOf(0.1) {
      fmt.Println("10% chance hit")
  }
  

Text

String normalisation helpers.

• Normalize(s string) string - Trim whitespace from each line and remove empty lines.

  clean := text.Normalize(" Line 1 \n\n  Line 2 ")
  _ = clean
  

• SanitizeToCamelCase(s string) string - Create a camelCase identifier suitable for HTML IDs.

  id := text.SanitizeToCamelCase("Example Title") // "exampleTitle"
  

System

Helpers for interacting with environment variables.

• GetEnvOrFail(name string) string - Retrieve a required environment variable or exit the program.

  token := system.GetEnvOrFail("API_TOKEN")
  _ = token
  

• ExpandEnvVar(s string) (string, error) - Expand $VAR style references and trim the result.

  path, _ := system.ExpandEnvVar("$HOME/tmp")
  

Pointers

Convenience functions for obtaining pointers to primitive values.

• FromFloat(f float64) *float64 - Return a pointer to the provided float.

  ptr := pointers.FromFloat(3.14)
  _ = ptr
  

Unexported helpers for strings, integers and booleans exist for internal tests.

Scheduler

Retry-aware scheduling helpers.

• Worker - Runs a periodic scan over pending jobs, applies exponential backoff, and persists attempt results via a repository interface.
• ClaimingRepository (optional) - Lets repositories atomically claim a job before side effects run; when claim is lost, the worker skips dispatch to avoid duplicate execution under contention.

Testing

The tool includes table-driven tests to ensure consistent behavior for a variety of inputs.

Run Tests:

go test ./test -v

Dependencies

• Goldmark - Markdown rendering and parsing.
• html-to-markdown - HTML-to-Markdown conversion and cleaning.
• net/html - HTML parsing and rendering.

Contributing

Contributions are welcome!

1. Fork the repository.
2. Create a new branch (feature/my-feature).
3. Commit changes and submit a pull request.

License

This project is licensed under the MIT License. See the LICENSE file for details.