README
¶
ThreatWinds Utils - Common Helper Functions
A collection of utility functions used across the ThreatWinds Go SDK and applications, providing simplified operations for type casting, file management, HTTP requests, and more.
🎯 Features
- 🔄 Robust Type Casting: Safe conversion between common Go types (int64, float64, bool, string) with the
castspackage. - 🌐 Simplified HTTP Requests: Generic-based
DoReqfunction for easy API consumption with built-in unmarshalling and error handling. - 📁 File & Folder Management: Helpers for creating, deleting, and listing files and directories.
- 🧬 JSON & ProtoJSON Helpers: Simplified serialization and deserialization, including support for Protocol Buffers.
- ⛓️ Pointer Helpers: Fast generation of pointers for primitive types.
📦 Installation
go get github.com/threatwinds/go-sdk/utils
🚀 Quick Start
Safe Type Casting
package main
import (
"fmt"
"github.com/threatwinds/go-sdk/utils"
)
func main() {
// Cast from interface to int64
var val interface{} = "123"
i64 := utils.CastInt64(val)
fmt.Println(i64) // 123
// Cast from interface to bool
var bVal interface{} = "true"
b := utils.CastBool(bVal)
fmt.Println(b) // true
}
Making an HTTP Request
package main
import (
"github.com/threatwinds/go-sdk/utils"
)
type MyResponse struct {
Status string `json:"status"`
}
func main() {
url := "https://api.example.com/data"
headers := map[string]string{"Authorization": "Bearer token"}
// DoReq[T] automatically unmarshals JSON response to struct T
resp, statusCode, err := utils.DoReq[MyResponse](url, nil, "GET", headers, false)
if err != nil {
// Wrap it as usual: your message, your args and your status are kept,
// and the upstream's x-error-id (if it sent one) is inherited, so the
// failure keeps one occurrence id across both services.
return catcher.Error("calling example-api failed", err, map[string]any{"status": 502})
}
}
On a response with a status >= 400, DoReq returns a *utils.RemoteError: a plain
error, deliberately not a *catcher.SdkError (which catcher.Error short-circuits
on, discarding the wrapping call's message, args and status). Use errors.As to
reach the upstream's Status and Detail, or utils.SdkErrorFromResponse if you
mean to relay the upstream's status to your own client.
🛠️ Package Modules
casts.go: Safe type conversions frominterface{}(Int64, Float64, Bool, String).request.go: High-level generic HTTP client with security defaults (TLS 1.2+).files.go&folders.go: Basic I/O operations and path manipulation helpers.json.go&protojson.go: Simplified wrappers for standard and protobuf JSON.yaml.go: Simplified YAML parsing and marshaling.gjson.go: Helper to get Go values fromgjson.Result.csv.go: Simplified CSV file reading.network.go: Utilities for IP and network-related operations.pointers.go: Functions likePointerOfto easily get addresses of literals.queue.go: Thread-safe generic queue implementation.retry.go: Lightweight retry logic for simple operations.
🤝 Contribution
Contributions are welcome! Please feel free to submit a Pull Request.
Documentation
¶
Index ¶
- func CastBool(value interface{}) bool
- func CastFloat64(value interface{}) float64
- func CastInt64(value interface{}) int64
- func CastString(value interface{}) string
- func CheckConnectivity(url string, timeout time.Duration) error
- func DoReq[response any](url string, data []byte, method string, headers map[string]string, ...) (response, int, error)
- func Download(url, file string, opts ...DownloadOption) error
- func DownloadStream(url string, opts ...DownloadOption) (io.ReadCloser, error)
- func GetMainIP() (string, error)
- func GetValueOf(value gjson.Result) interface{}
- func ListFiles(route string, filter string) []string
- func PointerOf[t any](s t) *t
- func ProtoMessageToBytes(object protoreflect.ProtoMessage) (*[]byte, error)
- func ProtoMessageToString(object protoreflect.ProtoMessage) (*string, error)
- func ReadCSV(url string) ([][]string, error)
- func ReadJSON[t any](f string) (*t, error)
- func ReadPbYaml(f string) ([]byte, error)
- func ReadYaml[t any](f string, jsonMode bool) (*t, error)
- func SanitizeField(s *string)
- func StringToProtoMessage(str *string, object protoreflect.ProtoMessage) error
- func ValidateFilePath(path string) error
- func ValidateReservedField(f string, allowEmpty bool) error
- type DownloadOption
- type Folder
- type RemoteError
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CastBool ¶
func CastBool(value interface{}) bool
CastBool attempts to cast an interface{} to a bool. It supports the following types: - bool: returns the value directly. - int, int64, float64: returns true if the value is non-zero, false otherwise. - string: attempts to parse the string as a boolean using strconv.ParseBool. For any other type or if parsing fails, it returns false.
Parameters:
- value: The value to be cast to bool.
Returns:
- The bool representation of the value, or false if the value cannot be cast.
func CastFloat64 ¶
func CastFloat64(value interface{}) float64
CastFloat64 attempts to cast an interface{} to a float64. It supports the following types: int, int64, float64, and string. If the value is a string, it tries to parse it as a float64. If the conversion is not possible, it returns 0.
Parameters:
- value: The value to be cast to float64.
Returns:
- The float64 representation of the value, or 0 if the value cannot be cast.
func CastInt64 ¶
func CastInt64(value interface{}) int64
CastInt64 attempts to cast a given interface{} value to an int64. It supports the following types: int, int64, float64, and string. If the value is a string, it tries to parse it as an int64. If the value cannot be cast to int64, it returns 0.
Parameters:
- value: The value to be cast to int64.
Returns:
- The int64 representation of the value, or 0 if the value cannot be cast.
func CastString ¶
func CastString(value interface{}) string
CastString attempts to cast an interface{} to a string. If the value is already a string, it returns the value directly. Otherwise, it converts the value to a string using fmt.Sprintf.
Parameters:
- value: The interface{} value to be cast to a string.
Returns:
- A string representation of the input value.
func CheckConnectivity ¶ added in v1.1.7
CheckConnectivity checks if a given URL is reachable within the specified timeout. It sends a HEAD request to minimize bandwidth usage.
The URL is whatever the caller wants reached — a ThreatWinds service, a vendor's endpoint, a captive-portal probe — so no error header is read: this function has no way to know it is talking to a peer that runs catcher.GinError, and adopting an x-error-id on the strength of the header name alone would let any endpoint name one of this org's occurrences. Both failures below therefore carry a locally generated id instead, which is honest about where the id came from and still gives an operator one value to grep by. A HEAD response has no body to read a richer detail from in any case.
func DoReq ¶
func DoReq[response any](url string, data []byte, method string, headers map[string]string, skipTlsVerification bool) (response, int, error)
DoReq sends an HTTP request and processes the response.
This function sends an HTTP request to the specified URL with the given method, data, and headers. It returns the response body unmarshalled into the specified response type, the HTTP status code, and an error if any occurred during the process.
Type Parameters:
- response: The type into which the response body will be unmarshalled.
Parameters:
- url: The URL to which the request is sent.
- data: The request payload as a byte slice.
- method: The HTTP method to use for the request (e.g., "GET", "POST").
- headers: A map of headers to include in the request.
Returns:
- response: The response body unmarshalled into the specified type.
- int: The HTTP status code of the response.
- error: An error if any occurred during the request or response processing, otherwise nil. For a response with a status >= 400 that error is a *RemoteError — a plain error, so wrapping it with catcher.Error/catcher.New builds the caller's own error normally, keeping the caller's message, args and status while inheriting the upstream's occurrence id. Use errors.As to reach the upstream's status and detail; use relay.SdkErrorFromResponse if you mean to relay the upstream's status as your own.
func Download ¶
func Download(url, file string, opts ...DownloadOption) error
Download downloads the content from the specified URL and saves it to the specified file. It returns an error if any error occurs during the process.
Parameters:
- url: The URL from which to download the content.
- file: The path to the file where the content should be saved.
- opts: Optional functional options to configure the download (e.g., WithHeaders, WithTimeout).
Returns:
- error: An error object if an error occurs, otherwise nil.
func DownloadStream ¶ added in v1.1.7
func DownloadStream(url string, opts ...DownloadOption) (io.ReadCloser, error)
DownloadStream downloads the content from the specified URL and returns it as an io.ReadCloser. The caller is responsible for closing the returned stream.
Parameters:
- url: The URL from which to download the content.
- opts: Optional functional options to configure the download (e.g., WithHeaders, WithTimeout).
Returns:
- io.ReadCloser: The response body as a stream.
- error: An error object if an error occurs, otherwise nil.
func GetMainIP ¶
GetMainIP retrieves the main IP address of the local machine by establishing a UDP connection to a remote server (Google's public DNS server in this case). It returns the IP address as a string and an error if any error occurs during the process.
Returns:
- string: The main IP address of the local machine.
- error: An error object if there is an issue obtaining the IP address.
func GetValueOf ¶
GetValueOf returns the Go representation of a gjson.Result value. It converts the gjson.Result to the appropriate Go type based on its type: - For gjson.String, it returns a string. - For gjson.Number, it returns an int if the value is an integer, or a float if it contains a decimal point or a comma. - For gjson.True, it returns true. - For gjson.False, it returns false. - For gjson.JSON, it returns the raw JSON string. - For any other type, it returns an empty string.
func ListFiles ¶
ListFiles walks through the directory specified by the route and returns a slice of file paths that match the given filter, sorted in natural numeric order by filename. The filter should be a file extension (e.g., ".txt").
Parameters:
- route: The root directory to start the file search.
- filter: The file extension to filter files by.
Returns:
- A slice of strings containing the paths of the files that match the filter.
If an error occurs during the file walk, it logs the error and panics if the error is not "no such file or directory".
func PointerOf ¶
func PointerOf[t any](s t) *t
PointerOf takes a value of any type and returns a pointer to that value. This is useful for creating pointers to literals or values that are not already pointers.
Example usage:
intValue := 42 intPointer := PointerOf(intValue)
Type Parameters:
t: The type of the value to be pointed to.
Parameters:
s: The value to create a pointer for.
Returns:
A pointer to the provided value.
func ProtoMessageToBytes ¶ added in v1.1.0
func ProtoMessageToBytes(object protoreflect.ProtoMessage) (*[]byte, error)
ProtoMessageToBytes converts a given ProtoMessage object to its binary representation. It returns a pointer to the binary representation and a pointer to an error if any error occurs during marshaling.
Parameters:
- object: The ProtoMessage object to be converted.
Returns:
- *[]byte: A pointer to the binary representation of the object.
- error: A pointer to an error if an error occurs, otherwise nil.
func ProtoMessageToString ¶ added in v1.1.0
func ProtoMessageToString(object protoreflect.ProtoMessage) (*string, error)
ProtoMessageToString converts a given ProtoMessage object to its JSON string representation. It returns a pointer to the JSON string and a pointer to an error if any error occurs during marshaling.
Parameters:
- object: The ProtoMessage object to be converted.
Returns:
- *string: A pointer to the JSON string representation of the object.
- *error: A pointer to an error if an error occurs, otherwise nil.
func ReadCSV ¶
ReadCSV reads a CSV file from the given URL and returns its contents as a slice of string slices. If an error occurs while opening or reading the file, it logs the error and returns nil.
Parameters:
- url: The path to the CSV file.
Returns:
- [][]string: The contents of the CSV file.
- error: An error object if an error occurs, otherwise nil.
func ReadJSON ¶
ReadJSON reads a JSON file and parses its content into a specified type. The function takes a file path as input and returns a pointer to the parsed value of the specified type and a pointer to an error if an error occurs.
Type Parameters:
t: The type into which the JSON content should be parsed.
Parameters:
f: The file path of the JSON file to be read.
Returns:
*t: A pointer to the parsed value of the specified type. error: An error object if any error occurs during the process.
func ReadPbYaml ¶
ReadPbYaml reads a YAML file, converts its content to JSON, and returns the JSON bytes. If an error occurs while reading the file or converting its content, it returns an error.
Parameters:
- f: The file path of the YAML file to be read.
Returns:
- []byte: The JSON bytes converted from the YAML file.
- error: An error object if an error occurs, otherwise nil.
func ReadYaml ¶
ReadYaml reads a YAML file and converts its content into a specified type. The function can also handle JSON mode if specified.
Type Parameters:
t: The type into which the YAML content will be converted.
Parameters:
f: The file path to the YAML file. jsonMode: A boolean flag indicating whether to use JSON mode for conversion.
Returns:
*t: A pointer to the converted content of type t. error: A pointer to an error object if an error occurs, otherwise nil.
func SanitizeField ¶
func SanitizeField(s *string)
SanitizeField removes characters that are unsafe in a gjson/proto field path. Letters, digits, dots and underscores are kept: underscores are standard JSON key characters and must survive so that rules can reference the original field name (e.g. log.correlationCandidate.<snake_case> markers, audit_verb, etc.).
func StringToProtoMessage ¶ added in v1.1.0
func StringToProtoMessage(str *string, object protoreflect.ProtoMessage) error
StringToProtoMessage parses a JSON-encoded string into a given ProtoMessage object.
Parameters:
- str: A pointer to the JSON-encoded string.
- object: The ProtoMessage object to unmarshal the JSON string into.
Returns:
- error: An error object if the unmarshalling fails, otherwise nil.
func ValidateFilePath ¶
ValidateFilePath validates the file path to ensure it does not contain any invalid characters or directory traversal attempts
func ValidateReservedField ¶
ValidateReservedField validates a field to ensure it is not empty or a reserved field.
Types ¶
type DownloadOption ¶ added in v1.1.7
type DownloadOption func(*downloadConfig)
DownloadOption defines a functional option for configuring the download request.
func WithHeaders ¶ added in v1.1.7
func WithHeaders(headers map[string]string) DownloadOption
WithHeaders sets the headers for the download request.
func WithSkipTlsVerification ¶ added in v1.1.14
func WithSkipTlsVerification(skip bool) DownloadOption
WithSkipTlsVerification sets the skipTlsVerification for the download request.
func WithTimeout ¶ added in v1.1.7
func WithTimeout(timeout time.Duration) DownloadOption
WithTimeout sets the timeout for the download request.
type Folder ¶ added in v1.0.10
type Folder string
func MkdirJoin ¶ added in v1.0.8
MkdirJoin creates a directory structure specified by the joined path of given parts and returns the complete path. It uses `os.MkdirAll` with a default permission of 0755 for creating directories. Returns the resulting path and an error if any occurs during directory creation.
type RemoteError ¶ added in v1.1.29
type RemoteError struct {
// Status is the upstream's HTTP status, exactly as received. It is not
// necessarily a status the relaying service should answer with; see
// relay.SdkErrorFromResponse's note on Args["status"].
Status int
// ID is the upstream's x-error-id when it supplied a well-formed one: the
// occurrence id that lets one failure be grepped across every service's
// logs. Empty when the upstream sent none, or sent one catcher.AdoptErrorID
// refused — this field means "the id the upstream gave us" and nothing
// else, so that a caller can tell an inherited occurrence from a local one.
// CatcherErrorID is what donates an id either way; see it for the fallback.
ID string
// Detail is the upstream's own error text — the first maxErrorDetailSize
// bytes of its response body, or its x-error header when the body is
// empty. Never the whole of an arbitrarily large body.
Detail string
// contains filtered or unexported fields
}
RemoteError is the error DoReq returns when an upstream answers with a status >= 400.
It is deliberately *not* a *catcher.SdkError. catcher.Error short-circuits when the cause it is handed is already an *SdkError: it returns that error unchanged and silently drops the wrapping call's message, args and status override (see catcher.Error's doc comment). A transport helper returning an *SdkError therefore converts every existing `catcher.Error("calling X failed", err, args)` in the org into a no-op wrap — the caller's message, its arg keys and the HTTP status it asked to answer with all vanish, replaced by the upstream's. Worse, it does so only on the HTTP-failure path and not on the transport-failure paths beside it, so a single call site logs two different shapes depending on how the request happened to fail.
A plain error keeps every caller's message, args and status exactly where they were, while catcher.ErrorIDCarrier (implemented below) still carries the upstream's occurrence id into the error the caller builds — which is the whole point of reading x-error-id in the first place.
A caller that genuinely wants to relay the upstream's own status and severity — a proxy, rather than a service making a call of its own — can opt in explicitly with relay.SdkErrorFromResponse.
func RemoteErrorFromResponse ¶ added in v1.1.29
func RemoteErrorFromResponse(resp *http.Response) *RemoteError
RemoteErrorFromResponse reads a failed HTTP response into a *RemoteError: its status, the occurrence id from x-error-id when the upstream sent one this process may adopt (see adoptErrorID), and its own error text from x-error.
It never touches resp.Body — the body may already have been consumed, may be nil on a hand-built response, and is the caller's to read and to close — so Detail comes from the x-error header alone. DoReq, which has the body in hand, gets the richer version through remoteErrorFrom.
A nil response, a nil Header map and absent headers are all handled: a nil response is reported as a 500, since "no response at all" is a local failure and 500 is the status catcher would otherwise default to.
A caller that means to relay the upstream's status and severity as its own — a proxy, rather than a service making a call of its own — wants relay.SdkErrorFromResponse, which builds on this.
func (*RemoteError) CatcherErrorID ¶ added in v1.1.29
func (e *RemoteError) CatcherErrorID() string
CatcherErrorID implements catcher.ErrorIDCarrier, so that a caller wrapping this error with catcher.Error or catcher.New builds its own error — its message, its args, its status — that nonetheless inherits the upstream's occurrence id, rather than minting a second, unrelated id for one failure.
It falls back to the id minted for this response when the upstream supplied none, so the answer to "which occurrence is this?" is never empty. The difference between the two cases is not lost — it is exactly what ID reports — but it is not the caller's problem at the point of wrapping: either way this failure has one id, and the caller's error gets that one rather than a second.
func (*RemoteError) Error ¶ added in v1.1.29
func (e *RemoteError) Error() string
Error renders the failure in the exact shape DoReq has always produced, so existing callers that log this text (or match on it) are unaffected. The only difference is that the detail is now bounded.
Source Files