Documentation
¶
Index ¶
- Constants
- func ApplyMemoryLimit() (soft int64, container int64, ok bool)
- func Bytes(s uint64) string
- func CreateDir(dir string) (string, error)
- func CreateDirFromFilename(destination string, s string) (string, error)
- func CreateFileWithDir(destination string) (string, error)
- func ExtractCPF(text string) (bool, string)
- func FileExists(path string) bool
- func FileType(path string) (string, error)
- func Float64ToTime(f float64) time.Time
- func FormatInt64Comma(n int64) string
- func FormatIntComma(n int) string
- func GetHash(data []byte) string
- func GetHashFromFile(file_path string) (string, error)
- func GetHashFromValues(values ...interface{}) string
- func GetMimeType(s string) (string, error)
- func HammingDistance(hash1, hash2 []byte) (int, error)
- func HumanateBytes(s uint64, base float64, sizes []string) string
- func IBytes(s uint64) string
- func IpsInCIDR(cidr string) ([]string, error)
- func LeftTrucate(s string, max int) string
- func MoveFile(sourcePath, destPath string) error
- func ParseBytes(s string) (uint64, error)
- func ParsePerceptionHash(hashStr string) ([]byte, error)
- func ReadTextFile(file_path string) (string, error)
- func RemoveFolder(path string) error
- func SafeFileName(s string) string
- func SafeFileNameWithRnd(s string) string
- func SanitizeUTF8(s string) string
- func ShuffleStr(slice []string)
- func SliceHasInt(slice []int, item int) bool
- func SliceHasStr(slice []string, item string) bool
- func TempFileName(base_path, prefix, suffix string) string
- func UniqueIntSlice(slice []int) []int
- func Unzip(src, dest string) error
- type HammingGroup
Constants ¶
const ( Byte = 1 << (iota * 10) KiByte MiByte GiByte TiByte PiByte EiByte )
IEC Sizes. kibis of bits
const ( IByte = 1 KByte = IByte * 1000 MByte = KByte * 1000 GByte = MByte * 1000 TByte = GByte * 1000 PByte = TByte * 1000 EByte = PByte * 1000 )
SI Sizes.
const IntelXSearchResultsFile = "intelx_search.json"
IntelXSearchResultsFile is the name of the JSON file the downloader ships inside the package alongside Info.csv, holding the raw records returned by the IntelX intelligent/search. The parser scans it for content like any data file, so intel that only appears in the search metadata is still captured.
Variables ¶
This section is empty.
Functions ¶
func ApplyMemoryLimit ¶ added in v0.1.48
ApplyMemoryLimit points the Go garbage collector at the memory limit of the cgroup this process runs in, and reports the soft limit it set together with the container limit it was derived from.
Without it the collector has no idea a hard cap exists: at the default GOGC=100 it lets the heap grow to twice the live set before collecting, so a live set of half the container's memory is enough for the kernel OOM killer to end the run — which is exactly what happens on a long migration, where the live set is large and the allocation rate is enormous. A soft limit makes the collector run more often as the heap approaches the cap instead, trading CPU for staying alive.
It is a soft limit by design: the runtime will exceed it rather than deadlock if the live heap genuinely does not fit, so this cannot mask a real leak, only absorb the collector's headroom.
An explicit GOMEMLIMIT in the environment always wins — the runtime has already applied it and an operator's number should not be second-guessed. Returns ok=false when there is nothing to do: no cgroup limit (a bare host, a non-Linux platform), or GOMEMLIMIT already set.
func Bytes ¶
Bytes produces a human readable representation of an SI size.
See also: ParseBytes.
Bytes(82854982) -> 83 MB
func CreateDir ¶
CreateDir creates a directory if it does not exist, returning the final normalized directory as a result.
func CreateDirFromFilename ¶
func CreateFileWithDir ¶
CreateFileWithDir creates a file, relative to a directory, returning the final normalized path as a result.
func ExtractCPF ¶ added in v0.1.20
func Float64ToTime ¶
Float64ToTime takes a float64 as number of seconds since unix epoch and returns time.Time
example field where this is used (expires field):
https://chromedevtools.github.io/devtools-protocol/tot/Network/#type-Cookie
func FormatInt64Comma ¶
func FormatIntComma ¶
func GetHashFromFile ¶
func GetHashFromValues ¶ added in v0.1.21
func GetHashFromValues(values ...interface{}) string
func GetMimeType ¶
func HammingDistance ¶
HammingDistance calculates the number of differing bits between two byte slices.
func IBytes ¶
IBytes produces a human readable representation of an IEC size.
See also: ParseBytes.
IBytes(82854982) -> 79 MiB
func IpsInCIDR ¶
IpsInCIDR returns a list of usable IP addresses in a given CIDR block excluding network and broadcast addresses for CIDRs larger than /31.
func LeftTrucate ¶
LeftTrucate a string if its more than max
func ParseBytes ¶
ParseBytes parses a string representation of bytes into the number of bytes it represents.
See Also: Bytes, IBytes.
ParseBytes("42 MB") -> 42000000, nil ParseBytes("42 mib") -> 44040192, nil
func ParsePerceptionHash ¶
ParsePerceptionHash converts a perception hash string "p:<hex>" to a byte slice.
func ReadTextFile ¶
func RemoveFolder ¶
func SafeFileName ¶
SafeFileName takes a string and returns a string safe to use as a file name.
func SafeFileNameWithRnd ¶ added in v0.1.21
SafeFileName takes a string and returns a string safe to use as a file name.
func SanitizeUTF8 ¶ added in v0.1.30
SanitizeUTF8 removes null bytes and invalid UTF-8 sequences from a string, ensuring it is safe for PostgreSQL UTF-8 encoding.
func SliceHasInt ¶
SliceHasInt checks if a slice has an int
func SliceHasStr ¶
SliceHasStr checks if a slice has a string
func TempFileName ¶
func UniqueIntSlice ¶
UniqueIntSlice returns a slice of unique ints
Types ¶
type HammingGroup ¶
HammingGroup represents a hash -> group assignment used for inmemory hammingdistance calulations.
Source Files