config

package

v1.4.0 Latest Latest Go to latest Published: Mar 3, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/andeya/pholcus

Links

Open Source Insights

Documentation ¶

Overview ¶

Package config provides software configuration, path, and runtime parameter loading and management.

Constants ¶

View Source

const (
	Version  string = "v1.3.5"                                      // Version number
	Author   string = "andeya"                                      // Author
	Name     string = "Pholcus Ghost Spider Data Collection"        // Software name
	FullName string = Name + "_" + Version + " (by " + Author + ")" // Full name
	Tag      string = "pholcus"                                     // Identifier
)

Software information.

View Source

const (
	WorkRoot      string = Tag + "_pkg"                   // Runtime directory name
	ConfigFile    string = WorkRoot + "/config.ini"       // Config file path
	CacheDir      string = WorkRoot + "/cache"            // Cache directory
	LogPath       string = WorkRoot + "/logs/pholcus.log" // Log file path
	LogAsync      bool   = true                           // Whether to output logs asynchronously
	PhantomJSTemp string = CacheDir                       // Surfer-Phantom: temp dir for JS files
	HistoryTag    string = "history"                      // History record identifier
	HistoryDir    string = WorkRoot + "/" + HistoryTag    // History dir for excel/csv output
	SpiderExt     string = ".pholcus.xml"                 // Dynamic rule extension (recommended)
	SpiderExtOld  string = ".pholcus.html"                // Dynamic rule extension (legacy)
)

Path constants derived from Tag.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type BeanstalkdConfig ¶ added in v1.4.0

type BeanstalkdConfig struct {
	Host string `ini:"host"`
	Tube string `ini:"tube"`
}

type Config ¶ added in v1.4.0

type Config struct {
	CrawlsCap int    `ini:"crawlcap"`
	PhantomJS string `ini:"phantomjs"`
	ProxyFile string `ini:"proxylib"`
	SpiderDir string `ini:"spiderdir"`
	FileDir   string `ini:"fileoutdir"`
	TextDir   string `ini:"textoutdir"`
	DBName    string `ini:"dbname"`

	Mgo        MgoConfig        `ini:"mgo"`
	MySQL      MySQLConfig      `ini:"mysql"`
	Beanstalkd BeanstalkdConfig `ini:"beanstalkd"`
	Kafka      KafkaConfig      `ini:"kafka"`
	Log        LogConfig        `ini:"log"`
	Run        RunConfig        `ini:"run"`
}

Config holds all runtime-configurable values, initialized with defaults. Fields are overwritten by LoadConfig() from the INI file via struct tags.

func Conf ¶ added in v1.4.0

func Conf() *Config

Conf returns the global configuration pointer, loading from INI on first access.

type KafkaConfig ¶ added in v1.4.0

type KafkaConfig struct {
	Brokers string `ini:"brokers"`
}

type LogConfig ¶ added in v1.4.0

type LogConfig struct {
	Cap              int64  `ini:"cap"`
	LevelStr         string `ini:"level"`
	ConsoleLevelStr  string `ini:"consolelevel"`
	FeedbackLevelStr string `ini:"feedbacklevel"`
	LineInfo         bool   `ini:"lineinfo"`
	Save             bool   `ini:"save"`
}

func (*LogConfig) ConsoleLevel ¶ added in v1.4.0

func (c *LogConfig) ConsoleLevel() int

ConsoleLevel returns the console log level, clamped to at least the global level.

func (*LogConfig) FeedbackLevel ¶ added in v1.4.0

func (c *LogConfig) FeedbackLevel() int

FeedbackLevel returns the feedback log level, clamped to at least the global level.

func (*LogConfig) Level ¶ added in v1.4.0

func (c *LogConfig) Level() int

Level returns the global log level as int.

type MgoConfig ¶ added in v1.4.0

type MgoConfig struct {
	ConnStr       string `ini:"connstring"`
	ConnCap       int    `ini:"conncap"`
	ConnGCSeconds int64  `ini:"conngcsecond"`
}

type MySQLConfig ¶ added in v1.4.0

type MySQLConfig struct {
	ConnStr          string `ini:"connstring"`
	ConnCap          int    `ini:"conncap"`
	MaxAllowedPacket int    `ini:"maxallowedpacket"`
}

type RunConfig ¶ added in v1.4.0

type RunConfig struct {
	Mode           int    `ini:"mode"`
	Port           int    `ini:"port"`
	Master         string `ini:"master"`
	ThreadNum      int    `ini:"thread"`
	Pausetime      int64  `ini:"pause"`
	OutType        string `ini:"outtype"`
	BatchCap       int    `ini:"batchcap"`
	Limit          int64  `ini:"limit"`
	ProxyMinute    int64  `ini:"proxyminute"`
	SuccessInherit bool   `ini:"success"`
	FailureInherit bool   `ini:"failure"`
}

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL