Documentation
¶
Overview ¶
Package net provides the core download functions
Package net provides net related functions
Index ¶
- Constants
- func DetectSizeWithRoundTripper(targetURL, output string, showProgress, noProxy, insecureSkipVerify bool, ...) (total int64, rangeSupport bool, err error)
- func DetectSizeWithRoundTripperAndAuth(targetURL, output string, showProgress, noProxy, insecureSkipVerify bool, ...) (total int64, rangeSupport bool, err error)
- func DetectSizeWithRoundTripperAndAuthStream(targetURL string, output io.Writer, ...) (total int64, rangeSupport bool, err error)
- func DownloadFileWithMultipleThreadKeepParts(targetURL, targetFilePath string, thread int, keepParts, showProgress bool) (err error)
- func ParseSuggestedFilename(header http.Header, filepath string) (filename string)
- func SetProxy(proxy, proxyAuth string, tr *http.Transport) (err error)
- type ContinueDownloader
- func (c *ContinueDownloader) DownloadWithContinue(targetURL, output string, index, continueAt, end int64, showProgress bool) (err error)
- func (c *ContinueDownloader) DownloadWithContinueAsStream(targetURL string, output io.Writer, index, continueAt, end int64, ...) (err error)
- func (c *ContinueDownloader) GetSuggestedFilename() string
- func (c *ContinueDownloader) WithBasicAuth(username, password string) *ContinueDownloader
- func (c *ContinueDownloader) WithContext(ctx context.Context) *ContinueDownloader
- func (c *ContinueDownloader) WithInsecureSkipVerify(insecureSkipVerify bool) *ContinueDownloader
- func (c *ContinueDownloader) WithMaxAttempts(maxAttempts int) *ContinueDownloader
- func (c *ContinueDownloader) WithProgressIndicator(progress *ProgressIndicator) *ContinueDownloader
- func (c *ContinueDownloader) WithRoundTripper(roundTripper http.RoundTripper) *ContinueDownloader
- func (c *ContinueDownloader) WithTimeout(timeout time.Duration) *ContinueDownloader
- func (c *ContinueDownloader) WithoutProxy(noProxy bool) *ContinueDownloader
- type DownloadError
- type FakeReader
- type HTTPDownloader
- type MultiProgressIndicator
- type MultiThreadDownloader
- func (d *MultiThreadDownloader) Download(targetURL, targetFilePath string, thread int) (err error)
- func (d *MultiThreadDownloader) DownloadWithContext(ctx context.Context, targetURL string, outputWriter io.Writer, thread int) (err error)
- func (d *MultiThreadDownloader) GetSuggestedFilename() string
- func (d *MultiThreadDownloader) WithBasicAuth(username, password string) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithBearerToken(bearerToken string) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithInsecureSkipVerify(insecureSkipVerify bool) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithKeepParts(keepParts bool) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithMaxAttempts(maxAttempts int) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithRoundTripper(roundTripper http.RoundTripper) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithShowProgress(showProgress bool) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithTimeout(timeout time.Duration) *MultiThreadDownloader
- func (d *MultiThreadDownloader) WithoutProxy(noProxy bool) *MultiThreadDownloader
- type ProgressIndicator
- type RetryClient
- type SuggestedFilenameAware
Constants ¶
const ( // ContentType is for the http header of content type ContentType = "Content-Type" // ApplicationForm is for the form submit ApplicationForm = "application/x-www-form-urlencoded" )
Variables ¶
This section is empty.
Functions ¶
func DetectSizeWithRoundTripper ¶ added in v0.0.68
func DetectSizeWithRoundTripper(targetURL, output string, showProgress, noProxy, insecureSkipVerify bool, roundTripper http.RoundTripper, timeout time.Duration) (total int64, rangeSupport bool, err error)
DetectSizeWithRoundTripper returns the size of target resource Deprecated, use DetectSizeWithRoundTripperAndAuth instead
func DetectSizeWithRoundTripperAndAuth ¶ added in v0.0.98
func DetectSizeWithRoundTripperAndAuth(targetURL, output string, showProgress, noProxy, insecureSkipVerify bool, roundTripper http.RoundTripper, username, password string, timeout time.Duration) (total int64, rangeSupport bool, err error)
DetectSizeWithRoundTripperAndAuth returns the size of target resource
func DetectSizeWithRoundTripperAndAuthStream ¶ added in v0.0.99
func DetectSizeWithRoundTripperAndAuthStream(targetURL string, output io.Writer, showProgress, noProxy, insecureSkipVerify bool, roundTripper http.RoundTripper, username, password string, timeout time.Duration) (total int64, rangeSupport bool, err error)
DetectSizeWithRoundTripperAndAuthStream returns the size of target resource
func DownloadFileWithMultipleThreadKeepParts ¶
func DownloadFileWithMultipleThreadKeepParts(targetURL, targetFilePath string, thread int, keepParts, showProgress bool) (err error)
DownloadFileWithMultipleThreadKeepParts downloads the files with multiple threads
func ParseSuggestedFilename ¶ added in v0.0.88
ParseSuggestedFilename parse the filename from resp header,More details from https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
Types ¶
type ContinueDownloader ¶ added in v0.0.32
type ContinueDownloader struct {
UserName, Password string
Timeout time.Duration
Context context.Context
// contains filtered or unexported fields
}
ContinueDownloader is a downloader which support continuously download
func (*ContinueDownloader) DownloadWithContinue ¶ added in v0.0.32
func (c *ContinueDownloader) DownloadWithContinue(targetURL, output string, index, continueAt, end int64, showProgress bool) (err error)
DownloadWithContinue downloads the files continuously. When the transfer breaks in the middle (e.g. an unexpected EOF) it automatically retries and resumes from the bytes that are already on the local disk via range requests, until the attempts are exhausted or the error is not retryable.
func (*ContinueDownloader) DownloadWithContinueAsStream ¶ added in v0.0.99
func (c *ContinueDownloader) DownloadWithContinueAsStream(targetURL string, output io.Writer, index, continueAt, end int64, showProgress bool) (err error)
DownloadWithContinueAsStream downloads the files continuously
func (*ContinueDownloader) GetSuggestedFilename ¶ added in v0.0.81
func (c *ContinueDownloader) GetSuggestedFilename() string
GetSuggestedFilename returns the suggested filename
func (*ContinueDownloader) WithBasicAuth ¶ added in v0.0.98
func (c *ContinueDownloader) WithBasicAuth(username, password string) *ContinueDownloader
WithBasicAuth sets the basic auth
func (*ContinueDownloader) WithContext ¶ added in v0.0.83
func (c *ContinueDownloader) WithContext(ctx context.Context) *ContinueDownloader
WithContext sets the context
func (*ContinueDownloader) WithInsecureSkipVerify ¶ added in v0.0.81
func (c *ContinueDownloader) WithInsecureSkipVerify(insecureSkipVerify bool) *ContinueDownloader
WithInsecureSkipVerify set if skip the insecure verify
func (*ContinueDownloader) WithMaxAttempts ¶ added in v0.0.102
func (c *ContinueDownloader) WithMaxAttempts(maxAttempts int) *ContinueDownloader
WithMaxAttempts sets the max times a download is attempted when the transfer breaks in the middle, zero or a negative value means no retry
func (*ContinueDownloader) WithProgressIndicator ¶ added in v0.0.102
func (c *ContinueDownloader) WithProgressIndicator(progress *ProgressIndicator) *ContinueDownloader
WithProgressIndicator sets a shared progress indicator which aggregates the progress of all the downloads that share it. Passing an indicator that was not initialized (Init) disables the progress of the downloads
func (*ContinueDownloader) WithRoundTripper ¶ added in v0.0.69
func (c *ContinueDownloader) WithRoundTripper(roundTripper http.RoundTripper) *ContinueDownloader
WithRoundTripper set WithRoundTripper
func (*ContinueDownloader) WithTimeout ¶ added in v0.0.83
func (c *ContinueDownloader) WithTimeout(timeout time.Duration) *ContinueDownloader
WithTimeout sets the timeout
func (*ContinueDownloader) WithoutProxy ¶ added in v0.0.69
func (c *ContinueDownloader) WithoutProxy(noProxy bool) *ContinueDownloader
WithoutProxy indicate no HTTP proxy use
type DownloadError ¶
DownloadError represents the error of HTTP download
type FakeReader ¶ added in v0.0.77
type FakeReader struct {
ExpectErr error
}
FakeReader is a fake reader for the test purpose
type HTTPDownloader ¶
type HTTPDownloader struct {
TargetFilePath string
URL string
ShowProgress bool
InsecureSkipVerify bool
Context context.Context
UserName string
Password string
NoProxy bool
Proxy string
ProxyAuth string
Header map[string]string
// PreStart returns false will don't continue
PreStart func(*http.Response) bool
Thread int
Title string
Timeout time.Duration
Debug bool
RoundTripper http.RoundTripper
// Append indicates to append the downloaded data to the target file
// instead of truncating it, used by the resumed downloads
Append bool
// ProgressBase is the number of bytes that were already downloaded
// before this download started, used by the progress indicator
ProgressBase int64
// contains filtered or unexported fields
}
HTTPDownloader is the downloader for http request
func (*HTTPDownloader) DownloadAsStream ¶ added in v0.0.99
func (h *HTTPDownloader) DownloadAsStream(writer io.Writer) (err error)
DownloadAsStream downloads the file as stream
func (*HTTPDownloader) DownloadFile ¶
func (h *HTTPDownloader) DownloadFile() (err error)
DownloadFile download a file with the progress
func (*HTTPDownloader) GetSuggestedFilename ¶ added in v0.0.81
func (h *HTTPDownloader) GetSuggestedFilename() string
GetSuggestedFilename returns the suggested filename which comes from the HTTP response header. Returns empty string if the filename is same with the given name.
type MultiProgressIndicator ¶ added in v0.0.102
type MultiProgressIndicator struct {
// contains filtered or unexported fields
}
MultiProgressIndicator renders one progress bar per download part, every bar stays on its own terminal line. All the bars and the relative cursor movements go through a single locked renderer, so the lines never garble each other - unlike bars rendered independently. It is only suitable for an interactive terminal, use a single shared ProgressIndicator instead when the output is piped or redirected.
func NewMultiProgressIndicator ¶ added in v0.0.102
func NewMultiProgressIndicator() *MultiProgressIndicator
NewMultiProgressIndicator creates an empty multi-bar group. The cursor is expected to stay at the beginning of a fresh line.
func (*MultiProgressIndicator) Close ¶ added in v0.0.102
func (m *MultiProgressIndicator) Close()
Close finishes all the bars and leaves the cursor below them.
func (*MultiProgressIndicator) Register ¶ added in v0.0.102
func (m *MultiProgressIndicator) Register(title string, total float64) *ProgressIndicator
Register allocates the next terminal line for a new bar and returns the indicator of it.
type MultiThreadDownloader ¶ added in v0.0.69
type MultiThreadDownloader struct {
// contains filtered or unexported fields
}
MultiThreadDownloader is a download with multi-thread
func (*MultiThreadDownloader) Download ¶ added in v0.0.69
func (d *MultiThreadDownloader) Download(targetURL, targetFilePath string, thread int) (err error)
Download starts to download the target URL
func (*MultiThreadDownloader) DownloadWithContext ¶ added in v0.0.99
func (d *MultiThreadDownloader) DownloadWithContext(ctx context.Context, targetURL string, outputWriter io.Writer, thread int) (err error)
DownloadWithContext starts to download the target URL with context
func (*MultiThreadDownloader) GetSuggestedFilename ¶ added in v0.0.81
func (d *MultiThreadDownloader) GetSuggestedFilename() string
GetSuggestedFilename returns the suggested filename
func (*MultiThreadDownloader) WithBasicAuth ¶ added in v0.0.98
func (d *MultiThreadDownloader) WithBasicAuth(username, password string) *MultiThreadDownloader
WithBasicAuth sets the basic auth
func (*MultiThreadDownloader) WithBearerToken ¶ added in v0.0.99
func (d *MultiThreadDownloader) WithBearerToken(bearerToken string) *MultiThreadDownloader
WithBearerToken sets the bearer token
func (*MultiThreadDownloader) WithInsecureSkipVerify ¶ added in v0.0.81
func (d *MultiThreadDownloader) WithInsecureSkipVerify(insecureSkipVerify bool) *MultiThreadDownloader
WithInsecureSkipVerify set if skip the insecure verify
func (*MultiThreadDownloader) WithKeepParts ¶ added in v0.0.69
func (d *MultiThreadDownloader) WithKeepParts(keepParts bool) *MultiThreadDownloader
WithKeepParts indicates if keeping the part files
func (*MultiThreadDownloader) WithMaxAttempts ¶ added in v0.0.102
func (d *MultiThreadDownloader) WithMaxAttempts(maxAttempts int) *MultiThreadDownloader
WithMaxAttempts sets the max times a download part is attempted when the transfer breaks in the middle, zero or a negative value means no retry
func (*MultiThreadDownloader) WithRoundTripper ¶ added in v0.0.69
func (d *MultiThreadDownloader) WithRoundTripper(roundTripper http.RoundTripper) *MultiThreadDownloader
WithRoundTripper sets RoundTripper
func (*MultiThreadDownloader) WithShowProgress ¶ added in v0.0.69
func (d *MultiThreadDownloader) WithShowProgress(showProgress bool) *MultiThreadDownloader
WithShowProgress indicate if show the download progress
func (*MultiThreadDownloader) WithTimeout ¶ added in v0.0.83
func (d *MultiThreadDownloader) WithTimeout(timeout time.Duration) *MultiThreadDownloader
WithTimeout sets the timeout
func (*MultiThreadDownloader) WithoutProxy ¶ added in v0.0.69
func (d *MultiThreadDownloader) WithoutProxy(noProxy bool) *MultiThreadDownloader
WithoutProxy indicates not use HTTP proxy
type ProgressIndicator ¶
type ProgressIndicator struct {
Writer io.Writer
Reader io.Reader
Title string
// Total is the expected total size in bytes. Zero or a negative value
// means the size is unknown, in that case the bar works in the spinner
// mode which still reports the transferred bytes.
Total float64
// contains filtered or unexported fields
}
ProgressIndicator hold the progress of io operation.
Multiple indicators are allowed to share one underlying progress bar, for example the parts of a multi-thread download can feed a single aggregated bar (see Share) when the output is not an interactive terminal. On an interactive terminal use MultiProgressIndicator instead, it renders one bar per part on dedicated terminal lines.
func (*ProgressIndicator) Close ¶ added in v0.0.32
func (i *ProgressIndicator) Close()
Close shutdowns the progress rendering. Only the owner of the underlying bar finishes it, shared indicators are not affected.
func (*ProgressIndicator) Init ¶
func (i *ProgressIndicator) Init()
Init set the default value for progress indicator
func (*ProgressIndicator) Read ¶
func (i *ProgressIndicator) Read(p []byte) (n int, err error)
Read reads the progress
func (*ProgressIndicator) Share ¶ added in v0.0.102
func (i *ProgressIndicator) Share() *ProgressIndicator
Share returns a new indicator which feeds the same progress bar with this indicator. The returned indicator does not own the bar, which means closing it does not finish the bar. Typically used by the multi-thread downloads: all the parts write into one aggregated bar. It returns nil if the current indicator was not initialized (Init), so that the progress can be disabled globally.
type RetryClient ¶
RetryClient is the wrap of http.Client
func NewRetryClient ¶ added in v0.0.83
func NewRetryClient(client http.Client) *RetryClient
NewRetryClient creates the instance of RetryClient
type SuggestedFilenameAware ¶ added in v0.0.81
type SuggestedFilenameAware interface {
GetSuggestedFilename() string
}
SuggestedFilenameAware is the interface for getting suggested filename