README
¶
URL Processor Package
The urlprocessor package provides a flexible and extensible system for processing URLs into structured paths, particularly useful for artifact storage and organization.
Overview
This package offers tools to transform URLs into organized file paths while supporting both raw URL preservation and clean structured paths. It's particularly optimized for handling GitHub release URLs but can be extended to support other URL patterns.
Main Components
Registry
The Registry type provides a centralized way to manage URL processors:
registry := urlprocessor.NewRegistry()
result, err := registry.ProcessURLString(urlStr, raw)
PathBuilder
An alternative implementation that provides URL processing capabilities:
builder := urlprocessor.New()
result, err := builder.ProcessURLString(urlStr, raw)
Processor Interface
Custom processors can be implemented using the Processor interface:
type Processor interface {
CanProcess(sourceURL *url.URL) bool
Process(sourceURL *url.URL, raw bool) string
}
Features
- GitHub URL Support: Special handling for GitHub release URLs
- Raw Mode: Preserve complete URL structure in output paths
- Clean Mode: Generate clean, structured paths
- Extensible: Easy to add custom processors for different URL patterns
- Default Fallback: Simple filename extraction for unrecognized URLs
Example Usage
// Create a new registry
registry := urlprocessor.NewRegistry()
// Process a GitHub release URL
url := "https://github.com/bazelbuild/bazel/releases/download/7.2.1/bazel-7.2.1-windows-x86_64.exe"
// Raw mode (preserves full path)
rawPath, _ := registry.ProcessURLString(url, true)
// Output: github.com/bazelbuild/bazel/releases/download/7.2.1/bazel-7.2.1-windows-x86_64.exe
// Clean mode (structured path)
cleanPath, _ := registry.ProcessURLString(url, false)
// Output: github.com/bazelbuild/bazel/7.2.1/bazel-7.2.1-windows-x86_64.exe
Adding Custom Processors
You can extend functionality by implementing the Processor interface or using the PathBuilder.AddProcessor method:
builder := urlprocessor.New()
builder.AddProcessor(
func(u *url.URL) bool { return strings.Contains(u.Host, "example.com") },
func(u *url.URL, raw bool) string { return path.Base(u.Path) },
)
Documentation
¶
Overview ¶
Package urlprocessor provides interfaces and implementations for processing URLs to create structured paths for artifacts.
Package urlprocessor provides URL processing capabilities for transforming repository URLs into structured paths for artifact storage.
Index ¶
- Constants
- type DefaultProcessor
- type GitHubProcessor
- type PathBuilder
- func (p *PathBuilder) AddProcessor(canHandle func(*url.URL) bool, process func(*url.URL, bool) string)
- func (p *PathBuilder) BuildStructuredPath(sourceURL *url.URL, artifactName, sourcePathStrip string, raw bool) (string, error)
- func (p *PathBuilder) ProcessURL(sourceURL *url.URL, raw bool) string
- func (p *PathBuilder) ProcessURLString(urlStr string, raw bool) (string, error)
- func (p *PathBuilder) StripPrefixFromURL(u *url.URL, prefix string) (*url.URL, error)
- type Processor
- type Registry
Constants ¶
const (
// GitHubHost is the standard GitHub hostname.
GitHubHost = "github.com"
)
Constants for URL processing.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DefaultProcessor ¶
type DefaultProcessor struct{}
DefaultProcessor is a fallback processor that simply extracts the filename.
func (*DefaultProcessor) CanProcess ¶
func (p *DefaultProcessor) CanProcess(sourceURL *url.URL) bool
CanProcess returns true for any URL, as this is the fallback processor.
type GitHubProcessor ¶
type GitHubProcessor struct {
// contains filtered or unexported fields
}
GitHubProcessor handles GitHub release URLs and creates structured paths.
func NewGitHubProcessor ¶ added in v0.3.1
func NewGitHubProcessor(logger *logrus.Logger) *GitHubProcessor
NewGitHubProcessor creates a new GitHubProcessor with optional logger.
func (*GitHubProcessor) CanProcess ¶
func (p *GitHubProcessor) CanProcess(sourceURL *url.URL) bool
CanProcess checks if the URL is from GitHub.
func (*GitHubProcessor) Process ¶
func (p *GitHubProcessor) Process(sourceURL *url.URL, raw bool) string
Process transforms a GitHub URL into a structured path. If raw=true, it preserves the complete URL structure. If raw=false, it creates a cleaner structure like github.com/owner/repo/version/filename.
type PathBuilder ¶
type PathBuilder struct {
// contains filtered or unexported fields
}
PathBuilder transforms URLs into structured storage paths. It provides both raw path preservation and cleaner structured paths.
func NewWithLogger ¶ added in v0.3.1
func NewWithLogger(logger *logrus.Logger) *PathBuilder
NewWithLogger creates a new PathBuilder with a logger and default processors.
func (*PathBuilder) AddProcessor ¶
func (p *PathBuilder) AddProcessor(canHandle func(*url.URL) bool, process func(*url.URL, bool) string)
AddProcessor adds a custom URL processor to the builder.
func (*PathBuilder) BuildStructuredPath ¶ added in v0.5.3
func (p *PathBuilder) BuildStructuredPath(sourceURL *url.URL, artifactName, sourcePathStrip string, raw bool) (string, error)
BuildStructuredPath creates the structured path component for an artifact with optional prefix stripping.
func (*PathBuilder) ProcessURL ¶
func (p *PathBuilder) ProcessURL(sourceURL *url.URL, raw bool) string
ProcessURL finds the appropriate processor for the URL and returns the resulting path.
func (*PathBuilder) ProcessURLString ¶
func (p *PathBuilder) ProcessURLString(urlStr string, raw bool) (string, error)
ProcessURLString parses a URL string and processes it.
func (*PathBuilder) StripPrefixFromURL ¶ added in v0.5.3
StripPrefixFromURL removes the configured prefix from the given URL's Host + Path. It returns a new URL instance representing the content after stripping, or the original URL if no stripping occurs.
type Processor ¶
type Processor interface {
// CanProcess checks if this processor can handle the given URL
CanProcess(sourceURL *url.URL) bool
// Process transforms a source URL into a structured path
// If raw is true, it preserves the complete URL structure
// If raw is false, it creates a clean, structured path
Process(sourceURL *url.URL, raw bool) string
}
Processor defines the interface for URL processors that can recognize and transform URLs into structured paths for storage.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
a centralized way to process URLs.
func NewRegistry ¶
func NewRegistry() *Registry
NewRegistry creates a new processor registry with default processors.
func (*Registry) AddProcessor ¶
AddProcessor adds a new processor to the registry.
func (*Registry) ProcessURL ¶
ProcessURL processes a URL with the appropriate processor.
Source Files