README
¶
GoOse
HTML Content / Article Extractor in Go
Description
GoOse is a powerful Go library and command-line tool for extracting article content and metadata from HTML pages. This is a Go port of the original "Goose" library, completely rewritten and modernized for contemporary Go development.
Key Features:
- 🚀 Extract clean article text from web pages
- 📰 Extract article metadata (title, description, keywords, images)
- 🖼️ Advanced image extraction and top image detection
- 🎥 Video content detection and extraction
- 🌐 Multi-language support with stopwords
- 🔧 Command-line interface for easy integration
- 📦 Clean library API for programmatic use
- ⚡ High performance with concurrent processing support
Originally licensed to Gravity.com under the Apache License 2.0. Go port written by Antonio Linari.
Installation
As a Library
go get github.com/advancedlogic/GoOse
As a CLI Tool
# Install directly
go install github.com/advancedlogic/GoOse/cmd/goose@latest
# Or build from source
git clone https://github.com/advancedlogic/GoOse.git
cd GoOse
make build
# Binary will be available at ./bin/goose
Quick Start
Command Line Usage
# Extract article from URL (text output)
goose convert https://example.com/article
# Extract article with JSON output
goose convert https://example.com/article --format json
# Save output to file
goose convert https://example.com/article --output article.txt
# Show version
goose version
# Show help
goose help
Library Usage
package main
import (
"fmt"
"log"
"github.com/advancedlogic/GoOse/pkg/goose"
)
func main() {
// Create a new GoOse instance
g := goose.New()
// Extract from URL
article, err := g.ExtractFromURL("https://edition.cnn.com/2012/07/08/opinion/banzi-ted-open-source/index.html")
if err != nil {
log.Fatal(err)
}
// Print extracted content
fmt.Println("Title:", article.Title)
fmt.Println("Description:", article.MetaDescription)
fmt.Println("Keywords:", article.MetaKeywords)
fmt.Println("Content:", article.CleanedText)
fmt.Println("URL:", article.FinalURL)
fmt.Println("Top Image:", article.TopImage)
fmt.Println("Authors:", article.Authors)
fmt.Println("Publish Date:", article.PublishDate)
}
Advanced Configuration
package main
import (
"github.com/advancedlogic/GoOse/pkg/goose"
)
func main() {
// Create configuration
config := goose.Configuration{
Debug: false,
TargetLanguage: "en",
UserAgent: "MyApp/1.0",
Timeout: 30, // seconds
}
// Create GoOse with custom configuration
g := goose.NewWithConfig(config)
// Extract from raw HTML
html := "<html><body><article><h1>Title</h1><p>Content...</p></article></body></html>"
article, err := g.ExtractFromRawHTML(html, "https://example.com")
if err != nil {
// Handle error
}
// Use the extracted article
_ = article
}
Project Structure
GoOse follows standard Go project layout:
├── cmd/goose/ # CLI application
├── pkg/goose/ # Public library API
├── internal/ # Private application code
│ ├── crawler/ # Web crawling logic
│ ├── extractor/ # Content extraction
│ ├── parser/ # HTML parsing utilities
│ ├── types/ # Shared data types
│ └── utils/ # Utility functions
├── docs/ # Documentation
├── sites/ # Test HTML files
└── Makefile # Build automation
Development
Prerequisites
- Go 1.21 or later
- Make (for build automation)
Getting Started
-
Clone the repository:
git clone https://github.com/advancedlogic/GoOse.git cd GoOse -
Install dependencies:
make deps -
Build the project:
make build -
Run tests:
make test -
Run all quality checks:
make qa
Available Make Commands
make help # Show all available commands
make build # Build the CLI binary
make install # Install CLI to GOPATH/bin
make test # Run all tests
make test-race # Run tests with race detection
make coverage # Generate coverage report
make format # Format source code
make lint # Run linters
make qa # Run all quality checks
make clean # Clean build artifacts
make tidy # Clean up go.mod and go.sum
Development Workflow
- Make changes to the code
- Run
make formatto format your code - Run
make qato ensure quality - Run
make testto verify functionality - Commit your changes
API Reference
Main Types
goose.Goose- Main extractor instancegoose.Article- Extracted article datagoose.Configuration- Extractor configuration
Key Methods
goose.New()- Create new extractor with default configgoose.NewWithConfig(config)- Create extractor with custom configExtractFromURL(url)- Extract article from URLExtractFromRawHTML(html, url)- Extract from HTML string
For complete API documentation, run:
go doc github.com/advancedlogic/GoOse/pkg/goose
Contributing
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Make your changes following the coding standards
- Run the full test suite (
make qa) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Please ensure your code:
- ✅ Passes all tests (
make test) - ✅ Follows Go formatting standards (
make format) - ✅ Passes linting checks (
make lint) - ✅ Has appropriate test coverage
- ✅ Includes documentation for public APIs
Roadmap
Current Status
- ✅ Modern Go modules support
- ✅ CLI interface with Cobra
- ✅ Comprehensive test coverage
- ✅ Standard Go project layout
- ✅ Build automation with Make
Planned Improvements
- Enhanced error handling and logging
- Plugin architecture for custom extractors
- Performance optimizations
- Additional output formats (XML, YAML)
- Docker containerization
- Advanced image processing
- Batch processing capabilities
License
Licensed under the Apache License, Version 2.0. See LICENSE for details.
Acknowledgments
- @Martin Angers for goquery
- @Fatih Arslan for set
- Go Team for the amazing language and net/html
- Original Goose contributors at Gravity.com
- Community contributors for ongoing improvements
Directories
Click to show internal directories.
Click to hide internal directories.