README
¶
gopher-parse-sitemap
A high effective golang library for parsing big-sized sitemaps and avoiding high memory usage. The sitemap parser was written on golang without external dependencies. See https://www.sitemaps.org/ for more information about the sitemap format.
Why yet another sitemaps parsing library?
Time by time needs to parse really huge sitemaps. If you just unmarshal the whole file to an array of structures it produces high memory usage and the application can crash due to OOM (out of memory error).
The solution is to handle sitemap entries on the fly. That is read one entity, consume it, repeat while there are unhandled items in the sitemap.
err := sitemap.ParseFromFile("./testdata/sitemap.xml", func(e Entry) error {
return fmt.Println(e.GetLocation())
})
I need parse only small and medium-sized sitemaps. Should I use this library?
Yes. Of course, you can just load a sitemap to memory.
result := make([]string, 0, 0)
err := sitemap.ParseIndexFromFile("./testdata/sitemap-index.xml", func(e IndexEntry) error {
result = append(result, e.GetLocation())
return nil
})
But if you are pretty sure that you don't need to handle big-sized sitemaps, may be better to choose a library with simpler and more suitable API. In that case, you can try projects like https://github.com/yterajima/go-sitemap, https://github.com/snabb/sitemap, and https://github.com/decaseal/go-sitemap-parser.
Install
Installation is pretty easy, just do:
go get -u github.com/oxffaa/gopher-parse-sitemap
After that import it:
import "github.com/oxffaa/gopher-parse-sitemap"
Well done, you can start to create something awesome.
Documentation
Please, see here for documentation.
Documentation
¶
Overview ¶
Package sitemap provides primitives for high effective parsing of huge sitemap files.
Index ¶
- func ParseFromFile(sitemapPath string, handlerSitemap func(entry.Sitemap) error, ...) error
- func ParseFromReader(reader io.Reader, handlerSitemap func(entry.Sitemap) error, ...) (err error)
- func ParseFromSite(url string, handlerSitemap func(entry.Sitemap) error, ...) error
- func SetHttpClientCompressedTrafficInstance(c *HttpClientCompressedTraffic)
- func SetHttpClientUserAgent(ua string)
- func SetIsCompressedTraffic(b bool)
- type HttpClientCompressedTraffic
- type HttpClientInterface
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ParseFromFile ¶
func ParseFromFile( sitemapPath string, handlerSitemap func(entry.Sitemap) error, handlerSitemapIndex func(entry.SitemapIndex) error, handlerUrl func(entry.Url) error, handlerUrlSet func(entry.UrlSet) error, interrupter func() error, ) error
ParseFromFile reads sitemap from a file, parses it and for each sitemap entry calls the consumer's functions.
Example ¶
* Examples
if err := ParseFromFile(
"./testdata/sitemap.xml",
nil,
nil,
func(e entry.Url) error {
fmt.Println(e.GetLocation())
return nil
},
nil,
nil,
); err != nil {
panic(err)
}
func ParseFromReader ¶
func ParseFromReader( reader io.Reader, handlerSitemap func(entry.Sitemap) error, handlerSitemapIndex func(entry.SitemapIndex) error, handlerUrl func(entry.Url) error, handlerUrlSet func(entry.UrlSet) error, interrupter func() error, ) (err error)
Parse parses data which provides by the reader and for each sitemap entry calls the consumer's functions.
func ParseFromSite ¶
func ParseFromSite( url string, handlerSitemap func(entry.Sitemap) error, handlerSitemapIndex func(entry.SitemapIndex) error, handlerUrl func(entry.Url) error, handlerUrlSet func(entry.UrlSet) error, interrupter func() error, ) error
ParseFromSite downloads sitemap from a site, parses it and for each sitemap entry calls the consumer's functions.
func SetHttpClientCompressedTrafficInstance ¶
func SetHttpClientCompressedTrafficInstance(c *HttpClientCompressedTraffic)
func SetHttpClientUserAgent ¶
func SetHttpClientUserAgent(ua string)
func SetIsCompressedTraffic ¶
func SetIsCompressedTraffic(b bool)
Types ¶
type HttpClientCompressedTraffic ¶
type HttpClientCompressedTraffic struct {
}
type HttpClientInterface ¶
func GetHttpClient ¶
func GetHttpClient() HttpClientInterface
func GetHttpClientCompressedTrafficInstance ¶
func GetHttpClientCompressedTrafficInstance() HttpClientInterface
Source Files
Directories