Documentation
¶
Overview ¶
Package walmart is the library behind the walmart command line: an HTTP client for Walmart's public web pages and typeahead host, an optional Affiliate API backend, and the typed records every command emits.
Walmart renders its public pages server-side and ships the data as a __NEXT_DATA__ JSON island, so this client GETs pages and reads that island; the typeahead host answers plain JSON. Walmart fronts its estate with the PerimeterX (HUMAN) bot manager, which walls almost everything from datacenter IPs: only the typeahead host and the homepage answer anonymously, while the product page (/ip/), keyword search (/search), the category pages (/cp/, /browse), and the store pages (/store/) return the "Robot or human?" challenge. So the client is two-layered: a reliable core over the open surfaces, and a best-effort layer over the walled ones that returns ErrBlocked when the wall is up and, if the operator has set Affiliate API credentials, falls back to that documented backend. Each surface lives in its own file (search.go, product.go, category.go, store.go, deals.go, suggest.go) with its parsing and record mapping; this file holds the shared web client.
Index ¶
- Constants
- Variables
- func Defaults(c *kit.Config)
- func Identity() kit.Identity
- func URLFor(kind, id string) string
- type Category
- type Client
- func (c *Client) CategoryBrowse(ctx context.Context, ref string, limit int) ([]*Listing, error)
- func (c *Client) CategoryTree(ctx context.Context, ref string, limit int) ([]*Category, error)
- func (c *Client) ClearCache() error
- func (c *Client) Deals(ctx context.Context, limit int) ([]*Deal, error)
- func (c *Client) FindStores(ctx context.Context, zip string, limit int) ([]*Store, error)
- func (c *Client) GetCategory(ctx context.Context, ref string) (*Category, error)
- func (c *Client) GetProduct(ctx context.Context, ref string) (*Product, error)
- func (c *Client) GetStore(ctx context.Context, ref string) (*Store, error)
- func (c *Client) Search(ctx context.Context, query string, limit int) ([]*Listing, error)
- func (c *Client) Suggest(ctx context.Context, prefix string, limit int) ([]*Suggestion, error)
- func (c *Client) Trending(ctx context.Context, limit int) ([]*Listing, error)
- type Config
- type Deal
- type Domain
- type Listing
- type Product
- type Ref
- type Store
- type Suggestion
Constants ¶
const ( // DefaultDelay is the minimum gap between requests. Walmart is touchy, so a // one-second pace reads steadily without leaning on it. DefaultDelay = 1 * time.Second DefaultRetries = 3 DefaultTimeout = 30 * time.Second DefaultCacheTTL = 24 * time.Hour )
Defaults for the polite client.
const BaseURL = "https://" + Host
BaseURL is the root every product, store, and category URL is built from.
const DefaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " +
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
DefaultUserAgent is sent with every page request. Walmart serves its public pages to a normal browser; a browser User-Agent is what keeps a logged-out reader looking like one. Override it with --user-agent.
const Host = "www.walmart.com"
Host is the Walmart hostname this client builds page URLs from and the host the URI driver in domain.go claims.
Variables ¶
var ( // ErrNotFound is a missing entity: an unknown item id, an unknown store, or a // bad category. Walmart serves these as a 404 or a not-found shell page. Exit // code 6. ErrNotFound = errors.New("not found") // ErrRateLimited is a sustained HTTP 429 after the client's own retries. Slow // down with --rate. Exit code 5. ErrRateLimited = errors.New("rate limited") // ErrBlocked is the PerimeterX bot wall: a 403 or 412 interstitial, the // "Robot or human?" challenge page served with a 200, or an implausibly small // body on a page that should be large. Walmart walls the product page, search, // the category pages, and the store pages from datacenter IPs, so this is // expected there. The message names the two remedies (a residential IP, or // Affiliate API credentials). Exit code 4. ErrBlocked = errors.New("blocked by Walmart's bot wall (retry from a residential network, " + "or set WALMART_CONSUMER_ID and WALMART_PRIVATE_KEY to use the Affiliate API)") )
The library reports its outcomes as a few sentinel errors. domain.go's mapErr translates each into the kit error kind that carries the matching exit code, so the standalone binary and a host agree on what a wall, a throttle, and a miss mean.
Functions ¶
func Defaults ¶
Defaults seeds the framework baseline with walmart's own values, so an unset --rate or --timeout uses the walmart default rather than the generic kit one. It is passed to kit.New via kit.WithDefaults.
Types ¶
type Category ¶
type Category struct {
ID string `json:"id" kit:"id"`
Name string `json:"name"`
Parent string `json:"parent,omitempty"` // parent category name (from the trail)
ParentID string `json:"parent_id,omitempty" table:"-" kit:"link,kind=walmart/category"` // edge up to the parent node
Children []string `json:"children,omitempty" table:"-" kit:"link,kind=walmart/category"` // edges down to the child nodes
Trail []string `json:"trail,omitempty" table:"-"` // the full ancestor path, root first, leaf last
URL string `json:"url"`
}
Category is a category node, emitted by category show and category tree. The ParentID and Children edges make the taxonomy walkable in both directions, so a host can recurse the department tree from any node to reconstruct it.
type Client ¶
type Client struct {
HTTP *http.Client
BaseURL string
UserAgent string
Delay time.Duration
Retries int
// SuggestURL is the typeahead endpoint. It defaults to the public host; tests
// point it at an httptest server.
SuggestURL string
// contains filtered or unexported fields
}
Client talks to Walmart's public web pages. It paces requests, retries the transient failures, detects the bot wall, and caches response bodies on disk keyed by the request URL.
func ClientFromConfig ¶
ClientFromConfig maps the framework config onto a walmart.Config and returns a client. The affiliate credentials are read from the environment, not flags, so they never land in shell history.
func (*Client) CategoryBrowse ¶
CategoryBrowse returns the items in a category.
func (*Client) CategoryTree ¶
CategoryTree returns the child categories of a node. With API credentials it reads the taxonomy; without them it scans the department links on the homepage (for the top level) or the category page, which works when that page is served.
func (*Client) ClearCache ¶
ClearCache removes the on-disk cache.
func (*Client) FindStores ¶
FindStores returns the stores near a ZIP code.
func (*Client) GetCategory ¶
GetCategory returns a category's metadata: its name and its trail. It reads the API taxonomy when configured, since the web page is walled.
func (*Client) GetProduct ¶
GetProduct returns one item by id (or /ip/ URL).
func (*Client) GetStore ¶
GetStore returns one store's profile by id (or /store/ URL), read from the page when Walmart serves it.
type Config ¶
type Config struct {
UserAgent string
// Zip is the ZIP context for the store locator and the default-store price on
// the API path. Empty leaves it to Walmart's default store.
Zip string
// The affiliate credentials enable the API fallback for the walled surfaces.
// ConsumerID with one of PrivateKey or PrivateKeyFile turns it on; empty
// leaves the fallback off, and the walled commands then report the bot wall
// rather than guessing.
ConsumerID string
PrivateKey string // PEM contents
PrivateKeyFile string // path to a PEM file, used when PrivateKey is empty
KeyVersion string
PublisherID string
// Delay is the minimum gap between requests. Zero means no pacing.
Delay time.Duration
Retries int
Timeout time.Duration
// BaseURL is the site root. Empty uses the public site; tests point it at an
// httptest server.
BaseURL string
// CacheDir is where responses are cached. Empty disables the cache, as does
// NoCache.
CacheDir string
CacheTTL time.Duration
NoCache bool
// Refresh fetches fresh copies and rewrites the cache, ignoring any hit.
Refresh bool
}
Config carries the knobs the client reads. It is built from the kit framework config in ClientFromConfig, so a --rate or --timeout on the command line and the same value resolved by a host both land here.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns the baseline configuration: a browser User-Agent, a one-second pace, three retries, a 30s timeout, the "1" key version, and a one-day cache.
type Deal ¶
type Deal struct {
ID string `json:"id" kit:"id"`
Title string `json:"title,omitempty" table:",truncate"`
Brand string `json:"brand,omitempty"`
Price float64 `json:"price,omitempty"`
Was float64 `json:"was,omitempty"` // price before the markdown
Save float64 `json:"save,omitempty"` // was minus price
Currency string `json:"currency,omitempty"`
Offer string `json:"offer,omitempty"` // the offer type, e.g. "Rollback" or "Clearance"
Rating float64 `json:"rating,omitempty"`
Reviews int `json:"reviews,omitempty"`
Image string `json:"image,omitempty" table:",truncate"`
URL string `json:"url"`
Item string `json:"item,omitempty" table:"-" kit:"link,kind=walmart/product"` // edge to the full product
}
Deal is a rolled-back or cleared item, emitted by deals.
type Domain ¶
type Domain struct{}
Domain is the walmart driver. It carries no state; the per-run client is built by the factory Register hands kit.
func (Domain) Classify ¶
Classify turns any accepted input into the canonical (type, id), so `ant resolve` and `ant url` touch no network.
func (Domain) Info ¶
func (Domain) Info() kit.DomainInfo
Info describes the scheme, the hostnames a pasted link is matched against, and the identity reused for the binary's help and version.
type Listing ¶
type Listing struct {
ID string `json:"id" kit:"id"`
Title string `json:"title,omitempty" table:",truncate"`
Brand string `json:"brand,omitempty"`
Price float64 `json:"price,omitempty"`
Currency string `json:"currency,omitempty"`
Was float64 `json:"was,omitempty"` // strikethrough / was price
Seller string `json:"seller,omitempty"`
Rating float64 `json:"rating,omitempty"` // average star rating
Reviews int `json:"reviews,omitempty"` // review count
Availability string `json:"availability,omitempty"`
Sponsored bool `json:"sponsored,omitempty"` // an ad tile rather than an organic result
Thumbnail string `json:"thumbnail,omitempty" table:",truncate"`
URL string `json:"url"`
Item string `json:"item,omitempty" table:"-" kit:"link,kind=walmart/product"` // edge to the full product
}
Listing is a summary row in a grid, emitted by search, category browse, and trending. Its id is the product's usItemId, so Item carries the same value as the graph edge to the full product record: a host walks a search result or a category page straight through to walmart://product/<id> and onward.
type Product ¶
type Product struct {
ID string `json:"id" kit:"id"`
Name string `json:"name,omitempty" table:",truncate"`
Brand string `json:"brand,omitempty"`
Model string `json:"model,omitempty"`
UPC string `json:"upc,omitempty"`
Price float64 `json:"price,omitempty"`
Currency string `json:"currency,omitempty"`
Was float64 `json:"was,omitempty"` // was price, when the item is marked down
List float64 `json:"list,omitempty"` // manufacturer list price / MSRP
Availability string `json:"availability,omitempty"`
Seller string `json:"seller,omitempty"`
Rating float64 `json:"rating,omitempty"`
Reviews int `json:"reviews,omitempty"`
Category string `json:"category,omitempty"` // the leaf department name
CategoryID string `json:"category_id,omitempty" table:"-" kit:"link,kind=walmart/category"` // edge to the leaf category
Trail []string `json:"trail,omitempty" table:"-"` // the full department path, root first, leaf last
Description string `json:"description,omitempty" table:",truncate"`
Image string `json:"image,omitempty" table:",truncate"`
Images []string `json:"images,omitempty" table:"-"` // the photo gallery, when more than one is listed
Variants []string `json:"variants,omitempty" table:"-" kit:"link,kind=walmart/product"` // edges to colour/size/configuration siblings
URL string `json:"url"`
}
Product is the full detail for one item, emitted by product. The fields are what the /ip page shows a logged-out visitor, the same fields the Affiliate API returns on the fallback path.
type Ref ¶
type Ref struct {
Input string `json:"input"`
Kind string `json:"kind"`
ID string `json:"id"`
URL string `json:"url"`
}
Ref is the result of `walmart ref id`: the canonical (kind, id) a reference resolves to, plus the live URL, all without touching the network.
type Store ¶
type Store struct {
ID string `json:"id" kit:"id"` // store number
Name string `json:"name,omitempty"`
Address string `json:"address,omitempty"`
City string `json:"city,omitempty"`
State string `json:"state,omitempty"`
Zip string `json:"zip,omitempty"`
Phone string `json:"phone,omitempty"`
Lat float64 `json:"lat,omitempty"`
Lon float64 `json:"lon,omitempty"`
Distance float64 `json:"distance,omitempty"` // miles from the queried ZIP, on store find
Hours string `json:"hours,omitempty"`
URL string `json:"url"`
}
Store is a Walmart store's public profile, emitted by store show and store find.
type Suggestion ¶
type Suggestion struct {
Query string `json:"query"` // the prefix that was queried
Term string `json:"term" kit:"id"` // a suggested completion
Image string `json:"image,omitempty"` // a thumbnail the typeahead pairs with some terms
}
Suggestion is one autocomplete term, emitted by suggest.