walmart

package
v0.0.0-...-1fab30b Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 29, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package walmart is the library behind the walmart command line: an HTTP client for Walmart's public web pages and typeahead host, an optional Affiliate API backend, and the typed records every command emits.

Walmart renders its public pages server-side and ships the data as a __NEXT_DATA__ JSON island, so this client GETs pages and reads that island; the typeahead host answers plain JSON. Walmart fronts its estate with the PerimeterX (HUMAN) bot manager, which walls almost everything from datacenter IPs: only the typeahead host and the homepage answer anonymously, while the product page (/ip/), keyword search (/search), the category pages (/cp/, /browse), and the store pages (/store/) return the "Robot or human?" challenge. So the client is two-layered: a reliable core over the open surfaces, and a best-effort layer over the walled ones that returns ErrBlocked when the wall is up and, if the operator has set Affiliate API credentials, falls back to that documented backend. Each surface lives in its own file (search.go, product.go, category.go, store.go, deals.go, suggest.go) with its parsing and record mapping; this file holds the shared web client.

Index

Constants

View Source
const (
	// DefaultDelay is the minimum gap between requests. Walmart is touchy, so a
	// one-second pace reads steadily without leaning on it.
	DefaultDelay    = 1 * time.Second
	DefaultRetries  = 3
	DefaultTimeout  = 30 * time.Second
	DefaultCacheTTL = 24 * time.Hour
)

Defaults for the polite client.

View Source
const BaseURL = "https://" + Host

BaseURL is the root every product, store, and category URL is built from.

View Source
const DefaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " +
	"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

DefaultUserAgent is sent with every page request. Walmart serves its public pages to a normal browser; a browser User-Agent is what keeps a logged-out reader looking like one. Override it with --user-agent.

View Source
const Host = "www.walmart.com"

Host is the Walmart hostname this client builds page URLs from and the host the URI driver in domain.go claims.

Variables

View Source
var (
	// ErrNotFound is a missing entity: an unknown item id, an unknown store, or a
	// bad category. Walmart serves these as a 404 or a not-found shell page. Exit
	// code 6.
	ErrNotFound = errors.New("not found")

	// ErrRateLimited is a sustained HTTP 429 after the client's own retries. Slow
	// down with --rate. Exit code 5.
	ErrRateLimited = errors.New("rate limited")

	// ErrBlocked is the PerimeterX bot wall: a 403 or 412 interstitial, the
	// "Robot or human?" challenge page served with a 200, or an implausibly small
	// body on a page that should be large. Walmart walls the product page, search,
	// the category pages, and the store pages from datacenter IPs, so this is
	// expected there. The message names the two remedies (a residential IP, or
	// Affiliate API credentials). Exit code 4.
	ErrBlocked = errors.New("blocked by Walmart's bot wall (retry from a residential network, " +
		"or set WALMART_CONSUMER_ID and WALMART_PRIVATE_KEY to use the Affiliate API)")
)

The library reports its outcomes as a few sentinel errors. domain.go's mapErr translates each into the kit error kind that carries the matching exit code, so the standalone binary and a host agree on what a wall, a throttle, and a miss mean.

Functions

func Defaults

func Defaults(c *kit.Config)

Defaults seeds the framework baseline with walmart's own values, so an unset --rate or --timeout uses the walmart default rather than the generic kit one. It is passed to kit.New via kit.WithDefaults.

func Identity

func Identity() kit.Identity

Identity is the fixed description of the walmart CLI, shared by the domain and the standalone composition root so help and version read the same everywhere.

func URLFor

func URLFor(kind, id string) string

URLFor builds the canonical Walmart URL for a (kind, id) pair.

Types

type Category

type Category struct {
	ID       string   `json:"id" kit:"id"`
	Name     string   `json:"name"`
	Parent   string   `json:"parent,omitempty"`                                               // parent category name (from the trail)
	ParentID string   `json:"parent_id,omitempty" table:"-" kit:"link,kind=walmart/category"` // edge up to the parent node
	Children []string `json:"children,omitempty" table:"-" kit:"link,kind=walmart/category"`  // edges down to the child nodes
	Trail    []string `json:"trail,omitempty" table:"-"`                                      // the full ancestor path, root first, leaf last
	URL      string   `json:"url"`
}

Category is a category node, emitted by category show and category tree. The ParentID and Children edges make the taxonomy walkable in both directions, so a host can recurse the department tree from any node to reconstruct it.

type Client

type Client struct {
	HTTP      *http.Client
	BaseURL   string
	UserAgent string
	Delay     time.Duration
	Retries   int

	// SuggestURL is the typeahead endpoint. It defaults to the public host; tests
	// point it at an httptest server.
	SuggestURL string
	// contains filtered or unexported fields
}

Client talks to Walmart's public web pages. It paces requests, retries the transient failures, detects the bot wall, and caches response bodies on disk keyed by the request URL.

func ClientFromConfig

func ClientFromConfig(cfg kit.Config) *Client

ClientFromConfig maps the framework config onto a walmart.Config and returns a client. The affiliate credentials are read from the environment, not flags, so they never land in shell history.

func NewClient

func NewClient(cfg Config) *Client

NewClient builds a client from cfg.

func (*Client) CategoryBrowse

func (c *Client) CategoryBrowse(ctx context.Context, ref string, limit int) ([]*Listing, error)

CategoryBrowse returns the items in a category.

func (*Client) CategoryTree

func (c *Client) CategoryTree(ctx context.Context, ref string, limit int) ([]*Category, error)

CategoryTree returns the child categories of a node. With API credentials it reads the taxonomy; without them it scans the department links on the homepage (for the top level) or the category page, which works when that page is served.

func (*Client) ClearCache

func (c *Client) ClearCache() error

ClearCache removes the on-disk cache.

func (*Client) Deals

func (c *Client) Deals(ctx context.Context, limit int) ([]*Deal, error)

Deals returns the current rollbacks.

func (*Client) FindStores

func (c *Client) FindStores(ctx context.Context, zip string, limit int) ([]*Store, error)

FindStores returns the stores near a ZIP code.

func (*Client) GetCategory

func (c *Client) GetCategory(ctx context.Context, ref string) (*Category, error)

GetCategory returns a category's metadata: its name and its trail. It reads the API taxonomy when configured, since the web page is walled.

func (*Client) GetProduct

func (c *Client) GetProduct(ctx context.Context, ref string) (*Product, error)

GetProduct returns one item by id (or /ip/ URL).

func (*Client) GetStore

func (c *Client) GetStore(ctx context.Context, ref string) (*Store, error)

GetStore returns one store's profile by id (or /store/ URL), read from the page when Walmart serves it.

func (*Client) Search

func (c *Client) Search(ctx context.Context, query string, limit int) ([]*Listing, error)

Search returns listings matching a keyword query.

func (*Client) Suggest

func (c *Client) Suggest(ctx context.Context, prefix string, limit int) ([]*Suggestion, error)

Suggest returns search-autocomplete terms for a typed prefix.

func (*Client) Trending

func (c *Client) Trending(ctx context.Context, limit int) ([]*Listing, error)

Trending returns the trending items.

type Config

type Config struct {
	UserAgent string

	// Zip is the ZIP context for the store locator and the default-store price on
	// the API path. Empty leaves it to Walmart's default store.
	Zip string

	// The affiliate credentials enable the API fallback for the walled surfaces.
	// ConsumerID with one of PrivateKey or PrivateKeyFile turns it on; empty
	// leaves the fallback off, and the walled commands then report the bot wall
	// rather than guessing.
	ConsumerID     string
	PrivateKey     string // PEM contents
	PrivateKeyFile string // path to a PEM file, used when PrivateKey is empty
	KeyVersion     string
	PublisherID    string

	// Delay is the minimum gap between requests. Zero means no pacing.
	Delay   time.Duration
	Retries int
	Timeout time.Duration

	// BaseURL is the site root. Empty uses the public site; tests point it at an
	// httptest server.
	BaseURL string

	// CacheDir is where responses are cached. Empty disables the cache, as does
	// NoCache.
	CacheDir string
	CacheTTL time.Duration
	NoCache  bool
	// Refresh fetches fresh copies and rewrites the cache, ignoring any hit.
	Refresh bool
}

Config carries the knobs the client reads. It is built from the kit framework config in ClientFromConfig, so a --rate or --timeout on the command line and the same value resolved by a host both land here.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns the baseline configuration: a browser User-Agent, a one-second pace, three retries, a 30s timeout, the "1" key version, and a one-day cache.

type Deal

type Deal struct {
	ID       string  `json:"id" kit:"id"`
	Title    string  `json:"title,omitempty" table:",truncate"`
	Brand    string  `json:"brand,omitempty"`
	Price    float64 `json:"price,omitempty"`
	Was      float64 `json:"was,omitempty"`  // price before the markdown
	Save     float64 `json:"save,omitempty"` // was minus price
	Currency string  `json:"currency,omitempty"`
	Offer    string  `json:"offer,omitempty"` // the offer type, e.g. "Rollback" or "Clearance"
	Rating   float64 `json:"rating,omitempty"`
	Reviews  int     `json:"reviews,omitempty"`
	Image    string  `json:"image,omitempty" table:",truncate"`
	URL      string  `json:"url"`
	Item     string  `json:"item,omitempty" table:"-" kit:"link,kind=walmart/product"` // edge to the full product
}

Deal is a rolled-back or cleared item, emitted by deals.

type Domain

type Domain struct{}

Domain is the walmart driver. It carries no state; the per-run client is built by the factory Register hands kit.

func (Domain) Classify

func (Domain) Classify(input string) (uriType, id string, err error)

Classify turns any accepted input into the canonical (type, id), so `ant resolve` and `ant url` touch no network.

func (Domain) Info

func (Domain) Info() kit.DomainInfo

Info describes the scheme, the hostnames a pasted link is matched against, and the identity reused for the binary's help and version.

func (Domain) Locate

func (Domain) Locate(uriType, id string) (string, error)

Locate is the inverse: the live https URL for a (type, id).

func (Domain) Register

func (Domain) Register(app *kit.App)

Register installs the client factory and every operation onto app. A resolver op (Single) names its own record type and answers `ant get`; a List op enumerates a parent resource's members and answers `ant ls`.

type Listing

type Listing struct {
	ID           string  `json:"id" kit:"id"`
	Title        string  `json:"title,omitempty" table:",truncate"`
	Brand        string  `json:"brand,omitempty"`
	Price        float64 `json:"price,omitempty"`
	Currency     string  `json:"currency,omitempty"`
	Was          float64 `json:"was,omitempty"` // strikethrough / was price
	Seller       string  `json:"seller,omitempty"`
	Rating       float64 `json:"rating,omitempty"`  // average star rating
	Reviews      int     `json:"reviews,omitempty"` // review count
	Availability string  `json:"availability,omitempty"`
	Sponsored    bool    `json:"sponsored,omitempty"` // an ad tile rather than an organic result
	Thumbnail    string  `json:"thumbnail,omitempty" table:",truncate"`
	URL          string  `json:"url"`
	Item         string  `json:"item,omitempty" table:"-" kit:"link,kind=walmart/product"` // edge to the full product
}

Listing is a summary row in a grid, emitted by search, category browse, and trending. Its id is the product's usItemId, so Item carries the same value as the graph edge to the full product record: a host walks a search result or a category page straight through to walmart://product/<id> and onward.

type Product

type Product struct {
	ID           string   `json:"id" kit:"id"`
	Name         string   `json:"name,omitempty" table:",truncate"`
	Brand        string   `json:"brand,omitempty"`
	Model        string   `json:"model,omitempty"`
	UPC          string   `json:"upc,omitempty"`
	Price        float64  `json:"price,omitempty"`
	Currency     string   `json:"currency,omitempty"`
	Was          float64  `json:"was,omitempty"`  // was price, when the item is marked down
	List         float64  `json:"list,omitempty"` // manufacturer list price / MSRP
	Availability string   `json:"availability,omitempty"`
	Seller       string   `json:"seller,omitempty"`
	Rating       float64  `json:"rating,omitempty"`
	Reviews      int      `json:"reviews,omitempty"`
	Category     string   `json:"category,omitempty"`                                               // the leaf department name
	CategoryID   string   `json:"category_id,omitempty" table:"-" kit:"link,kind=walmart/category"` // edge to the leaf category
	Trail        []string `json:"trail,omitempty" table:"-"`                                        // the full department path, root first, leaf last
	Description  string   `json:"description,omitempty" table:",truncate"`
	Image        string   `json:"image,omitempty" table:",truncate"`
	Images       []string `json:"images,omitempty" table:"-"`                                   // the photo gallery, when more than one is listed
	Variants     []string `json:"variants,omitempty" table:"-" kit:"link,kind=walmart/product"` // edges to colour/size/configuration siblings
	URL          string   `json:"url"`
}

Product is the full detail for one item, emitted by product. The fields are what the /ip page shows a logged-out visitor, the same fields the Affiliate API returns on the fallback path.

type Ref

type Ref struct {
	Input string `json:"input"`
	Kind  string `json:"kind"`
	ID    string `json:"id"`
	URL   string `json:"url"`
}

Ref is the result of `walmart ref id`: the canonical (kind, id) a reference resolves to, plus the live URL, all without touching the network.

func Classify

func Classify(ref string) Ref

Classify reads a reference (a URL, a path, or a bare id) and reports what it points at. Kind is one of product, store, category, or unknown.

type Store

type Store struct {
	ID       string  `json:"id" kit:"id"` // store number
	Name     string  `json:"name,omitempty"`
	Address  string  `json:"address,omitempty"`
	City     string  `json:"city,omitempty"`
	State    string  `json:"state,omitempty"`
	Zip      string  `json:"zip,omitempty"`
	Phone    string  `json:"phone,omitempty"`
	Lat      float64 `json:"lat,omitempty"`
	Lon      float64 `json:"lon,omitempty"`
	Distance float64 `json:"distance,omitempty"` // miles from the queried ZIP, on store find
	Hours    string  `json:"hours,omitempty"`
	URL      string  `json:"url"`
}

Store is a Walmart store's public profile, emitted by store show and store find.

type Suggestion

type Suggestion struct {
	Query string `json:"query"`           // the prefix that was queried
	Term  string `json:"term" kit:"id"`   // a suggested completion
	Image string `json:"image,omitempty"` // a thumbnail the typeahead pairs with some terms
}

Suggestion is one autocomplete term, emitted by suggest.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL