operations

package module
v0.0.0-...-3409afb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 11, 2026 License: Apache-2.0 Imports: 16 Imported by: 35

README

Package cloudeng.io/webapi/operations

import cloudeng.io/webapi/operations

Package operations provides support for invoking various operations on web APIs.

Functions

Func RunCrawl
func RunCrawl[ScanT, EndpointT any](ctx context.Context, crawler *Crawler[ScanT, EndpointT], handler CrawlHandler[EndpointT]) error

RunCrawl is a convenience function that runs a crawler and calls the supplied handler for each Object crawled.

Types

Type Auth
type Auth interface {
	// WithAuthorization adds an authorization header or other required
	// authorization information to the provided http.Request.
	WithAuthorization(context.Context, *http.Request) error
}

Auth represents an authorization mechanism.

Type CrawlHandler
type CrawlHandler[EndpointT any] func(context.Context, []content.Object[EndpointT, Response]) error
Type Crawler
type Crawler[ScanT any, EndpointT any] struct {
	// contains filtered or unexported fields
}

Crawler is a generic crawler that can be used to iterate over a paginated API endpoint (using a Scanner) that enumerates objects that can be downloaded using a Fetcher. The Fetcher is responsible for decoding the results of each response from the paginated API and downloading the objects.

Functions
func NewCrawler[ScanT, EndpointT any](scanner *Scanner[ScanT], fetcher Fetcher[ScanT, EndpointT]) *Crawler[ScanT, EndpointT]

NewCrawler creates a new crawler that scans the API using the provided Scanner with the result of each scan being passed to the Fetcher to download each item returned by the scan.

Methods
func (c *Crawler[ScanT, EndpointT]) Run(ctx context.Context, ch chan<- []content.Object[EndpointT, Response]) error

Run runs the crawler. It consists of a scan loop that calls a Fetcher for each scan response.

Type Encoding
type Encoding int

Encoding represents the encoding scheme used for the response body.

Constants
JSONEncoding
JSONEncoding Encoding = iota
Methods
func (e Encoding) ContentType() string

ContentType returns the content type associated with this encoding.

Type Endpoint
type Endpoint[T any] struct {
	// contains filtered or unexported fields
}

Endpoint represents an API endpoint that can be invoked using GET. The response body is unmarshaled into the specified type T. Use PutEndpoint for operations where both the request and response bodies can be typed.

Functions
func NewEndpoint[T any](opts ...Option) *Endpoint[T]

NewEndpoint returns a new endpoint for the specified type.

Methods
func (ep *Endpoint[T]) Get(ctx context.Context, url string) (T, []byte, Encoding, error)

Get invokes a GET request on this endpoint (without a body).

func (ep *Endpoint[T]) IssueRequest(ctx context.Context, req *http.Request) (T, []byte, Encoding, *http.Response, error)

IssueRequest invokes an arbitrary request on this endpoint using the supplied http.Request. The Body in the http.Response has already been read and its contents returned as the second return value.

Type Error
type Error struct {
	Err        error
	Status     string
	StatusCode int
	Attempts   int
}
Methods
func (err *Error) Error() string
Type FS
type FS interface {
	content.FS
	filewalk.FS
}

FS defines a filesystem interface to be broadly used by webapi packages and clients. It is defined in operations for convenience.

Type Fetcher
type Fetcher[ScanT, EndpointT any] interface {
	Fetch(context.Context, ScanT, chan<- []content.Object[EndpointT, Response]) error
}

Fetcher is a generic interface for fetching objects from an API endpoint as part of a crawl. The Fetcher extracts/decodes items to be fetched from the result of a Scan and then downloads each item.

Type Marshal
type Marshal func(any) ([]byte, error)

Marshal represents a function that can be used to marshal a request body.

Type Option
type Option func(o *options)

Option represents an option that can be used when creating new Endpoints and Streams.

Functions
func WithAuth(a Auth) Option

WithAuth specifies the instance of Auth to use when making requests.

func WithHTTPClient(client *http.Client) Option

WithHTTPClient specifies the http.Client to use for making requests. If not specified, http.DefaultClient is used.

func WithLogger(logger *slog.Logger) Option

WithLogger specifies the logger to use for logging request and response information. If not specified, no logging is performed.

func WithMarshal(marshal Marshal, e Encoding) Option

WithMarshal specifies a custom marshaling function to use for encoding request bodies. The default is json.Marshal.

func WithRateController(rc *ratecontrol.Controller, statusCodes ...int) Option

WithRateController sets the rate controller to use to enforce rate control and backoff.

func WithSigner(signer Signer) Option

WithSigner specifies a Signer function to use for signing requests.

func WithSuccessCodes(codes ...int) Option

WithSuccessCodes specifies the HTTP status codes that should be considered successful responses. If not specified, only http.StatusOK (200) is considered a successful response for Get operations and http.StatusOK (200) http.StatusAccepted or for Put/Post operations.

func WithUnmarshal(u Unmarshal, e Encoding) Option

WithUnmarshal specifies a custom unmarshaling function to use for decoding response bodies. The default is json.Unmarshal.

Type Paginator
type Paginator[T any] interface {
	// Next is called with the returned type and response for that operation.
	// The first URL to use in a scan is generated by calling Next with an empty
	// payload and nil *http.Response
	Next(ctx context.Context, t T, r *http.Response) (req *http.Request, done bool, err error)
}

Paginator represents the ability to generate the next request (URL with optional body) given the response from the previous request. Paginators are typically used with Scanners to iterate over a paginated API.

Type PutEndpoint
type PutEndpoint[RequestT, ResponseT any] struct {
	// contains filtered or unexported fields
}

PutEndpoint represents an API endpoint that supports PUT requests with a request body of type RequestT and a response body of type ResponseT.

Functions
func NewPutEndpoint[RequestT, ResponseT any](opts ...Option) *PutEndpoint[RequestT, ResponseT]
Methods
func (ep *PutEndpoint[RequestT, ResponseT]) IssueRequest(ctx context.Context, req *http.Request, data RequestT) (ResponseT, []byte, Encoding, *http.Response, error)

IssueRequest invokes an arbitrary request on this endpoint using the supplied http.Request except that the Request body is overridden with encoding of the supplied data.

func (ep *PutEndpoint[RequestT, ResponseT]) Post(ctx context.Context, url string, data RequestT) (ResponseT, []byte, Encoding, error)

Post invokes a POST request on this endpoint with a request of type RequestT and a response of type ResponseT.

func (ep *PutEndpoint[RequestT, ResponseT]) Put(ctx context.Context, url string, data RequestT) (ResponseT, []byte, Encoding, error)

Put invokes a PUT request on this endpoint with a request of type RequestT and a response of type ResponseT.

Type Response
type Response struct {
	// The raw bytes of the response.
	Bytes []byte
	// The encoding used for Bytes.
	Encoding Encoding
	// When the response was received.
	When time.Time

	// Fields copied from the http.Response.
	Headers                http.Header
	Trailers               http.Header
	ContentLength          int64
	StatusCode             int
	ProtoMajor, ProtoMinir int
	TransferEncoding       []string

	// Any error encountered during the operation.
	Error error

	// Checkpoint is an opaque value that can be used to resume an
	// operation at a later time. This is used generally used by implementations
	// of Crawler/Fetcher.
	Checkpoint []byte

	// Current and Total, if non-zero, provide an indication of progress.
	Current int64
	Total   int64
}

Response contains metadata for the result of an operation and is used for the response field of the content.Object returned by the Fetcher.

Methods
func (r *Response) FromHTTPResponse(hr *http.Response)
Type Scanner
type Scanner[T any] struct {
	// contains filtered or unexported fields
}

Scanner provides the ability to iterate over a paginated API a page at a time.

Functions
func NewScanner[T any](paginator Paginator[T], opts ...Option) *Scanner[T]

NewScanner creates a new Scanner using the supplied paginator. The options are used to create the underlying Endpoint.

Methods
func (sc *Scanner[T]) Body() []byte

Body returns the body for the current page.

func (sc *Scanner[T]) Err() error

Err returns the first error encountered during scanning.

func (sc *Scanner[T]) HTTPResponse() *http.Response

HTTPResponse returns the response for the current page.

func (sc *Scanner[T]) Response() T

Response returns the response for the current page.

func (sc *Scanner[T]) Scan(ctx context.Context) bool

Scan iterates over the paginated API. It returns true if there is another page to scan, false otherwise.

Type Signer
type Signer func(ctx context.Context, header http.Header, payload []byte) error

Signer represents a function that can be used to sign requests, e.g. by adding appropriate headers. This is used for operations that require signing of requests. Signer is called with the payload to be signed and the header to which signature information should be added.

Type Unmarshal
type Unmarshal func([]byte, any) error

Unmarshal represents a function that can be used to unmarshal a response body.

Documentation

Overview

Package operations provides support for invoking various operations on web APIs.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RunCrawl 

func RunCrawl[ScanT, EndpointT any](ctx context.Context, crawler *Crawler[ScanT, EndpointT], handler CrawlHandler[EndpointT]) error

RunCrawl is a convenience function that runs a crawler and calls the supplied handler for each Object crawled.

Types

type Auth 

type Auth interface {
	// WithAuthorization adds an authorization header or other required
	// authorization information to the provided http.Request.
	WithAuthorization(context.Context, *http.Request) error
}

Auth represents an authorization mechanism.

type CrawlHandler 

type CrawlHandler[EndpointT any] func(context.Context, []content.Object[EndpointT, Response]) error

type Crawler 

type Crawler[ScanT any, EndpointT any] struct {
	// contains filtered or unexported fields
}

Crawler is a generic crawler that can be used to iterate over a paginated API endpoint (using a Scanner) that enumerates objects that can be downloaded using a Fetcher. The Fetcher is responsible for decoding the results of each response from the paginated API and downloading the objects.

func NewCrawler 

func NewCrawler[ScanT, EndpointT any](scanner *Scanner[ScanT], fetcher Fetcher[ScanT, EndpointT]) *Crawler[ScanT, EndpointT]

NewCrawler creates a new crawler that scans the API using the provided Scanner with the result of each scan being passed to the Fetcher to download each item returned by the scan.

func (*Crawler[ScanT, EndpointT]) Run 

func (c *Crawler[ScanT, EndpointT]) Run(ctx context.Context, ch chan<- []content.Object[EndpointT, Response]) error

Run runs the crawler. It consists of a scan loop that calls a Fetcher for each scan response.

type Encoding 

type Encoding int

Encoding represents the encoding scheme used for the response body. 

const (
	JSONEncoding Encoding = iota
)

func (Encoding) ContentType 

func (e Encoding) ContentType() string

ContentType returns the content type associated with this encoding.

type Endpoint 

type Endpoint[T any] struct {
	// contains filtered or unexported fields
}

Endpoint represents an API endpoint that can be invoked using GET. The response body is unmarshaled into the specified type T. Use PutEndpoint for operations where both the request and response bodies can be typed.

func NewEndpoint 

func NewEndpoint[T any](opts ...Option) *Endpoint[T]

NewEndpoint returns a new endpoint for the specified type.

func (*Endpoint[T]) Get 

func (ep *Endpoint[T]) Get(ctx context.Context, url string) (T, []byte, Encoding, error)

Get invokes a GET request on this endpoint (without a body).

func (*Endpoint[T]) IssueRequest 

func (ep *Endpoint[T]) IssueRequest(ctx context.Context, req *http.Request) (T, []byte, Encoding, *http.Response, error)

IssueRequest invokes an arbitrary request on this endpoint using the supplied http.Request. The Body in the http.Response has already been read and its contents returned as the second return value.

type Error 

type Error struct {
	Err        error
	Status     string
	StatusCode int
	Attempts   int
}

func (*Error) Error 

func (err *Error) Error() string

type FS 

type FS interface {
	content.FS
	filewalk.FS
}

FS defines a filesystem interface to be broadly used by webapi packages and clients. It is defined in operations for convenience.

type Fetcher 

type Fetcher[ScanT, EndpointT any] interface {
	Fetch(context.Context, ScanT, chan<- []content.Object[EndpointT, Response]) error
}

Fetcher is a generic interface for fetching objects from an API endpoint as part of a crawl. The Fetcher extracts/decodes items to be fetched from the result of a Scan and then downloads each item.

type Marshal 

type Marshal func(any) ([]byte, error)

Marshal represents a function that can be used to marshal a request body.

type Option 

type Option func(o *options)

Option represents an option that can be used when creating new Endpoints and Streams.

func WithAuth 

func WithAuth(a Auth) Option

WithAuth specifies the instance of Auth to use when making requests.

func WithHTTPClient 

func WithHTTPClient(client *http.Client) Option

WithHTTPClient specifies the http.Client to use for making requests. If not specified, http.DefaultClient is used.

func WithLogger 

func WithLogger(logger *slog.Logger) Option

WithLogger specifies the logger to use for logging request and response information. If not specified, no logging is performed.

func WithMarshal 

func WithMarshal(marshal Marshal, e Encoding) Option

WithMarshal specifies a custom marshaling function to use for encoding request bodies. The default is json.Marshal.

func WithRateController 

func WithRateController(rc *ratecontrol.Controller, statusCodes ...int) Option

WithRateController sets the rate controller to use to enforce rate control and backoff.

func WithSigner 

func WithSigner(signer Signer) Option

WithSigner specifies a Signer function to use for signing requests.

func WithSuccessCodes 

func WithSuccessCodes(codes ...int) Option

WithSuccessCodes specifies the HTTP status codes that should be considered successful responses. If not specified, only http.StatusOK (200) is considered a successful response for Get operations and http.StatusOK (200) http.StatusAccepted or for Put/Post operations.

func WithUnmarshal 

func WithUnmarshal(u Unmarshal, e Encoding) Option

WithUnmarshal specifies a custom unmarshaling function to use for decoding response bodies. The default is json.Unmarshal.

type Paginator 

type Paginator[T any] interface {
	// Next is called with the returned type and response for that operation.
	// The first URL to use in a scan is generated by calling Next with an empty
	// payload and nil *http.Response
	Next(ctx context.Context, t T, r *http.Response) (req *http.Request, done bool, err error)
}

Paginator represents the ability to generate the next request (URL with optional body) given the response from the previous request. Paginators are typically used with Scanners to iterate over a paginated API.

type PutEndpoint 

type PutEndpoint[RequestT, ResponseT any] struct {
	// contains filtered or unexported fields
}

PutEndpoint represents an API endpoint that supports PUT requests with a request body of type RequestT and a response body of type ResponseT.

func NewPutEndpoint 

func NewPutEndpoint[RequestT, ResponseT any](opts ...Option) *PutEndpoint[RequestT, ResponseT]

func (*PutEndpoint[RequestT, ResponseT]) IssueRequest 

func (ep *PutEndpoint[RequestT, ResponseT]) IssueRequest(ctx context.Context, req *http.Request, data RequestT) (ResponseT, []byte, Encoding, *http.Response, error)

IssueRequest invokes an arbitrary request on this endpoint using the supplied http.Request except that the Request body is overridden with encoding of the supplied data.

func (*PutEndpoint[RequestT, ResponseT]) Post 

func (ep *PutEndpoint[RequestT, ResponseT]) Post(ctx context.Context, url string, data RequestT) (ResponseT, []byte, Encoding, error)

Post invokes a POST request on this endpoint with a request of type RequestT and a response of type ResponseT.

func (*PutEndpoint[RequestT, ResponseT]) Put 

func (ep *PutEndpoint[RequestT, ResponseT]) Put(ctx context.Context, url string, data RequestT) (ResponseT, []byte, Encoding, error)

Put invokes a PUT request on this endpoint with a request of type RequestT and a response of type ResponseT.

type Response 

type Response struct {
	// The raw bytes of the response.
	Bytes []byte
	// The encoding used for Bytes.
	Encoding Encoding
	// When the response was received.
	When time.Time

	// Fields copied from the http.Response.
	Headers                http.Header
	Trailers               http.Header
	ContentLength          int64
	StatusCode             int
	ProtoMajor, ProtoMinir int
	TransferEncoding       []string

	// Any error encountered during the operation.
	Error error

	// Checkpoint is an opaque value that can be used to resume an
	// operation at a later time. This is used generally used by implementations
	// of Crawler/Fetcher.
	Checkpoint []byte

	// Current and Total, if non-zero, provide an indication of progress.
	Current int64
	Total   int64
}

Response contains metadata for the result of an operation and is used for the response field of the content.Object returned by the Fetcher.

func (*Response) FromHTTPResponse 

func (r *Response) FromHTTPResponse(hr *http.Response)

type Scanner 

type Scanner[T any] struct {
	// contains filtered or unexported fields
}

Scanner provides the ability to iterate over a paginated API a page at a time.

func NewScanner 

func NewScanner[T any](paginator Paginator[T], opts ...Option) *Scanner[T]

NewScanner creates a new Scanner using the supplied paginator. The options are used to create the underlying Endpoint.

func (*Scanner[T]) Body 

func (sc *Scanner[T]) Body() []byte

Body returns the body for the current page.

func (*Scanner[T]) Err 

func (sc *Scanner[T]) Err() error

Err returns the first error encountered during scanning.

func (*Scanner[T]) HTTPResponse 

func (sc *Scanner[T]) HTTPResponse() *http.Response

HTTPResponse returns the response for the current page.

func (*Scanner[T]) Response 

func (sc *Scanner[T]) Response() T

Response returns the response for the current page.

func (*Scanner[T]) Scan 

func (sc *Scanner[T]) Scan(ctx context.Context) bool

Scan iterates over the paginated API. It returns true if there is another page to scan, false otherwise.

type Signer 

type Signer func(ctx context.Context, header http.Header, payload []byte) error

Signer represents a function that can be used to sign requests, e.g. by adding appropriate headers. This is used for operations that require signing of requests. Signer is called with the payload to be signed and the header to which signature information should be added.

type Unmarshal 

type Unmarshal func([]byte, any) error

Unmarshal represents a function that can be used to unmarshal a response body.

Source Files

View all Source files

Directories

Path Synopsis
Package apicrawlcmd provides support for building command line tools that implement API crawls.
 Package apicrawlcmd provides support for building command line tools that implement API crawls.
Package apitokens provides types and functions for managing API tokens and is built on top of the cloudeng.io/cmdutil/keys package and its InmemoryKeyStore.
 Package apitokens provides types and functions for managing API tokens and is built on top of the cloudeng.io/cmdutil/keys package and its InmemoryKeyStore.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.