agent

package
v1.0.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package agent provides a typed, policy-gated session layer above RobotGo's low-level compatibility API. It deliberately contains no protocol adapter.

Index

Constants

View Source 
const AuditSchemaVersion = "2"

AuditSchemaVersion identifies the payload-free audit event contract.

View Source 
const CatalogSchemaVersion = "3"

CatalogSchemaVersion identifies the operation catalog JSON contract.

View Source 
const (
	// ConditionSchemaVersion identifies the serialized visual-condition contract.
	ConditionSchemaVersion = "1"
)
View Source 
const (
	// ObservationSchemaVersion identifies the serialized observation contract.
	ObservationSchemaVersion = "1"
)

Variables

View Source 
var (
	ErrSessionBusy     = errors.New("another agent session is already active")
	ErrSessionClosed   = errors.New("agent session is closed")
	ErrPolicyDenied    = errors.New("agent policy denied the action")
	ErrStaleTarget     = errors.New("agent observation target is stale")
	ErrVerification    = errors.New("agent action verification failed")
	ErrAuditDelivery   = errors.New("agent audit delivery failed")
	ErrConditionNotMet = errors.New("agent visual condition was not met")
)
View Source 
var (

	// ErrObservationClosed reports access to a zeroed observation buffer.
	ErrObservationClosed = errors.New("agent observation is closed")
)

Functions

This section is empty.

Types

type ActionError 

type ActionError struct {
	Code      ErrorCode `json:"code"`
	Operation Operation `json:"operation,omitempty"`
	Message   string    `json:"message"`
	// contains filtered or unexported fields
}

ActionError is safe to serialize: Message never contains action payloads.

func (*ActionError) Error 

func (e *ActionError) Error() string

func (*ActionError) Unwrap 

func (e *ActionError) Unwrap() error

type ActionRequest 

type ActionRequest struct {
	Operation    Operation                `json:"operation"`
	Confirmed    bool                     `json:"confirmed,omitempty"`
	Move         *MoveAction              `json:"move,omitempty"`
	Click        *ClickAction             `json:"click,omitempty"`
	TypeText     *TypeTextAction          `json:"type_text,omitempty"`
	Precondition *ObservationPrecondition `json:"precondition,omitempty"`
	Verification *VerificationRequest     `json:"verification,omitempty"`
}

ActionRequest is a strict JSON-serializable action union. Exactly one action payload must be present and must match Operation.

type ActionResult 

type ActionResult struct {
	ActionID                  string              `json:"action_id"`
	Operation                 Operation           `json:"operation"`
	Status                    ActionStatus        `json:"status"`
	Backend                   string              `json:"backend,omitempty"`
	DurationMillis            int64               `json:"duration_ms"`
	Error                     *ActionError        `json:"error,omitempty"`
	PreconditionObservationID string              `json:"precondition_observation_id,omitempty"`
	PostObservationID         string              `json:"post_observation_id,omitempty"`
	Verification              *VerificationResult `json:"verification,omitempty"`
}

ActionResult reports one planned or attempted action without retaining its input payload.

type ActionStatus 

type ActionStatus string

ActionStatus identifies the outcome of an action request. 

const (
	ActionPlanned    ActionStatus = "planned"
	ActionSucceeded  ActionStatus = "succeeded"
	ActionFailed     ActionStatus = "failed"
	ActionUnverified ActionStatus = "unverified"
)

type AuditEvent 

type AuditEvent struct {
	SchemaVersion             string             `json:"schema_version"`
	Sequence                  uint64             `json:"sequence"`
	Kind                      AuditKind          `json:"kind"`
	Timestamp                 time.Time          `json:"timestamp"`
	Operation                 Operation          `json:"operation,omitempty"`
	ActionID                  string             `json:"action_id,omitempty"`
	ObservationID             string             `json:"observation_id,omitempty"`
	PreconditionObservationID string             `json:"precondition_observation_id,omitempty"`
	PostObservationID         string             `json:"post_observation_id,omitempty"`
	ActionStatus              ActionStatus       `json:"action_status,omitempty"`
	VerificationStatus        VerificationStatus `json:"verification_status,omitempty"`
	VerificationAttempts      uint32             `json:"verification_attempts,omitempty"`
	ConditionID               string             `json:"condition_id,omitempty"`
	ConditionStatus           ConditionStatus    `json:"condition_status,omitempty"`
	ConditionAttempts         uint32             `json:"condition_attempts,omitempty"`
	ErrorCode                 ErrorCode          `json:"error_code,omitempty"`
}

AuditEvent deliberately contains no request payload, coordinates, text, pixels, capture digest, backend error detail, or restore token.

type AuditKind 

type AuditKind string

AuditKind identifies a sanitized session lifecycle event. 

const (
	AuditObservationStarted   AuditKind = "observation.started"
	AuditObservationFinished  AuditKind = "observation.finished"
	AuditActionStarted        AuditKind = "action.started"
	AuditActionFinished       AuditKind = "action.finished"
	AuditVerificationFinished AuditKind = "verification.finished"
	AuditConditionStarted     AuditKind = "condition.started"
	AuditConditionFinished    AuditKind = "condition.finished"
)

type AuditSink 

type AuditSink interface {
	Record(ctx context.Context, event AuditEvent) error
}

AuditSink receives synchronous, payload-free events. Implementations should honor ctx and return promptly; an intent-event error prevents desktop I/O. A completion-event error is returned alongside the completed result so a caller never needs to guess whether a desktop mutation occurred. A sink must not call back into the Session that invoked it.

type CancellationSupport 

type CancellationSupport string

CancellationSupport describes where cancellation is enforceable. 

const (
	CancellationPreflightOnly CancellationSupport = "preflight-only"
	CancellationCooperative   CancellationSupport = "cooperative"
)

type CaptureMetadata 

type CaptureMetadata struct {
	Region CaptureRegion `json:"region"`
	SHA256 string        `json:"sha256"`
	Width  int           `json:"width"`
	Height int           `json:"height"`
}

CaptureMetadata contains bounded, serializable capture lineage. Pixels are owned privately by Observation and never form part of this value.

type CaptureRegion 

type CaptureRegion struct {
	X         int `json:"x"`
	Y         int `json:"y"`
	Width     int `json:"width"`
	Height    int `json:"height"`
	DisplayID int `json:"display_id"`
}

CaptureRegion identifies one global logical rectangle on an explicit display. Width and Height are positive and bounded by session policy.

type ClickAction 

type ClickAction struct {
	Button MouseButton `json:"button"`
	Double bool        `json:"double,omitempty"`
}

ClickAction clicks one validated pointer button.

type ColorCondition 

type ColorCondition struct {
	Red       uint8   `json:"red"`
	Green     uint8   `json:"green"`
	Blue      uint8   `json:"blue"`
	Tolerance float64 `json:"tolerance"`
}

ColorCondition describes one RGB target. Tolerance is a normalized Euclidean RGB distance in the inclusive range 0 through 1. Alpha is ignored.

type ConditionStatus 

type ConditionStatus string

ConditionStatus identifies whether a visual condition matched. 

const (
	ConditionMatched    ConditionStatus = "matched"
	ConditionNotMatched ConditionStatus = "not-matched"
)

type Config 

type Config struct {
	Policy    Policy    `json:"policy"`
	AuditSink AuditSink `json:"-"`
}

Config defines immutable session policy.

type DiagnosticFeature 

type DiagnosticFeature struct {
	Name        string `json:"name"`
	Available   bool   `json:"available"`
	Fallback    bool   `json:"fallback"`
	Backend     string `json:"backend,omitempty"`
	Reason      string `json:"reason,omitempty"`
	Remediation string `json:"remediation,omitempty"`
}

DiagnosticFeature is a stable, sanitized feature snapshot.

type ErrorCode 

type ErrorCode string

ErrorCode is a stable machine-readable action failure category. 

const (
	ErrorInvalidInput     ErrorCode = "invalid-input"
	ErrorPolicyDenied     ErrorCode = "policy-denied"
	ErrorUnsupported      ErrorCode = "unsupported"
	ErrorPermissionDenied ErrorCode = "permission-denied"
	ErrorSessionClosed    ErrorCode = "session-closed"
	ErrorSessionBusy      ErrorCode = "session-busy"
	ErrorCanceled         ErrorCode = "canceled"
	ErrorTimedOut         ErrorCode = "timed-out"
	ErrorBackendFailure   ErrorCode = "backend-failure"
	ErrorStaleTarget      ErrorCode = "stale-target"
	ErrorVerification     ErrorCode = "verification-failed"
	ErrorAuditDelivery    ErrorCode = "audit-delivery-failed"
	ErrorConditionNotMet  ErrorCode = "condition-not-met"
)

type FindColorRequest 

type FindColorRequest struct {
	ObservationID string         `json:"observation_id"`
	Condition     ColorCondition `json:"condition"`
	Confirmed     bool           `json:"confirmed,omitempty"`
}

FindColorRequest searches only a previously captured, live observation. It never causes an implicit desktop capture.

type FindColorResult 

type FindColorResult struct {
	SchemaVersion string          `json:"schema_version"`
	ConditionID   string          `json:"condition_id"`
	ObservationID string          `json:"observation_id"`
	Status        ConditionStatus `json:"status"`
	Match         *VisualMatch    `json:"match,omitempty"`
}

FindColorResult contains no pixels, capture digest, or target color.

type MouseButton 

type MouseButton string

MouseButton is a validated pointer button name. 

const (
	MouseButtonLeft   MouseButton = "left"
	MouseButtonMiddle MouseButton = "center"
	MouseButtonRight  MouseButton = "right"
)

type MoveAction 

type MoveAction struct {
	X         int `json:"x"`
	Y         int `json:"y"`
	DisplayID int `json:"display_id"`
}

MoveAction moves the pointer to global coordinates that must fall within the live bounds of the explicitly selected display.

type Observation 

type Observation struct {
	SchemaVersion string             `json:"schema_version"`
	ObservationID string             `json:"observation_id"`
	CreatedAt     time.Time          `json:"created_at"`
	Diagnostics   RuntimeDiagnostics `json:"diagnostics"`
	Capture       *CaptureMetadata   `json:"capture,omitempty"`
	// contains filtered or unexported fields
}

Observation owns optional sensitive pixels and sanitized diagnostics.

func (*Observation) Close 

func (observation *Observation) Close() error

Close zeroes the optional capture. Sanitized metadata remains readable so audit and stale-lineage errors can still identify the closed observation.

func (*Observation) Image 

func (observation *Observation) Image() (*image.RGBA, error)

Image returns a defensive copy of the optional sensitive capture.

type ObservationPrecondition 

type ObservationPrecondition struct {
	ObservationID string `json:"observation_id"`
}

ObservationPrecondition requires the target capture to remain byte-identical immediately before mutation.

type ObserveRequest 

type ObserveRequest struct {
	Confirmed bool           `json:"confirmed,omitempty"`
	Capture   *CaptureRegion `json:"capture,omitempty"`
}

ObserveRequest asks for sanitized diagnostics and, optionally, an in-memory capture. Capture pixels are never included in JSON.

type Operation 

type Operation string

Operation identifies one strict agent operation. 

const (
	OperationMove      Operation = "pointer.move"
	OperationClick     Operation = "pointer.click"
	OperationTypeText  Operation = "keyboard.type-text"
	OperationObserve   Operation = "desktop.observe"
	OperationFindColor Operation = "desktop.find-color"
	OperationWaitColor Operation = "desktop.wait-color"
)

type OperationCapability 

type OperationCapability struct {
	Operation             Operation           `json:"operation"`
	Available             bool                `json:"available"`
	PolicyAllowed         bool                `json:"policy_allowed"`
	Backend               string              `json:"backend,omitempty"`
	Fallback              bool                `json:"fallback"`
	Risk                  RiskClass           `json:"risk"`
	ConfirmationRequired  bool                `json:"confirmation_required"`
	Cancellation          CancellationSupport `json:"cancellation"`
	ProcessGlobalBackend  bool                `json:"process_global_backend"`
	ExclusiveAgentSession bool                `json:"exclusive_agent_session"`
	Reason                string              `json:"reason,omitempty"`
	Remediation           string              `json:"remediation,omitempty"`
	OptionalCapture       bool                `json:"optional_capture,omitempty"`
	CaptureAvailable      bool                `json:"capture_available,omitempty"`
	CapturePolicyAllowed  bool                `json:"capture_policy_allowed,omitempty"`
	CaptureFallback       bool                `json:"capture_fallback,omitempty"`
	CaptureBackend        string              `json:"capture_backend,omitempty"`
}

OperationCapability is one stable entry in an operation catalog.

type OperationCatalog 

type OperationCatalog struct {
	SchemaVersion string                `json:"schema_version"`
	Operations    []OperationCapability `json:"operations"`
}

OperationCatalog is an immutable snapshot of operation availability.

type Policy 

type Policy struct {
	AllowedOperations          []Operation `json:"allowed_operations"`
	ConfirmOperations          []Operation `json:"confirm_operations,omitempty"`
	AllowedDisplayIDs          []int       `json:"allowed_display_ids,omitempty"`
	MaxActions                 uint64      `json:"max_actions"`
	MaxTextRunes               int         `json:"max_text_runes"`
	AllowDoubleClick           bool        `json:"allow_double_click,omitempty"`
	MaxObservations            uint64      `json:"max_observations,omitempty"`
	MaxCapturePixels           uint64      `json:"max_capture_pixels,omitempty"`
	MaxQueries                 uint64      `json:"max_queries,omitempty"`
	WaitAttempts               uint32      `json:"wait_attempts,omitempty"`
	WaitIntervalMillis         int         `json:"wait_interval_ms,omitempty"`
	WaitTimeoutMillis          int         `json:"wait_timeout_ms,omitempty"`
	VerificationAttempts       uint32      `json:"verification_attempts,omitempty"`
	VerificationIntervalMillis int         `json:"verification_interval_ms,omitempty"`
	VerificationTimeoutMillis  int         `json:"verification_timeout_ms,omitempty"`
	// contains filtered or unexported fields
}

Policy constrains every observation, query, and mutation performed by a Session. Empty allow lists deny access; callers must opt in explicitly.

type RiskClass 

type RiskClass string

RiskClass describes the policy impact of an operation. 

const (
	RiskSensitiveRead      RiskClass = "sensitive-read"
	RiskReversibleMutation RiskClass = "reversible-mutation"
)

type RuntimeDiagnostics 

type RuntimeDiagnostics struct {
	GOOS           string              `json:"goos"`
	GOARCH         string              `json:"goarch"`
	CGOEnabled     bool                `json:"cgo_enabled"`
	Implementation string              `json:"implementation"`
	DisplayServer  string              `json:"display_server"`
	Compositor     string              `json:"compositor,omitempty"`
	Features       []DiagnosticFeature `json:"features"`
}

RuntimeDiagnostics is safe to serialize and never opens desktop consent.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session serializes policy-gated desktop mutations. The underlying RobotGo input backends remain process-global, so only one agent Session may exist.

func NewSession 

func NewSession(config Config) (*Session, error)

NewSession creates the single active agent session for this process. Runtime capability discovery is bounded and never opens a consent dialog.

func (*Session) Catalog 

func (s *Session) Catalog() OperationCatalog

Catalog returns a defensive copy of the session's immutable catalog.

func (*Session) Close 

func (s *Session) Close() error

Close prevents future actions, waits for an active synchronous mutation, and releases the process-wide agent-session claim. It is safe to call repeatedly.

func (*Session) DryRun 

func (s *Session) DryRun(ctx context.Context, request ActionRequest) (ActionResult, error)

DryRun performs the same shape, policy, quota, capability, and cancellation preflight as Execute without injecting input or consuming action quota. A supplied observation precondition is recaptured and consumes observation quota because stale-target validation is a real sensitive read.

func (*Session) Execute 

func (s *Session) Execute(ctx context.Context, request ActionRequest) (ActionResult, error)

Execute validates and serially performs one typed desktop mutation.

func (*Session) FindColor 

func (s *Session) FindColor(ctx context.Context, request FindColorRequest) (FindColorResult, error)

FindColor evaluates a color condition against an explicit session-owned observation without touching the desktop.

func (*Session) Observe 

func (s *Session) Observe(ctx context.Context, request ObserveRequest) (*Observation, error)

Observe returns a diagnostics snapshot and optional bounded capture. It is serialized with actions so an observation cannot race a session mutation.

func (*Session) ReleaseObservation 

func (s *Session) ReleaseObservation(id string) error

ReleaseObservation idempotently removes one valid observation from the session and zeroes its optional capture. It can release observations returned directly by Observe or referenced by wait/verification results.

func (*Session) WaitColor 

func (s *Session) WaitColor(ctx context.Context, request WaitColorRequest) (WaitColorResult, error)

WaitColor captures and evaluates an explicit region until it matches or the bounded policy is exhausted. A successful result references the sole retained observation by ID; callers should pass it to ReleaseObservation when no longer needed. Every other temporary frame is zeroed before return.

type TypeTextAction 

type TypeTextAction struct {
	Text string `json:"text"`
}

TypeTextAction types UTF-8 text. The text is never copied into results.

type VerificationCondition 

type VerificationCondition string

VerificationCondition identifies a bounded post-action capture predicate. 

const (
	VerificationCaptureChanged   VerificationCondition = "capture-changed"
	VerificationCaptureUnchanged VerificationCondition = "capture-unchanged"
)

type VerificationRequest 

type VerificationRequest struct {
	Condition VerificationCondition `json:"condition"`
}

VerificationRequest compares post-action captures with the precondition observation. Attempt count and interval come from immutable session policy.

type VerificationResult 

type VerificationResult struct {
	Status    VerificationStatus    `json:"status"`
	Condition VerificationCondition `json:"condition"`
	Attempts  uint32                `json:"attempts"`
}

VerificationResult never includes capture pixels or digests.

type VerificationStatus 

type VerificationStatus string

VerificationStatus is the machine-readable post-action outcome. 

const (
	VerificationPassed VerificationStatus = "passed"
	VerificationFailed VerificationStatus = "failed"
)

type VisualMatch 

type VisualMatch struct {
	X         int `json:"x"`
	Y         int `json:"y"`
	DisplayID int `json:"display_id"`
}

VisualMatch identifies the first matching global logical coordinate.

type WaitColorRequest 

type WaitColorRequest struct {
	Region    CaptureRegion  `json:"region"`
	Condition ColorCondition `json:"condition"`
	Confirmed bool           `json:"confirmed,omitempty"`
}

WaitColorRequest polls one explicit capture region using the immutable attempt, interval, timeout, and quota limits in Policy.

type WaitColorResult 

type WaitColorResult struct {
	SchemaVersion string          `json:"schema_version"`
	ConditionID   string          `json:"condition_id"`
	Status        ConditionStatus `json:"status"`
	Attempts      uint32          `json:"attempts"`
	ObservationID string          `json:"observation_id,omitempty"`
	Match         *VisualMatch    `json:"match,omitempty"`
}

WaitColorResult contains bounded polling evidence but no pixels, capture digest, or target color. A matched result identifies the retained observation.

Source Files

View all Source files

Directories

Path Synopsis
Package mcpserver exposes an agent Session through a small, local-only MCP tool surface.
 Package mcpserver exposes an agent Session through a small, local-only MCP tool surface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.