desktop

package
v2.18.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2026 License: GPL-3.0 Imports: 16 Imported by: 0

Documentation

Overview

Package desktop captures the screen and sends pointer and keyboard input to the desktop of the current user session.

A Driver builds clicks, drags, key combinations and typing from the few primitives a Backend implements for each operating system. Positions are pixels of the screenshots the driver captures, on the primary display.

Index

Constants

This section is empty.

Variables

View Source
var ErrUnsupported = errors.New("desktop automation is supported on macOS and Windows only")

ErrUnsupported reports an operating system without a desktop backend.

Functions

func Crop

func Crop(src *image.RGBA, rect image.Rectangle) *image.RGBA

Crop returns the part of an image inside a rectangle, clipped to the image.

func EncodePNG

func EncodePNG(img image.Image) ([]byte, error)

EncodePNG encodes an image as PNG, favoring speed over size.

func FitSize

func FitSize(width, height, longEdge, maxPixels int) (int, int)

FitSize returns the largest size with the same aspect ratio that fits a long edge and a pixel count; a zero limit is ignored. The size never grows.

func Launch

func Launch(dir, command string, args []string) error

Launch starts an application in a working directory without waiting for it. The application keeps running after the calling process exits.

func RequestPermissions

func RequestPermissions()

RequestPermissions does nothing where desktop automation is unsupported.

func Resize

func Resize(src *image.RGBA, width, height int) *image.RGBA

Resize scales an image down to a size by averaging the source pixels each target pixel covers.

Types

type Backend

type Backend interface {
	// Capture returns the primary display in physical pixels.
	Capture() (*image.RGBA, error)
	// MoveTo moves the pointer to a physical pixel position.
	MoveTo(x, y int) error
	// Position reports the pointer's physical pixel position.
	Position() (x, y int, err error)
	// Button presses or releases a button at the pointer. clicks counts
	// the clicks of a multi-click so far, starting at 1.
	Button(button Button, down bool, clicks int) error
	// Wheel scrolls by wheel notches; positive values scroll right and down.
	Wheel(dx, dy int) error
	// Key presses or releases a key.
	Key(key Key, down bool) error
	// Type types text at the keyboard focus.
	Type(text string) error
	// LastInput reports when the desktop last received pointer or keyboard
	// input from any source, the driver included, or the zero time when it
	// cannot tell.
	LastInput() time.Time
	// Close releases the backend.
	Close() error
}

Backend is the operating system's side of a Driver.

type Button

type Button string

Button is a pointer button.

const (
	ButtonLeft   Button = "left"
	ButtonRight  Button = "right"
	ButtonMiddle Button = "middle"
)

Pointer buttons.

func ParseButton

func ParseButton(name string) (Button, error)

ParseButton accepts a button name, defaulting to the left button.

type Diagnostics

type Diagnostics struct {
	OS     string
	Width  int
	Height int
	// Problems are the reasons automation cannot work, such as a missing
	// permission; empty when the desktop is usable.
	Problems []Problem
}

Diagnostics describes whether the current session can be automated.

func Check

func Check() Diagnostics

Check reports whether the desktop of the current session can be automated.

func (Diagnostics) Err

func (d Diagnostics) Err() error

Err summarizes the problems as one error, or returns nil.

type Driver

type Driver struct {
	// contains filtered or unexported fields
}

Driver performs desktop actions on a Backend.

func New

func New(backend Backend) *Driver

New returns a driver for a backend.

func Open

func Open() (*Driver, error)

Open returns a driver for the desktop of the current session.

func (*Driver) AssumeInputSent

func (d *Driver) AssumeInputSent(at time.Time)

AssumeInputSent records that input was sent at, such as by an earlier driver on the same desktop, so the desktop reporting it is not taken for a person's.

func (*Driver) Click

func (d *Driver) Click(ctx context.Context, at *image.Point, button Button, count int, modifiers []Key) error

Click clicks a button count times at a position, or at the pointer when at is nil, while holding modifiers.

func (*Driver) Close

func (d *Driver) Close() error

Close releases the driver.

func (*Driver) CursorPosition

func (d *Driver) CursorPosition() (image.Point, error)

CursorPosition reports the pointer position.

func (*Driver) Drag

func (d *Driver) Drag(ctx context.Context, path []image.Point, modifiers []Key) error

Drag holds the left button from the first point of a path to the last, moving in small steps so applications see the drag.

func (*Driver) HoldKeys

func (d *Driver) HoldKeys(ctx context.Context, keys []Key, duration time.Duration) error

HoldKeys holds keys down for a duration.

func (*Driver) InputSentAt

func (d *Driver) InputSentAt() time.Time

InputSentAt reports when the driver last sent input.

func (*Driver) Move

func (d *Driver) Move(ctx context.Context, at image.Point) error

Move moves the pointer.

func (*Driver) MoveHolding

func (d *Driver) MoveHolding(ctx context.Context, at image.Point, modifiers []Key) error

MoveHolding moves the pointer while holding modifiers, as when hovering with a key pressed.

func (*Driver) PersonInputSince

func (d *Driver) PersonInputSince(since time.Time) bool

PersonInputSince reports whether input the driver did not send, such as a person's, reached the desktop after since.

func (*Driver) PressButton

func (d *Driver) PressButton(button Button) error

PressButton presses a button at the pointer.

func (*Driver) PressKeys

func (d *Driver) PressKeys(ctx context.Context, keys []Key, repeat int) error

PressKeys presses keys together, then releases them, repeat times.

func (*Driver) ReleaseButton

func (d *Driver) ReleaseButton(button Button) error

ReleaseButton releases a button at the pointer.

func (*Driver) Screenshot

func (d *Driver) Screenshot() (*image.RGBA, error)

Screenshot captures the primary display.

func (*Driver) Scroll

func (d *Driver) Scroll(ctx context.Context, at *image.Point, dx, dy int, modifiers []Key) error

Scroll scrolls wheel notches at a position, or at the pointer when at is nil, while holding modifiers.

func (*Driver) Type

func (d *Driver) Type(ctx context.Context, text string) error

Type types text. Line breaks are sent as the Enter key, which applications treat more consistently than a typed newline.

func (*Driver) WaitForIdle

func (d *Driver) WaitForIdle(ctx context.Context, idle, poll time.Duration, onWait func()) error

WaitForIdle returns once nobody but the driver has used the desktop for idle, checking every poll. onWait runs once if the driver has to wait.

type Fingerprint

type Fingerprint uint64

Fingerprint is a perceptual hash of an image: similar images have fingerprints a small Distance apart.

func FingerprintAround

func FingerprintAround(img *image.RGBA, at image.Point, radius int) Fingerprint

FingerprintAround hashes the square of an image centered on a point, which tells apart screens that differ only near where an action lands.

func FingerprintOf

func FingerprintOf(img *image.RGBA) Fingerprint

FingerprintOf hashes an image by whether brightness rises between neighboring cells of a coarse grid.

func (Fingerprint) Distance

func (f Fingerprint) Distance(other Fingerprint) int

Distance counts the bits two fingerprints differ in, from 0 to 64.

type Key

type Key string

Key is a keyboard key: one of the named keys below, a function key "f1" to "f24", or a single printable ASCII character such as "a" or "/".

const (
	KeyShift       Key = "shift"
	KeyCtrl        Key = "ctrl"
	KeyAlt         Key = "alt"
	KeyMeta        Key = "meta" // Command on macOS, Windows key on Windows.
	KeyEnter       Key = "enter"
	KeyTab         Key = "tab"
	KeyEscape      Key = "escape"
	KeyBackspace   Key = "backspace"
	KeyDelete      Key = "delete"
	KeySpace       Key = "space"
	KeyInsert      Key = "insert"
	KeyHome        Key = "home"
	KeyEnd         Key = "end"
	KeyPageUp      Key = "pageup"
	KeyPageDown    Key = "pagedown"
	KeyUp          Key = "up"
	KeyDown        Key = "down"
	KeyLeft        Key = "left"
	KeyRight       Key = "right"
	KeyCapsLock    Key = "capslock"
	KeyPrintScreen Key = "printscreen"
)

Named keys.

func ParseKey

func ParseKey(name string) (Key, error)

ParseKey accepts a key name in any common spelling, ignoring case.

func ParseKeys

func ParseKeys(names []string) ([]Key, error)

ParseKeys parses each key name.

func (Key) Char

func (k Key) Char() (rune, bool)

Char returns the character of a single-character key.

func (Key) FunctionNumber

func (k Key) FunctionNumber() (int, bool)

FunctionNumber returns n for the function key "fn".

func (Key) USKey

func (k Key) USKey() (base rune, shift bool, ok bool)

USKey returns the unshifted character that produces a character key on a US layout and whether Shift is needed.

type Problem

type Problem struct {
	// Code identifies the kind of problem, such as screen_locked, for
	// programs that act on it.
	Code string `json:"code"`
	// Message explains the problem and how to fix it.
	Message string `json:"message"`
}

Problem is one reason the desktop cannot be automated.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL