maps

package
v1.148.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package maps keeps the basemap a deployment serves for street maps (#2068): the operator's settings, the regions extracted from an OpenStreetMap build into the operator's bucket, the background fetch that extracts them, and the read the public archive route and the maps knowledge page make.

A map is an HTML asset that loads the runtime the portal serves under /portal/vendor/maplibre/ and reads one of these archives by byte range from /portal/maps/. No map view and no thumbnail calls a third party; only the fetch does, and only when an administrator saves a region.

Index

Constants

View Source
const (
	StateQueued   = "queued"
	StateFetching = "fetching"
	StateReady    = "ready"
	StateFailed   = "failed"
)

The states a region's last fetch can be in.

View Source
const (
	OriginFetch  = "fetch"  // extracted by the platform from the source build
	OriginUpload = "upload" // a .pmtiles file the operator put in the bucket
)

The ways a region came to be.

View Source
const (
	ObjectPrefix = "maps/"
	UploadPrefix = ObjectPrefix + "uploads/"
)

Where archives live in the bucket. A fetched region's archive is named by its build and the time it was written, so a refresh writes beside the archive being served and the swap is a row update; an uploaded archive is any .pmtiles file directly under the uploads prefix.

View Source
const (
	ProtomapsIndexURL = "https://build-metadata.protomaps.dev/builds.json"
	ProtomapsBuildURL = "https://build.protomaps.com/"
)

Where the current Protomaps build is found. Protomaps keeps a week of daily builds and one per version and says their URLs may change, so a fetch resolves the newest daily build from the index when it starts rather than storing a file name.

View Source
const DefaultEvery = 30 * time.Second

DefaultEvery is the wait between fetch passes. A save on this replica wakes its pass at once; another replica's save is picked up within this.

View Source
const DefaultMaxZoom = 14

DefaultMaxZoom is the deepest zoom a region is extracted to unless the operator says otherwise. The Protomaps build goes to 15; the map runtime draws the zoom-14 tiles at 15 and beyond without loss a reader would notice, and the contiguous United States is about 8.8 GB at 14 against 18.8 GB at 15.

View Source
const DefaultPresetID = "united-states"

DefaultPresetID is the preset the settings page offers first.

View Source
const FetchLockKey int64 = 4713210301

FetchLockKey is the advisory lock a fetch pass holds, so one replica does the work. Distinct from every other advisory lock key the platform takes.

View Source
const RegionsPath = "/portal/maps/regions"

RegionsPath is where the public route lists the ready regions.

View Source
const SettingsSection = "maps"

SettingsSection is the platform_settings row the maps settings live in.

Variables

View Source
var (
	ErrRegionNotFound = errors.New("maps: no such region")
	ErrRegionExists   = errors.New("maps: a region with this id already exists")
	ErrRegionBusy     = errors.New("maps: the region is being fetched")
)

Errors the region store reports.

View Source
var ErrUnavailable = errors.New("maps: no such archive")

ErrUnavailable is an archive the public route does not serve: maps are off, or the region has no ready archive.

Functions

func ArchivePath

func ArchivePath(id string) string

ArchivePath is where the public route serves a region's archive.

func ValidRegionID

func ValidRegionID(id string) bool

ValidRegionID reports whether id can name a region: it is the archive's file name in the public route, so it is lowercase letters, digits and dashes.

Types

type Archive

type Archive struct {
	Bucket string `json:"-"`
	Key    string `json:"-"`
	Size   int64  `json:"size_bytes"`
	// Build is the OpenStreetMap data the archive was built from, as the
	// source's metadata dates it (the replication time), or the source build's
	// name when it carries none.
	Build   string         `json:"build"`
	MinZoom int            `json:"min_zoom"`
	MaxZoom int            `json:"max_zoom"`
	Bounds  pmtiles.Bounds `json:"bounds"`
	ReadyAt time.Time      `json:"ready_at"`
}

Archive is the archive a region serves.

func (Archive) ETag

func (a Archive) ETag() string

ETag is the archive's validator for a range read: the object key, which a refresh changes.

type Bucket

type Bucket struct {
	Objects Objects
	Name    string
}

Bucket is an opened bucket: its client and its name.

type Deps

type Deps struct {
	DB       *sql.DB
	Settings SettingsStore
	Regions  RegionStore
	Open     OpenBucket
	// Client is the outbound client a fetch reads its source through.
	Client *http.Client
	// Changed is called after anything the maps knowledge page shows has
	// changed: the settings, or a region becoming ready or going away.
	Changed func(ctx context.Context)
	// Every is the wait between fetch passes on a replica no one woke.
	Every time.Duration
	// IndexURL and BuildURL locate the Protomaps builds; empty is Protomaps'.
	IndexURL string
	BuildURL string
	// RetryPause is the wait before a failed range read is tried again; nil
	// is 1s, 4s, 9s.
	RetryPause func(attempt int) time.Duration
}

Deps is what a Service is built from.

type Estimate

type Estimate struct {
	SizeBytes int64          `json:"size_bytes"`
	Tiles     uint64         `json:"tiles"`
	Build     string         `json:"build"`
	MinZoom   int            `json:"min_zoom"`
	MaxZoom   int            `json:"max_zoom"`
	Bounds    pmtiles.Bounds `json:"bounds"`
}

Estimate is what a fetch of a region would write.

type EstimateInput

type EstimateInput struct {
	Preset string          `json:"preset,omitempty" example:"united-states"`
	Bounds *pmtiles.Bounds `json:"bounds,omitempty"`
}

EstimateInput asks what a region would cost before it is saved.

type InputError

type InputError struct {
	// contains filtered or unexported fields
}

InputError is a request the caller can correct; its message says how.

func (*InputError) Error

func (e *InputError) Error() string

type Objects

type Objects interface {
	PutObjectStream(ctx context.Context, bucket, key string, body io.Reader, contentType string) (int64, error)
	GetObjectRange(ctx context.Context, bucket, key string, offset, length int64) ([]byte, int64, error)
	DeleteObject(ctx context.Context, bucket, key string) error
	ListDirectory(ctx context.Context, bucket, prefix string) ([]s3adapter.ObjectEntry, bool, error)
}

Objects is what the maps surface does with the bucket.

type OpenBucket

type OpenBucket func(ctx context.Context, s Settings) (Bucket, error)

OpenBucket opens the bucket the settings name, resolving their empty connection and bucket to the managed-resources ones.

type PostgresRegions

type PostgresRegions struct {
	// contains filtered or unexported fields
}

PostgresRegions is the map_regions table.

func NewPostgresRegions

func NewPostgresRegions(db *sql.DB) *PostgresRegions

NewPostgresRegions returns the region store over db.

func (*PostgresRegions) ClaimNext

func (s *PostgresRegions) ClaimNext(ctx context.Context) (*Region, error)

ClaimNext marks the longest-waiting queued region fetching and returns it, or nil when none is queued.

func (*PostgresRegions) Create

func (s *PostgresRegions) Create(ctx context.Context, r Region) error

Create adds a region queued for its first fetch.

func (*PostgresRegions) Delete

func (s *PostgresRegions) Delete(ctx context.Context, id string) (*Region, error)

Delete removes a region and returns what it was, so its archive can be removed from the bucket.

func (*PostgresRegions) DeleteUploadsExcept

func (s *PostgresRegions) DeleteUploadsExcept(ctx context.Context, keep []string) ([]Region, error)

DeleteUploadsExcept removes the uploaded regions whose object key is not in keep, which is what the bucket holds now, and returns them.

func (*PostgresRegions) Fail

func (s *PostgresRegions) Fail(ctx context.Context, id, reason string) error

Fail records why a fetch stopped. The archive being served is untouched.

func (*PostgresRegions) Finish

func (s *PostgresRegions) Finish(ctx context.Context, id string, a Archive) (*Archive, bool, error)

Finish makes a completed fetch's archive the one served and returns the archive it replaced. ok is false when the region was deleted or requeued while the fetch ran; the caller then removes the archive it wrote.

func (*PostgresRegions) Get

func (s *PostgresRegions) Get(ctx context.Context, id string) (*Region, error)

Get returns one region.

func (*PostgresRegions) List

func (s *PostgresRegions) List(ctx context.Context) ([]Region, error)

List returns every region, by name.

func (*PostgresRegions) Progress

func (s *PostgresRegions) Progress(ctx context.Context, id string, done, total int64) error

Progress records how much of a fetch is written.

func (*PostgresRegions) PutUpload

func (s *PostgresRegions) PutUpload(ctx context.Context, r Region) error

PutUpload records an archive the operator uploaded, as its header describes it, or as failed with the reason its header could not be read. The object key is recorded either way, so the next scan knows the file. A region of the same id the platform fetches is left alone.

func (*PostgresRegions) Requeue

func (s *PostgresRegions) Requeue(ctx context.Context, id string, maxZoom int) error

Requeue asks for a fresh fetch of a region at maxZoom. A region already being fetched is refused; an uploaded one is not fetched.

func (*PostgresRegions) ResetStale

func (s *PostgresRegions) ResetStale(ctx context.Context) error

ResetStale queues again every region left fetching. It is called only by a worker holding the fetch lock, so a region in that state belongs to a worker that died.

type PostgresSettings

type PostgresSettings struct {
	// contains filtered or unexported fields
}

PostgresSettings is the section's row in platform_settings.

func NewPostgresSettings

func NewPostgresSettings(db *sql.DB) *PostgresSettings

NewPostgresSettings returns the settings store over db.

func (*PostgresSettings) Get

Get returns the stored settings, or the defaults when none were saved.

func (*PostgresSettings) Set

func (s *PostgresSettings) Set(ctx context.Context, in Settings, author string) error

Set upserts the settings.

type Preset

type Preset struct {
	ID     string         `json:"id"`
	Name   string         `json:"name"`
	Bounds pmtiles.Bounds `json:"bounds"`
}

Preset is a named region an operator can pick instead of drawing a box.

func PresetByID

func PresetByID(id string) (Preset, bool)

PresetByID returns the preset with the id.

func Presets

func Presets() []Preset

Presets returns the regions offered by name.

type Reader

type Reader struct {
	Settings SettingsStore
	Regions  RegionStore
}

Reader reads what the maps surface offers a map, from the stores alone: the knowledge page's reconcile reads it at startup, before any service runs.

func NewReader

func NewReader(db *sql.DB) *Reader

NewReader returns the reader over db's stores, or nil without a database.

func (*Reader) State

func (rd *Reader) State(ctx context.Context) (State, error)

State returns whether maps are enabled and the regions a map can load. Disabled, it lists none.

type Ready

type Ready struct {
	ID      string         `json:"id"`
	Name    string         `json:"name"`
	URL     string         `json:"url"`
	Bounds  pmtiles.Bounds `json:"bounds"`
	MinZoom int            `json:"min_zoom"`
	MaxZoom int            `json:"max_zoom"`
	Build   string         `json:"build"`
	Size    int64          `json:"size_bytes"`
}

Ready is a region a map can load: what the public listing and the knowledge page show.

type Region

type Region struct {
	ID      string         `json:"id"`
	Name    string         `json:"name"`
	Preset  string         `json:"preset,omitempty"`
	Origin  string         `json:"origin"`
	Bounds  pmtiles.Bounds `json:"bounds"`
	MaxZoom int            `json:"max_zoom"`
	// State is the last fetch's: queued, fetching, ready or failed. A failed
	// refresh of a region that was ready keeps serving Archive.
	State         string     `json:"state"`
	ProgressBytes int64      `json:"progress_bytes"`
	TotalBytes    int64      `json:"total_bytes"`
	Error         string     `json:"error,omitempty"`
	RequestedAt   time.Time  `json:"requested_at"`
	StartedAt     *time.Time `json:"started_at,omitempty"`
	FinishedAt    *time.Time `json:"finished_at,omitempty"`
	Archive       *Archive   `json:"archive,omitempty"`
	CreatedBy     string     `json:"created_by,omitempty"`
}

Region is one basemap region and the archive serving it.

type RegionInput

type RegionInput struct {
	Preset string          `json:"preset,omitempty" example:"united-states"`
	ID     string          `json:"id,omitempty" example:"bay-area"`
	Name   string          `json:"name,omitempty" example:"Bay Area"`
	Bounds *pmtiles.Bounds `json:"bounds,omitempty"`
}

RegionInput adds a region: a preset by id, or a named box.

type RegionStore

type RegionStore interface {
	List(ctx context.Context) ([]Region, error)
	Get(ctx context.Context, id string) (*Region, error)
	Create(ctx context.Context, r Region) error
	Delete(ctx context.Context, id string) (*Region, error)
	Requeue(ctx context.Context, id string, maxZoom int) error
	ResetStale(ctx context.Context) error
	ClaimNext(ctx context.Context) (*Region, error)
	Progress(ctx context.Context, id string, done, total int64) error
	Finish(ctx context.Context, id string, a Archive) (previous *Archive, ok bool, err error)
	Fail(ctx context.Context, id, reason string) error
	PutUpload(ctx context.Context, r Region) error
	DeleteUploadsExcept(ctx context.Context, keep []string) ([]Region, error)
}

RegionStore persists regions.

type ResolvedBucket

type ResolvedBucket func(s Settings) string

ResolvedBucket names the bucket archives go to under s.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service is the maps surface: the settings and regions an administrator manages, the background fetch, and the reads the public routes make.

func New

func New(d Deps) *Service

New returns the maps service.

func (*Service) AddRegion

func (s *Service) AddRegion(ctx context.Context, in RegionInput, author string) (*Region, error)

AddRegion saves a region and queues its fetch at the settings' zoom.

func (*Service) DeleteRegion

func (s *Service) DeleteRegion(ctx context.Context, id string) error

DeleteRegion removes a region and its archive. An uploaded region is removed by deleting its file; deleting the row alone would bring it back on the next scan.

func (*Service) Estimate

func (s *Service) Estimate(ctx context.Context, in EstimateInput) (Estimate, error)

Estimate plans an extract without writing it and reports its size.

func (*Service) Pass

func (s *Service) Pass(ctx context.Context) error

Pass is one iteration of the fetch loop: under the fetch lock, it records what the uploads prefix holds and fetches every queued region.

func (*Service) RefreshRegion

func (s *Service) RefreshRegion(ctx context.Context, id string) (*Region, error)

RefreshRegion queues a fetch of the newest build, at the settings' zoom. The archive being served keeps serving until the new one is complete.

func (*Service) SaveSettings

func (s *Service) SaveSettings(ctx context.Context, in SettingsInput, author string) error

SaveSettings stores the settings and wakes the fetch.

func (*Service) ServedArchive

func (s *Service) ServedArchive(ctx context.Context, id string) (Archive, Objects, error)

ServedArchive returns the archive a ready region serves and the bucket client to read it with.

func (*Service) Start

func (s *Service) Start(ctx context.Context)

Start runs the fetch loop until Stop.

func (*Service) State

func (s *Service) State(ctx context.Context) (State, error)

State returns whether maps are enabled and the regions a map can load.

func (*Service) Stop

func (s *Service) Stop()

Stop ends the fetch loop and waits for it. A fetch in flight is canceled and left fetching, so the next pass on any replica queues it again; the part it wrote is removed.

func (*Service) View

func (s *Service) View(ctx context.Context, bucketOf ResolvedBucket) (View, error)

View returns the settings and every region.

type Settings

type Settings struct {
	// Enabled turns the maps surface on: the archive routes answer, the fetch
	// runs, and the knowledge page lists the regions. Off by default.
	Enabled bool `json:"enabled"`
	// S3Connection names the S3 connection archives are written to. Empty is
	// the managed-resources connection.
	S3Connection string `json:"s3_connection"`
	// Bucket is the bucket archives are written to. Empty is the
	// managed-resources bucket.
	Bucket string `json:"bucket"`
	// MaxZoom is the deepest zoom a fetch extracts.
	MaxZoom int `json:"max_zoom"`
	// SourceURL is the archive a fetch extracts from. Empty is the current
	// Protomaps daily build, resolved when each fetch starts.
	SourceURL string `json:"source_url"`

	UpdatedBy string    `json:"-"`
	UpdatedAt time.Time `json:"-"`
}

Settings is the operator's maps configuration.

func DefaultSettings

func DefaultSettings() Settings

DefaultSettings is what a deployment that never saved the section has.

func (Settings) Validate

func (s Settings) Validate() string

Validate refuses settings a fetch could not act on, returning the reason or "".

type SettingsInput

type SettingsInput struct {
	Enabled      bool   `json:"enabled"`
	S3Connection string `json:"s3_connection"`
	Bucket       string `json:"bucket"`
	MaxZoom      int    `json:"max_zoom" example:"14"`
	SourceURL    string `json:"source_url"`
}

SettingsInput is a settings write.

type SettingsStore

type SettingsStore interface {
	Get(ctx context.Context) (Settings, error)
	Set(ctx context.Context, s Settings, author string) error
}

SettingsStore persists the section.

type SettingsView

type SettingsView struct {
	Enabled      bool   `json:"enabled"`
	S3Connection string `json:"s3_connection"`
	Bucket       string `json:"bucket"`
	MaxZoom      int    `json:"max_zoom"`
	SourceURL    string `json:"source_url"`
	// ResolvedBucket is the bucket archives are written to: Bucket, or the
	// managed-resources bucket when that is empty.
	ResolvedBucket string     `json:"resolved_bucket"`
	UpdatedBy      string     `json:"updated_by,omitempty"`
	UpdatedAt      *time.Time `json:"updated_at,omitempty"`
}

SettingsView is the stored settings with the defaults they resolve to.

type State

type State struct {
	Enabled bool    `json:"enabled"`
	Regions []Ready `json:"regions"`
}

State is what the maps surface offers a map right now.

type View

type View struct {
	Settings SettingsView `json:"settings"`
	Regions  []Region     `json:"regions"`
	Presets  []Preset     `json:"presets"`
	// UploadPrefix is where in the bucket an operator puts a .pmtiles file to
	// serve it without a fetch.
	UploadPrefix string `json:"upload_prefix"`
	// DefaultSource is the source a fetch reads when source_url is empty.
	DefaultSource string `json:"default_source"`
}

View is the maps section as the settings page shows it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL