Documentation
¶
Overview ¶
Package maps keeps the basemap a deployment serves for street maps (#2068): the operator's settings, the regions extracted from an OpenStreetMap build into the operator's bucket, the background fetch that extracts them, and the read the public archive route and the maps knowledge page make.
A map is an HTML asset that loads the runtime the portal serves under /portal/vendor/maplibre/ and reads one of these archives by byte range from /portal/maps/. No map view and no thumbnail calls a third party; only the fetch does, and only when an administrator saves a region.
Index ¶
- Constants
- Variables
- func ArchivePath(id string) string
- func ValidRegionID(id string) bool
- type Archive
- type Bucket
- type Deps
- type Estimate
- type EstimateInput
- type InputError
- type Objects
- type OpenBucket
- type PostgresRegions
- func (s *PostgresRegions) ClaimNext(ctx context.Context) (*Region, error)
- func (s *PostgresRegions) Create(ctx context.Context, r Region) error
- func (s *PostgresRegions) Delete(ctx context.Context, id string) (*Region, error)
- func (s *PostgresRegions) DeleteUploadsExcept(ctx context.Context, keep []string) ([]Region, error)
- func (s *PostgresRegions) Fail(ctx context.Context, id, reason string) error
- func (s *PostgresRegions) Finish(ctx context.Context, id string, a Archive) (*Archive, bool, error)
- func (s *PostgresRegions) Get(ctx context.Context, id string) (*Region, error)
- func (s *PostgresRegions) List(ctx context.Context) ([]Region, error)
- func (s *PostgresRegions) Progress(ctx context.Context, id string, done, total int64) error
- func (s *PostgresRegions) PutUpload(ctx context.Context, r Region) error
- func (s *PostgresRegions) Requeue(ctx context.Context, id string, maxZoom int) error
- func (s *PostgresRegions) ResetStale(ctx context.Context) error
- type PostgresSettings
- type Preset
- type Reader
- type Ready
- type Region
- type RegionInput
- type RegionStore
- type ResolvedBucket
- type Service
- func (s *Service) AddRegion(ctx context.Context, in RegionInput, author string) (*Region, error)
- func (s *Service) DeleteRegion(ctx context.Context, id string) error
- func (s *Service) Estimate(ctx context.Context, in EstimateInput) (Estimate, error)
- func (s *Service) Pass(ctx context.Context) error
- func (s *Service) RefreshRegion(ctx context.Context, id string) (*Region, error)
- func (s *Service) SaveSettings(ctx context.Context, in SettingsInput, author string) error
- func (s *Service) ServedArchive(ctx context.Context, id string) (Archive, Objects, error)
- func (s *Service) Start(ctx context.Context)
- func (s *Service) State(ctx context.Context) (State, error)
- func (s *Service) Stop()
- func (s *Service) View(ctx context.Context, bucketOf ResolvedBucket) (View, error)
- type Settings
- type SettingsInput
- type SettingsStore
- type SettingsView
- type State
- type View
Constants ¶
const ( StateQueued = "queued" StateFetching = "fetching" StateReady = "ready" StateFailed = "failed" )
The states a region's last fetch can be in.
const ( OriginFetch = "fetch" // extracted by the platform from the source build OriginUpload = "upload" // a .pmtiles file the operator put in the bucket )
The ways a region came to be.
const ( ObjectPrefix = "maps/" UploadPrefix = ObjectPrefix + "uploads/" )
Where archives live in the bucket. A fetched region's archive is named by its build and the time it was written, so a refresh writes beside the archive being served and the swap is a row update; an uploaded archive is any .pmtiles file directly under the uploads prefix.
const ( ProtomapsIndexURL = "https://build-metadata.protomaps.dev/builds.json" ProtomapsBuildURL = "https://build.protomaps.com/" )
Where the current Protomaps build is found. Protomaps keeps a week of daily builds and one per version and says their URLs may change, so a fetch resolves the newest daily build from the index when it starts rather than storing a file name.
const DefaultEvery = 30 * time.Second
DefaultEvery is the wait between fetch passes. A save on this replica wakes its pass at once; another replica's save is picked up within this.
const DefaultMaxZoom = 14
DefaultMaxZoom is the deepest zoom a region is extracted to unless the operator says otherwise. The Protomaps build goes to 15; the map runtime draws the zoom-14 tiles at 15 and beyond without loss a reader would notice, and the contiguous United States is about 8.8 GB at 14 against 18.8 GB at 15.
const DefaultPresetID = "united-states"
DefaultPresetID is the preset the settings page offers first.
const FetchLockKey int64 = 4713210301
FetchLockKey is the advisory lock a fetch pass holds, so one replica does the work. Distinct from every other advisory lock key the platform takes.
const RegionsPath = "/portal/maps/regions"
RegionsPath is where the public route lists the ready regions.
const SettingsSection = "maps"
SettingsSection is the platform_settings row the maps settings live in.
Variables ¶
var ( ErrRegionNotFound = errors.New("maps: no such region") ErrRegionExists = errors.New("maps: a region with this id already exists") ErrRegionBusy = errors.New("maps: the region is being fetched") )
Errors the region store reports.
ErrUnavailable is an archive the public route does not serve: maps are off, or the region has no ready archive.
Functions ¶
func ArchivePath ¶
ArchivePath is where the public route serves a region's archive.
func ValidRegionID ¶
ValidRegionID reports whether id can name a region: it is the archive's file name in the public route, so it is lowercase letters, digits and dashes.
Types ¶
type Archive ¶
type Archive struct {
Bucket string `json:"-"`
Key string `json:"-"`
Size int64 `json:"size_bytes"`
// Build is the OpenStreetMap data the archive was built from, as the
// source's metadata dates it (the replication time), or the source build's
// name when it carries none.
Build string `json:"build"`
MinZoom int `json:"min_zoom"`
MaxZoom int `json:"max_zoom"`
Bounds pmtiles.Bounds `json:"bounds"`
ReadyAt time.Time `json:"ready_at"`
}
Archive is the archive a region serves.
type Deps ¶
type Deps struct {
DB *sql.DB
Settings SettingsStore
Regions RegionStore
Open OpenBucket
// Client is the outbound client a fetch reads its source through.
Client *http.Client
// Changed is called after anything the maps knowledge page shows has
// changed: the settings, or a region becoming ready or going away.
Changed func(ctx context.Context)
// Every is the wait between fetch passes on a replica no one woke.
Every time.Duration
// IndexURL and BuildURL locate the Protomaps builds; empty is Protomaps'.
IndexURL string
BuildURL string
// RetryPause is the wait before a failed range read is tried again; nil
// is 1s, 4s, 9s.
RetryPause func(attempt int) time.Duration
}
Deps is what a Service is built from.
type Estimate ¶
type Estimate struct {
SizeBytes int64 `json:"size_bytes"`
Tiles uint64 `json:"tiles"`
Build string `json:"build"`
MinZoom int `json:"min_zoom"`
MaxZoom int `json:"max_zoom"`
Bounds pmtiles.Bounds `json:"bounds"`
}
Estimate is what a fetch of a region would write.
type EstimateInput ¶
type EstimateInput struct {
Preset string `json:"preset,omitempty" example:"united-states"`
Bounds *pmtiles.Bounds `json:"bounds,omitempty"`
}
EstimateInput asks what a region would cost before it is saved.
type InputError ¶
type InputError struct {
// contains filtered or unexported fields
}
InputError is a request the caller can correct; its message says how.
func (*InputError) Error ¶
func (e *InputError) Error() string
type Objects ¶
type Objects interface {
PutObjectStream(ctx context.Context, bucket, key string, body io.Reader, contentType string) (int64, error)
GetObjectRange(ctx context.Context, bucket, key string, offset, length int64) ([]byte, int64, error)
DeleteObject(ctx context.Context, bucket, key string) error
ListDirectory(ctx context.Context, bucket, prefix string) ([]s3adapter.ObjectEntry, bool, error)
}
Objects is what the maps surface does with the bucket.
type OpenBucket ¶
OpenBucket opens the bucket the settings name, resolving their empty connection and bucket to the managed-resources ones.
type PostgresRegions ¶
type PostgresRegions struct {
// contains filtered or unexported fields
}
PostgresRegions is the map_regions table.
func NewPostgresRegions ¶
func NewPostgresRegions(db *sql.DB) *PostgresRegions
NewPostgresRegions returns the region store over db.
func (*PostgresRegions) ClaimNext ¶
func (s *PostgresRegions) ClaimNext(ctx context.Context) (*Region, error)
ClaimNext marks the longest-waiting queued region fetching and returns it, or nil when none is queued.
func (*PostgresRegions) Create ¶
func (s *PostgresRegions) Create(ctx context.Context, r Region) error
Create adds a region queued for its first fetch.
func (*PostgresRegions) Delete ¶
Delete removes a region and returns what it was, so its archive can be removed from the bucket.
func (*PostgresRegions) DeleteUploadsExcept ¶
DeleteUploadsExcept removes the uploaded regions whose object key is not in keep, which is what the bucket holds now, and returns them.
func (*PostgresRegions) Fail ¶
func (s *PostgresRegions) Fail(ctx context.Context, id, reason string) error
Fail records why a fetch stopped. The archive being served is untouched.
func (*PostgresRegions) Finish ¶
Finish makes a completed fetch's archive the one served and returns the archive it replaced. ok is false when the region was deleted or requeued while the fetch ran; the caller then removes the archive it wrote.
func (*PostgresRegions) List ¶
func (s *PostgresRegions) List(ctx context.Context) ([]Region, error)
List returns every region, by name.
func (*PostgresRegions) PutUpload ¶
func (s *PostgresRegions) PutUpload(ctx context.Context, r Region) error
PutUpload records an archive the operator uploaded, as its header describes it, or as failed with the reason its header could not be read. The object key is recorded either way, so the next scan knows the file. A region of the same id the platform fetches is left alone.
func (*PostgresRegions) Requeue ¶
Requeue asks for a fresh fetch of a region at maxZoom. A region already being fetched is refused; an uploaded one is not fetched.
func (*PostgresRegions) ResetStale ¶
func (s *PostgresRegions) ResetStale(ctx context.Context) error
ResetStale queues again every region left fetching. It is called only by a worker holding the fetch lock, so a region in that state belongs to a worker that died.
type PostgresSettings ¶
type PostgresSettings struct {
// contains filtered or unexported fields
}
PostgresSettings is the section's row in platform_settings.
func NewPostgresSettings ¶
func NewPostgresSettings(db *sql.DB) *PostgresSettings
NewPostgresSettings returns the settings store over db.
type Preset ¶
type Preset struct {
ID string `json:"id"`
Name string `json:"name"`
Bounds pmtiles.Bounds `json:"bounds"`
}
Preset is a named region an operator can pick instead of drawing a box.
func PresetByID ¶
PresetByID returns the preset with the id.
type Reader ¶
type Reader struct {
Settings SettingsStore
Regions RegionStore
}
Reader reads what the maps surface offers a map, from the stores alone: the knowledge page's reconcile reads it at startup, before any service runs.
type Ready ¶
type Ready struct {
ID string `json:"id"`
Name string `json:"name"`
URL string `json:"url"`
Bounds pmtiles.Bounds `json:"bounds"`
MinZoom int `json:"min_zoom"`
MaxZoom int `json:"max_zoom"`
Build string `json:"build"`
Size int64 `json:"size_bytes"`
}
Ready is a region a map can load: what the public listing and the knowledge page show.
type Region ¶
type Region struct {
ID string `json:"id"`
Name string `json:"name"`
Preset string `json:"preset,omitempty"`
Origin string `json:"origin"`
Bounds pmtiles.Bounds `json:"bounds"`
MaxZoom int `json:"max_zoom"`
// State is the last fetch's: queued, fetching, ready or failed. A failed
// refresh of a region that was ready keeps serving Archive.
State string `json:"state"`
ProgressBytes int64 `json:"progress_bytes"`
TotalBytes int64 `json:"total_bytes"`
Error string `json:"error,omitempty"`
RequestedAt time.Time `json:"requested_at"`
StartedAt *time.Time `json:"started_at,omitempty"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
Archive *Archive `json:"archive,omitempty"`
CreatedBy string `json:"created_by,omitempty"`
}
Region is one basemap region and the archive serving it.
type RegionInput ¶
type RegionInput struct {
Preset string `json:"preset,omitempty" example:"united-states"`
ID string `json:"id,omitempty" example:"bay-area"`
Name string `json:"name,omitempty" example:"Bay Area"`
Bounds *pmtiles.Bounds `json:"bounds,omitempty"`
}
RegionInput adds a region: a preset by id, or a named box.
type RegionStore ¶
type RegionStore interface {
List(ctx context.Context) ([]Region, error)
Get(ctx context.Context, id string) (*Region, error)
Create(ctx context.Context, r Region) error
Delete(ctx context.Context, id string) (*Region, error)
Requeue(ctx context.Context, id string, maxZoom int) error
ResetStale(ctx context.Context) error
ClaimNext(ctx context.Context) (*Region, error)
Progress(ctx context.Context, id string, done, total int64) error
Finish(ctx context.Context, id string, a Archive) (previous *Archive, ok bool, err error)
Fail(ctx context.Context, id, reason string) error
PutUpload(ctx context.Context, r Region) error
DeleteUploadsExcept(ctx context.Context, keep []string) ([]Region, error)
}
RegionStore persists regions.
type ResolvedBucket ¶
ResolvedBucket names the bucket archives go to under s.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service is the maps surface: the settings and regions an administrator manages, the background fetch, and the reads the public routes make.
func (*Service) DeleteRegion ¶
DeleteRegion removes a region and its archive. An uploaded region is removed by deleting its file; deleting the row alone would bring it back on the next scan.
func (*Service) Pass ¶
Pass is one iteration of the fetch loop: under the fetch lock, it records what the uploads prefix holds and fetches every queued region.
func (*Service) RefreshRegion ¶
RefreshRegion queues a fetch of the newest build, at the settings' zoom. The archive being served keeps serving until the new one is complete.
func (*Service) SaveSettings ¶
SaveSettings stores the settings and wakes the fetch.
func (*Service) ServedArchive ¶
ServedArchive returns the archive a ready region serves and the bucket client to read it with.
type Settings ¶
type Settings struct {
// Enabled turns the maps surface on: the archive routes answer, the fetch
// runs, and the knowledge page lists the regions. Off by default.
Enabled bool `json:"enabled"`
// S3Connection names the S3 connection archives are written to. Empty is
// the managed-resources connection.
S3Connection string `json:"s3_connection"`
// Bucket is the bucket archives are written to. Empty is the
// managed-resources bucket.
Bucket string `json:"bucket"`
// MaxZoom is the deepest zoom a fetch extracts.
MaxZoom int `json:"max_zoom"`
// SourceURL is the archive a fetch extracts from. Empty is the current
// Protomaps daily build, resolved when each fetch starts.
SourceURL string `json:"source_url"`
UpdatedBy string `json:"-"`
UpdatedAt time.Time `json:"-"`
}
Settings is the operator's maps configuration.
func DefaultSettings ¶
func DefaultSettings() Settings
DefaultSettings is what a deployment that never saved the section has.
type SettingsInput ¶
type SettingsInput struct {
Enabled bool `json:"enabled"`
S3Connection string `json:"s3_connection"`
Bucket string `json:"bucket"`
MaxZoom int `json:"max_zoom" example:"14"`
SourceURL string `json:"source_url"`
}
SettingsInput is a settings write.
type SettingsStore ¶
type SettingsStore interface {
Get(ctx context.Context) (Settings, error)
Set(ctx context.Context, s Settings, author string) error
}
SettingsStore persists the section.
type SettingsView ¶
type SettingsView struct {
Enabled bool `json:"enabled"`
S3Connection string `json:"s3_connection"`
Bucket string `json:"bucket"`
MaxZoom int `json:"max_zoom"`
SourceURL string `json:"source_url"`
// ResolvedBucket is the bucket archives are written to: Bucket, or the
// managed-resources bucket when that is empty.
ResolvedBucket string `json:"resolved_bucket"`
UpdatedBy string `json:"updated_by,omitempty"`
UpdatedAt *time.Time `json:"updated_at,omitempty"`
}
SettingsView is the stored settings with the defaults they resolve to.
type View ¶
type View struct {
Settings SettingsView `json:"settings"`
Regions []Region `json:"regions"`
Presets []Preset `json:"presets"`
// UploadPrefix is where in the bucket an operator puts a .pmtiles file to
// serve it without a fetch.
UploadPrefix string `json:"upload_prefix"`
// DefaultSource is the source a fetch reads when source_url is empty.
DefaultSource string `json:"default_source"`
}
View is the maps section as the settings page shows it.