engine

package module
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 9, 2020 License: BSD-3-Clause Imports: 45 Imported by: 0

README

GoDoc Golang Status Android Status Coverage Status Go Report Card

OONI probe measurement engine

This repository contains OONI probe's measurement engine. That is, the piece of software that implements OONI nettests.

API

You can browse ooni/probe-engine's API online at godoc.org. We currently don't provide any API stability guarantees.

This repository also allows to build miniooni, a small command line client useful to test the functionality in here without integrating with OONI probe. You can browse the manual of this tool online at godoc.org. We currently don't promise that the miniooni CLI will be stable over time.

Integrating ooni/probe-engine

This software uses Go modules and requires Go v1.13+. We also depend on Measurement Kit, a C++14 library implementing many OONI tests, a.k.a. MK.

Note that passing the -tags nomk flag to Go will disable linking Measurement Kit into the resulting Go binaries. You may want that in cases where you only want to use experiments written in Go.

We plan on gradually rewriting all OONI tests in Go, therefore the dependency on Measurement Kit will eventually be removed. A future version of this document will provide platform specific instructions for installing Measurement Kit and linking to it.

Building miniooni

go build -v -tags nomk ./cmd/miniooni/

Omit -tags nomk to link with MK.

Building Android bindings

./build-android.bash

When building Android bindings, we automatically omit linking with MK.

Release procedure

  1. make sure that dependencies are up to date

  2. make sure that resources are up to date

  3. commit, tag, and push

  4. create new release on GitHub

Updating dependencies

  1. update direct dependencies using:
for name in `grep -v indirect go.mod | awk '/^\t/{print $1}'`; do \
  go get -u -v $name;                                             \
done
  1. pin to a specific psiphon version (we usually track the staging-client branch) using:
go get -v github.com/Psiphon-Labs/psiphon-tunnel-core@COMMITHASH

  1. clone psiphon-tunnel-core, checkout the tip of the staging-client branch and generate a go.mod by running go mod init && go mod tidy in the toplevel dir

  2. rewrite go.mod such that it contains only your direct dependencies followed by the exact content of psiphon-tunnel-core's go.mod

  3. run go mod tidy

  4. make sure you don't downgrade bolt and goselect because this will break downstream builds on MIPS:

go get -u -v github.com/Psiphon-Labs/bolt github.com/creack/goselect

This allows us to pin all psiphon dependencies precisely.

Documentation

Overview

Package engine contains the engine API

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AllExperiments 

func AllExperiments() []string

AllExperiments returns the name of all experiments

Types

type Experiment 

type Experiment struct {
	// contains filtered or unexported fields
}

Experiment is an experiment instance.

func NewExperiment added in v0.8.0 

func NewExperiment(sess *Session, measurer model.ExperimentMeasurer) *Experiment

NewExperiment creates a new experiment given a measurer. The preferred way to create an experiment is the ExperimentBuilder. Though this function allows the programmer to create a custom, external experiment.

func (*Experiment) CloseReport 

func (e *Experiment) CloseReport() (err error)

CloseReport is an idempotent method that closes an open report if one has previously been opened, otherwise it does nothing.

func (*Experiment) LoadMeasurement 

func (e *Experiment) LoadMeasurement(data []byte) (*model.Measurement, error)

LoadMeasurement loads a measurement from a byte stream. The measurement must be a measurement for this experiment.

func (*Experiment) Measure 

func (e *Experiment) Measure(input string) (*model.Measurement, error)

Measure performs a measurement with input. We assume that you have configured the available test helpers, either manually or by calling the session's MaybeLookupBackends() method.

func (*Experiment) Name 

func (e *Experiment) Name() string

Name returns the experiment name.

func (*Experiment) OpenReport 

func (e *Experiment) OpenReport() (err error)

OpenReport is an idempotent method to open a report. We assume that you have configured the available collectors, either manually or through using the session's MaybeLookupBackends method.

func (*Experiment) ReportID 

func (e *Experiment) ReportID() string

ReportID returns the open reportID, if we have opened a report successfully before, or an empty string, otherwise.

func (*Experiment) SaveMeasurement 

func (e *Experiment) SaveMeasurement(measurement *model.Measurement, filePath string) error

SaveMeasurement saves a measurement on the specified file path.

func (*Experiment) SubmitAndUpdateMeasurement 

func (e *Experiment) SubmitAndUpdateMeasurement(measurement *model.Measurement) error

SubmitAndUpdateMeasurement submits a measurement and updates the fields whose value has changed as part of the submission.

type ExperimentBuilder 

type ExperimentBuilder struct {
	// contains filtered or unexported fields
}

ExperimentBuilder is an experiment builder.

func (*ExperimentBuilder) NeedsInput 

func (b *ExperimentBuilder) NeedsInput() bool

NeedsInput returns whether the experiment needs input

func (*ExperimentBuilder) NewExperiment added in v0.8.0 

func (b *ExperimentBuilder) NewExperiment() *Experiment

NewExperiment creates the experiment

func (*ExperimentBuilder) Options 

func (b *ExperimentBuilder) Options() (map[string]OptionInfo, error)

Options returns info about all options

func (*ExperimentBuilder) SetCallbacks 

func (b *ExperimentBuilder) SetCallbacks(callbacks model.ExperimentCallbacks)

SetCallbacks sets the interactive callbacks

func (*ExperimentBuilder) SetOptionBool 

func (b *ExperimentBuilder) SetOptionBool(key string, value bool) error

SetOptionBool sets a bool option

func (*ExperimentBuilder) SetOptionInt 

func (b *ExperimentBuilder) SetOptionInt(key string, value int64) error

SetOptionInt sets an int option

func (*ExperimentBuilder) SetOptionString 

func (b *ExperimentBuilder) SetOptionString(key, value string) error

SetOptionString sets a string option

type FileSystemKVStore added in v0.4.0 

type FileSystemKVStore struct {
	// contains filtered or unexported fields
}

FileSystemKVStore is a directory based KVStore

func NewFileSystemKVStore added in v0.4.0 

func NewFileSystemKVStore(basedir string) (kvs *FileSystemKVStore, err error)

NewFileSystemKVStore creates a new FileSystemKVStore.

func (*FileSystemKVStore) Get added in v0.4.0 

func (kvs *FileSystemKVStore) Get(key string) ([]byte, error)

Get returns the specified key's value

func (*FileSystemKVStore) Set added in v0.4.0 

func (kvs *FileSystemKVStore) Set(key string, value []byte) error

Set sets the value of a specific key

type KVStore added in v0.4.0 

type KVStore interface {
	Get(key string) (value []byte, err error)
	Set(key string, value []byte) (err error)
}

KVStore is a simple, atomic key-value store. The user of probe-engine should supply an implementation of this interface, which will be used by probe-engine to store specific data.

type OptionInfo 

type OptionInfo struct {
	Doc  string
	Type string
}

OptionInfo contains info about an option

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is a measurement session

func NewSession 

func NewSession(config SessionConfig) (*Session, error)

NewSession creates a new session or returns an error

func (*Session) ASNDatabasePath added in v0.8.0 

func (s *Session) ASNDatabasePath() string

ASNDatabasePath returns the path where the ASN database path should be if you have called s.FetchResourcesIdempotent.

func (*Session) AddAvailableHTTPSBouncer 

func (s *Session) AddAvailableHTTPSBouncer(baseURL string)

AddAvailableHTTPSBouncer adds an HTTPS bouncer to the list of bouncers that we'll try to contact.

func (*Session) AddAvailableHTTPSCollector 

func (s *Session) AddAvailableHTTPSCollector(baseURL string)

AddAvailableHTTPSCollector adds an HTTPS collector to the list of collectors that we'll try to use.

func (*Session) CABundlePath added in v0.8.0 

func (s *Session) CABundlePath() string

CABundlePath is like ASNDatabasePath but for the CA bundle path.

func (*Session) Close added in v0.8.0 

func (s *Session) Close() error

Close ensures that we close all the idle connections that the HTTP clients we are currently using may have created. Not calling this function may likely cause memory leaks in your application because of open idle connections.

func (*Session) CountryDatabasePath added in v0.8.0 

func (s *Session) CountryDatabasePath() string

CountryDatabasePath is like ASNDatabasePath but for the country DB path.

func (*Session) DefaultHTTPClient added in v0.8.0 

func (s *Session) DefaultHTTPClient() *http.Client

DefaultHTTPClient returns the session's default HTTP client.

func (*Session) ExplicitProxy added in v0.8.0 

func (s *Session) ExplicitProxy() bool

ExplicitProxy returns true if the user has explicitly set a proxy (as opposed to using the HTTP_PROXY envvar).

func (*Session) GetTestHelpersByName added in v0.8.0 

func (s *Session) GetTestHelpersByName(name string) ([]model.Service, bool)

GetTestHelpersByName returns the available test helpers that use the specified name, or false if there's none.

func (*Session) Logger added in v0.8.0 

func (s *Session) Logger() model.Logger

Logger returns the logger used by the session.

func (*Session) MaybeLookupBackends 

func (s *Session) MaybeLookupBackends() error

MaybeLookupBackends is a caching OONI backends lookup call.

func (*Session) MaybeLookupLocation 

func (s *Session) MaybeLookupLocation() error

MaybeLookupLocation is a caching location lookup call.

func (*Session) NewExperimentBuilder 

func (s *Session) NewExperimentBuilder(name string) (*ExperimentBuilder, error)

NewExperimentBuilder returns a new experiment builder for the experiment with the given name, or an error if there's no such experiment with the given name

func (*Session) NewOrchestraClient added in v0.8.0 

func (s *Session) NewOrchestraClient(ctx context.Context) (model.ExperimentOrchestraClient, error)

NewOrchestraClient creates a new orchestra client. This client is registered and logged in with the OONI orchestra. An error is returned on failure.

func (*Session) Platform added in v0.3.0 

func (s *Session) Platform() string

Platform returns the current platform. The platform is one of:

- android - ios - linux - macos - windows - unknown

When running on the iOS simulator, the returned platform is macos rather than ios if CGO is disabled. This is a known issue, that however should have a very limited impact.

func (*Session) ProbeASN 

func (s *Session) ProbeASN() uint

ProbeASN returns the probe ASN as an integer.

func (*Session) ProbeASNString 

func (s *Session) ProbeASNString() string

ProbeASNString returns the probe ASN as a string.

func (*Session) ProbeCC 

func (s *Session) ProbeCC() string

ProbeCC returns the probe CC.

func (*Session) ProbeIP 

func (s *Session) ProbeIP() string

ProbeIP returns the probe IP.

func (*Session) ProbeNetworkName 

func (s *Session) ProbeNetworkName() string

ProbeNetworkName returns the probe network name.

func (*Session) QueryTestListsURLs added in v0.3.0 

func (s *Session) QueryTestListsURLs(conf *TestListsURLsConfig) (*TestListsURLsResult, error)

QueryTestListsURLs queries the test-lists/urls API.

func (*Session) ResolverASN added in v0.5.0 

func (s *Session) ResolverASN() uint

ResolverASN returns the resolver ASN

func (*Session) ResolverASNString added in v0.5.0 

func (s *Session) ResolverASNString() string

ResolverASNString returns the resolver ASN as a string

func (*Session) ResolverIP 

func (s *Session) ResolverIP() string

ResolverIP returns the resolver IP

func (*Session) ResolverNetworkName added in v0.5.0 

func (s *Session) ResolverNetworkName() string

ResolverNetworkName returns the resolver network name.

func (*Session) SetIncludeProbeASN 

func (s *Session) SetIncludeProbeASN(value bool)

SetIncludeProbeASN controls whether to include the ASN

func (*Session) SetIncludeProbeCC 

func (s *Session) SetIncludeProbeCC(value bool)

SetIncludeProbeCC controls whether to include the country code

func (*Session) SetIncludeProbeIP 

func (s *Session) SetIncludeProbeIP(value bool)

SetIncludeProbeIP controls whether to include the IP

func (*Session) SoftwareName added in v0.8.0 

func (s *Session) SoftwareName() string

SoftwareName returns the application name.

func (*Session) SoftwareVersion added in v0.8.0 

func (s *Session) SoftwareVersion() string

SoftwareVersion returns the application version.

func (*Session) TempDir added in v0.8.0 

func (s *Session) TempDir() string

TempDir returns the temporary directory.

func (*Session) UserAgent added in v0.8.0 

func (s *Session) UserAgent() string

UserAgent constructs the user agent to be used in this session.

type SessionConfig 

type SessionConfig struct {
	AssetsDir       string
	KVStore         KVStore
	Logger          model.Logger
	ProxyURL        *url.URL
	SoftwareName    string
	SoftwareVersion string
	TempDir         string
}

SessionConfig contains the Session config

type TestListsURLsConfig added in v0.3.0 

type TestListsURLsConfig struct {
	BaseURL    string   // URL to use (empty means default)
	Categories []string // Categories to query for (empty means all)
	Limit      int64    // Max number of URLs (<= 0 means no limit)
}

TestListsURLsConfig config config for test-lists/urls API.

func (*TestListsURLsConfig) AddCategory added in v0.3.0 

func (c *TestListsURLsConfig) AddCategory(s string)

AddCategory adds a category to the list of categories to query. Not adding any categories will query for URLs in all categories.

type TestListsURLsResult added in v0.3.0 

type TestListsURLsResult struct {
	Result []model.URLInfo
}

TestListsURLsResult contains the results of calling the test-lists/urls OONI orchestra API.

func (*TestListsURLsResult) At added in v0.3.0 

func (r *TestListsURLsResult) At(idx int64) (out *model.URLInfo)

At returns the URL at the given index or nil

func (*TestListsURLsResult) Count added in v0.3.0 

func (r *TestListsURLsResult) Count() int64

Count returns the number of returned URLs

Source Files

View all Source files

Directories

Path Synopsis
Package bouncer contains a OONI bouncer client implementation.
 Package bouncer contains a OONI bouncer client implementation.
cmd
miniooni command
Command miniooni is simple binary for testing purposes.
 Command miniooni is simple binary for testing purposes.
Package collector contains a OONI collector client implementation.
 Package collector contains a OONI collector client implementation.
experiment
dash
Package dash contains the dash network experiment.
 Package dash contains the dash network experiment.
example
Package example contains a simple example of experiment.
 Package example contains a simple example of experiment.
fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
 Package fbmessenger contains the Facebook Messenger network experiment.
handler
Package handler contains experiment events handler
 Package handler contains experiment events handler
hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
 Package hhfm contains the HTTP Header Field Manipulation network experiment.
hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
 Package hirl contains the HTTP Invalid Request Line network experiment.
httpheader
Package httpheader contains code to set common HTTP headers.
 Package httpheader contains code to set common HTTP headers.
mkevent
Package mkevent processes MK events
 Package mkevent processes MK events
mkhelper
Package mkhelper contains common code to get the proper helper and configure it into settings.
 Package mkhelper contains common code to get the proper helper and configure it into settings.
mkrunner
Package mkrunner contains code to run an MK based test
 Package mkrunner contains code to run an MK based test
mktesting
Package mktesting contains common code to run MK based tests.
 Package mktesting contains common code to run MK based tests.
ndt
Package ndt contains the ndt network experiment.
 Package ndt contains the ndt network experiment.
ndt7
Package ndt7 contains the ndt7 network experiment.
 Package ndt7 contains the ndt7 network experiment.
psiphon
Package psiphon implements the psiphon network experiment.
 Package psiphon implements the psiphon network experiment.
sniblocking
Package sniblocking contains the SNI blocking network experiment.
 Package sniblocking contains the SNI blocking network experiment.
telegram
Package telegram contains the Telegram network experiment.
 Package telegram contains the Telegram network experiment.
tor
Package tor contains the tor experiment.
 Package tor contains the tor experiment.
web_connectivity
Package web_connectivity contains the Web Connectivity network experiment.
 Package web_connectivity contains the Web Connectivity network experiment.
whatsapp
Package whatsapp contains the WhatsApp network experiment.
 Package whatsapp contains the WhatsApp network experiment.
geoiplookup
iplookup
Package iplookup implements probe IP lookup.
 Package iplookup implements probe IP lookup.
iplookup/avast
Package avast lookups the IP using avast.
 Package avast lookups the IP using avast.
iplookup/invalid
Package invalid returns an invalid IP.
 Package invalid returns an invalid IP.
iplookup/ubuntu
Package ubuntu lookups the IP using Ubuntu.
 Package ubuntu lookups the IP using Ubuntu.
mmdblookup
Package mmdblookup performs probe ASN, CC, NetworkName lookups.
 Package mmdblookup performs probe ASN, CC, NetworkName lookups.
resolverlookup
Package resolverlookup discovers the resolver's IP
 Package resolverlookup discovers the resolver's IP
internal
fetch
Package fetch is used to fetch resources.
 Package fetch is used to fetch resources.
jsonapi
Package jsonapi interacts with HTTP JSON APIs.
 Package jsonapi interacts with HTTP JSON APIs.
kvstore
Package kvstore contains key-value stores
 Package kvstore contains key-value stores
mockable
Package mockable contains mockable objects
 Package mockable contains mockable objects
netxlogger
Package netxlogger is a logger for netx events.
 Package netxlogger is a logger for netx events.
oonidatamodel
Package oonidatamodel contains the OONI data model.
 Package oonidatamodel contains the OONI data model.
oonitemplates
Package oonitemplates contains templates for experiments.
 Package oonitemplates contains templates for experiments.
orchestra
Package orchestra contains common orchestra code.
 Package orchestra contains common orchestra code.
orchestra/login
Package login contains code to login with OONI orchestra.
 Package login contains code to login with OONI orchestra.
orchestra/metadata
Package metadata contains metadata about a probe
 Package metadata contains metadata about a probe
orchestra/register
Package register contains code to register to the OONI orchestra.
 Package register contains code to register to the OONI orchestra.
orchestra/statefile
Package statefile defines the state file
 Package statefile defines the state file
orchestra/testlists/psiphon
Package psiphon implements fetching psiphon config using orchestra
 Package psiphon implements fetching psiphon config using orchestra
orchestra/testlists/tor
Package tor contains code to fetch targets for the tor experiment.
 Package tor contains code to fetch targets for the tor experiment.
orchestra/testlists/urls
Package urls queries orchestra test-lists/urls API
 Package urls queries orchestra test-lists/urls API
orchestra/testorchestra
Package testorchestra contains code to simplify testing
 Package testorchestra contains code to simplify testing
orchestra/update
Package update contains code to update the probe state with orchestra
 Package update contains code to update the probe state with orchestra
platform
Package platform returns the platform name.
 Package platform returns the platform name.
resources
Package resources contains code to download resources.
 Package resources contains code to download resources.
tlsx
Package tlsx contains TLS extensions
 Package tlsx contains TLS extensions
urlpath
Package urlpath contains code to manipulate URL paths
 Package urlpath contains code to manipulate URL paths
Package measurementkit allows to use Measurement Kit.
 Package measurementkit allows to use Measurement Kit.
mkcgo
Package mkcgo contains CGO bindings to Measurement Kit.
 Package mkcgo contains CGO bindings to Measurement Kit.
Package model defines shared data structures.
 Package model defines shared data structures.
Package netx contains OONI's net extensions.
 Package netx contains OONI's net extensions.
handlers
Package handlers contains default modelx.Handler handlers.
 Package handlers contains default modelx.Handler handlers.
internal/connid
Package connid contains code to generate the connectionID
 Package connid contains code to generate the connectionID
internal/dialer
Package dialer contains the dialer's API.
 Package dialer contains the dialer's API.
internal/dialer/connx
Package connx contains net.Conn extensions
 Package connx contains net.Conn extensions
internal/dialer/dialerbase
Package dialerbase contains the base dialer functionality.
 Package dialerbase contains the base dialer functionality.
internal/dialer/dnsdialer
Package dnsdialer contains a dialer with DNS lookups.
 Package dnsdialer contains a dialer with DNS lookups.
internal/dialer/tlsdialer
Package tlsdialer contains the TLS dialer
 Package tlsdialer contains the TLS dialer
internal/dialid
internal/errwrapper
Package errwrapper contains our error wrapper
 Package errwrapper contains our error wrapper
internal/httptransport
Package httptransport contains HTTP transport extensions.
 Package httptransport contains HTTP transport extensions.
internal/httptransport/bodytracer
Package bodytracer contains the HTTP body tracer.
 Package bodytracer contains the HTTP body tracer.
internal/httptransport/tracetripper
Package tracetripper contains the tracing round tripper
 Package tracetripper contains the tracing round tripper
internal/httptransport/transactioner
Package transactioner contains the transaction assigning round tripper
 Package transactioner contains the transaction assigning round tripper
internal/resolver
Package resolver contains code to create a resolver
 Package resolver contains code to create a resolver
internal/resolver/bogondetector
Package bogondetector contains code to determine if an IP is private/bogon.
 Package bogondetector contains code to determine if an IP is private/bogon.
internal/resolver/brokenresolver
Package brokenresolver is a broken resolver
 Package brokenresolver is a broken resolver
internal/resolver/chainresolver
Package chainresolver allows to chain two resolvers
 Package chainresolver allows to chain two resolvers
internal/resolver/dnstransport/dnsoverhttps
Package dnsoverhttps implements DNS over HTTPS.
 Package dnsoverhttps implements DNS over HTTPS.
internal/resolver/dnstransport/dnsovertcp
Package dnsovertcp implements DNS over TCP.
 Package dnsovertcp implements DNS over TCP.
internal/resolver/dnstransport/dnsoverudp
Package dnsoverudp implements DNS over UDP.
 Package dnsoverudp implements DNS over UDP.
internal/resolver/ooniresolver
Package ooniresolver is OONI's DNS resolver.
 Package ooniresolver is OONI's DNS resolver.
internal/resolver/parentresolver
Package parentresolver contains the parent resolver
 Package parentresolver contains the parent resolver
internal/resolver/systemresolver
Package systemresolver contains the system resolver
 Package systemresolver contains the system resolver
internal/transactionid
Package transactionid contains code to share the transactionID
 Package transactionid contains code to share the transactionID
modelx
Package modelx contains the data modelx.
 Package modelx contains the data modelx.
Package oonimkall implements measurement-kit's FFI API.
 Package oonimkall implements measurement-kit's FFI API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.