engine

package module
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 29, 2020 License: BSD-3-Clause Imports: 49 Imported by: 0

README

GoDoc Golang Status Android Status Coverage Status Go Report Card

OONI probe measurement engine

This repository contains OONI probe's measurement engine. That is, the piece of software that implements OONI nettests.

API

You can browse ooni/probe-engine's API online at godoc.org. We currently don't provide any API stability guarantees.

This repository also allows to build miniooni, a small command line client useful to test the functionality in here without integrating with OONI probe. You can browse the manual of this tool online at godoc.org. We currently don't promise that the miniooni CLI will be stable over time.

Integrating ooni/probe-engine

This software uses Go modules and requires Go v1.14+. We also depend on Measurement Kit, a C++14 library implementing many OONI tests, a.k.a. MK.

Note that passing the -tags nomk flag to Go will disable linking Measurement Kit into the resulting Go binaries. You may want that in cases where you only want to use experiments written in Go.

We plan on gradually rewriting all OONI tests in Go, therefore the dependency on Measurement Kit will eventually be removed. A future version of this document will provide platform specific instructions for installing Measurement Kit and linking to it.

Building miniooni

go build -v -tags nomk ./cmd/miniooni/

Omit -tags nomk to link with MK.

Building Android bindings

./build-android.bash

When building Android bindings, we automatically omit linking with MK. We automatically build Android bindings whenever commits are pushed to the mobile-staging branch. Such builds could be integrated by using:

maven { url "https://dl.bintray.com/ooni/android/" }

and

implementation "org.ooni:oonimkall:VERSION"

Where VERSION is like 2020.03.30-231914 corresponding to the moment in time in which the version has been built.

Release procedure

  1. make sure that dependencies are up to date

  2. make sure that resources are up to date

  3. commit, tag, and push

  4. create new release on GitHub

Updating dependencies

  1. update direct dependencies using:
for name in `grep -v indirect go.mod | awk '/^\t/{print $1}'`; do \
  go get -u -v $name;                                             \
done
  1. pin to a specific psiphon version (we usually track the staging-client branch) using:
go get -v github.com/Psiphon-Labs/psiphon-tunnel-core@COMMITHASH

  1. clone psiphon-tunnel-core, checkout the tip of the staging-client branch and generate a go.mod by running go mod init && go mod tidy in the toplevel dir

  2. rewrite go.mod such that it contains only your direct dependencies followed by the exact content of psiphon-tunnel-core's go.mod

  3. run go mod tidy

  4. make sure you don't downgrade bolt and goselect because this will break downstream builds on MIPS:

go get -u -v github.com/Psiphon-Labs/bolt github.com/creack/goselect

This allows us to pin all psiphon dependencies precisely.

Documentation

Overview

Package engine contains the engine API

Index

Constants

View Source 
const (
	// InputRequired indicates that the experiment requires
	// external input to run. If this input is not provided to
	// the experiment, it will not know what to do.
	InputRequired = InputPolicy("required")

	// InputOptional indicates that the experiment handles input,
	// if any; otherwise it fetchs input/uses a default.
	InputOptional = InputPolicy("optional")

	// InputNone indicates that the experiment does not want any
	// input and ignores the input if provided with it.
	InputNone = InputPolicy("none")
)

Variables

View Source 
var ErrAlreadyUsingProxy = errors.New(
	"session: cannot create a new tunnel of this kind: we are already using a proxy",
)

ErrAlreadyUsingProxy indicates that we cannot create a tunnel with a specific name because we already configured a proxy.

Functions

func AllExperiments 

func AllExperiments() []string

AllExperiments returns the name of all experiments

Types

type Experiment 

type Experiment struct {
	// contains filtered or unexported fields
}

Experiment is an experiment instance.

func NewExperiment added in v0.8.0 

func NewExperiment(sess *Session, measurer model.ExperimentMeasurer) *Experiment

NewExperiment creates a new experiment given a measurer. The preferred way to create an experiment is the ExperimentBuilder. Though this function allows the programmer to create a custom, external experiment.

func (*Experiment) CloseReport 

func (e *Experiment) CloseReport() (err error)

CloseReport is an idempotent method that closes an open report if one has previously been opened, otherwise it does nothing.

func (*Experiment) KibiBytesReceived added in v0.9.0 

func (e *Experiment) KibiBytesReceived() float64

KibiBytesReceived accounts for the KibiBytes received by the HTTP clients managed by this session so far, including experiments.

func (*Experiment) KibiBytesSent added in v0.9.0 

func (e *Experiment) KibiBytesSent() float64

KibiBytesSent is like KibiBytesReceived but for the bytes sent.

func (*Experiment) LoadMeasurement 

func (e *Experiment) LoadMeasurement(data []byte) (*model.Measurement, error)

LoadMeasurement loads a measurement from a byte stream. The measurement must be a measurement for this experiment.

func (*Experiment) Measure 

func (e *Experiment) Measure(input string) (*model.Measurement, error)

Measure performs a measurement with input. We assume that you have configured the available test helpers, either manually or by calling the session's MaybeLookupBackends() method.

func (*Experiment) MeasureWithContext added in v0.9.0 

func (e *Experiment) MeasureWithContext(
	ctx context.Context, input string,
) (measurement *model.Measurement, err error)

MeasureWithContext is like Measure but with context.

func (*Experiment) Name 

func (e *Experiment) Name() string

Name returns the experiment name.

func (*Experiment) OpenReport 

func (e *Experiment) OpenReport() (err error)

OpenReport is an idempotent method to open a report. We assume that you have configured the available probe services, either manually or through using the session's MaybeLookupBackends method.

func (*Experiment) ReportID 

func (e *Experiment) ReportID() string

ReportID returns the open reportID, if we have opened a report successfully before, or an empty string, otherwise.

func (*Experiment) SaveMeasurement 

func (e *Experiment) SaveMeasurement(measurement *model.Measurement, filePath string) error

SaveMeasurement saves a measurement on the specified file path.

func (*Experiment) SubmitAndUpdateMeasurement 

func (e *Experiment) SubmitAndUpdateMeasurement(measurement *model.Measurement) error

SubmitAndUpdateMeasurement submits a measurement and updates the fields whose value has changed as part of the submission.

type ExperimentBuilder 

type ExperimentBuilder struct {
	// contains filtered or unexported fields
}

ExperimentBuilder is an experiment builder.

func (*ExperimentBuilder) InputPolicy added in v0.11.0 

func (b *ExperimentBuilder) InputPolicy() InputPolicy

InputPolicy returns the experiment input policy

func (*ExperimentBuilder) Interruptible added in v0.9.0 

func (b *ExperimentBuilder) Interruptible() bool

Interruptible tells you whether this is an interruptible experiment. This kind of experiments (e.g. ndt7) may be interrupted mid way.

func (*ExperimentBuilder) NewExperiment added in v0.8.0 

func (b *ExperimentBuilder) NewExperiment() *Experiment

NewExperiment creates the experiment

func (*ExperimentBuilder) Options 

func (b *ExperimentBuilder) Options() (map[string]OptionInfo, error)

Options returns info about all options

func (*ExperimentBuilder) SetCallbacks 

func (b *ExperimentBuilder) SetCallbacks(callbacks model.ExperimentCallbacks)

SetCallbacks sets the interactive callbacks

func (*ExperimentBuilder) SetOptionBool 

func (b *ExperimentBuilder) SetOptionBool(key string, value bool) error

SetOptionBool sets a bool option

func (*ExperimentBuilder) SetOptionInt 

func (b *ExperimentBuilder) SetOptionInt(key string, value int64) error

SetOptionInt sets an int option

func (*ExperimentBuilder) SetOptionString 

func (b *ExperimentBuilder) SetOptionString(key, value string) error

SetOptionString sets a string option

type FileSystemKVStore added in v0.4.0 

type FileSystemKVStore struct {
	// contains filtered or unexported fields
}

FileSystemKVStore is a directory based KVStore

func NewFileSystemKVStore added in v0.4.0 

func NewFileSystemKVStore(basedir string) (kvs *FileSystemKVStore, err error)

NewFileSystemKVStore creates a new FileSystemKVStore.

func (*FileSystemKVStore) Get added in v0.4.0 

func (kvs *FileSystemKVStore) Get(key string) ([]byte, error)

Get returns the specified key's value

func (*FileSystemKVStore) Set added in v0.4.0 

func (kvs *FileSystemKVStore) Set(key string, value []byte) error

Set sets the value of a specific key

type InputPolicy added in v0.11.0 

type InputPolicy string

InputPolicy describes the experiment policy with respect to input. That is whether it requires input, optionally accepts input, does not want input.

type KVStore added in v0.4.0 

type KVStore interface {
	Get(key string) (value []byte, err error)
	Set(key string, value []byte) (err error)
}

KVStore is a simple, atomic key-value store. The user of probe-engine should supply an implementation of this interface, which will be used by probe-engine to store specific data.

type OptionInfo 

type OptionInfo struct {
	Doc  string
	Type string
}

OptionInfo contains info about an option

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is a measurement session

func NewSession 

func NewSession(config SessionConfig) (*Session, error)

NewSession creates a new session or returns an error

func (*Session) ASNDatabasePath added in v0.8.0 

func (s *Session) ASNDatabasePath() string

ASNDatabasePath returns the path where the ASN database path should be if you have called s.FetchResourcesIdempotent.

func (*Session) CABundlePath added in v0.8.0 

func (s *Session) CABundlePath() string

CABundlePath is like ASNDatabasePath but for the CA bundle path.

func (*Session) Close added in v0.8.0 

func (s *Session) Close() error

Close ensures that we close all the idle connections that the HTTP clients we are currently using may have created. Not calling this function may likely cause memory leaks in your application because of open idle connections.

func (*Session) CountryDatabasePath added in v0.8.0 

func (s *Session) CountryDatabasePath() string

CountryDatabasePath is like ASNDatabasePath but for the country DB path.

func (*Session) DefaultHTTPClient added in v0.8.0 

func (s *Session) DefaultHTTPClient() *http.Client

DefaultHTTPClient returns the session's default HTTP client.

func (*Session) GetTestHelpersByName added in v0.8.0 

func (s *Session) GetTestHelpersByName(name string) ([]model.Service, bool)

GetTestHelpersByName returns the available test helpers that use the specified name, or false if there's none.

func (*Session) KibiBytesReceived added in v0.9.0 

func (s *Session) KibiBytesReceived() float64

KibiBytesReceived accounts for the KibiBytes received by the HTTP clients managed by this session so far, including experiments.

func (*Session) KibiBytesSent added in v0.9.0 

func (s *Session) KibiBytesSent() float64

KibiBytesSent is like KibiBytesReceived but for the bytes sent.

func (*Session) Logger added in v0.8.0 

func (s *Session) Logger() model.Logger

Logger returns the logger used by the session.

func (*Session) MaybeLookupBackends 

func (s *Session) MaybeLookupBackends() error

MaybeLookupBackends is a caching OONI backends lookup call.

func (*Session) MaybeLookupLocation 

func (s *Session) MaybeLookupLocation() error

MaybeLookupLocation is a caching location lookup call.

func (*Session) MaybeStartTunnel added in v0.10.0 

func (s *Session) MaybeStartTunnel(ctx context.Context, name string) error

MaybeStartTunnel starts the requested tunnel.

This function silently succeeds if we're already using a tunnel with the same name or if the requested tunnel name is the empty string. This function fails, tho, when we already have a proxy or a tunnel with another name and we try to open a tunnel. This function of course also fails if we cannot start the requested tunnel. All in all, if you request for a tunnel name that is not the empty string and you get a nil error, you can be confident that session.ProxyURL() gives you the tunnel URL.

The tunnel will be closed by session.Close().

func (*Session) NewExperimentBuilder 

func (s *Session) NewExperimentBuilder(name string) (*ExperimentBuilder, error)

NewExperimentBuilder returns a new experiment builder for the experiment with the given name, or an error if there's no such experiment with the given name

func (*Session) NewOrchestraClient added in v0.8.0 

func (s *Session) NewOrchestraClient(ctx context.Context) (model.ExperimentOrchestraClient, error)

NewOrchestraClient creates a new orchestra client. This client is registered and logged in with the OONI orchestra. An error is returned on failure.

func (*Session) Platform added in v0.3.0 

func (s *Session) Platform() string

Platform returns the current platform. The platform is one of:

- android - ios - linux - macos - windows - unknown

When running on the iOS simulator, the returned platform is macos rather than ios if CGO is disabled. This is a known issue, that however should have a very limited impact.

func (*Session) ProbeASN 

func (s *Session) ProbeASN() uint

ProbeASN returns the probe ASN as an integer.

func (*Session) ProbeASNString 

func (s *Session) ProbeASNString() string

ProbeASNString returns the probe ASN as a string.

func (*Session) ProbeCC 

func (s *Session) ProbeCC() string

ProbeCC returns the probe CC.

func (*Session) ProbeIP 

func (s *Session) ProbeIP() string

ProbeIP returns the probe IP.

func (*Session) ProbeNetworkName 

func (s *Session) ProbeNetworkName() string

ProbeNetworkName returns the probe network name.

func (*Session) ProxyURL added in v0.10.0 

func (s *Session) ProxyURL() *url.URL

ProxyURL returns the Proxy URL, or nil if not set

func (*Session) QueryTestListsURLs added in v0.3.0 

func (s *Session) QueryTestListsURLs(conf *TestListsURLsConfig) (*TestListsURLsResult, error)

QueryTestListsURLs queries the test-lists/urls API.

func (*Session) ResolverASN added in v0.5.0 

func (s *Session) ResolverASN() uint

ResolverASN returns the resolver ASN

func (*Session) ResolverASNString added in v0.5.0 

func (s *Session) ResolverASNString() string

ResolverASNString returns the resolver ASN as a string

func (*Session) ResolverIP 

func (s *Session) ResolverIP() string

ResolverIP returns the resolver IP

func (*Session) ResolverNetworkName added in v0.5.0 

func (s *Session) ResolverNetworkName() string

ResolverNetworkName returns the resolver network name.

func (*Session) SoftwareName added in v0.8.0 

func (s *Session) SoftwareName() string

SoftwareName returns the application name.

func (*Session) SoftwareVersion added in v0.8.0 

func (s *Session) SoftwareVersion() string

SoftwareVersion returns the application version.

func (*Session) TempDir added in v0.8.0 

func (s *Session) TempDir() string

TempDir returns the temporary directory.

func (*Session) TorArgs added in v0.11.0 

func (s *Session) TorArgs() []string

TorArgs returns the configured extra args for the tor binary. If not set we will not pass in any extra arg. Applies to `-OTunnel=tor` mainly.

func (*Session) TorBinary added in v0.11.0 

func (s *Session) TorBinary() string

TorBinary returns the configured path to the tor binary. If not set we will attempt to use "tor". Applies to `-OTunnel=tor` mainly.

func (*Session) TunnelBootstrapTime added in v0.10.0 

func (s *Session) TunnelBootstrapTime() time.Duration

TunnelBootstrapTime returns the time required to bootstrap the tunnel we're using, or zero if we're using no tunnel.

func (*Session) UserAgent added in v0.8.0 

func (s *Session) UserAgent() string

UserAgent constructs the user agent to be used in this session.

type SessionConfig 

type SessionConfig struct {
	AssetsDir              string
	AvailableProbeServices []model.Service
	KVStore                KVStore
	Logger                 model.Logger
	PrivacySettings        model.PrivacySettings
	ProxyURL               *url.URL
	SoftwareName           string
	SoftwareVersion        string
	TempDir                string
	TorArgs                []string
	TorBinary              string
}

SessionConfig contains the Session config

type TestListsURLsConfig added in v0.3.0 

type TestListsURLsConfig struct {
	BaseURL    string   // URL to use (empty means default)
	Categories []string // Categories to query for (empty means all)
	Limit      int64    // Max number of URLs (<= 0 means no limit)
}

TestListsURLsConfig config config for test-lists/urls API.

func (*TestListsURLsConfig) AddCategory added in v0.3.0 

func (c *TestListsURLsConfig) AddCategory(s string)

AddCategory adds a category to the list of categories to query. Not adding any categories will query for URLs in all categories.

type TestListsURLsResult added in v0.3.0 

type TestListsURLsResult struct {
	Result []model.URLInfo
}

TestListsURLsResult contains the results of calling the test-lists/urls OONI orchestra API.

func (*TestListsURLsResult) At added in v0.3.0 

func (r *TestListsURLsResult) At(idx int64) (out *model.URLInfo)

At returns the URL at the given index or nil

func (*TestListsURLsResult) Count added in v0.3.0 

func (r *TestListsURLsResult) Count() int64

Count returns the number of returned URLs

Source Files

View all Source files

Directories

Path Synopsis
Package atomicx contains atomic extensions.
 Package atomicx contains atomic extensions.
cmd
miniooni command
Command miniooni is simple binary for testing purposes.
 Command miniooni is simple binary for testing purposes.
experiment
dash
Package dash contains the dash network experiment.
 Package dash contains the dash network experiment.
example
Package example contains a simple example of experiment.
 Package example contains a simple example of experiment.
fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
 Package fbmessenger contains the Facebook Messenger network experiment.
handler
Package handler contains experiment events handler
 Package handler contains experiment events handler
hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
 Package hhfm contains the HTTP Header Field Manipulation network experiment.
hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
 Package hirl contains the HTTP Invalid Request Line network experiment.
mkevent
Package mkevent processes MK events
 Package mkevent processes MK events
mkhelper
Package mkhelper contains common code to get the proper helper and configure it into settings.
 Package mkhelper contains common code to get the proper helper and configure it into settings.
mkrunner
Package mkrunner contains code to run an MK based test
 Package mkrunner contains code to run an MK based test
mktesting
Package mktesting contains common code to run MK based tests.
 Package mktesting contains common code to run MK based tests.
ndt7
Package ndt7 contains the ndt7 network experiment.
 Package ndt7 contains the ndt7 network experiment.
psiphon
Package psiphon implements the psiphon network experiment.
 Package psiphon implements the psiphon network experiment.
sniblocking
Package sniblocking contains the SNI blocking network experiment.
 Package sniblocking contains the SNI blocking network experiment.
telegram
Package telegram contains the Telegram network experiment.
 Package telegram contains the Telegram network experiment.
tor
Package tor contains the tor experiment.
 Package tor contains the tor experiment.
urlgetter
Package urlgetter implements a nettest that fetches a URL.
 Package urlgetter implements a nettest that fetches a URL.
web_connectivity
Package web_connectivity contains the Web Connectivity network experiment.
 Package web_connectivity contains the Web Connectivity network experiment.
whatsapp
Package whatsapp contains the WhatsApp network experiment.
 Package whatsapp contains the WhatsApp network experiment.
geoiplookup
iplookup
Package iplookup implements probe IP lookup.
 Package iplookup implements probe IP lookup.
iplookup/avast
Package avast lookups the IP using avast.
 Package avast lookups the IP using avast.
iplookup/invalid
Package invalid returns an invalid IP.
 Package invalid returns an invalid IP.
iplookup/ubuntu
Package ubuntu lookups the IP using Ubuntu.
 Package ubuntu lookups the IP using Ubuntu.
mmdblookup
Package mmdblookup performs probe ASN, CC, NetworkName lookups.
 Package mmdblookup performs probe ASN, CC, NetworkName lookups.
resolverlookup
Package resolverlookup discovers the resolver's IP
 Package resolverlookup discovers the resolver's IP
internal
fetch
Package fetch is used to fetch resources.
 Package fetch is used to fetch resources.
httpheader
Package httpheader contains code to set common HTTP headers.
 Package httpheader contains code to set common HTTP headers.
humanizex
Package humanizex is like dustin/go-humanize
 Package humanizex is like dustin/go-humanize
jsonapi
Package jsonapi interacts with HTTP JSON APIs.
 Package jsonapi interacts with HTTP JSON APIs.
kvstore
Package kvstore contains key-value stores
 Package kvstore contains key-value stores
mlablocate
Package mlablocate contains a locate.measurementlab.net client.
 Package mlablocate contains a locate.measurementlab.net client.
mlablocatev2
Package mlablocatev2 use m-lab locate services API v2.
 Package mlablocatev2 use m-lab locate services API v2.
mockable
Package mockable contains mockable objects
 Package mockable contains mockable objects
netxlogger
Package netxlogger is a logger for netx events.
 Package netxlogger is a logger for netx events.
oonidatamodel
Package oonidatamodel contains the OONI data model.
 Package oonidatamodel contains the OONI data model.
oonitemplates
Package oonitemplates contains templates for experiments.
 Package oonitemplates contains templates for experiments.
orchestra
Package orchestra contains common orchestra code.
 Package orchestra contains common orchestra code.
orchestra/login
Package login contains code to login with OONI orchestra.
 Package login contains code to login with OONI orchestra.
orchestra/metadata
Package metadata contains metadata about a probe
 Package metadata contains metadata about a probe
orchestra/register
Package register contains code to register to the OONI orchestra.
 Package register contains code to register to the OONI orchestra.
orchestra/statefile
Package statefile defines the state file
 Package statefile defines the state file
orchestra/testlists/psiphon
Package psiphon implements fetching psiphon config using orchestra
 Package psiphon implements fetching psiphon config using orchestra
orchestra/testlists/tor
Package tor contains code to fetch targets for the tor experiment.
 Package tor contains code to fetch targets for the tor experiment.
orchestra/testlists/urls
Package urls queries orchestra test-lists/urls API
 Package urls queries orchestra test-lists/urls API
orchestra/testorchestra
Package testorchestra contains code to simplify testing
 Package testorchestra contains code to simplify testing
orchestra/update
Package update contains code to update the probe state with orchestra
 Package update contains code to update the probe state with orchestra
platform
Package platform returns the platform name.
 Package platform returns the platform name.
psiphonx
Package psiphonx is a wrapper around the psiphon-tunnel-core.
 Package psiphonx is a wrapper around the psiphon-tunnel-core.
resources
Package resources contains code to download resources.
 Package resources contains code to download resources.
runtimex
Package runtimex contains runtime extensions.
 Package runtimex contains runtime extensions.
sessionresolver
Package sessionresolver contains the resolver used by the session.
 Package sessionresolver contains the resolver used by the session.
sessiontunnel
tlsx
Package tlsx contains TLS extensions
 Package tlsx contains TLS extensions
torx
Package torx contains code to control tor.
 Package torx contains code to control tor.
urlpath
Package urlpath contains code to manipulate URL paths
 Package urlpath contains code to manipulate URL paths
Package libminiooni implements the cmd/miniooni CLI.
 Package libminiooni implements the cmd/miniooni CLI.
Package measurementkit allows to use Measurement Kit.
 Package measurementkit allows to use Measurement Kit.
mkcgo
Package mkcgo contains CGO bindings to Measurement Kit.
 Package mkcgo contains CGO bindings to Measurement Kit.
Package model defines shared data structures.
 Package model defines shared data structures.
Package netx contains OONI's net extensions.
 Package netx contains OONI's net extensions.
archival
Package archival contains data formats used for archival.
 Package archival contains data formats used for archival.
bytecounter
dialer
handlers
Package handlers contains default modelx.Handler handlers.
 Package handlers contains default modelx.Handler handlers.
httptransport
Package httptransport contains HTTP transport extensions.
 Package httptransport contains HTTP transport extensions.
internal/connid
Package connid contains code to generate the connectionID
 Package connid contains code to generate the connectionID
internal/dialid
internal/errwrapper
Package errwrapper contains our error wrapper
 Package errwrapper contains our error wrapper
internal/transactionid
Package transactionid contains code to share the transactionID
 Package transactionid contains code to share the transactionID
modelx
Package modelx contains the data modelx.
 Package modelx contains the data modelx.
oldhttptransport
Package oldhttptransport contains HTTP transport extensions.
 Package oldhttptransport contains HTTP transport extensions.
resolver
selfcensor
Package selfcensor contains code that triggers censorship.
 Package selfcensor contains code that triggers censorship.
trace
Package oonimkall implements measurement-kit's FFI API.
 Package oonimkall implements measurement-kit's FFI API.
Package probeservices contains code to contact OONI probe services.
 Package probeservices contains code to contact OONI probe services.
Package version contains version information
 Package version contains version information

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.