engine

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 10, 2020 License: BSD-3-Clause Imports: 32 Imported by: 0

README

GoDoc Build Status Coverage Status Go Report Card

OONI probe measurement engine

This repository contains OONI probe's measurement engine. That is, the piece of software that implements OONI nettests.

API

You can browse ooni/probe-engine's API online at godoc.org. We currently don't provide any API stability guarantees.

This repository also allows to build miniooni, a small command line client useful to test the functionality in here without integrating with OONI probe. You can browse the manual of this tool online at godoc.org. We currently don't promise that the miniooni CLI will be stable over time.

Integrating ooni/probe-engine

This software uses Go modules and requires Go v1.13+. We also depend on Measurement Kit, a C++14 library implementing many OONI tests.

Note that export CGO_ENABLED=0 will disable C/C++ extensions and therefore will prevent Measurement Kit tests from being linked into the resulting Go binaries. You may want that in some cases, e.g. when you only want to use OONI tests written in Go.

We plan on gradually rewriting all OONI tests in Go, therefore the dependency on Measurement Kit will eventually be removed.

A future version of this document will provide platform specific instruction for installing Measurement Kit and building.

Release procedure

  1. make sure that dependencies are up to date

  2. make sure that resources are up to date

  3. commit, tag, and push

  4. create new release on GitHub

Updating dependencies

  1. update direct dependencies using:
for name in `grep -v indirect go.mod | awk '/^\t/{print $1}'`; do \
  go get -u -v $name;                                             \
done
  1. pin to a specific psiphon version (we usually track the staging-client branch) using:
go get -v github.com/Psiphon-Labs/psiphon-tunnel-core@COMMITHASH

  1. clone psiphon-tunnel-core and generate a go.mod by running go mod init && go mod tidy in the toplevel dir

  2. rewrite go.mod such that it contains only your direct dependencies followed by the exact content of psiphon-tunnel-core's go.mod

  3. run go mod tidy

This allows us to pin all psiphon dependencies precisely.

Documentation

Overview

Package engine contains the engine API

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AllExperiments 

func AllExperiments() []string

AllExperiments returns the name of all experiments

Types

type Callbacks 

type Callbacks interface {
	// OnDataUsage provides information about data usage.
	OnDataUsage(dloadKiB, uploadKiB float64)

	// OnProgress provides information about an experiment progress.
	OnProgress(percentage float64, message string)
}

Callbacks contains event handling callbacks

This is a copy of experiment/handler.Callbacks. Go will make sure the interface will match for us. This means we can have this set of callbacks as part of the toplevel engine API.

type Experiment 

type Experiment struct {
	// contains filtered or unexported fields
}

Experiment is an experiment instance.

func (*Experiment) CloseReport 

func (e *Experiment) CloseReport() error

CloseReport is an idempotent method that closes and open report if one has previously been opened, otherwise it does nothing.

func (*Experiment) LoadMeasurement 

func (e *Experiment) LoadMeasurement(data []byte) (*Measurement, error)

LoadMeasurement loads a measurement from a byte stream. The measurement must be a measurement for this experiment.

func (*Experiment) Measure 

func (e *Experiment) Measure(input string) (*Measurement, error)

Measure performs a measurement with input. We assume that you have configured the available test helpers, either manually or by calling the session's MaybeLookupBackends() method.

func (*Experiment) Name 

func (e *Experiment) Name() string

Name returns the experiment name.

func (*Experiment) OpenReport 

func (e *Experiment) OpenReport() error

OpenReport is an idempotent method to open a report. We assume that you have configured the available collectors, either manually or through using the session's MaybeLookupBackends method.

func (*Experiment) ReportID 

func (e *Experiment) ReportID() string

ReportID returns the open reportID, if we have opened a report successfully before, or an empty string, otherwise.

func (*Experiment) SaveMeasurement 

func (e *Experiment) SaveMeasurement(measurement *Measurement, path string) error

SaveMeasurement saves the measurement at the specified path.

func (*Experiment) SubmitAndUpdateMeasurement 

func (e *Experiment) SubmitAndUpdateMeasurement(measurement *Measurement) error

SubmitAndUpdateMeasurement submits a measurement and updates the fields whose value has changed as part of the submission.

type ExperimentBuilder 

type ExperimentBuilder struct {
	// contains filtered or unexported fields
}

ExperimentBuilder is an experiment builder.

func (*ExperimentBuilder) Build 

func (b *ExperimentBuilder) Build() *Experiment

Build builds the experiment

func (*ExperimentBuilder) NeedsInput 

func (b *ExperimentBuilder) NeedsInput() bool

NeedsInput returns whether the experiment needs input

func (*ExperimentBuilder) Options 

func (b *ExperimentBuilder) Options() (map[string]OptionInfo, error)

Options returns info about all options

func (*ExperimentBuilder) SetCallbacks 

func (b *ExperimentBuilder) SetCallbacks(callbacks Callbacks)

SetCallbacks sets the interactive callbacks

func (*ExperimentBuilder) SetOptionBool 

func (b *ExperimentBuilder) SetOptionBool(key string, value bool) error

SetOptionBool sets a bool option

func (*ExperimentBuilder) SetOptionInt 

func (b *ExperimentBuilder) SetOptionInt(key string, value int64) error

SetOptionInt sets an int option

func (*ExperimentBuilder) SetOptionString 

func (b *ExperimentBuilder) SetOptionString(key, value string) error

SetOptionString sets a string option

type FileSystemKVStore added in v0.4.0 

type FileSystemKVStore struct {
	// contains filtered or unexported fields
}

FileSystemKVStore is a directory based KVStore

func NewFileSystemKVStore added in v0.4.0 

func NewFileSystemKVStore(basedir string) (kvs *FileSystemKVStore, err error)

NewFileSystemKVStore creates a new FileSystemKVStore.

func (*FileSystemKVStore) Get added in v0.4.0 

func (kvs *FileSystemKVStore) Get(key string) ([]byte, error)

Get returns the specified key's value

func (*FileSystemKVStore) Set added in v0.4.0 

func (kvs *FileSystemKVStore) Set(key string, value []byte) error

Set sets the value of a specific key

type KVStore added in v0.4.0 

type KVStore interface {
	Get(key string) (value []byte, err error)
	Set(key string, value []byte) (err error)
}

KVStore is a simple, atomic key-value store. The user of probe-engine should supply an implementation of this interface, which will be used by probe-engine to store specific data.

type Measurement 

type Measurement struct {
	// contains filtered or unexported fields
}

Measurement is a OONI measurement

func (*Measurement) AddAnnotations 

func (m *Measurement) AddAnnotations(annotations map[string]string)

AddAnnotations adds annotation to the measurement

func (*Measurement) MakeGenericTestKeys 

func (m *Measurement) MakeGenericTestKeys() (map[string]interface{}, error)

MakeGenericTestKeys casts the m.TestKeys to a map[string]interface{}.

Ideally, all tests should have a clear Go structure, well defined, that will be stored in m.TestKeys as an interface. This is not already the case and it's just valid for tests written in Go. Until all tests will be written in Go, we'll keep this glue here to make sure we convert from the engine format to the cli format.

This function will first attempt to cast directly to map[string]interface{}, which is possible for MK tests, and then use JSON serialization and de-serialization only if that's required.

type OptionInfo 

type OptionInfo struct {
	Doc  string
	Type string
}

OptionInfo contains info about an option

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is a measurement session

func NewSession 

func NewSession(config SessionConfig) (*Session, error)

NewSession creates a new session or returns an error

func (*Session) AddAvailableHTTPSBouncer 

func (sess *Session) AddAvailableHTTPSBouncer(baseURL string)

AddAvailableHTTPSBouncer adds an HTTPS bouncer to the list of bouncers that we'll try to contact.

func (*Session) AddAvailableHTTPSCollector 

func (sess *Session) AddAvailableHTTPSCollector(baseURL string)

AddAvailableHTTPSCollector adds an HTTPS collector to the list of collectors that we'll try to use.

func (*Session) MaybeLookupBackends 

func (sess *Session) MaybeLookupBackends() error

MaybeLookupBackends is a caching OONI backends lookup call.

func (*Session) MaybeLookupLocation 

func (sess *Session) MaybeLookupLocation() error

MaybeLookupLocation is a caching location lookup call.

func (*Session) NewExperimentBuilder 

func (sess *Session) NewExperimentBuilder(name string) (*ExperimentBuilder, error)

NewExperimentBuilder returns a new experiment builder for the experiment with the given name, or an error if there's no such experiment with the given name

func (*Session) Platform added in v0.3.0 

func (sess *Session) Platform() string

Platform returns the current platform. The platform is one of:

- android - ios - linux - macos - windows - unknown

When running on the iOS simulator, the returned platform is macos rather than ios if CGO is disabled. This is a known issue, that however should have a very limited impact.

func (*Session) ProbeASN 

func (sess *Session) ProbeASN() uint

ProbeASN returns the ASN of the probe's network.

func (*Session) ProbeASNString 

func (sess *Session) ProbeASNString() string

ProbeASNString is like ProbeASN but returns the "AS<%d>" string where the number is the number returned by ProbeASN.

func (*Session) ProbeCC 

func (sess *Session) ProbeCC() string

ProbeCC returns the probe country code.

func (*Session) ProbeIP 

func (sess *Session) ProbeIP() string

ProbeIP returns the probe IP.

func (*Session) ProbeNetworkName 

func (sess *Session) ProbeNetworkName() string

ProbeNetworkName returns the probe network name.

func (*Session) QueryTestListsURLs added in v0.3.0 

func (sess *Session) QueryTestListsURLs(
	conf *TestListsURLsConfig,
) (*TestListsURLsResult, error)

QueryTestListsURLs queries the test-lists/urls API.

func (*Session) ResolverASN added in v0.5.0 

func (sess *Session) ResolverASN() uint

ResolverASN returns the resolver ASN

func (*Session) ResolverASNString added in v0.5.0 

func (sess *Session) ResolverASNString() string

ResolverASNString returns the resolver ASN as as string

func (*Session) ResolverIP 

func (sess *Session) ResolverIP() string

ResolverIP returns the resolver IP.

func (*Session) ResolverNetworkName added in v0.5.0 

func (sess *Session) ResolverNetworkName() string

ResolverNetworkName returns the resolver network name

func (*Session) SetIncludeProbeASN 

func (sess *Session) SetIncludeProbeASN(value bool)

SetIncludeProbeASN controls whether to include the ASN

func (*Session) SetIncludeProbeCC 

func (sess *Session) SetIncludeProbeCC(value bool)

SetIncludeProbeCC controls whether to include the country code

func (*Session) SetIncludeProbeIP 

func (sess *Session) SetIncludeProbeIP(value bool)

SetIncludeProbeIP controls whether to include the IP

type SessionConfig 

type SessionConfig struct {
	AssetsDir       string
	Logger          log.Logger
	KVStore         KVStore
	ProxyURL        *url.URL
	SoftwareName    string
	SoftwareVersion string
	TempDir         string
	TLSConfig       *tls.Config
}

SessionConfig contains the Session config

type TestListsURLsConfig added in v0.3.0 

type TestListsURLsConfig struct {
	BaseURL    string   // URL to use (empty means default)
	Categories []string // Categories to query for (empty means all)
	Limit      int64    // Max number of URLs (<= 0 means no limit)
}

TestListsURLsConfig config config for test-lists/urls API.

func (*TestListsURLsConfig) AddCategory added in v0.3.0 

func (c *TestListsURLsConfig) AddCategory(s string)

AddCategory adds a category to the list of categories to query. Not adding any categories will query for URLs in all categories.

type TestListsURLsResult added in v0.3.0 

type TestListsURLsResult struct {
	Result []model.URLInfo
}

TestListsURLsResult contains the results of calling the test-lists/urls OONI orchestra API.

func (*TestListsURLsResult) At added in v0.3.0 

func (r *TestListsURLsResult) At(idx int64) (out *model.URLInfo)

At returns the URL at the given index or nil

func (*TestListsURLsResult) Count added in v0.3.0 

func (r *TestListsURLsResult) Count() int64

Count returns the number of returned URLs

Source Files

View all Source files

Directories

Path Synopsis
Package bouncer contains a OONI bouncer client implementation.
 Package bouncer contains a OONI bouncer client implementation.
cmd
miniooni command
Command miniooni is simple binary for testing purposes.
 Command miniooni is simple binary for testing purposes.
Package collector contains a OONI collector client implementation.
 Package collector contains a OONI collector client implementation.
Package experiment contains network experiment.
 Package experiment contains network experiment.
dash
Package dash contains the dash network experiment.
 Package dash contains the dash network experiment.
example
Package example contains a simple example of experiment.
 Package example contains a simple example of experiment.
fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
 Package fbmessenger contains the Facebook Messenger network experiment.
handler
Package handler contains experiment events handler
 Package handler contains experiment events handler
hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
 Package hhfm contains the HTTP Header Field Manipulation network experiment.
hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
 Package hirl contains the HTTP Invalid Request Line network experiment.
httpheader
Package httpheader contains code to set common HTTP headers.
 Package httpheader contains code to set common HTTP headers.
mkevent
Package mkevent processes MK events
 Package mkevent processes MK events
mkhelper
Package mkhelper contains common code to get the proper helper and configure it into settings.
 Package mkhelper contains common code to get the proper helper and configure it into settings.
mkrunner
Package mkrunner contains code to run an MK based test
 Package mkrunner contains code to run an MK based test
ndt
Package ndt contains the ndt network experiment.
 Package ndt contains the ndt network experiment.
ndt7
Package ndt7 contains the ndt7 network experiment.
 Package ndt7 contains the ndt7 network experiment.
psiphon
Package psiphon implements the psiphon network experiment.
 Package psiphon implements the psiphon network experiment.
sniblocking
Package sniblocking contains the SNI blocking network experiment.
 Package sniblocking contains the SNI blocking network experiment.
telegram
Package telegram contains the Telegram network experiment.
 Package telegram contains the Telegram network experiment.
tor
Package tor contains the tor experiment.
 Package tor contains the tor experiment.
web_connectivity
Package web_connectivity contains the Web Connectivity network experiment.
 Package web_connectivity contains the Web Connectivity network experiment.
whatsapp
Package whatsapp contains the WhatsApp network experiment.
 Package whatsapp contains the WhatsApp network experiment.
geoiplookup
iplookup
Package iplookup implements probe IP lookup.
 Package iplookup implements probe IP lookup.
iplookup/akamai
Package akamai lookups the IP using akamai.
 Package akamai lookups the IP using akamai.
iplookup/avast
Package avast lookups the IP using avast.
 Package avast lookups the IP using avast.
iplookup/invalid
Package invalid returns an invalid IP.
 Package invalid returns an invalid IP.
iplookup/ubuntu
Package ubuntu lookups the IP using Ubuntu.
 Package ubuntu lookups the IP using Ubuntu.
mmdblookup
Package mmdblookup performs probe ASN, CC, NetworkName lookups.
 Package mmdblookup performs probe ASN, CC, NetworkName lookups.
resolverlookup
Package resolverlookup discovers the resolver's IP
 Package resolverlookup discovers the resolver's IP
Package httpx contains HTTP extensions.
 Package httpx contains HTTP extensions.
httplog
Package httplog implements HTTP event logging.
 Package httplog implements HTTP event logging.
httptracex
Package httptracex contains httptrace extensions.
 Package httptracex contains httptrace extensions.
netx
Package netx contains network extensions.
 Package netx contains network extensions.
retryx
Package retryx helps to retry operations
 Package retryx helps to retry operations
internal
fetch
Package fetch is used to fetch resources.
 Package fetch is used to fetch resources.
jsonapi
Package jsonapi interacts with HTTP JSON APIs.
 Package jsonapi interacts with HTTP JSON APIs.
kvstore
Package kvstore contains key-value stores
 Package kvstore contains key-value stores
netxlogger
Package netxlogger is a logger for netx events.
 Package netxlogger is a logger for netx events.
oonidatamodel
Package oonidatamodel contains the OONI data model.
 Package oonidatamodel contains the OONI data model.
oonitemplates
Package oonitemplates contains templates for experiments.
 Package oonitemplates contains templates for experiments.
orchestra
Package orchestra contains common orchestra code.
 Package orchestra contains common orchestra code.
orchestra/login
Package login contains code to login with OONI orchestra.
 Package login contains code to login with OONI orchestra.
orchestra/metadata
Package metadata contains metadata about a probe
 Package metadata contains metadata about a probe
orchestra/register
Package register contains code to register to the OONI orchestra.
 Package register contains code to register to the OONI orchestra.
orchestra/statefile
Package statefile defines the state file
 Package statefile defines the state file
orchestra/testlists/psiphon
Package psiphon implements fetching psiphon config using orchestra
 Package psiphon implements fetching psiphon config using orchestra
orchestra/testlists/tor
Package tor contains code to fetch targets for the tor experiment.
 Package tor contains code to fetch targets for the tor experiment.
orchestra/testlists/urls
Package urls queries orchestra test-lists/urls API
 Package urls queries orchestra test-lists/urls API
orchestra/testorchestra
Package testorchestra contains code to simplify testing
 Package testorchestra contains code to simplify testing
orchestra/update
Package update contains code to update the probe state with orchestra
 Package update contains code to update the probe state with orchestra
platform
Package platform returns the platform name.
 Package platform returns the platform name.
tlsx
Package tlsx contains TLS extensions
 Package tlsx contains TLS extensions
urlpath
Package urlpath contains code to manipulate URL paths
 Package urlpath contains code to manipulate URL paths
Package log defines the logging interface used in this library.
 Package log defines the logging interface used in this library.
Package measurementkit allows to use Measurement Kit.
 Package measurementkit allows to use Measurement Kit.
Package model defines shared data structures.
 Package model defines shared data structures.
Package resources contains code to download resources.
 Package resources contains code to download resources.
Package session models a measurement session.
 Package session models a measurement session.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.