README
¶
OONI Probe Client Library and CLI
The Open Observatory of Network Interference (OONI) is a non-profit free software project that aims to empower decentralized efforts in documenting Internet censorship around the world.
This repository contains core OONI tools written in Go:
-
the CLI client (cmd/ooniprobe);
-
the test helper server (internal/cmd/oohelperd);
-
the mobile library (pkg/oonimkall);
-
and all the related support packages (inside internal).
Every top-level directory in this repository contains an explanatory README file. You
may also notice that some internal packages live under internal/engine
while most others are top-level. This is part of a long-standing refactoring started
when we merged https://github.com/ooni/probe-engine into this repository. We'll slowly
ensure that all packages inside engine are moved out of it and inside internal.
Semantic versioning policy
The mobile library is a public package for technical reasons. Go mobile tools require
a public package to build from. Yet, we don't consider API breakages happening in
such a package to be sufficient to bump our major version number. For us, the mobile
library is just a mean to implement OONI Probe Android and OONI Probe iOS. We'll
only bump the major version number if we implement any set of breaking changes of
the ./cmd/ooniprobe's CLI.
License
SPDX-License-Identifier: GPL-3.0-or-later
User setup
Please, follow the instructions at ooni.org/install/cli
to install ooniprobe. If we do not support your use case, please let us know. Once
ooniprobe is installed, try ooniprobe help to get interactive help.
Reporting issues
Report issues at github.com/ooni/probe.
Please, make sure you add the ooni/probe-cli label.
Build instructions
ooniprobe
Be sure you have golang >= 1.17 and a C compiler (Mingw-w64 for Windows). You can build using:
go build -v ./cmd/ooniprobe
This will generate a binary called ooniprobe in the current directory.
Android bindings
Make sure you have GNU make installed, then run:
./mk android
to build bindings for Android. (Add OONI_PSIPHON_TAGS="" if you
cannot clone private repositories in the https://github.com/ooni namespace.)
The generated bindings are (manually) pushed to the Maven Central package repository. The instructions explaining how to integrate these bindings are published along with the release notes.
iOS bindings
Make sure you have GNU make installed, then run:
./mk ios
to build bindings for iOS. (Add OONI_PSIPHON_TAGS="" if you
cannot clone private repositories in the https://github.com/ooni namespace.)
The generated bindings are (manually) added to GitHub releases. The instructions explaining how to integrate these bindings are published along with the release notes.
miniooni
Miniooni is the experimental OONI client used for research. Compile using:
go build -v ./internal/cmd/miniooni
This will generate a binary called miniooni in the current directory.
oohelperd
Oohelperd is the test helper server. Compile using:
go build -v ./internal/cmd/oohelperd
This will generate a binary called oohelperd in the current directory.
Specifications
Every nettest (aka experiment) implemented in this repository has a companion spec in the ooni/spec repository.
Contributing
Please, see CONTRIBUTING.md.
Updating dependencies
go get -u -v ./... && go mod tidy
Releasing
Create an issue according to the routine release template and perform any item inside the check-list.
We build releases using ./mk, which requires GNU make. Try
the ./mk help|less command for detailed usage.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
ooniprobe
command
|
|
|
ooniprobe/internal/autorun
Package autorun contains code to manage automatic runs
|
Package autorun contains code to manage automatic runs |
|
ooniprobe/internal/enginex
Package enginex contains ooni/probe-engine extensions.
|
Package enginex contains ooni/probe-engine extensions. |
|
ooniprobe/internal/log/handlers/cli/progress
Package progress provides a simple terminal progress bar.
|
Package progress provides a simple terminal progress bar. |
|
ooniprobe/internal/log/handlers/syslog
Package syslog contains a syslog handler.
|
Package syslog contains a syslog handler. |
|
ooniprobe/internal/oonitest
Package oonitest contains code used for testing.
|
Package oonitest contains code used for testing. |
|
internal
|
|
|
archival
Package implements a Saver type that saves network, TCP, DNS, and TLS events.
|
Package implements a Saver type that saves network, TCP, DNS, and TLS events. |
|
atomicx
Package atomicx extends sync/atomic.
|
Package atomicx extends sync/atomic. |
|
bytecounter
Package bytecounter contains code to track the number of bytes sent and received by a probe.
|
Package bytecounter contains code to track the number of bytes sent and received by a probe. |
|
cmd/apitool
command
Command apitool is a simple tool to fetch individual OONI measurements.
|
Command apitool is a simple tool to fetch individual OONI measurements. |
|
cmd/e2epostprocess
command
|
|
|
cmd/getresources
command
Command getresources downloads the resources
|
Command getresources downloads the resources |
|
cmd/jafar
command
Jafar is a censorship simulation tool used for testing OONI.
|
Jafar is a censorship simulation tool used for testing OONI. |
|
cmd/jafar/badproxy
Package badproxy implements misbehaving proxies.
|
Package badproxy implements misbehaving proxies. |
|
cmd/jafar/flagx
Package flagx contains extensions for the standard library flag package.
|
Package flagx contains extensions for the standard library flag package. |
|
cmd/jafar/httpproxy
Package httpproxy contains a censoring HTTP proxy.
|
Package httpproxy contains a censoring HTTP proxy. |
|
cmd/jafar/iptables
Package iptables contains code for managing firewall rules.
|
Package iptables contains code for managing firewall rules. |
|
cmd/jafar/resolver
Package resolver contains a censoring DNS resolver.
|
Package resolver contains a censoring DNS resolver. |
|
cmd/jafar/tlsproxy
Package tlsproxy contains a censoring TLS proxy.
|
Package tlsproxy contains a censoring TLS proxy. |
|
cmd/jafar/uncensored
Package uncensored contains code used by Jafar to evade its own censorship efforts by taking alternate routes.
|
Package uncensored contains code used by Jafar to evade its own censorship efforts by taking alternate routes. |
|
cmd/miniooni
command
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x.
|
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x. |
|
cmd/oohelper
command
Command oohelper contains a simple command line client for the Web Connectivity test helper.
|
Command oohelper contains a simple command line client for the Web Connectivity test helper. |
|
cmd/oohelperd
command
Command oohelperd contains the Web Connectivity test helper.
|
Command oohelperd contains the Web Connectivity test helper. |
|
cmd/oonireport
command
Command oonireport uploads reports stored on disk to the OONI collector.
|
Command oonireport uploads reports stored on disk to the OONI collector. |
|
cmd/printversion
command
Command printversion prints the current version of this repository.
|
Command printversion prints the current version of this repository. |
|
engine
Package engine contains the engine API.
|
Package engine contains the engine API. |
|
engine/experiment/dash
Package dash implements the DASH network experiment.
|
Package dash implements the DASH network experiment. |
|
engine/experiment/dnscheck
Package dnscheck contains the DNS check experiment.
|
Package dnscheck contains the DNS check experiment. |
|
engine/experiment/dnsping
Package dnsping is the experimental dnsping experiment.
|
Package dnsping is the experimental dnsping experiment. |
|
engine/experiment/example
Package example contains a simple example experiment.
|
Package example contains a simple example experiment. |
|
engine/experiment/fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
|
Package fbmessenger contains the Facebook Messenger network experiment. |
|
engine/experiment/hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
|
Package hhfm contains the HTTP Header Field Manipulation network experiment. |
|
engine/experiment/hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
|
Package hirl contains the HTTP Invalid Request Line network experiment. |
|
engine/experiment/httphostheader
Package httphostheader contains the HTTP host header network experiment.
|
Package httphostheader contains the HTTP host header network experiment. |
|
engine/experiment/ndt7
Package ndt7 contains the ndt7 network experiment.
|
Package ndt7 contains the ndt7 network experiment. |
|
engine/experiment/psiphon
Package psiphon implements the psiphon network experiment.
|
Package psiphon implements the psiphon network experiment. |
|
engine/experiment/quicping
Package quicping implements the quicping network experiment.
|
Package quicping implements the quicping network experiment. |
|
engine/experiment/riseupvpn
Package riseupvpn contains the RiseupVPN network experiment.
|
Package riseupvpn contains the RiseupVPN network experiment. |
|
engine/experiment/run
Package run contains code to run other experiments.
|
Package run contains code to run other experiments. |
|
engine/experiment/signal
Package signal contains the Signal network experiment.
|
Package signal contains the Signal network experiment. |
|
engine/experiment/simplequicping
Package simplequicping is the experimental simplequicping experiment.
|
Package simplequicping is the experimental simplequicping experiment. |
|
engine/experiment/sniblocking
Package sniblocking contains the SNI blocking network experiment.
|
Package sniblocking contains the SNI blocking network experiment. |
|
engine/experiment/stunreachability
Package stunreachability contains the STUN reachability experiment.
|
Package stunreachability contains the STUN reachability experiment. |
|
engine/experiment/tcpping
Package tcpping is the experimental tcpping experiment.
|
Package tcpping is the experimental tcpping experiment. |
|
engine/experiment/telegram
Package telegram contains the Telegram network experiment.
|
Package telegram contains the Telegram network experiment. |
|
engine/experiment/tlsping
Package tlsping is the experimental tlsping experiment.
|
Package tlsping is the experimental tlsping experiment. |
|
engine/experiment/tlstool
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments.
|
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments. |
|
engine/experiment/tlstool/internal
Package internal contains the implementation of tlstool.
|
Package internal contains the implementation of tlstool. |
|
engine/experiment/tor
Package tor contains the tor experiment.
|
Package tor contains the tor experiment. |
|
engine/experiment/torsf
Package torsf contains the torsf experiment.
|
Package torsf contains the torsf experiment. |
|
engine/experiment/urlgetter
Package urlgetter implements a nettest that fetches a URL.
|
Package urlgetter implements a nettest that fetches a URL. |
|
engine/experiment/vanillator
Package vanillator contains the vanilla_tor experiment.
|
Package vanillator contains the vanilla_tor experiment. |
|
engine/experiment/webconnectivity
Package webconnectivity implements OONI's Web Connectivity experiment.
|
Package webconnectivity implements OONI's Web Connectivity experiment. |
|
engine/experiment/webconnectivity/internal
Package internal contains internal code.
|
Package internal contains internal code. |
|
engine/experiment/webstepsx
Package webstepsx contains a websteps implementation based on the internal/measurex package.
|
Package webstepsx contains a websteps implementation based on the internal/measurex package. |
|
engine/experiment/whatsapp
Package whatsapp contains the WhatsApp network experiment.
|
Package whatsapp contains the WhatsApp network experiment. |
|
engine/geolocate
Package geolocate implements IP lookup, resolver lookup, and geolocation.
|
Package geolocate implements IP lookup, resolver lookup, and geolocation. |
|
engine/httpheader
Package httpheader contains code to set common HTTP headers.
|
Package httpheader contains code to set common HTTP headers. |
|
engine/internal/httpfailure
Package httpfailure groups a bunch of extra HTTP failures.
|
Package httpfailure groups a bunch of extra HTTP failures. |
|
engine/internal/sessionresolver
Package sessionresolver contains the resolver used by the session.
|
Package sessionresolver contains the resolver used by the session. |
|
engine/legacy/assetsdir
Package assetsdir contains code to cleanup the assets dir.
|
Package assetsdir contains code to cleanup the assets dir. |
|
engine/mockable
Package mockable contains mockable objects
|
Package mockable contains mockable objects |
|
engine/netx
Package netx contains code to perform network measurements.
|
Package netx contains code to perform network measurements. |
|
engine/netx/archival
Package archival contains data formats used for archival.
|
Package archival contains data formats used for archival. |
|
engine/netx/dialer
Package dialer allows you to create a net.Dialer-compatible DialContext-enabled dialer with error wrapping, optional logging, optional network-events saving, and optional proxying.
|
Package dialer allows you to create a net.Dialer-compatible DialContext-enabled dialer with error wrapping, optional logging, optional network-events saving, and optional proxying. |
|
engine/netx/httptransport
Package httptransport contains HTTP transport extensions.
|
Package httptransport contains HTTP transport extensions. |
|
engine/probeservices
Package probeservices contains code to contact OONI probe services.
|
Package probeservices contains code to contact OONI probe services. |
|
engine/probeservices/testorchestra
Package testorchestra helps with testing the OONI orchestra API.
|
Package testorchestra helps with testing the OONI orchestra API. |
|
fakefill
Package fakefill contains code to fill structs for testing.
|
Package fakefill contains code to fill structs for testing. |
|
fsx
Package fsx contains io/fs extensions.
|
Package fsx contains io/fs extensions. |
|
httpx
Package httpx contains http extensions.
|
Package httpx contains http extensions. |
|
humanize
Package humanize is like dustin/go-humanize.
|
Package humanize is like dustin/go-humanize. |
|
kvstore
Package kvstore implements model.KeyValueStore.
|
Package kvstore implements model.KeyValueStore. |
|
measurex
Package measurex contains measurement extensions.
|
Package measurex contains measurement extensions. |
|
mlablocate
Package mlablocate contains a locate.measurementlab.net client implementing v1 of the locate API.
|
Package mlablocate contains a locate.measurementlab.net client implementing v1 of the locate API. |
|
mlablocatev2
Package mlablocatev2 implements m-lab locate services API v2.
|
Package mlablocatev2 implements m-lab locate services API v2. |
|
model
Package model contains the shared interfaces and data structures.
|
Package model contains the shared interfaces and data structures. |
|
model/mocks
Package mocks contains mocks for internal/model interfaces.
|
Package mocks contains mocks for internal/model interfaces. |
|
multierror
Package multierror contains code to manage multiple errors.
|
Package multierror contains code to manage multiple errors. |
|
netxlite
Package netxlite contains network extensions.
|
Package netxlite contains network extensions. |
|
netxlite/filtering
Package filtering allows to implement self-censorship.
|
Package filtering allows to implement self-censorship. |
|
netxlite/internal/gencertifi
command
|
|
|
netxlite/internal/generrno
command
|
|
|
netxlite/quictesting
Package quictesting contains code useful to test QUIC.
|
Package quictesting contains code useful to test QUIC. |
|
ooapi
Package ooapi contains a client for the OONI API.
|
Package ooapi contains a client for the OONI API. |
|
ooapi/apimodel
Package apimodel describes the data types used by OONI's API.
|
Package apimodel describes the data types used by OONI's API. |
|
ooapi/internal/generator
command
Command generator generates code in the ooapi package.
|
Command generator generates code in the ooapi package. |
|
ooapi/internal/openapi
Package openapi contains data structures for Swagger v2.0.
|
Package openapi contains data structures for Swagger v2.0. |
|
platform
Package platform allows you to obtain the platform name.
|
Package platform allows you to obtain the platform name. |
|
ptx
Package ptx contains code to use pluggable transports.
|
Package ptx contains code to use pluggable transports. |
|
randx
Package randx contains math/rand extensions.
|
Package randx contains math/rand extensions. |
|
runtimex
Package runtimex contains runtime extensions.
|
Package runtimex contains runtime extensions. |
|
shellx
Package shellx runs external commands.
|
Package shellx runs external commands. |
|
stuninput
Package stuninput contains stun targets as well as code to format such targets according to various conventions.
|
Package stuninput contains stun targets as well as code to format such targets according to various conventions. |
|
torlogs
Package torlogs contains code to read tor logs.
|
Package torlogs contains code to read tor logs. |
|
tunnel
Package tunnel allows to create tunnels to speak with OONI backends and other services.
|
Package tunnel allows to create tunnels to speak with OONI backends and other services. |
|
tunnel/mocks
Package mocks contains mocks for tunnel.
|
Package mocks contains mocks for tunnel. |
|
tutorial/experiment/torsf/chapter01
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/experiment/torsf/chapter02
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/generator
command
Command generator generates or re-generates the tutorial chapters.
|
Command generator generates or re-generates the tutorial chapters. |
|
tutorial/measurex/chapter01
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter02
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter03
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter04
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter05
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter06
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter07
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter08
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter09
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter10
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter11
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter12
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter13
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/measurex/chapter14
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter01
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter02
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter03
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter04
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter05
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter06
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter07
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
tutorial/netxlite/chapter08
command
-=-=- StartHere -=-=-
|
-=-=- StartHere -=-=- |
|
version
Package version contains version information
|
Package version contains version information |
|
pkg
|
|
|
oonimkall
Package oonimkall implements APIs used by OONI mobile apps.
|
Package oonimkall implements APIs used by OONI mobile apps. |
Click to show internal directories.
Click to hide internal directories.