README
¶
BOXO: IPFS SDK for GO
A set of libraries for building IPFS applications and implementations in GO.
About
Boxo is a component library for building IPFS applications and implementations in Go.
Some scenarios in which you may find Boxo helpful:
- You are building an application that interacts with the IPFS network
- You are building an IPFS implementation
- You want to reuse some components of IPFS such as its Kademlia DHT, Bitswap, data encoding, etc.
- You want to experiment with IPFS
Boxo powers Kubo, which is the most popular IPFS implementation, so its code has been battle-tested on the IPFS network for years, and is well-understood by the community.
Motivation
TL;DR The goal of this repo is to help people build things. Previously users struggled to find existing useful code or to figure out how to use what they did find. We observed many running Kubo and using its HTTP RPC API. This repo aims to do better. We're taking the libraries that many were already effectively relying on in production and making them more easily discoverable and usable.
The maintainers primarily aim to help people trying to build with IPFS in Go that were previously either giving up or relying on the Kubo HTTP RPC API. Some of these people will end up being better served by IPFS tooling in other languages (e.g., Javascript, Rust, Java, Python), but for those who are either looking to write in Go or to leverage the set of IPFS tooling we already have in Go we’d like to make their lives easier.
We’d also like to make life easier on ourselves as the maintainers by reducing the maintenance burden that comes from being the owners on many repos and then use that time to contribute more to the community in the form of easier to use libraries, better implementations, improved protocols, new protocols, etc.
Boxo is not exhaustive nor comprehensive--there are plenty of useful IPFS protocols, specs, libraries, etc. that are not in Boxo. The goal of Boxo is to provide cohesive and well-maintained components for common IPFS use cases.
More details can also be found in the Rationale FAQ.
Scope
What kind of components does Boxo have?
Boxo includes high-quality components useful for interacting with IPFS protocols, public and private IPFS networks, and content-addressed data, such as:
- Content routing (DHT, delegated content routing, providing)
- Data transfer (gateways, Bitswap, incremental verification)
- Naming and mutability (name resolution, IPNS)
- Interacting with public and private IPFS networks
- Working with content-addressed data
Boxo aims to provide a cohesive interface into these components. Note that not all of the underlying components necessarily reside in this repository.
Does Boxo == IPFS?
No. This repo houses some IPFS functionality written in Go that has been useful in practice, and is maintained by a group that has long term commitments to the IPFS project
Is everything related to IPFS in the Go ecosystem in this repo?
No. Not everything related to IPFS is intended to be in Boxo. View it as a starter toolbox (potentially among multiple). If you’d like to build an IPFS implementation with Go, here are some tools you might want that are maintained by a group that has long term commitments to the IPFS project. There are certainly repos that others maintain that aren't included here (e.g., ipfs/go-car) which are still useful to IPFS implementations. It's expected and fine for new IPFS functionality to be developed that won't be part of Boxo.
Consuming
Getting started
See examples.
Deprecations and Breaking Changes
See RELEASE.md.
Development
Should I add my IPFS component to Boxo?
We happily accept external contributions! However, Boxo maintains a high quality bar, so code accepted into Boxo must meet some minimum maintenance criteria:
- Actively maintained
- Must be actively used by, or will be included in software that is actively used by, a significant number of users or production systems. Code that is not actively used cannot be properly maintained.
- Must have multiple engineers who are willing and able to maintain the relevant code in Boxo for a long period of time.
- If either of these changes, Boxo maintainers will consider removing the component from Boxo.
- Adequately tested
- At least with unit tests
- Ideally also including integration tests with other components
- Adequately documented
- Godocs at minimum
- Complex components should have their own doc.go or README.md describing the component, its use cases, tradeoffs, design rationale, etc.
- If the maintainers are not Boxo maintainers, then the component must include a CODEOWNERS file with at least two code owners who can commit to reviewing PRs
If you have some experimental component that you think would benefit the IPFS community, we suggest you build the component in your own repository until it's clear that there's community demand for it, and then open an issue/PR in this repository to discuss including it in Boxo.
Release Process
See RELEASE.md.
Why is the code coverage so bad?
The code coverage of this repo is not currently representative of the actual test coverage of this code. Much of the code in this repo is currently covered by integration tests in Kubo. We are in the process of moving those tests here, and as that continues the code coverage will significantly increase.
General
Help
If you suspect a bug or have technical questions, feel free to open an issue.
For regular support, try Community chat's #ipfs-implementers room or help/boxo at IPFS discussion forums.
What is the response time for issues or PRs filed?
New issues and PRs to this repo are usually looked at on a weekly basis as part of Shipyard's GO Triage triage. However, the response time may vary.
What are some projects that depend on this project?
The exhaustive list is https://github.com/ipfs/boxo/network/dependents. Some notable projects include:
- Kubo, an IPFS implementation in Go
- Lotus, a Filecoin implementation in Go
- rainbow, a specialized IPFS gateway
- ipfs-check, checks IPFS data availability
- someguy, a dedicated Delegated Routing V1 server and client
- IPFS Desktop, a desktop application for running an IPFS node
Governance and Access
See CODEOWNERS for the current maintainers list. Governance for graduating additional maintainers hasn't been established. Repo permissions are all managed through ipfs/github-mgmt.
Why is this named "Boxo"?
See https://github.com/ipfs/boxo/issues/215.
Additional Docs and FAQs
See the wiki.
License
Directories
¶
| Path | Synopsis |
|---|---|
|
Package autoconf provides a client for fetching and caching IPFS network configurations.
|
Package autoconf provides a client for fetching and caching IPFS network configurations. |
|
Package bitswap implements the [Bitswap protocol] for exchanging blocks between IPFS peers.
|
Package bitswap implements the [Bitswap protocol] for exchanging blocks between IPFS peers. |
|
client
Package client implements the IPFS exchange interface with the [Bitswap] bilateral exchange protocol (see [interaction pattern]).
|
Package client implements the IPFS exchange interface with the [Bitswap] bilateral exchange protocol (see [interaction pattern]). |
|
client/internal/messagequeue
Package messagequeue implements a queue of want messages to send to peers.
|
Package messagequeue implements a queue of want messages to send to peers. |
|
client/wantlist
Package wantlist implements an object for bitswap that contains the keys that a given peer wants.
|
Package wantlist implements an object for bitswap that contains the keys that a given peer wants. |
|
message
Package message implements [Bitswap protocol] messages.
|
Package message implements [Bitswap protocol] messages. |
|
message/pb
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames.
|
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames. |
|
metrics
Package metrics provides metric constructors for Bitswap operations, including histograms for received, sent, and duplicate block sizes and for message send times.
|
Package metrics provides metric constructors for Bitswap operations, including histograms for received, sent, and duplicate block sizes and for message send times. |
|
network
Package network defines interfaces for [Bitswap protocol] network operations.
|
Package network defines interfaces for [Bitswap protocol] network operations. |
|
network/httpnet
Package httpnet implements an Exchange network that sends and receives Exchange messages from peers' HTTP endpoints.
|
Package httpnet implements an Exchange network that sends and receives Exchange messages from peers' HTTP endpoints. |
|
server
Package server implements the [Bitswap protocol] server that handles incoming block requests from peers (see [interaction pattern]).
|
Package server implements the [Bitswap protocol] server that handles incoming block requests from peers (see [interaction pattern]). |
|
server/internal/decision
Package decision implements the decision engine for the bitswap service.
|
Package decision implements the decision engine for the bitswap service. |
|
Package blockservice implements a BlockService interface that provides a single GetBlock/AddBlock interface that seamlessly retrieves data either locally or from a remote peer through the exchange.
|
Package blockservice implements a BlockService interface that provides a single GetBlock/AddBlock interface that seamlessly retrieves data either locally or from a remote peer through the exchange. |
|
Package blockstore implements a thin wrapper over a datastore, giving a clean interface for Getting and Putting block objects.
|
Package blockstore implements a thin wrapper over a datastore, giving a clean interface for Getting and Putting block objects. |
|
Package bootstrap manages network bootstrapping for IPFS nodes.
|
Package bootstrap manages network bootstrapping for IPFS nodes. |
|
Package chunk implements streaming block splitters.
|
Package chunk implements streaming block splitters. |
|
gen
command
This file generates bytehash LUT
|
This file generates bytehash LUT |
|
cmd
|
|
|
boxo-migrate
module
|
|
|
deprecator
module
|
|
|
migrate
module
|
|
|
dag
|
|
|
walker
Package walker provides memory-efficient DAG traversal with deduplication.
|
Package walker provides memory-efficient DAG traversal with deduplication. |
|
datastore
|
|
|
dshelp
Package dshelp provides utilities for parsing and creating datastore keys used by go-ipfs
|
Package dshelp provides utilities for parsing and creating datastore keys used by go-ipfs |
|
examples
module
|
|
|
Package exchange defines the IPFS exchange interface
|
Package exchange defines the IPFS exchange interface |
|
offline
Package offline implements an object that implements the exchange interface but returns nil values to every request.
|
Package offline implements an object that implements the exchange interface but returns nil values to every request. |
|
Package fetcher defines interfaces for reading data from a DAG.
|
Package fetcher defines interfaces for reading data from a DAG. |
|
Package files provides abstractions for working with files, directories, and other file-like objects.
|
Package files provides abstractions for working with files, directories, and other file-like objects. |
|
Package filestore implements a Blockstore which is able to read certain blocks of data directly from its original location in the filesystem.
|
Package filestore implements a Blockstore which is able to read certain blocks of data directly from its original location in the filesystem. |
|
pb
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames.
|
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames. |
|
posinfo
Package posinfo wraps offset information used by ipfs filestore nodes
|
Package posinfo wraps offset information used by ipfs filestore nodes |
|
Package gateway provides an HTTP gateway for serving IPFS content over HTTP.
|
Package gateway provides an HTTP gateway for serving IPFS content over HTTP. |
|
assets/test
command
|
|
|
internal
|
|
|
ipnstest
Package ipnstest builds IPNS records at the wire level for tests, bypassing ipns.NewRecord so callers can construct records that record creation would sanitize or that verification would reject (for example a negative TTL).
|
Package ipnstest builds IPNS records at the wire level for tests, bypassing ipns.NewRecord so callers can construct records that record creation would sanitize or that verification would reject (for example a negative TTL). |
|
ipld
|
|
|
merkledag
Package merkledag implements the IPFS Merkle DAG data structures.
|
Package merkledag implements the IPFS Merkle DAG data structures. |
|
merkledag/pb
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames.
|
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames. |
|
merkledag/traverse
Package traverse provides merkledag traversal functions
|
Package traverse provides merkledag traversal functions |
|
unixfs
Package unixfs implements the [UnixFS] data format for describing files, directories, and symlinks in IPFS.
|
Package unixfs implements the [UnixFS] data format for describing files, directories, and symlinks in IPFS. |
|
unixfs/hamt
Package hamt implements a Hash Array Mapped Trie over ipfs merkledag nodes.
|
Package hamt implements a Hash Array Mapped Trie over ipfs merkledag nodes. |
|
unixfs/importer
Package importer implements utilities used to create IPFS DAGs from files and readers.
|
Package importer implements utilities used to create IPFS DAGs from files and readers. |
|
unixfs/importer/balanced
Package balanced provides methods to build balanced DAGs, which are generalistic DAGs in which all leaves (nodes representing chunks of data) are at the same distance from the root.
|
Package balanced provides methods to build balanced DAGs, which are generalistic DAGs in which all leaves (nodes representing chunks of data) are at the same distance from the root. |
|
unixfs/importer/trickle
Package trickle allows to build trickle DAGs.
|
Package trickle allows to build trickle DAGs. |
|
unixfs/io
Package io implements convenience objects for working with the IPFS UnixFS data format.
|
Package io implements convenience objects for working with the IPFS UnixFS data format. |
|
unixfs/mod
Package mod provides DAG modification utilities for UnixFS files.
|
Package mod provides DAG modification utilities for UnixFS files. |
|
unixfs/pb
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames.
|
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames. |
|
Package ipns implements IPNS record creation, marshaling, and validation as specified in the [IPNS Record specification].
|
Package ipns implements IPNS record creation, marshaling, and validation as specified in the [IPNS Record specification]. |
|
pb
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames.
|
These commands work around namespace conflicts that occur when multiple repositories depend on .proto files with generic filenames. |
|
Package keystore provides a key management interface and a filesystem-backed implementation for storing cryptographic private keys.
|
Package keystore provides a key management interface and a filesystem-backed implementation for storing cryptographic private keys. |
|
Package mfs implements an in-memory model of a mutable IPFS filesystem.
|
Package mfs implements an in-memory model of a mutable IPFS filesystem. |
|
Package namesys defines Resolver and Publisher interfaces for IPNS paths, that is, IPFS paths in the form of /ipns/<name_to_be_resolved>.
|
Package namesys defines Resolver and Publisher interfaces for IPNS paths, that is, IPFS paths in the form of /ipns/<name_to_be_resolved>. |
|
republisher
Package republisher provides a utility to automatically re-publish IPNS records related to the keys in a Keystore.
|
Package republisher provides a utility to automatically re-publish IPNS records related to the keys in a Keystore. |
|
Package path contains utilities to work with ipfs paths.
|
Package path contains utilities to work with ipfs paths. |
|
resolver
Package resolver implements utilities for resolving paths within ipfs.
|
Package resolver implements utilities for resolving paths within ipfs. |
|
Package peering maintains persistent connections to specified peers.
|
Package peering maintains persistent connections to specified peers. |
|
pinning
|
|
|
pinner
Package pin implements structures and methods to keep track of which objects a user wants to keep stored locally.
|
Package pin implements structures and methods to keep track of which objects a user wants to keep stored locally. |
|
pinner/dsindex
Package dsindex provides secondary indexing functionality for a datastore.
|
Package dsindex provides secondary indexing functionality for a datastore. |
|
pinner/dspinner
Package dspinner implements structures and methods to keep track of which objects a user wants to keep stored locally.
|
Package dspinner implements structures and methods to keep track of which objects a user wants to keep stored locally. |
|
remote/client/cmd
command
|
|
|
Package provider provides interfaces and tooling for (Re)providers.
|
Package provider provides interfaces and tooling for (Re)providers. |
|
Package retrieval tracks the state of an IPFS content retrieval as it moves through path resolution, provider discovery, connection, and data transfer.
|
Package retrieval tracks the state of an IPFS content retrieval as it moves through path resolution, provider discovery, connection, and data transfer. |
|
routing
|
|
|
http/client
Package client implements an HTTP client for the [Delegated Routing V1] API (IPIP-337).
|
Package client implements an HTTP client for the [Delegated Routing V1] API (IPIP-337). |
|
http/server
Package server implements an HTTP server for the [Delegated Routing V1] API (IPIP-337).
|
Package server implements an HTTP server for the [Delegated Routing V1] API (IPIP-337). |
|
mock
Package mockrouting provides a virtual routing server.
|
Package mockrouting provides a virtual routing server. |
|
offline
Package offline implements Routing with a client which is only able to perform offline operations.
|
Package offline implements Routing with a client which is only able to perform offline operations. |
|
Package tar extracts tar archives to the local filesystem.
|
Package tar extracts tar archives to the local filesystem. |
|
Package tracing creates OpenTelemetry span exporters from environment variables.
|
Package tracing creates OpenTelemetry span exporters from environment variables. |
|
Package util implements various utility functions used within ipfs that do not currently have a better place to live.
|
Package util implements various utility functions used within ipfs that do not currently have a better place to live. |
|
Package verifcid validates CIDs against configurable hash function allowlists.
|
Package verifcid validates CIDs against configurable hash function allowlists. |