sources

package
v0.22.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2026 License: MIT Imports: 0 Imported by: 0

Documentation

Overview

Package sources is the single source of truth for all external data URLs that atb downloads. Every URL the tool fetches from the internet is defined here, grouped by feature.

OSF project: https://osf.io/h7wzy/files/osfstorage AllTheBacteria: https://allthebacteria.org

To update a URL: change it here and it propagates everywhere. To audit what atb downloads: read this file.

Index

Constants

View Source
const (
	SketchSkmFilename = "atb_sketchlib.skm"
	SketchSkdFilename = "atb_sketchlib.skd"
	SketchSubdir      = "sketch"
)

SketchSkmFilename and SketchSkdFilename are local filenames for the sketch database files, stored in <data-dir>/sketch/.

View Source
const AGCArchiveMapFilename = "assemblies_filelist.txt.gz"

AGCArchiveMapFilename is the local cache filename for the archive map. It keeps the .gz extension because the artifact is cached exactly as downloaded.

View Source
const AGCArchiveMapURL = "https://osf.io/download/gtqrx/"

AGCArchiveMapURL is the accession->batch list for the balanced v202505 collection, published on OSF as a gzipped two-column text file (assemblies_filelist.txt.gz: accession, batch name without .agc). atb caches the gzip as downloaded and decompresses on read.

View Source
const AGCArchiveSubdir = "agc"

AGCArchiveSubdir is the <data-dir> subdirectory holding cached archives and the archive map. Mirrors SketchSubdir.

View Source
const AGCArchivesFolder = "agc_batches"

AGCArchivesFolder is the folder holding .agc batches on the collection nodes.

View Source
const AGCBatchMetadataURL = "https://osf.io/download/8y9r2/"

AGCBatchMetadataURL is the OSF /download/ URL of the batch metadata TSV (8y9r2, batches_202505_metadata.tsv.gz): a gzipped TSV with batch_name and old_name columns. old_name carries the species; the numbered batch filename does not, so `atb agc index` joins this on batch_name to fill the species column of the combined index.

View Source
const AGCIndexFilename = "atb_agc_files.tsv"

AGCIndexFilename is the local cache filename for the AGC index TSV, stored in <data-dir>/agc/ alongside the cached archives.

View Source
const AGCIndexURL = "https://osf.io/download/6a719381a2e9e3d202b91f7d/"

AGCIndexURL is the OSF /download/ URL of the published combined AGC index TSV (crawl + metadata join, produced by `atb agc index`). When set, the runtime downloads this single published TSV; an empty value falls back to crawling the collection nodes and joining the metadata on demand.

View Source
const AGCRepo = "refresh-bio/agc"

AGCRepo is the GitHub repository for agc releases.

View Source
const AGCVersion = "v3.2.3"

AGCVersion is the pinned release tag of the agc binary. agc 3.x reads v1/v2/v3 archives, so this single pin covers every .agc file in existence today.

View Source
const AssemblyBaseURL = "https://allthebacteria-assemblies.s3.eu-west-2.amazonaws.com/"

AssemblyBaseURL is the S3 bucket hosting individual genome FASTA files. File pattern: {sample_accession}.fa.gz Used by: atb download, atb sketch query --download

View Source
const IndexFilename = "all_atb_files.tsv"

IndexFilename is the local cache filename for the OSF file index.

View Source
const IndexURL = "https://osf.io/download/r6gcp/"

IndexURL is the master TSV index listing ~3,000 files across the entire AllTheBacteria project on OSF, with download URLs and MD5 checksums. Used by: atb osf ls, atb osf download Source: https://osf.io/r6gcp (all_atb_files.tsv)

View Source
const OSFAPIBase = "https://api.osf.io/v2"

OSFAPIBase is the root of the OSF REST API (v2).

View Source
const SketchSkdURL = "https://osf.io/download/92qmr/"

SketchSkdURL is the sketch data file (.skd) for the aggregated ATB sketch database (~4.1 GB). Used by: atb sketch fetch

View Source
const SketchSkmURL = "https://osf.io/download/nwfkc/"

SketchSkmURL is the sketch metadata file (.skm) for the aggregated ATB sketch database covering all genomes up to Aug 2024 (~122 MB). Used by: atb sketch fetch

View Source
const SketchlibRepo = "bacpop/sketchlib.rust"

SketchlibRepo is the GitHub repository for sketchlib releases.

View Source
const SketchlibVersion = "v0.2.4"

SketchlibVersion is the pinned version of the sketchlib binary.

Variables

View Source
var AGCCollectionNodes = []AGCNode{
	{ID: "4jq8u"},
	{ID: "jmeqg"},
	{ID: "kzcnr"},
}

AGCCollectionNodes are the OSF nodes that together host the balanced v202505 collection; each has an agc_batches/ folder of numbered .agc batches.

View Source
var CoreTables = []string{
	"assembly.parquet",
	"assembly_stats.parquet",
	"checkm2.parquet",
	"sylph.parquet",
	"run.parquet",
	"mlst.parquet",
	"amrfinderplus.parquet",
}

CoreTables lists the tables downloaded by default with `atb fetch`.

View Source
var TableURLs = map[string]string{

	"assembly.parquet":       "https://osf.io/download/4ku2n/",
	"assembly_stats.parquet": "https://osf.io/download/69c51e86801fecc5d6146396/",
	"checkm2.parquet":        "https://osf.io/download/69c51e93cba7111bb21d27f2/",
	"sylph.parquet":          "https://osf.io/download/69c51f90cba7111bb21d2905/",
	"run.parquet":            "https://osf.io/download/69c51f68376eb79a651d2d85/",
	"mlst.parquet":           "https://osf.io/download/69c66d33fa3d973d94254f46/",
	"amrfinderplus.parquet":  "https://osf.io/download/69f1e5debb4f674d5fd949ad/",

	"ena_20250506.parquet":    "https://osf.io/download/69c51f3ab4f99c692d54cf73/",
	"ena_20240801.parquet":    "https://osf.io/download/69c51f002e72f67915145d0e/",
	"ena_20240625.parquet":    "https://osf.io/download/69c51ec99ce80b96ac54cd08/",
	"ena_202505_used.parquet": "https://osf.io/download/69c51f475eedad376954ce7b/",
	"ena_661k.parquet":        "https://osf.io/download/69c51f57376eb79a651d2d83/",
}

TableURLs maps parquet table filenames to their OSF download URLs.

Functions

func OSFNodeFilesURL added in v0.18.0

func OSFNodeFilesURL(nodeID string) string

OSFNodeFilesURL returns the osfstorage root listing URL for an OSF node, the entry point for crawling its folders.

Types

type AGCNode added in v0.18.0

type AGCNode struct {
	ID string
}

AGCNode identifies one OSF node in the collection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL