Documentation
¶
Overview ¶
Package sources is the single source of truth for all external data URLs that atb downloads. Every URL the tool fetches from the internet is defined here, grouped by feature.
OSF project: https://osf.io/h7wzy/files/osfstorage AllTheBacteria: https://allthebacteria.org
To update a URL: change it here and it propagates everywhere. To audit what atb downloads: read this file.
Index ¶
Constants ¶
const ( SketchSkmFilename = "atb_sketchlib.skm" SketchSkdFilename = "atb_sketchlib.skd" SketchSubdir = "sketch" )
SketchSkmFilename and SketchSkdFilename are local filenames for the sketch database files, stored in <data-dir>/sketch/.
const AGCArchiveMapFilename = "assemblies_filelist.txt.gz"
AGCArchiveMapFilename is the local cache filename for the archive map. It keeps the .gz extension because the artifact is cached exactly as downloaded.
const AGCArchiveMapURL = "https://osf.io/download/gtqrx/"
AGCArchiveMapURL is the accession->batch list for the balanced v202505 collection, published on OSF as a gzipped two-column text file (assemblies_filelist.txt.gz: accession, batch name without .agc). atb caches the gzip as downloaded and decompresses on read.
const AGCArchiveSubdir = "agc"
AGCArchiveSubdir is the <data-dir> subdirectory holding cached archives and the archive map. Mirrors SketchSubdir.
const AGCArchivesFolder = "agc_batches"
AGCArchivesFolder is the folder holding .agc batches on the collection nodes.
const AGCBatchMetadataURL = "https://osf.io/download/8y9r2/"
AGCBatchMetadataURL is the OSF /download/ URL of the batch metadata TSV (8y9r2, batches_202505_metadata.tsv.gz): a gzipped TSV with batch_name and old_name columns. old_name carries the species; the numbered batch filename does not, so `atb agc index` joins this on batch_name to fill the species column of the combined index.
const AGCIndexFilename = "atb_agc_files.tsv"
AGCIndexFilename is the local cache filename for the AGC index TSV, stored in <data-dir>/agc/ alongside the cached archives.
const AGCIndexURL = "https://osf.io/download/6a719381a2e9e3d202b91f7d/"
AGCIndexURL is the OSF /download/ URL of the published combined AGC index TSV (crawl + metadata join, produced by `atb agc index`). When set, the runtime downloads this single published TSV; an empty value falls back to crawling the collection nodes and joining the metadata on demand.
const AGCRepo = "refresh-bio/agc"
AGCRepo is the GitHub repository for agc releases.
const AGCVersion = "v3.2.3"
AGCVersion is the pinned release tag of the agc binary. agc 3.x reads v1/v2/v3 archives, so this single pin covers every .agc file in existence today.
const AssemblyBaseURL = "https://allthebacteria-assemblies.s3.eu-west-2.amazonaws.com/"
AssemblyBaseURL is the S3 bucket hosting individual genome FASTA files. File pattern: {sample_accession}.fa.gz Used by: atb download, atb sketch query --download
const IndexFilename = "all_atb_files.tsv"
IndexFilename is the local cache filename for the OSF file index.
const IndexURL = "https://osf.io/download/r6gcp/"
IndexURL is the master TSV index listing ~3,000 files across the entire AllTheBacteria project on OSF, with download URLs and MD5 checksums. Used by: atb osf ls, atb osf download Source: https://osf.io/r6gcp (all_atb_files.tsv)
const OSFAPIBase = "https://api.osf.io/v2"
OSFAPIBase is the root of the OSF REST API (v2).
const SketchSkdURL = "https://osf.io/download/92qmr/"
SketchSkdURL is the sketch data file (.skd) for the aggregated ATB sketch database (~4.1 GB). Used by: atb sketch fetch
const SketchSkmURL = "https://osf.io/download/nwfkc/"
SketchSkmURL is the sketch metadata file (.skm) for the aggregated ATB sketch database covering all genomes up to Aug 2024 (~122 MB). Used by: atb sketch fetch
const SketchlibRepo = "bacpop/sketchlib.rust"
SketchlibRepo is the GitHub repository for sketchlib releases.
const SketchlibVersion = "v0.2.4"
SketchlibVersion is the pinned version of the sketchlib binary.
Variables ¶
var AGCCollectionNodes = []AGCNode{
{ID: "4jq8u"},
{ID: "jmeqg"},
{ID: "kzcnr"},
}
AGCCollectionNodes are the OSF nodes that together host the balanced v202505 collection; each has an agc_batches/ folder of numbered .agc batches.
var CoreTables = []string{
"assembly.parquet",
"assembly_stats.parquet",
"checkm2.parquet",
"sylph.parquet",
"run.parquet",
"mlst.parquet",
"amrfinderplus.parquet",
}
CoreTables lists the tables downloaded by default with `atb fetch`.
var TableURLs = map[string]string{
"assembly.parquet": "https://osf.io/download/4ku2n/",
"assembly_stats.parquet": "https://osf.io/download/69c51e86801fecc5d6146396/",
"checkm2.parquet": "https://osf.io/download/69c51e93cba7111bb21d27f2/",
"sylph.parquet": "https://osf.io/download/69c51f90cba7111bb21d2905/",
"run.parquet": "https://osf.io/download/69c51f68376eb79a651d2d85/",
"mlst.parquet": "https://osf.io/download/69c66d33fa3d973d94254f46/",
"amrfinderplus.parquet": "https://osf.io/download/69f1e5debb4f674d5fd949ad/",
"ena_20250506.parquet": "https://osf.io/download/69c51f3ab4f99c692d54cf73/",
"ena_20240801.parquet": "https://osf.io/download/69c51f002e72f67915145d0e/",
"ena_20240625.parquet": "https://osf.io/download/69c51ec99ce80b96ac54cd08/",
"ena_202505_used.parquet": "https://osf.io/download/69c51f475eedad376954ce7b/",
"ena_661k.parquet": "https://osf.io/download/69c51f57376eb79a651d2d83/",
}
TableURLs maps parquet table filenames to their OSF download URLs.
Functions ¶
func OSFNodeFilesURL ¶ added in v0.18.0
OSFNodeFilesURL returns the osfstorage root listing URL for an OSF node, the entry point for crawling its folders.