Documentation
¶
Overview ¶
Package sources is the single source of truth for all external data URLs that atb downloads. Every URL the tool fetches from the internet is defined here, grouped by feature.
OSF project: https://osf.io/h7wzy/files/osfstorage AllTheBacteria: https://allthebacteria.org
To update a URL: change it here and it propagates everywhere. To audit what atb downloads: read this file.
Index ¶
Constants ¶
const ( SketchSkmFilename = "atb_sketchlib.skm" SketchSkdFilename = "atb_sketchlib.skd" SketchSubdir = "sketch" )
SketchSkmFilename and SketchSkdFilename are local filenames for the sketch database files, stored in <data-dir>/sketch/.
const AGCArchiveBaseURL = "https://data-access.cesgo.org/index.php/s/w8ylLnVjCokP0B5/download?path=%2F&files="
AGCArchiveBaseURL is the prefix for downloading a single .agc archive; ArchiveURL appends "<archive>.agc".
const AGCArchiveMapFilename = "agc_file_list.txt"
AGCArchiveMapFilename is the local cache filename for the archive map.
const AGCArchiveMapURL = "https://data-access.cesgo.org/index.php/s/9WuPWJL25cYYZml/download"
AGCArchiveMapURL is the whitespace-delimited sample->archive map (~90 MB): column 1 is the sample accession, column 2 is the archive name (no .agc).
const AGCArchiveSubdir = "agc"
AGCArchiveSubdir is the <data-dir> subdirectory holding cached archives and the archive map. Mirrors SketchSubdir.
const AGCBatchesFolder = "agc_batches"
AGCBatchesFolder is the folder on the node holding the .agc batch files.
const AGCIndexFilename = "atb_agc_files.tsv"
AGCIndexFilename is the local cache filename for the AGC index TSV, stored in <data-dir>/agc/ alongside the cached archives.
const AGCIndexURL = "https://osf.io/download/6a477a94899134067adf99c9/"
AGCIndexURL is the OSF /download/<guid>/ URL of the published AGC index TSV, hosted on the main ATB node h7wzy alongside the master index. atb downloads this single file (like the master IndexURL) instead of crawling the z7q5y agc_batches/ folder page by page. Set it back to "" to fall back to the live crawl (see useHostedAGCIndex). The batch rows inside still point at z7q5y.
const AGCRepo = "refresh-bio/agc"
AGCRepo is the GitHub repository for agc releases.
const AGCTestNodeID = "z7q5y"
AGCTestNodeID is the OSF node hosting the AGC test batches ("ATB testing").
const AGCVersion = "v3.2.3"
AGCVersion is the pinned release tag of the agc binary. agc 3.x reads v1/v2/v3 archives, so this single pin covers every .agc file in existence today.
const AssemblyBaseURL = "https://allthebacteria-assemblies.s3.eu-west-2.amazonaws.com/"
AssemblyBaseURL is the S3 bucket hosting individual genome FASTA files. File pattern: {sample_accession}.fa.gz Used by: atb download, atb sketch query --download
const IndexFilename = "all_atb_files.tsv"
IndexFilename is the local cache filename for the OSF file index.
const IndexURL = "https://osf.io/download/r6gcp/"
IndexURL is the master TSV index listing ~3,000 files across the entire AllTheBacteria project on OSF, with download URLs and MD5 checksums. Used by: atb osf ls, atb osf download Source: https://osf.io/r6gcp (all_atb_files.tsv)
const OSFAPIBase = "https://api.osf.io/v2"
OSFAPIBase is the root of the OSF REST API (v2).
const SketchSkdURL = "https://osf.io/download/92qmr/"
SketchSkdURL is the sketch data file (.skd) for the aggregated ATB sketch database (~4.1 GB). Used by: atb sketch fetch
const SketchSkmURL = "https://osf.io/download/nwfkc/"
SketchSkmURL is the sketch metadata file (.skm) for the aggregated ATB sketch database covering all genomes up to Aug 2024 (~122 MB). Used by: atb sketch fetch
const SketchlibRepo = "bacpop/sketchlib.rust"
SketchlibRepo is the GitHub repository for sketchlib releases.
const SketchlibVersion = "v0.2.4"
SketchlibVersion is the pinned version of the sketchlib binary.
Variables ¶
var CoreTables = []string{
"assembly.parquet",
"assembly_stats.parquet",
"checkm2.parquet",
"sylph.parquet",
"run.parquet",
"mlst.parquet",
"amrfinderplus.parquet",
}
CoreTables lists the tables downloaded by default with `atb fetch`.
var TableURLs = map[string]string{
"assembly.parquet": "https://osf.io/download/4ku2n/",
"assembly_stats.parquet": "https://osf.io/download/69c51e86801fecc5d6146396/",
"checkm2.parquet": "https://osf.io/download/69c51e93cba7111bb21d27f2/",
"sylph.parquet": "https://osf.io/download/69c51f90cba7111bb21d2905/",
"run.parquet": "https://osf.io/download/69c51f68376eb79a651d2d85/",
"mlst.parquet": "https://osf.io/download/69c66d33fa3d973d94254f46/",
"amrfinderplus.parquet": "https://osf.io/download/69f1e5debb4f674d5fd949ad/",
"ena_20250506.parquet": "https://osf.io/download/69c51f3ab4f99c692d54cf73/",
"ena_20240801.parquet": "https://osf.io/download/69c51f002e72f67915145d0e/",
"ena_20240625.parquet": "https://osf.io/download/69c51ec99ce80b96ac54cd08/",
"ena_202505_used.parquet": "https://osf.io/download/69c51f475eedad376954ce7b/",
"ena_661k.parquet": "https://osf.io/download/69c51f57376eb79a651d2d83/",
}
TableURLs maps parquet table filenames to their OSF download URLs.
Functions ¶
func ArchiveURL ¶
ArchiveURL returns the download URL for the named archive (without ".agc").
func OSFNodeFilesURL ¶ added in v0.18.0
OSFNodeFilesURL returns the osfstorage root listing URL for an OSF node, the entry point for crawling its folders.
Types ¶
This section is empty.