sources

package
v0.18.0-beta.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 10, 2026 License: MIT Imports: 0 Imported by: 0

Documentation

Overview

Package sources is the single source of truth for all external data URLs that atb downloads. Every URL the tool fetches from the internet is defined here, grouped by feature.

OSF project: https://osf.io/h7wzy/files/osfstorage AllTheBacteria: https://allthebacteria.org

To update a URL: change it here and it propagates everywhere. To audit what atb downloads: read this file.

Index

Constants

View Source
const (
	SketchSkmFilename = "atb_sketchlib.skm"
	SketchSkdFilename = "atb_sketchlib.skd"
	SketchSubdir      = "sketch"
)

SketchSkmFilename and SketchSkdFilename are local filenames for the sketch database files, stored in <data-dir>/sketch/.

View Source
const AGCArchiveBaseURL = "https://data-access.cesgo.org/index.php/s/w8ylLnVjCokP0B5/download?path=%2F&files="

AGCArchiveBaseURL is the prefix for downloading a single .agc archive; ArchiveURL appends "<archive>.agc".

View Source
const AGCArchiveMapFilename = "agc_file_list.txt.zip"

AGCArchiveMapFilename is the local cache filename for the archive map. It keeps the .zip extension because the artifact is cached exactly as downloaded.

View Source
const AGCArchiveMapURL = "https://osf.io/download/2xjt8/"

AGCArchiveMapURL is the accession->batch list for the full v202505 collection, published on OSF as a ZIP of a ~148 MB two-column text file (accession, batch name without .agc). atb caches the ZIP as downloaded and decompresses on read.

View Source
const AGCArchiveSubdir = "agc"

AGCArchiveSubdir is the <data-dir> subdirectory holding cached archives and the archive map. Mirrors SketchSubdir.

View Source
const AGCArchivesFolder = "agc_archives"

AGCArchivesFolder is the folder holding .agc batches on the collection nodes. The legacy z7q5y test node uses AGCBatchesFolder instead.

View Source
const AGCBatchesFolder = "agc_batches"

AGCBatchesFolder is the folder on the node holding the .agc batch files.

View Source
const AGCIndexFilename = "atb_agc_files.tsv"

AGCIndexFilename is the local cache filename for the AGC index TSV, stored in <data-dir>/agc/ alongside the cached archives.

View Source
const AGCIndexURL = "https://osf.io/download/6a477a94899134067adf99c9/"

AGCIndexURL is the OSF /download/<guid>/ URL of the published AGC index TSV, hosted on the main ATB node h7wzy alongside the master index. atb downloads this single file (like the master IndexURL) instead of crawling the z7q5y agc_batches/ folder page by page. Set it back to "" to fall back to the live crawl (see useHostedAGCIndex). The batch rows inside still point at z7q5y.

View Source
const AGCRepo = "refresh-bio/agc"

AGCRepo is the GitHub repository for agc releases.

View Source
const AGCTestNodeID = "z7q5y"

AGCTestNodeID is the OSF node hosting the AGC test batches ("ATB testing").

View Source
const AGCVersion = "v3.2.3"

AGCVersion is the pinned release tag of the agc binary. agc 3.x reads v1/v2/v3 archives, so this single pin covers every .agc file in existence today.

View Source
const AssemblyBaseURL = "https://allthebacteria-assemblies.s3.eu-west-2.amazonaws.com/"

AssemblyBaseURL is the S3 bucket hosting individual genome FASTA files. File pattern: {sample_accession}.fa.gz Used by: atb download, atb sketch query --download

View Source
const IndexFilename = "all_atb_files.tsv"

IndexFilename is the local cache filename for the OSF file index.

View Source
const IndexURL = "https://osf.io/download/r6gcp/"

IndexURL is the master TSV index listing ~3,000 files across the entire AllTheBacteria project on OSF, with download URLs and MD5 checksums. Used by: atb osf ls, atb osf download Source: https://osf.io/r6gcp (all_atb_files.tsv)

View Source
const OSFAPIBase = "https://api.osf.io/v2"

OSFAPIBase is the root of the OSF REST API (v2).

View Source
const SketchSkdURL = "https://osf.io/download/92qmr/"

SketchSkdURL is the sketch data file (.skd) for the aggregated ATB sketch database (~4.1 GB). Used by: atb sketch fetch

View Source
const SketchSkmURL = "https://osf.io/download/nwfkc/"

SketchSkmURL is the sketch metadata file (.skm) for the aggregated ATB sketch database covering all genomes up to Aug 2024 (~122 MB). Used by: atb sketch fetch

View Source
const SketchlibRepo = "bacpop/sketchlib.rust"

SketchlibRepo is the GitHub repository for sketchlib releases.

View Source
const SketchlibVersion = "v0.2.4"

SketchlibVersion is the pinned version of the sketchlib binary.

Variables

View Source
var AGCCollectionNodes = []AGCNode{
	{ID: "6g8by", Part: "major"},
	{ID: "9fqeh", Part: "unknown"},
	{ID: "xrzub", Part: "dustbin"},
}

AGCCollectionNodes are the OSF nodes that together host the full collection.

View Source
var CoreTables = []string{
	"assembly.parquet",
	"assembly_stats.parquet",
	"checkm2.parquet",
	"sylph.parquet",
	"run.parquet",
	"mlst.parquet",
	"amrfinderplus.parquet",
}

CoreTables lists the tables downloaded by default with `atb fetch`.

View Source
var TableURLs = map[string]string{

	"assembly.parquet":       "https://osf.io/download/4ku2n/",
	"assembly_stats.parquet": "https://osf.io/download/69c51e86801fecc5d6146396/",
	"checkm2.parquet":        "https://osf.io/download/69c51e93cba7111bb21d27f2/",
	"sylph.parquet":          "https://osf.io/download/69c51f90cba7111bb21d2905/",
	"run.parquet":            "https://osf.io/download/69c51f68376eb79a651d2d85/",
	"mlst.parquet":           "https://osf.io/download/69c66d33fa3d973d94254f46/",
	"amrfinderplus.parquet":  "https://osf.io/download/69f1e5debb4f674d5fd949ad/",

	"ena_20250506.parquet":    "https://osf.io/download/69c51f3ab4f99c692d54cf73/",
	"ena_20240801.parquet":    "https://osf.io/download/69c51f002e72f67915145d0e/",
	"ena_20240625.parquet":    "https://osf.io/download/69c51ec99ce80b96ac54cd08/",
	"ena_202505_used.parquet": "https://osf.io/download/69c51f475eedad376954ce7b/",
	"ena_661k.parquet":        "https://osf.io/download/69c51f57376eb79a651d2d83/",
}

TableURLs maps parquet table filenames to their OSF download URLs.

Functions

func ArchiveURL

func ArchiveURL(archive string) string

ArchiveURL returns the download URL for the named archive (without ".agc").

func OSFNodeFilesURL added in v0.18.0

func OSFNodeFilesURL(nodeID string) string

OSFNodeFilesURL returns the osfstorage root listing URL for an OSF node, the entry point for crawling its folders.

func PartForNode

func PartForNode(nodeID string) string

PartForNode returns the collection-part label for a node id, or "" when the id is not a collection node (e.g. the legacy z7q5y test node).

Types

type AGCNode added in v0.18.0

type AGCNode struct {
	ID   string
	Part string
}

AGCNode identifies one OSF node in the collection together with the human-facing "part" label shown in `atb agc locate` output.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL