Documentation
¶
Index ¶
- Constants
- Variables
- func CollectionCacheSource(nodes []sources.AGCNode) string
- func SpeciesFromArchive(name string) string
- func WriteAGCIndexTSV(idx *Index, w io.Writer) error
- type Entry
- type Index
- func CrawlAGCCollection(rootURLFor func(nodeID string) string, nodes []sources.AGCNode) (*Index, error)
- func CrawlAGCIndex(rootURL, nodeID string) (*Index, error)
- func CrawlAGCNode(client *http.Client, startURL, nodeID string) (*Index, error)
- func FetchAGCCollection(cacheDir string, rootURLFor func(nodeID string) string, ...) (*Index, error)
- func FetchAGCIndex(cacheDir, rootURL, nodeID string, force bool) (*Index, error)
- func FetchAGCIndexFromURL(cacheDir, url string, force bool) (*Index, error)
- func FetchIndex(cacheDir string, force bool) (*Index, error)
- func ParseIndex(r io.Reader) (*Index, error)
- type ProjectSummary
Constants ¶
const CacheMaxAge = 7 * 24 * time.Hour
Variables ¶
var ErrFolderNotFound = errors.New("folder not found on OSF node")
ErrFolderNotFound is returned (wrapped) when a node listing has no folder of the requested name - distinct from a network or decode failure, so a still-provisioning collection node can be skipped without masking real errors.
Functions ¶
func CollectionCacheSource ¶ added in v0.18.0
CollectionCacheSource is the cache source-marker for a combined collection crawl over nodes. It is exposed so a warm cache can be recognised across the exact node set that produced it: changing the set changes the marker and invalidates the cache. Tests also use it to seed a warm cache offline.
func SpeciesFromArchive ¶
SpeciesFromArchive derives the species prefix from an AGC archive name by splitting on the literal "_global_ordered_" token. This keeps GTDB letter-suffixes ("Streptococcus_suis_AA") and the special "subthreshold_remainder" batch intact. The ".agc" extension is optional.
Types ¶
type Index ¶
type Index struct {
Entries []Entry
}
func CrawlAGCCollection ¶ added in v0.18.0
func CrawlAGCCollection(rootURLFor func(nodeID string) string, nodes []sources.AGCNode) (*Index, error)
CrawlAGCCollection crawls the AGCArchivesFolder of every node in nodes and concatenates the results into one Index. rootURLFor maps a node id to its osfstorage listing URL (sources.OSFNodeFilesURL in production; a test double otherwise). A node whose agc_archives/ folder does not exist yet is skipped - it is still provisioning - rather than failing the whole crawl; any other error (network, HTTP, decode) is returned so a real outage is not silently hidden. An existing but partially populated folder simply contributes fewer rows.
func CrawlAGCIndex ¶
CrawlAGCIndex resolves the agc_batches/ folder from an OSF node's root listing (rootURL, build it with sources.OSFNodeFilesURL) and crawls every page of that folder into an Index, with no caching side effect. Each entry's ProjectID is stamped with nodeID. This is the network half of FetchAGCIndex, exposed so `atb agc index` can crawl on demand and write the TSV wherever the user wants it.
func CrawlAGCNode ¶ added in v0.18.0
CrawlAGCNode walks an OSF folder listing starting at startURL, following the "next" link until exhausted, and returns an Index of every .agc file found. Each entry's ProjectID is stamped with nodeID. A nil client gets a default.
func FetchAGCCollection ¶ added in v0.18.0
func FetchAGCCollection(cacheDir string, rootURLFor func(nodeID string) string, nodes []sources.AGCNode, force bool) (*Index, error)
FetchAGCCollection returns the combined batch index across the collection nodes, caching the merged TSV under <cacheDir>/atb_agc_files.tsv with a source marker derived from the node set (CollectionCacheSource). A cached copy younger than CacheMaxAge whose marker still matches is reused; otherwise the nodes are crawled and the result cached atomically. rootURLFor is sources.OSFNodeFilesURL in production. Set force=true to bypass a fresh cache.
func FetchAGCIndex ¶
FetchAGCIndex returns the AGC batch index for an OSF node, mirroring FetchIndex: a cached TSV is reused while younger than CacheMaxAge and while its source marker still matches rootURL, otherwise the node's agc_batches/ folder is crawled and the result written atomically to <cacheDir>/atb_agc_files.tsv (alongside a refreshed marker). rootURL is the node's osfstorage listing (build it with sources.OSFNodeFilesURL); it is a parameter so the crawl is testable against a local server. nodeID is stamped onto every entry's ProjectID. Set force=true to bypass a fresh cache.
func FetchAGCIndexFromURL ¶ added in v0.18.0
FetchAGCIndexFromURL returns the AGC batch index by downloading a pre-built TSV from url, mirroring FetchIndex: a cached copy under <cacheDir>/atb_agc_files.tsv is reused while younger than CacheMaxAge and while its source marker still matches url, otherwise the file is downloaded and written atomically (alongside a refreshed marker) before parsing. This is the hosted counterpart to FetchAGCIndex's live crawl — once the index has been published as a single OSF file (sources.AGCIndexURL) there is no need to walk the node's agc_batches/ folder page by page. Set force=true to bypass a fresh cache.
func FetchIndex ¶
FetchIndex returns a parsed index, using a cached copy if fresh enough. Set force=true to always re-download.
func ParseIndex ¶
ParseIndex reads the TSV index from r. Expected columns: project, project_id, filename, url, md5, size(MB)
func (*Index) Filter ¶
Filter returns entries matching the given criteria. If project is non-empty, only entries whose Project starts with that prefix are included. If pattern is non-empty, it is compiled as a regex and matched against "project/filename".
func (*Index) MatchProject ¶
MatchProject returns entries whose project matches a case-insensitive substring.
func (*Index) Projects ¶
func (idx *Index) Projects() []ProjectSummary
Projects returns a summary of each unique project with file count and total size.
Source Files