Documentation
¶
Index ¶
- Constants
- func SpeciesFromArchive(name string) string
- func WriteAGCIndexTSV(idx *Index, w io.Writer) error
- type Entry
- type Index
- func CrawlAGCIndex(rootURL, nodeID string) (*Index, error)
- func CrawlAGCNode(client *http.Client, startURL, nodeID string) (*Index, error)
- func FetchAGCIndex(cacheDir, rootURL, nodeID string, force bool) (*Index, error)
- func FetchAGCIndexFromURL(cacheDir, url string, force bool) (*Index, error)
- func FetchIndex(cacheDir string, force bool) (*Index, error)
- func ParseIndex(r io.Reader) (*Index, error)
- type ProjectSummary
Constants ¶
const CacheMaxAge = 7 * 24 * time.Hour
Variables ¶
This section is empty.
Functions ¶
func SpeciesFromArchive ¶
SpeciesFromArchive derives the species prefix from an AGC archive name by splitting on the literal "_global_ordered_" token. This keeps GTDB letter-suffixes ("Streptococcus_suis_AA") and the special "subthreshold_remainder" batch intact. The ".agc" extension is optional.
Types ¶
type Index ¶
type Index struct {
Entries []Entry
}
func CrawlAGCIndex ¶
CrawlAGCIndex resolves the agc_batches/ folder from an OSF node's root listing (rootURL, build it with sources.OSFNodeFilesURL) and crawls every page of that folder into an Index, with no caching side effect. Each entry's ProjectID is stamped with nodeID. This is the network half of FetchAGCIndex, exposed so `atb agc index` can crawl on demand and write the TSV wherever the user wants it.
func CrawlAGCNode ¶ added in v0.18.0
CrawlAGCNode walks an OSF folder listing starting at startURL, following the "next" link until exhausted, and returns an Index of every .agc file found. Each entry's ProjectID is stamped with nodeID. A nil client gets a default.
func FetchAGCIndex ¶
FetchAGCIndex returns the AGC batch index for an OSF node, mirroring FetchIndex: a cached TSV is reused while younger than CacheMaxAge and while its source marker still matches rootURL, otherwise the node's agc_batches/ folder is crawled and the result written atomically to <cacheDir>/atb_agc_files.tsv (alongside a refreshed marker). rootURL is the node's osfstorage listing (build it with sources.OSFNodeFilesURL); it is a parameter so the crawl is testable against a local server. nodeID is stamped onto every entry's ProjectID. Set force=true to bypass a fresh cache.
func FetchAGCIndexFromURL ¶ added in v0.18.0
FetchAGCIndexFromURL returns the AGC batch index by downloading a pre-built TSV from url, mirroring FetchIndex: a cached copy under <cacheDir>/atb_agc_files.tsv is reused while younger than CacheMaxAge and while its source marker still matches url, otherwise the file is downloaded and written atomically (alongside a refreshed marker) before parsing. This is the hosted counterpart to FetchAGCIndex's live crawl — once the index has been published as a single OSF file (sources.AGCIndexURL) there is no need to walk the node's agc_batches/ folder page by page. Set force=true to bypass a fresh cache.
func FetchIndex ¶
FetchIndex returns a parsed index, using a cached copy if fresh enough. Set force=true to always re-download.
func ParseIndex ¶
ParseIndex reads the TSV index from r. Expected columns: project, project_id, filename, url, md5, size(MB)
func (*Index) Filter ¶
Filter returns entries matching the given criteria. If project is non-empty, only entries whose Project starts with that prefix are included. If pattern is non-empty, it is compiled as a regex and matched against "project/filename".
func (*Index) MatchProject ¶
MatchProject returns entries whose project matches a case-insensitive substring.
func (*Index) Projects ¶
func (idx *Index) Projects() []ProjectSummary
Projects returns a summary of each unique project with file count and total size.
Source Files