Documentation
¶
Index ¶
- Constants
- Variables
- func CollectionCacheSource(nodes []sources.AGCNode) string
- func FetchBatchMetadata(url string) (map[string]string, error)
- func FirstDuplicateBatch(idx *Index) (string, bool)
- func ParseBatchMetadata(r io.Reader) (map[string]string, error)
- func SpeciesFromOldName(oldName string) string
- func WriteAGCIndexTSV(idx *Index, w io.Writer) error
- type Entry
- type Index
- func BuildAGCCollectionIndex(rootURLFor func(nodeID string) string, nodes []sources.AGCNode, ...) (idx *Index, unmatched []string, err error)
- func CrawlAGCCollection(rootURLFor func(nodeID string) string, nodes []sources.AGCNode) (*Index, error)
- func CrawlAGCNode(client *http.Client, startURL, nodeID string) (*Index, error)
- func FetchAGCCollection(cacheDir string, rootURLFor func(nodeID string) string, ...) (*Index, error)
- func FetchAGCIndexFromURL(cacheDir, url string, force bool) (*Index, error)
- func FetchIndex(cacheDir string, force bool) (*Index, error)
- func ParseIndex(r io.Reader) (*Index, error)
- type ProjectSummary
Constants ¶
const CacheMaxAge = 7 * 24 * time.Hour
Variables ¶
var ErrFolderNotFound = errors.New("folder not found on OSF node")
ErrFolderNotFound is returned (wrapped) when a node listing has no folder of the requested name - distinct from a network or decode failure, so a still-provisioning collection node can be skipped without masking real errors.
Functions ¶
func CollectionCacheSource ¶ added in v0.18.0
CollectionCacheSource is the cache source-marker for a combined collection crawl over nodes. It is exposed so a warm cache can be recognised across the exact node set that produced it: changing the set changes the marker and invalidates the cache. Tests also use it to seed a warm cache offline.
func FetchBatchMetadata ¶ added in v0.18.0
FetchBatchMetadata downloads the gzipped batch metadata TSV from url and parses it into a batch_name -> old_name map. The file is small (~20 KB) and read fully into memory, gunzipped in-process.
func FirstDuplicateBatch ¶ added in v0.18.0
FirstDuplicateBatch returns the first (project_id, filename) pair that appears more than once in idx, or "", false when every batch is distinct. A duplicate is never legitimate: it means the crawl saw the same batch twice, so the publish path uses this to fail closed rather than ship a silently corrupt index.
func ParseBatchMetadata ¶ added in v0.18.0
ParseBatchMetadata reads the batch metadata TSV and returns a map from batch stem to old_name, the two columns the species join needs. The header names the columns, so column order is not assumed; a trailing ".agc" is stripped from batch_name so the key joins the crawled ".agc" filenames whether or not the metadata carries the extension. Rows missing either field are skipped.
func SpeciesFromOldName ¶ added in v0.18.0
SpeciesFromOldName derives the species from a batch's metadata old_name. Three forms occur: "<Species>_global_ordered.partNNN", "unknown.partNNN", and "mixed_species.partNNN" (each optionally ending ".agc"). One rule covers all three: strip a trailing ".agc", cut at the first ".part", then trim a trailing "_global_ordered" (so GTDB letter-suffixed species like "Streptococcus_suis_AA" survive). The result is non-empty for every published batch.
Types ¶
type Index ¶
type Index struct {
Entries []Entry
}
func BuildAGCCollectionIndex ¶ added in v0.18.0
func BuildAGCCollectionIndex(rootURLFor func(nodeID string) string, nodes []sources.AGCNode, metadataURL string) (idx *Index, unmatched []string, err error)
BuildAGCCollectionIndex crawls the collection nodes and joins the batch metadata on the batch stem so every entry carries its species in Project. The numbered batch filename does not encode a species, so the join is the only source of it. Batches with no metadata match keep an empty Project and are returned, sorted, in unmatched: `atb agc index` fails closed on any unmatched batch, while the runtime fallback tolerates them. rootURLFor and metadataURL are parameters so the join is testable offline.
func CrawlAGCCollection ¶ added in v0.18.0
func CrawlAGCCollection(rootURLFor func(nodeID string) string, nodes []sources.AGCNode) (*Index, error)
CrawlAGCCollection crawls the AGCArchivesFolder of every node in nodes and concatenates the results into one Index. rootURLFor maps a node id to its osfstorage listing URL (sources.OSFNodeFilesURL in production; a test double otherwise). A node whose agc_batches/ folder does not exist yet is skipped - it is still provisioning - rather than failing the whole crawl; any other error (network, HTTP, decode) is returned so a real outage is not silently hidden. An existing but partially populated folder simply contributes fewer rows.
func CrawlAGCNode ¶ added in v0.18.0
CrawlAGCNode walks an OSF folder listing starting at startURL, following the "next" link until exhausted, and returns an Index of every .agc file found. Each entry's ProjectID is stamped with nodeID. A nil client gets a default.
func FetchAGCCollection ¶ added in v0.18.0
func FetchAGCCollection(cacheDir string, rootURLFor func(nodeID string) string, nodes []sources.AGCNode, metadataURL string, force bool) (*Index, error)
FetchAGCCollection returns the combined batch index across the collection nodes, joined against the batch metadata for the species column, caching the merged TSV under <cacheDir>/atb_agc_files.tsv with a source marker derived from the node set (CollectionCacheSource). A cached copy younger than CacheMaxAge whose marker still matches is reused; otherwise the nodes are crawled, joined, and cached atomically. This runtime fallback is tolerant of a batch the metadata does not yet cover (its species stays empty). rootURLFor and metadataURL are sources.OSFNodeFilesURL and sources.AGCBatchMetadataURL in production. Set force=true to bypass a fresh cache.
func FetchAGCIndexFromURL ¶ added in v0.18.0
FetchAGCIndexFromURL returns the AGC batch index by downloading a pre-built TSV from url, mirroring FetchIndex: a cached copy under <cacheDir>/atb_agc_files.tsv is reused while younger than CacheMaxAge and while its source marker still matches url, otherwise the file is downloaded and written atomically (alongside a refreshed marker) before parsing. It reads the index that has been published as a single OSF file (sources.AGCIndexURL), so there is no need to crawl the collection nodes page by page. Set force=true to bypass a fresh cache.
func FetchIndex ¶
FetchIndex returns a parsed index, using a cached copy if fresh enough. Set force=true to always re-download.
func ParseIndex ¶
ParseIndex reads the TSV index from r. Expected columns: project, project_id, filename, url, md5, size(MB)
func (*Index) Filter ¶
Filter returns entries matching the given criteria. If project is non-empty, only entries whose Project starts with that prefix are included. If pattern is non-empty, it is compiled as a regex and matched against "project/filename".
func (*Index) MatchProject ¶
MatchProject returns entries whose project matches a case-insensitive substring.
func (*Index) Projects ¶
func (idx *Index) Projects() []ProjectSummary
Projects returns a summary of each unique project with file count and total size.
Source Files