fs

package
v0.68.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 4, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package fs ships the native file-system action fs.download, the fs.walk stream source, and fs.filter, fs.read, fs.sort, fs.write, and the terminal sinks out.stdout and out.file as stream operators.

The pipeline carries fs.FileMeta between stages; fs.read replaces it with fs.FileContent (metadata + bytes) so downstream stages can inspect and transform file bodies.

Typical use: 

fs.walk:ext="go" -> fs.filter:tests=true -> fs.read -> fs.sort:by="size" -> collect

Index

Constants

View Source 
const ID = "fs"

Variables

View Source 
var BinaryExtensions = map[string]bool{
	".exe": true, ".dll": true, ".so": true, ".dylib": true,
	".bin": true, ".obj": true, ".o": true, ".a": true,
	".png": true, ".jpg": true, ".jpeg": true, ".gif": true,
	".ico": true, ".webp": true, ".pdf": true,
	".zip": true, ".tar": true, ".gz": true, ".7z": true,
	".db": true, ".sqlite": true, ".sqlite3": true,
	".pyc": true, ".pyo": true, ".wasm": true,
}
View Source 
var DefaultSkipDirs = []string{
	".git", "node_modules", "vendor", "target", "dist", "build",
	"_srcpack", ".next", ".turbo", ".cache", ".idea", ".vscode",
}
View Source 
var Download = action.New[DownloadRequest, DownloadResult]("fs.download", downloadHTTP).
	Description("Stream HTTP response bodies to safe relative file paths with byte progress and resumable Range requests").
	Tag("fs", "network").
	Build()

Download streams HTTP bodies to disk, reporting real byte progress through Kernel's supported xctx.ReportProgress API. It accepts either url/path fields for one file or a files array for a generic manifest.

Functions

func Bundle 

func Bundle(_ map[string]string) compiler.Bundle

func Filter 

func Filter(cfg FilterConfig) action.StreamOp[FileMeta, FileMeta]

Filter returns a stream operator that drops files matching any of the configured exclusion rules.

func FilterOperator 

func FilterOperator() action.NamedOperator

func GenerateStatsSummary 

func GenerateStatsSummary(entries []StatEntry, maxDepth int) string

func Library 

func Library() action.Library

func OutFile 

func OutFile[T any](cfg OutFileConfig) action.StreamOp[T, T]

OutFile aggregates the whole stream into a single file. Content items (FileContent) are written byte-for-byte; everything else uses fmt.Sprint or Stringer.

func OutFileOperator 

func OutFileOperator() action.NamedOperator

func Read 

func Read(cfg ReadConfig) action.StreamOp[FileMeta, FileContent]

Read returns a stream operator that loads each file's bytes (or a selected subset of lines) into FileContent.

func ReadOperator 

func ReadOperator() action.NamedOperator

func Sort 

func Sort(cfg SortConfig) action.StreamOp[FileMeta, FileMeta]

Sort buffers the stream and yields items in sorted order. Sorting is stable within equal keys.

func SortOperator 

func SortOperator() action.NamedOperator

func Stdout 

func Stdout(up iter.Seq2[FileMeta, error]) iter.Seq2[FileMeta, error]

Stdout prints each file's RelPath to stdout and passes the item through. It is the file-listing sink for `fs.walk -> out.stdout`.

func StdoutOperator 

func StdoutOperator() action.NamedOperator

func WalkSource 

func WalkSource() action.AnyStreamAction

func Write 

func Write(cfg WriteConfig) action.StreamOp[FileContent, FileContent]

Write persists each FileContent under Dir, preserving RelPath. It is the per-item counterpart to out.file, which aggregates the whole stream into one file.

RelPath is validated through xfs.Rel and all file operations use xfs.Root so symlinks cannot escape Dir.

func WriteOperator 

func WriteOperator() action.NamedOperator

Types

type DownloadFile added in v0.67.2 

type DownloadFile struct {
	URL           string            `json:"url"`
	Path          string            `json:"path"`
	ExpectedBytes int64             `json:"expected_bytes,omitempty"`
	Headers       map[string]string `json:"headers,omitempty"`
}

DownloadFile is one HTTP resource in a generic download manifest. Path is a slash-separated relative path below DownloadRequest.OutputDir.

type DownloadFileResult added in v0.67.2 

type DownloadFileResult struct {
	Path             string `json:"path"`
	Bytes            int64  `json:"bytes"`
	TransferredBytes int64  `json:"transferred_bytes"`
	Resumed          bool   `json:"resumed"`
	Skipped          bool   `json:"skipped,omitempty"`
}

DownloadFileResult describes the verified final file. TransferredBytes is only bytes received during this invocation; Bytes is the complete file size.

type DownloadRequest added in v0.67.2 

type DownloadRequest struct {
	URL           string            `json:"url,omitempty"`
	Path          string            `json:"path,omitempty"`
	ExpectedBytes int64             `json:"expected_bytes,omitempty"`
	Headers       map[string]string `json:"headers,omitempty"`
	OutputDir     string            `json:"output_dir,omitempty"`
	Files         []DownloadFile    `json:"files,omitempty"`
	ReceiptPath   string            `json:"receipt_path,omitempty"`
	Provenance    map[string]any    `json:"provenance,omitempty"`
}

DownloadRequest downloads one URL or a manifest of URLs without buffering response bodies in memory. A .part file is retained on transfer failure and is resumed with HTTP Range on the next run.

type DownloadResult added in v0.67.2 

type DownloadResult struct {
	Files            []DownloadFileResult `json:"files"`
	TotalBytes       int64                `json:"total_bytes"`
	TransferredBytes int64                `json:"transferred_bytes"`
	ReceiptPath      string               `json:"receipt_path,omitempty"`
}

DownloadResult summarizes one action invocation and is also the payload stored in the optional receipt (with caller-supplied provenance).

type FileContent 

type FileContent struct {
	FileMeta
	Content []byte
}

FileContent is FileMeta plus the file body. fs.read replaces the stream item type from FileMeta to FileContent.

type FileMeta 

type FileMeta struct {
	Path    string    `json:"path"`
	RelPath string    `json:"rel_path"`
	Size    int64     `json:"size"`
	ModTime time.Time `json:"mod_time"`
	Lang    string    `json:"lang"`
}

FileMeta is the item type that flows through the fs pipeline.

type FilterConfig 

type FilterConfig struct {
	Ext       string `json:"ext"       cli:"ext"`
	Tests     bool   `json:"tests"`
	Generated bool   `json:"generated"`
	Deps      bool   `json:"deps"`
	Examples  bool   `json:"examples"`
	Exclude   string `json:"exclude"`
}

FilterConfig controls fs.filter. All fields are optional; the zero value passes every file through.

type OutFileConfig 

type OutFileConfig struct {
	Path      string `json:"path"       cli:"o,output"`
	Append    bool   `json:"append"     cli:"append"`
	FlushEach bool   `json:"flush_each"`
}

OutFileConfig controls out.file.

type ReadConfig 

type ReadConfig struct {
	// Lines restricts the content to line ranges. Accepted forms:
	//
	//	"10"       single line
	//	"10-20"    inclusive range
	//	"10-"      from line 10 to end
	//	"1,3,5-7"  union of ranges
	//
	// Empty means the whole file.
	Lines string `json:"lines" cli:"lines"`
}

ReadConfig controls fs.read.

type SortConfig 

type SortConfig struct {
	// By is one of "rel_path" (default), "size", "mod_time".
	By string `json:"by" cli:"by"`
	// MaxItems bounds the buffer. Zero uses a safe default of 100000.
	MaxItems int `json:"max_items" cli:"max_items"`
}

SortConfig controls fs.sort.

type StatEntry 

type StatEntry struct {
	RelPath   string
	Size      int64
	FuncCount int
}

type WalkConfig 

type WalkConfig struct {
	Dir           string   `json:"dir"            cli:"dir,d"`
	Dirs          []string `json:"dirs"           cli:"dirs"`
	Files         []string `json:"files"          cli:"files,file,f"`
	Skip          []string `json:"skip"           cli:"skip"`
	IncludeHidden bool     `json:"include_hidden" cli:"include_hidden"`
	IncludeAll    bool     `json:"include_all"    cli:"include_all"`
	Ext           string   `json:"ext"            cli:"ext"`
	MaxDepth      int      `json:"max_depth"      cli:"max_depth"`
}

type WriteConfig 

type WriteConfig struct {
	Dir       string `json:"dir"       cli:"dir,d"`
	Mode      uint32 `json:"mode"      cli:"mode"`
	Overwrite bool   `json:"overwrite" cli:"overwrite"`
}

WriteConfig controls fs.write.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.