Documentation
¶
Index ¶
- Constants
- func AnalyzeTextContent(data []byte) (string, error)
- func DetectByContent(data []byte) (string, error)
- func DetectByExtension(filename string) string
- func DetectFileType(data []byte, filename string) (string, error)
- func LooksLikeCSV(data []byte) bool
- func ProcessCSV(ctx context.Context, uploadID, source string, reader io.Reader, ...) (int, error)
- func ProcessJSON(ctx context.Context, uploadID, source string, reader io.Reader, ...) (int, error)
- func UploadExternalData(ctx context.Context, source string, reader io.Reader, filename string, ...) (string, int, error)
- type StoreFunc
Constants ¶
const ( DefaultBatchSize = 500 DefaultWorkerCount = 4 DefaultBufferSize = 64 * 1024 // 64KB buffer DefaultChannelSize = 1000 ContextCheckInterval = 100 )
Configuration constants for performance tuning
Variables ¶
This section is empty.
Functions ¶
func AnalyzeTextContent ¶
func DetectByContent ¶
func DetectByExtension ¶
func LooksLikeCSV ¶
func ProcessCSV ¶
func ProcessCSV(ctx context.Context, uploadID, source string, reader io.Reader, store StoreFunc) (int, error)
ProcessCSV now uses parallel processing with workers
func ProcessJSON ¶
func ProcessJSON(ctx context.Context, uploadID, source string, reader io.Reader, store StoreFunc) (int, error)
ProcessJSON streams a JSON array of external transactions, decoding and storing one element at a time so memory use stays constant regardless of file size (the previous implementation decoded the entire array into a slice first, an unbounded allocation driven by the uploaded file).
func UploadExternalData ¶
func UploadExternalData(ctx context.Context, source string, reader io.Reader, filename string, store StoreFunc) (string, int, error)
UploadExternalData handles the process of uploading external data by detecting file type, parsing, and storing it. Parameters: - ctx: The context for controlling execution. - source: The source of the external data. - reader: An io.Reader for reading the data. - filename: The name of the file being uploaded. - store: The callback function to store processed transactions. Returns: - string: The ID of the upload. - int: The total number of records processed. - error: If any step of the process fails.
Source Files