README
¶
Mongoimport
CLI and go library for importing data from CSV, JSON or XML files into MongoDB.
go run github.com/romnn/mongoimport/cmd/mongoimport --db-user=root --db-password=example csv <path-to-csv-files>
You can also download pre built binaries from the releases page.
For a list of options, run
go run github.com/romnn/mongoimport/cmd/mongoimport csv --help
Usage as a library
Using the tool as a standalone CLI tool is great for quick loading of a few files. However, you might need more fine-grained control over what files are imported into which collection or perform additional pre/post processing (e.g. parsing timestamps). For this use case, we offer a very extensivle and modular API for configuring your imports.
import "github.com/romnn/mongoimport"
// example t.b.a
For more examples, see examples/.
Development
All commits are automatically built and tested via github actions. In order to pass the required checks it is strongly recommended to install the repositories pre commit hooks (assuming you are in the repositories root):
pip install pre-commit invoke bump2version
pre-commit install
The pre commit hooks will run a number of go tools. Try to run pre-commit run --all-files and check for missing tools. You might need to install some of those:
go get -u golang.org/x/lint/golint
go get -u github.com/fzipp/gocyclo
In order to be found, make sure to include $GOPATH/bin in your $PATH ($GOPATH has to be set of course like export GOPATH="$HOME/go for example).
Before committing, pre-commit is run to make sure all checks pass!
Documentation
¶
Index ¶
Constants ¶
const Version = "0.1.12"
Version is incremented using bump2version
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Datasource ¶
type Datasource struct {
Options
Disabled bool
Description string
FileProvider files.FileProvider
// contains filtered or unexported fields
}
Datasource ...
type Import ¶
type Import struct {
Options
Connection *MongoConnection
Sources []*Datasource
MaxParallelism int
// contains filtered or unexported fields
}
Import ...
type ImportJob ¶
type ImportJob struct {
Source *Datasource
Loader *loaders.Loader
File string
InsertionBatchSize int
IgnoreErrors bool
Collection *mongo.Collection
}
ImportJob ...
type ImportResult ¶
type ImportResult struct {
TotalFiles int
TotalSources int
Description string
Succeeded int
Failed int
Elapsed time.Duration
PartialResults []SourceResult
}
ImportResult ...
type MongoConnection ¶
type MongoConnection struct {
DatabaseName string
AuthDatabaseName string
User string
Password string
Host string
Port uint
}
MongoConnection ...
type Options ¶
type Options struct {
DatabaseName string
Collection string
Loader loaders.Loader
PostLoad PostLoadHook
PreDump PreDumpHook
UpdateFilter UpdateFilterHook
EmptyCollection *opt.Flag
Sanitize *opt.Flag
FailOnErrors *opt.Flag
CollectErrors *opt.Flag
IndividualProgress *opt.Flag
ShowCurrentFile *opt.Flag
InsertionBatchSize int
}
Options ...
type PartialResult ¶
type PartialResult struct {
File string
Collection string
Source *Datasource
Succeeded int
Failed int
Elapsed time.Duration
Errors []error
}
PartialResult ...
type PostLoadHook ¶
PostLoadHook ...
type SourceResult ¶
type SourceResult struct {
TotalFiles int
Collection string
Description string
Succeeded int
Failed int
Elapsed time.Duration
PartialResults []PartialResult
}
SourceResult ...
type UpdateFilterHook ¶
type UpdateFilterHook func(loaded interface{}) ([]interface{}, error)
UpdateFilterHook ...
Source Files
Directories