The highest tagged major version is
README
¶
dataset 
dataset is a command line tool, Go package, and an experimental C shared library for working with JSON objects as collections. Collections can be stored on disc, on AWS S3 or Google Cloud Storage. JSON objects are stored in collections as plain UTF-8 text. This means the objects can be accessed with common Unix text processing tools as well as most programming languages with text processing support.
The dataset command line tool supports common data manage operations such as initialization of collections, creation, reading, updating and deleting JSON objects in the collection. Some of its enhanced features include the ability to generate data grids and frames, the ability to import and export JSON object to and from CSV files and Google Sheets.
dataset is written in the Go programming language. It can be used as a Go package by other Go based software. Go supports generating C shared libraries. By compiling the Go source you can create a libdataset C shared library. The C shared library is currently being used by the DLD Group in Caltech Library expermentally from Python. This approach looks promising to support other languages (e.g. Julia can easily use dataset via its ccall function, while R, Octave and NodeJS would probably need some C++ wrapping code).
See getting-started-with-datataset.md for a tour and tutorial.
Design choices
dataset isn't a database or repository system. It is guided by
the idea that you should be able to work with text files (e.g. the JSON
object documents) with standard Unix text utilities. It is intended
to be simple to use with minimal setup (e.g.
dataset init mycollection.ds would create a new collection called
'mycollection.ds'). It is built around a few abstractions --
dataset stores JSON objects in collections, collections are a folder(s)
containing a JSON object documents and any attachments, a
collections.json file describes the mapping of keys to folder locations).
It takes minimal system resources and keeps all content,
except JSON object attachments, in plain UTF-8 text.
A typical library processing pattern is to write a "harvester" which then stores it results in a dataset collection. The harvesters we use are written either as simple shell scripts, Python programs or Go programs. Once you have your JSON objects in a dataset collection it is easy to iterate over them and augment them further.
Care has been taken to keep dataset simple enough and light weight enough that it will run on a machine as small as a Raspberry Pi while being equally comfortable on a more resource rich server or desktop environment.
Features
dataset supports
- Basic storage actions (create, read, update and delete)
- listing of collection keys (including filtering and sorting)
- import/export of CSV files and Google Sheets
- An experimental full text search interface based on Blevesearch
- The ability to reshape data by performing simple object joins
- The ability to create data grids and frames from collections based on keys lists and dot paths into the JSON objects stored
You can work with dataset collections via the command line tool, via Go using the dataset package or in Python 3.6 using a python package. dataset is useful for general data science applications which need intermediate JSON object management but not a full blown database.
Limitations of dataset
dataset has many limitations, some are listed below
- it is not a multi-process, multi-user data store (it's files on "disc" without any locking)
- it is not a repository management system
- it is not a general purpose database system
- it does not supply version control on collections or objects
Example
Below is a simple example of shell based interaction with dataset a collection using the command line dataset tool.
# Create a collection "friends.ds", the ".ds" lets the bin/dataset command know that's the collection to use.
dataset init friends.ds
# if successful then you should see an OK otherwise an error message
# Create a JSON document
dataset friends.ds create frieda '{"name":"frieda","email":"frieda@inverness.example.org"}'
# If successful then you should see an OK otherwise an error message
# Read a JSON document
dataset friends.ds read frieda
# Path to JSON document
dataset friends.ds path frieda
# Update a JSON document
dataset friends.ds update frieda '{"name":"frieda","email":"frieda@zbs.example.org", "count": 2}'
# If successful then you should see an OK or an error message
# List the keys in the collection
dataset friends.ds keys
# Get keys filtered for the name "frieda"
dataset friends.ds keys '(eq .name "frieda")'
# Join frieda-profile.json with "frieda" adding unique key/value pairs
dataset friends.ds join append frieda frieda-profile.json
# Join frieda-profile.json overwriting in commont key/values adding unique key/value pairs
# from frieda-profile.json
dataset friends.ds join overwrite frieda frieda-profile.json
# Delete a JSON document
dataset friends.ds delete frieda
# Import data from a CSV file using column 1 as key
dataset -quiet -nl=false friends.ds import-csv my-data.csv 1
# To remove the collection just use the Unix shell command
rm -fR friends.ds
Releases
Compiled versions are provided for Linux (amd64), Mac OS X (amd64), Windows 10 (amd64) and Raspbian (ARM7). See https://github.com/caltechlibrary/dataset/releases.
Documentation
¶
Overview ¶
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
bucket.go is part of the dataset pacakge includes the operations needed for processing collections of JSON documents and their attachments using the bucket layout.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset provides a common approach for storing JSON object documents on local disc or on S3 and Google Cloud Storage. It is intended as a single user system for intermediate processing of JSON content for analysis or batch processing. It is not a database management system (if you need a JSON database system I would suggest looking at Couchdb, Mongo and Redis as a starting point).
The approach dataset takes to storing buckets is to maintain a JSON document with keys (document names) and bucket assignments. JSON documents (and possibly their attachments) are then stored based on that assignment. Conversely the collection.json document is used to find and retrieve documents from the collection. The layout of the metadata is as follows
+ Collection
- Collection/collection.json - metadata for retrieval
- Collection/[Buckets] - usually an "aa" to "zz" list of buckets
- Collection/[Bucket]/[Document]
A key feature of dataset is to be Posix shell friendly. This has lead to storing the JSON documents in a directory structure that standard Posix tooling can traverse. It has also mean that the JSON documents themselves remain on "disc" as plain text. This has facilitated integration with many other applications, programming langauages and systems.
Attachments are non-JSON documents explicitly "attached" that share the same basename but are placed in a tar ball (e.g. document Jane.Doe.json attachements would be stored in Jane.Doe.tar).
Additional operations beside storing and reading JSON documents are also supported. These include creating lists (arrays) of JSON documents from a list of keys, listing keys in the collection, counting documents in the collection, indexing and searching by indexes.
The primary use case driving the development of dataset is harvesting API content for library systems (e.g. EPrints, Invenio, ArchivesSpace, ORCID, CrossRef, OCLC). The harvesting needed to be done in such a way as to leverage existing Posix tooling (e.g. grep, sed, etc) for processing and analysis.
Initial use case:
Caltech Library has many repository, catelog and record management systems (e.g. EPrints, Invenion, ArchivesSpace, Islandora, Invenio). It is common practice to harvest data from these systems for analysis or processing. Harvested records typically come in XML or JSON format. JSON has proven a flexibly way for working with the data and in our more modern tools the common format we use to move data around. We needed a way to standardize how we stored these JSON records for intermediate processing to allow us to use the growing ecosystem of JSON related tooling available under Posix/Unix compatible systems.
Aproach to file system layout ¶
+ /dataset (directory on file system)
- collection (directory on file system)
- collection.json - metadata about collection
- maps the filename of the JSON blob stored to a bucket in the collection
- e.g. file "mydocs.jons" stored in bucket "aa" would have a map of {"mydocs.json": "aa"}
- keys.json - a list of keys in the collection (it is the default select list)
- BUCKETS - a sequence of alphabet names for buckets holding JSON documents and their attachments
- Buckets let supporting common commands like ls, tree, etc. when the doc count is high
- SELECT_LIST.json - a JSON document holding an array of keys
- the default select list is "keys", it is not mutable by Push, Pop, Shift and Unshift
- select lists cannot be named "keys" or "collection"
BUCKETS are names without meaning normally using Alphabetic characters. A dataset defined with four buckets might looks like aa, ab, ba, bb. These directories will contains JSON documents and a tar file if the document has attachments.
Operations ¶
+ Collection level
- InitCollection (collection) - creates or opens collection structure on disc, creates collection.json and keys.json if new
- Open (collection) - opens an existing collections and reads collection.json into memory
- Close (collection) - writes changes to collection.json to disc if dirty
- Keys (collection) - list of keys in the collection
+ JSON document level
- Create (JSON document) - saves a new JSON blob or overwrites and existing one on disc with given blob name, updates keys.json if needed
- Read (JSON document)) - finds the JSON document in the buckets and returns the JSON document contents
- Update (JSON document) - updates an existing blob on disc (record must already exist)
- Delete (JSON document) - removes a JSON blob from its disc
- Path (JSON document) - returns the path to the JSON document
+ Select list level
- Count (select list) - returns the number of keys in a select list
Example ¶
Common operations using the *dataset* command line tool
+ create collection + create a JSON document to collection + read a JSON document + update a JSON document + delete a JSON document
Example Bash script usage
# Create a collection "mystuff.ds" inside the directory called demo
dataset init mystuff.ds
# if successful an expression to export the collection name is show
export DATASET="mystuff.ds"
# Create a JSON document
dataset create freda.json '{"name":"freda","email":"freda@inverness.example.org"}'
# If successful then you should see an OK or an error message
# Read a JSON document
dataset read freda.json
# Path to JSON document
dataset path freda.json
# Update a JSON document
dataset update freda.json '{"name":"freda","email":"freda@zbs.example.org"}'
# If successful then you should see an OK or an error message
# List the keys in the collection
dataset keys
# Delete a JSON document
dataset delete freda.json
# To remove the collection just use the Unix shell command
# /bin/rm -fR mystuff.ds
Common operations shown in Golang ¶
+ create collection + create a JSON document to collection + read a JSON document + update a JSON document + delete a JSON document
Example Go code
// Create a collection "mystuff" inside the directory called demo
collection, err := dataset.InitCollection("mystuff.ds")
if err != nil {
log.Fatalf("%s", err)
}
defer collection.Close()
// Create a JSON document
docName := "freda.json"
document := map[string]string{"name":"freda","email":"freda@inverness.example.org"}
if err := collection.Create(docName, document); err != nil {
log.Fatalf("%s", err)
}
// Attach an image file to freda.json in the collection
if buf, err := ioutil.ReadAll("images/freda.png"); err != nil {
collection.Attach("freda", "images/freda.png", buf)
} else {
log.Fatalf("%s", err)
}
// Read a JSON document
if err := collection.Read(docName, document); err != nil {
log.Fatalf("%s", err)
}
// Update a JSON document
document["email"] = "freda@zbs.example.org"
if err := collection.Update(docName, document); err != nil {
log.Fatalf("%s", err)
}
// Delete a JSON document
if err := collection.Delete(docName); err != nil {
log.Fatalf("%s", err)
}
Working with attachments in Go
collection, err := dataset.Open("dataset/mystuff")
if err != nil {
log.Fatalf("%s", err)
}
defer collection.Close()
// Add a helloworld.txt file to freda.json record as an attachment.
if err := collection.Attach("freda", "docs/helloworld.txt", []byte("Hello World!!!!")); err != nil {
log.Fatalf("%s", err)
}
// Attached files aditional files from the filesystem by their relative file path
if err := collection.AttachFiles("freda", "docs/presentation-article.pdf", "docs/charts-and-figures.zip", "docs/transcript.fdx") {
log.Fatalf("%s", err)
}
// List the attached files for freda.json
if filenames, err := collection.Attachments("freda"); err != nil {
log.Fatalf("%s", err)
} else {
fmt.Printf("%s\n", strings.Join(filenames, "\n"))
}
// Get an array of attachments (reads in content into memory as an array of Attachment Structs)
allAttachments, err := collection.GetAttached("freda")
if err != nil {
log.Fatalf("%s", err)
}
fmt.Printf("all attachments: %+v\n", allAttachments)
// Get two attachments docs/transcript.fdx, docs/helloworld.txt
twoAttachments, _ := collection.GetAttached("fred", "docs/transcript.fdx", "docs/helloworld.txt")
fmt.Printf("two attachments: %+v\n", twoAttachments)
// Get attached files writing them out to disc relative to your working directory
if err := collection.GetAttachedFiles("freda"); err != nil {
log.Fatalf("%s", err)
}
// Get two selection attached files writing them out to disc relative to your working directory
if err := collection.GetAttached("fred", "docs/transcript.fdx", "docs/helloworld.txt"); err != nil {
log.Fatalf("%s", err)
}
// Remove docs/transcript.fdx and docs/helloworld.txt from freda.json attachments
if err := collection.Detach("fred", "docs/transcript.fdx", "docs/helloworld.txt"); err != nil {
log.Fatalf("%s", err)
}
// Remove all attached files from freda.json
if err := collection.Detach("fred")
log.Fatalf("%s", err)
}
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Package dataset includes the operations needed for processing collections of JSON documents and their attachments.
Authors R. S. Doiel, <rsdoiel@library.caltech.edu> and Tom Morrel, <tmorrell@library.caltech.edu>
Copyright (c) 2018, Caltech All rights not granted herein are expressly reserved by Caltech.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Index ¶
- Constants
- Variables
- func Analyzer(collectionName string) error
- func CSVFormatter(out io.Writer, results *bleve.SearchResult, colNames []string, ...) error
- func CollectionLayout(p string) int
- func Deindexer(idxName string, keys []string, batchSize int) error
- func Delete(name string) error
- func Find(idxAlias bleve.IndexAlias, queryString string, options map[string]string) (*bleve.SearchResult, error)
- func Formatter(out io.Writer, results *bleve.SearchResult, tmpl *template.Template, ...) error
- func IsCollection(p string) bool
- func JSONFormatter(out io.Writer, results *bleve.SearchResult, prettyPrint bool) error
- func Migrate(collectionName string, newLayout int) error
- func Repair(collectionName string) error
- type Attachment
- type Collection
- func (c *Collection) AttachFile(keyName, fName string, buf io.Reader) error
- func (c *Collection) AttachFiles(name string, fileNames ...string) error
- func (c *Collection) Attachments(name string) ([]string, error)
- func (c *Collection) Clone(keys []string, cloneName string) error
- func (c *Collection) CloneSample(sampleSize int, trainingCollectionName string, testCollectionName string) error
- func (c *Collection) Close() error
- func (c *Collection) Create(name string, data map[string]interface{}) error
- func (c *Collection) CreateJSON(key string, src []byte) error
- func (c *Collection) Deindexer(idxName string, keys []string, batchSize int) error
- func (c *Collection) Delete(name string) error
- func (c *Collection) DeleteFrame(name string) error
- func (c *Collection) DocPath(name string) (string, error)
- func (c *Collection) ExportCSV(fp io.Writer, eout io.Writer, filterExpr string, dotExpr []string, ...) (int, error)
- func (c *Collection) Frame(name string, keys []string, dotPaths []string, verbose bool) (*DataFrame, error)
- func (c *Collection) FrameLabels(name string, labels []string) error
- func (c *Collection) FrameTypes(name string, columnTypes []string) error
- func (c *Collection) Frames() []string
- func (c *Collection) GetAttachedFiles(name string, filterNames ...string) error
- func (c *Collection) Grid(keys []string, dotPaths []string, verbose bool) ([][]interface{}, error)
- func (c *Collection) HasKey(key string) bool
- func (c *Collection) ImportCSV(buf io.Reader, skipHeaderRow bool, idCol int, useUUID bool, verboseLog bool) (int, error)
- func (c *Collection) ImportTable(table [][]string, skipHeaderRow bool, idCol int, ...) (int, error)
- func (c *Collection) Indexer(idxName string, idxMapName string, keys []string, batchSize int) error
- func (c *Collection) KeyFilter(keyList []string, filterExpr string) ([]string, error)
- func (c *Collection) KeySortByExpression(keys []string, expr string) ([]string, error)
- func (c *Collection) Keys() []string
- func (c *Collection) Length() int
- func (c *Collection) Prune(name string, filterNames ...string) error
- func (c *Collection) Read(name string, data map[string]interface{}) error
- func (c *Collection) ReadJSON(name string) ([]byte, error)
- func (c *Collection) Reframe(name string, keys []string, verbose bool) error
- func (c *Collection) Update(name string, data map[string]interface{}) error
- func (c *Collection) UpdateJSON(name string, src []byte) error
- type DataFrame
- type IndexList
- type KeyValue
- type KeyValues
Constants ¶
const ( // Version of the dataset package Version = `v0.0.45` // License is a formatted from for dataset package based command line tools License = `` /* 1530-byte string literal not displayed */ // Sort directions ASC = iota DESC = iota )
const ( // Assume an unknown layout is zero, then add consts in order of adoption UNKNOWN_LAYOUT = iota // Buckets is the first file layout implemented when dataset started BUCKETS_LAYOUT = iota // Pairtree is the perferred file layout moving forward PAIRTREE_LAYOUT = iota )
Supported file layout types
const (
DefaultAlphabet = `abcdefghijklmnopqrstuvwxyz`
)
Variables ¶
var DefaultBucketNames = []string{}/* 676 elements not displayed */
DefaultBucketNames provides a A-Z list of bucket names with a length 2
Functions ¶
func Analyzer ¶ added in v0.0.3
Analyzer checks the collection version and either calls bucketAnalyzer or pairtreeAnalyzer as appropriate.
func CSVFormatter ¶ added in v0.0.3
func CSVFormatter(out io.Writer, results *bleve.SearchResult, colNames []string, skipHeaderRow bool) error
CSVFormatter writes out CSV representation using encoding/csv
func CollectionLayout ¶ added in v0.0.45
CollectionLayout returns the numeric type association with the collection (e.g BUCKETS_LAYOUT, PAIRTREE_LAYOUT).
func Deindexer ¶ added in v0.0.33
Deindexer deletes the keys from an index. Returns an error if a problem occurs.
func Find ¶
func Find(idxAlias bleve.IndexAlias, queryString string, options map[string]string) (*bleve.SearchResult, error)
Find takes a Bleve index name and query string, opens the index, and writes the results to the os.File provided. Function returns an error if their are problems.
func Formatter ¶ added in v0.0.3
func Formatter(out io.Writer, results *bleve.SearchResult, tmpl *template.Template, tName string, pageData map[string]string) error
Formatter writes out a format based on the specified template name merging any additional pageData provided
func IsCollection ¶ added in v0.0.45
IsCollection checks to see if a given path contains a collection.json file
func JSONFormatter ¶ added in v0.0.3
JSONFormatter writes out JSON representation using encoding/json
Types ¶
type Attachment ¶
type Attachment struct {
// Name is the filename and path to be used inside the generated tar file
Name string `json:"name"`
// Body is a byte array for storing the content associated with Name
Body []byte `json:"-"`
// Size
Size int64 `json:"size"`
}
Attachment is a structure for holding non-JSON content you wish to store alongside a JSON document in a collection
type Collection ¶
type Collection struct {
// Version of collection being stored
Version string `json:"version"`
// Name of collection
Name string `json:"name"`
// Type allows for transitioning from bucket layout to pairtree layout for collections.
Layout int `json:"layout"`
// Buckets is a list of bucket names used by collection (depreciated, will be removed after migration to pairtree)
Buckets []string `json:"buckets,omitempty"`
// KeyMap holds the document key to path in the collection
KeyMap map[string]string `json:"keymap"`
// Store holds the storage system information (e.g. local disc, S3, GS)
// and related methods for interacting with it
Store *storage.Store `json:"-"`
// FrameMap is a list of frame names and with rel path to the frame defined in the collection
FrameMap map[string]string `json:"frames"`
}
Collection is the container holding buckets which in turn hold JSON docs
func InitCollection ¶ added in v0.0.8
func InitCollection(name string, layoutType int) (*Collection, error)
InitCollection - creates a new collection with default alphabet and names of length 2. NOTE: layoutType is provided to allow for future changes in the file layout of a collection.
func Open ¶
func Open(name string) (*Collection, error)
Open reads in a collection's metadata and returns and new collection structure and err
func (*Collection) AttachFile ¶ added in v0.0.33
func (c *Collection) AttachFile(keyName, fName string, buf io.Reader) error
AttachFile is for attaching a single non-JSON document to a dataset record. It will replace ANY existing attached content (i.e. it creates an new tarball holding on this single document) This is a limitation of our storage package supporting a minimal set of operation across all storage environments (e.g. S3/Google Cloud Storage do not support append to file, only replacement). It takes the document key, name and an io.Reader reading content in and appending the results to the tar file updating the internal _Attributes metadata as needed.
func (*Collection) AttachFiles ¶
func (c *Collection) AttachFiles(name string, fileNames ...string) error
AttachFiles attaches non-JSON documents to a JSON document in the collection. Attachments are stored in a tar file, if tar file exits then attachment(s) are appended to tar file.
func (*Collection) Attachments ¶
func (c *Collection) Attachments(name string) ([]string, error)
Attachments returns a list of files in the attached tarball for a given name in the collection
func (*Collection) Clone ¶ added in v0.0.39
func (c *Collection) Clone(keys []string, cloneName string) error
Clone copies the current collection records into a newly initialized collection given a list of keys and new collection name. Returns an error value if there is a problem. Clone does NOT copy attachments, only the JSON records.
func (*Collection) CloneSample ¶ added in v0.0.39
func (c *Collection) CloneSample(sampleSize int, trainingCollectionName string, testCollectionName string) error
CloneSample takes the current collection, a sample size, a training collection name and a test collection name. The training collection will be created and receive a random sample of the records from the current collection based on the sample size provided. Sample size must be greater than zero and less than the total number of records in the current collection.
If the test collection name is not an empty string it will be created and any records not in the training collection will be cloned from the current collection into the test collection.
func (*Collection) Close ¶
func (c *Collection) Close() error
Close closes a collection, writing the updated keys to disc
func (*Collection) Create ¶
func (c *Collection) Create(name string, data map[string]interface{}) error
Create a JSON doc from an map[string]interface{} and adds it to a collection, if problem returns an error name must be unique. Document must be an JSON object (not an array).
func (*Collection) CreateJSON ¶ added in v0.0.33
func (c *Collection) CreateJSON(key string, src []byte) error
CreateJSON adds a JSON doc to a collection, if a problem occurs it returns an error
func (*Collection) Deindexer ¶ added in v0.0.33
func (c *Collection) Deindexer(idxName string, keys []string, batchSize int) error
Deindexer removes items from an index on a collection.
func (*Collection) Delete ¶
func (c *Collection) Delete(name string) error
Delete removes a JSON doc from a collection
func (*Collection) DeleteFrame ¶ added in v0.0.41
func (c *Collection) DeleteFrame(name string) error
DeleteFrame removes a frame from a collection, returns an error if frame can't be deleted.
func (*Collection) DocPath ¶
func (c *Collection) DocPath(name string) (string, error)
DocPath returns a full path to a key or an error if not found
func (*Collection) ExportCSV ¶ added in v0.0.3
func (c *Collection) ExportCSV(fp io.Writer, eout io.Writer, filterExpr string, dotExpr []string, colNames []string, verboseLog bool) (int, error)
ExportCSV takes a reader and iterates over the rows and exports then as a CSV file
func (*Collection) Frame ¶ added in v0.0.41
func (c *Collection) Frame(name string, keys []string, dotPaths []string, verbose bool) (*DataFrame, error)
Frame takes a set of collection keys and dotpaths, builds a grid and assembles the grid and metadata returning a new CollectionFrame and error. Frames are assoicated with the collection.
func (*Collection) FrameLabels ¶ added in v0.0.41
func (c *Collection) FrameLabels(name string, labels []string) error
FrameLabels sets the labels for a frame, the number of labels must match the number of dot paths (columns) in the frame.
func (*Collection) FrameTypes ¶ added in v0.0.41
func (c *Collection) FrameTypes(name string, columnTypes []string) error
FrameTypes sets the types associated with a frame's columns, types list must match the number of dot paths (columns) in the frame.
func (*Collection) Frames ¶ added in v0.0.41
func (c *Collection) Frames() []string
Frames retrieves a list of available frames associated with a collection
func (*Collection) GetAttachedFiles ¶
func (c *Collection) GetAttachedFiles(name string, filterNames ...string) error
GetAttachedFiles returns an error if encountered, side effect is to write file to destination directory If no filterNames provided then return all attachments or error
func (*Collection) Grid ¶ added in v0.0.41
func (c *Collection) Grid(keys []string, dotPaths []string, verbose bool) ([][]interface{}, error)
Grid takes a set of collection keys and builds a grid (a 2D array cells) from the array of keys and dot paths provided
func (*Collection) HasKey ¶ added in v0.0.3
func (c *Collection) HasKey(key string) bool
HasKey returns true if key is in collection's KeyMap, false otherwise
func (*Collection) ImportCSV ¶ added in v0.0.3
func (c *Collection) ImportCSV(buf io.Reader, skipHeaderRow bool, idCol int, useUUID bool, verboseLog bool) (int, error)
ImportCSV takes a reader and iterates over the rows and imports them as a JSON records into dataset.
func (*Collection) ImportTable ¶ added in v0.0.4
func (c *Collection) ImportTable(table [][]string, skipHeaderRow bool, idCol int, useUUID, overwrite, verboseLog bool) (int, error)
ImportTable takes a [][]string and iterates over the rows and imports them as a JSON records into dataset.
func (*Collection) Indexer ¶
Indexer generates or updates and a Bleve index based on the index map filename provided, a list of keys and batch size.
func (*Collection) KeyFilter ¶ added in v0.0.33
func (c *Collection) KeyFilter(keyList []string, filterExpr string) ([]string, error)
KeyFilter takes a list of keys and filter expression and returns the list of keys passing through the filter or an error
func (*Collection) KeySortByExpression ¶ added in v0.0.33
func (c *Collection) KeySortByExpression(keys []string, expr string) ([]string, error)
KeySortByExpression takes a array of keys and a sort expression and turns a sorted list of keys.
func (*Collection) Keys ¶
func (c *Collection) Keys() []string
Keys returns a list of keys in a collection
func (*Collection) Length ¶ added in v0.0.6
func (c *Collection) Length() int
Length returns the number of keys in a collection
func (*Collection) Prune ¶ added in v0.0.33
func (c *Collection) Prune(name string, filterNames ...string) error
Prune a non-JSON document from a JSON document in the collection.
func (*Collection) Read ¶
func (c *Collection) Read(name string, data map[string]interface{}) error
Read finds the record in a collection, updates the data interface provide and if problem returns an error name must exist or an error is returned
func (*Collection) ReadJSON ¶ added in v0.0.33
func (c *Collection) ReadJSON(name string) ([]byte, error)
ReadJSON finds a the record in the collection and returns the JSON source
func (*Collection) Reframe ¶ added in v0.0.41
func (c *Collection) Reframe(name string, keys []string, verbose bool) error
Reframe will re-generate contents of a frame based on the current records in a collection. If a list of keys is supplied then the regenerated frame will be based on the new set of keys provided
func (*Collection) Update ¶
func (c *Collection) Update(name string, data map[string]interface{}) error
Update JSON doc in a collection from the provided data interface (note: JSON doc must exist or returns an error )
func (*Collection) UpdateJSON ¶ added in v0.0.33
func (c *Collection) UpdateJSON(name string, src []byte) error
UpdateJSON a JSON doc in a collection, returns an error if there is a problem
type DataFrame ¶ added in v0.0.41
type DataFrame struct {
// Explicit at creator
Name string `json:"frame_name"`
CollectionName string `json:"collection_name"`
DotPaths []string `json:"dot_paths"`
// NOTE: Keys may be deprecaited as _Key as column zero of the grid is a requirement of Frames.
Keys []string `json:"keys"`
Grid [][]interface{} `json:"grid"`
Created time.Time `json:"created"`
Updated time.Time `json:"updated,omitempty"`
// Derived or explicitly set after creation
Labels []string `json:"labels,omitempty"`
ColumnTypes []string `json:"column_types,omitempty"`
}
type IndexList ¶ added in v0.0.33
type IndexList struct {
Names []string
Fields []string
Alias bleve.IndexAlias
Indexes []bleve.Index
}
func OpenIndexes ¶ added in v0.0.3
OpenIndexes opens a list of index names and returns an index alias, a combined list of fields and error
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
analyzers
|
|
|
cmd
|
|
|
dataset
command
dataset is a command line utility to manage content stored in a dataset collection.
|
dataset is a command line utility to manage content stored in a dataset collection. |
|
gsheets.go is a part of the dataset package written to allow import/export of records to/from dataset collections.
|
gsheets.go is a part of the dataset package written to allow import/export of records to/from dataset collections. |