README
¶
Try it Out
This example imports a couple hours worth of historical data from https://www.githubarchive.org/ into a local elasticsearch server for example.
cd tools/importgithub
# assuming elasticsearch on localhost elase --host=myeshost
go build && ./importgithub
# using dataux.conf from root of this project
go build
./dataux --config=dataux.conf
# now that dataux is running use mysql-client to connect
mysql -h 127.0.0.1 -P 4000
now run some queries
use datauxtest;
show tables;
describe github_watch;
select cardinality(`actor`) AS users_who_watched, min(`repository.id`) as oldest_repo from github_watch;
SELECT actor, `repository.name`, `repository.stargazers_count`, `repository.language`
FROM github_watch where `repository.language` = "Go";
select actor, repository.name from github_watch where repository.stargazers_count BETWEEN "1000" AND 1100;
SELECT actor, repository.organization AS org
FROM github_watch
WHERE repository.created_at BETWEEN "2008-10-21T17:20:37Z" AND "2008-10-21T19:20:37Z";
select actor, repository.name from github_watch where repository.name IN ("node", "docker","d3","myicons", "bootstrap") limit 100;
select cardinality(`actor`) AS users_who_watched, count(*) as ct, min(`repository.id`) as oldest_repo
FROM github_watch
WHERE repository.description LIKE "database";
SQL -> Elasticsearch
| ES API | SQL Query |
|---|---|
| Aliases | show tables; |
| Mapping | describe mytable; |
| hits.total for filter | select count(*) from table WHERE exists(a); |
| aggs min, max, avg, sum | select min(year), max(year), avg(year), sum(year) from table WHERE exists(a); |
| filter: terms | select * from table WHERE year IN (2015,2014,2013); |
| filter: gte, range | select * from table WHERE year BETWEEN 2012 AND 2014 |
Documentation
¶
Index ¶
- Constants
- Variables
- type ResultReader
- type ResultReaderNext
- type Source
- type SqlToEs
- func (m *SqlToEs) Close() error
- func (m *SqlToEs) Columns() []string
- func (m *SqlToEs) Host() string
- func (m *SqlToEs) WalkAggs(cur expr.Node) (q esMap, _ error)
- func (m *SqlToEs) WalkExecSource(p *plan.Source) (exec.Task, error)
- func (m *SqlToEs) WalkGroupBy() error
- func (m *SqlToEs) WalkNode(cur expr.Node, q *esMap) (value.Value, error)
- func (m *SqlToEs) WalkSelectList() error
- func (m *SqlToEs) WalkSourceSelect(planner plan.Planner, p *plan.Source) (plan.Task, error)
Constants ¶
View Source
const (
// SourceType defines the "source" type from qlbridge datasource
SourceType = "elasticsearch"
)
Variables ¶
View Source
var (
// DefaultLimit is default page size limit.
DefaultLimit = 1000
)
Functions ¶
This section is empty.
Types ¶
type ResultReader ¶
type ResultReader struct {
*exec.TaskBase
Docs []u.JsonHelper
Vals [][]driver.Value
Total int
Aggs u.JsonHelper
ScrollId string
Req *SqlToEs
// contains filtered or unexported fields
}
Elasticsearch ResultProvider, adapts the elasticsearch http json
to dataux/driver values
func NewResultReader ¶
func NewResultReader(req *SqlToEs) *ResultReader
func (*ResultReader) Close ¶
func (m *ResultReader) Close() error
func (*ResultReader) Columns ¶
func (m *ResultReader) Columns() []string
func (*ResultReader) Run ¶
func (m *ResultReader) Run() error
Run() Fetch api response, wait for response, then convert response into rows (static) and allow Next() to start iterating through them.
type ResultReaderNext ¶
type ResultReaderNext struct {
*ResultReader
}
A wrapper, allowing us to implement sql/driver Next() interface
which is different than qlbridge/datasource Next()
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source is the elasticsearch datasource
type SqlToEs ¶
type SqlToEs struct {
// contains filtered or unexported fields
}
SqlToEs Sql To Elasticsearch Request Object Map sql queries into Elasticsearch Json Requests
func NewSqlToEs ¶
NewSqlToEs creates request for translating SqlToEs
func (*SqlToEs) WalkAggs ¶
WalkAggs() aggregate expressions when used ast part of <select_list>
- For Aggregates (functions) it builds aggs
- For Projectsion (non-functions) it does nothing, that will be done later during projection
func (*SqlToEs) WalkExecSource ¶
func (*SqlToEs) WalkNode ¶
Walk() an expression, and its AND/OR/() logic to create an appropriately
nested json document for elasticsearch queries TODO: think we need to separate Value Nodes from those that return es types?
func (*SqlToEs) WalkSelectList ¶
Aggregations from the <select_list>
SELECT <select_list> FROM ... WHERE
Source Files
Click to show internal directories.
Click to hide internal directories.