Documentation
¶
Index ¶
- type CsvFileIterator
- type DataFlowTester
- func (t *DataFlowTester) FlushTable(tableName string)
- func (t *DataFlowTester) ImportCsv(csvRelPath string, tableName string)
- func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})
- func (t *DataFlowTester) VerifyTable(tableName string, csvRelPath string, pkfields []string, targetfields []string)
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CsvFileIterator ¶
type CsvFileIterator struct {
// contains filtered or unexported fields
}
CsvFileIterator make iterating rows from csv file easier, it reads tuple from csv file and turn it into a `map[string]interface{}` for you.
Example CSV format (exported by dbeaver):
"id","name","json","created_at"
123,"foobar","{""url"": ""https://example.com""}","2022-05-05 09:56:43.438000000"
Example ¶
iter := NewCsvFileIterator("/path/to/foobar.csv")
defer iter.Close()
for iter.HasNext() {
row := iter.Fetch()
println(row["name"]) // foobar
println(row["json"]) // {"url": "https://example.com"}
}
func NewCsvFileIterator ¶
func NewCsvFileIterator(csvPath string) *CsvFileIterator
NewCsvFileIterator create a `*CsvFileIterator` based on path to csv file
func (*CsvFileIterator) Fetch ¶
func (ci *CsvFileIterator) Fetch() map[string]interface{}
Fetch returns current row
func (*CsvFileIterator) HasNext ¶
func (ci *CsvFileIterator) HasNext() bool
HasNext returns a boolean to indicate whether there was any row to be `Fetch`
type DataFlowTester ¶
type DataFlowTester struct {
Cfg *viper.Viper
Db *gorm.DB
T *testing.T
Name string
Plugin core.PluginMeta
Log core.Logger
}
- Create a folder under your plugin root folder. i.e. `plugins/gitlab/e2e/ to host all your e2e-tests`
- Create a folder named `tables` to hold all data in `csv` format
- Create e2e test-cases to cover all possible data-flow routes
Example code:
See [Gitlab Project Data Flow Test](plugins/gitlab/e2e/project_test.go) for detail
DataFlowTester use `N`
Example ¶
var t *testing.T // stub
var gitlab core.PluginMeta
dataflowTester := NewDataFlowTester(t, "gitlab", gitlab)
taskData := &tasks.GitlabTaskData{
Options: &tasks.GitlabOptions{
ProjectId: 3472737,
},
}
// import raw data table
dataflowTester.ImportCsv("./tables/_raw_gitlab_api_projects.csv", "_raw_gitlab_api_project")
// verify extraction
dataflowTester.FlushTable("_tool_gitlab_projects")
dataflowTester.Subtask(tasks.ExtractProjectMeta, taskData)
dataflowTester.VerifyTable(
"_tool_gitlab_projects",
"tables/_tool_gitlab_projects.csv",
[]string{"gitlab_id"},
[]string{
"name",
"description",
"default_branch",
"path_with_namespace",
"web_url",
"creator_id",
"visibility",
"open_issues_count",
"star_count",
"forked_from_project_id",
"forked_from_project_web_url",
"created_date",
"updated_date",
"_raw_data_params",
"_raw_data_table",
"_raw_data_id",
"_raw_data_remark",
},
)
func NewDataFlowTester ¶
func NewDataFlowTester(t *testing.T, pluginName string, pluginMeta core.PluginMeta) *DataFlowTester
NewDataFlowTester create a *DataFlowTester to help developer test their subtasks data flow
func (*DataFlowTester) FlushTable ¶
func (t *DataFlowTester) FlushTable(tableName string)
FlushTable deletes all records from specified table
func (*DataFlowTester) ImportCsv ¶
func (t *DataFlowTester) ImportCsv(csvRelPath string, tableName string)
ImportCsv imports records from specified csv file into target table, note that existing data would be deleted first.
func (*DataFlowTester) Subtask ¶
func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})
Subtask executes specified subtasks
func (*DataFlowTester) VerifyTable ¶
func (t *DataFlowTester) VerifyTable(tableName string, csvRelPath string, pkfields []string, targetfields []string)
VerifyTable reads rows from csv file and compare with records from database one by one. You must specified the Primary Key Fields with `pkfields` so DataFlowTester could select the exact record from database, as well as which fields to compare with by specifying `targetfields` parameter.
Source Files