Documentation
¶
Index ¶
- type DataFlowTester
- func (t *DataFlowTester) CreateSnapshot(dst schema.Tabler, opts TableOptions)
- func (t *DataFlowTester) ExportRawTable(rawTableName string, csvRelPath string)
- func (t *DataFlowTester) FlushRawTable(rawTableName string)
- func (t *DataFlowTester) FlushTabler(dst schema.Tabler)
- func (t *DataFlowTester) ImportCsvIntoRawTable(csvRelPath string, rawTableName string)
- func (t *DataFlowTester) ImportCsvIntoTabler(csvRelPath string, dst schema.Tabler)
- func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})
- func (t *DataFlowTester) SubtaskContext(taskData interface{}) core.SubTaskContext
- func (t *DataFlowTester) VerifyTable(dst schema.Tabler, csvRelPath string, targetFields []string)
- func (t *DataFlowTester) VerifyTableWithOptions(dst schema.Tabler, opts TableOptions)
- type TableOptions
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DataFlowTester ¶
type DataFlowTester struct {
Cfg *viper.Viper
Db *gorm.DB
Dal dal.Dal
T *testing.T
Name string
Plugin core.PluginMeta
Log core.Logger
}
DataFlowTester use `N`
- Create a folder under your plugin root folder. i.e. `plugins/gitlab/e2e/ to host all your e2e-tests`
- Create a folder named `tables` to hold all data in `csv` format
- Create e2e test-cases to cover all possible data-flow routes
Example code:
See [Gitlab Project Data Flow Test](plugins/gitlab/e2e/project_test.go) for detail
Example ¶
var t *testing.T // stub
var gitlab core.PluginMeta
dataflowTester := NewDataFlowTester(t, "gitlab", gitlab)
taskData := &tasks.GitlabTaskData{
Options: &tasks.GitlabOptions{
ProjectId: 666888,
},
}
// import raw data table
dataflowTester.ImportCsvIntoRawTable("./tables/_raw_gitlab_api_projects.csv", "_raw_gitlab_api_project")
// verify extraction
dataflowTester.FlushTabler(gitlabModels.GitlabProject{})
dataflowTester.Subtask(tasks.ExtractProjectMeta, taskData)
dataflowTester.VerifyTable(
gitlabModels.GitlabProject{},
"tables/_tool_gitlab_projects.csv",
[]string{
"gitlab_id",
"name",
"description",
"default_branch",
"path_with_namespace",
"web_url",
"creator_id",
"visibility",
"open_issues_count",
"star_count",
"forked_from_project_id",
"forked_from_project_web_url",
"created_date",
"updated_date",
"_raw_data_params",
"_raw_data_table",
"_raw_data_id",
"_raw_data_remark",
},
)
func NewDataFlowTester ¶
func NewDataFlowTester(t *testing.T, pluginName string, pluginMeta core.PluginMeta) *DataFlowTester
NewDataFlowTester create a *DataFlowTester to help developer test their subtasks data flow
func (*DataFlowTester) CreateSnapshot ¶
func (t *DataFlowTester) CreateSnapshot(dst schema.Tabler, opts TableOptions)
CreateSnapshot reads rows from database and write them into .csv file.
func (*DataFlowTester) ExportRawTable ¶
func (t *DataFlowTester) ExportRawTable(rawTableName string, csvRelPath string)
ExportRawTable reads rows from raw table and write them into .csv file.
func (*DataFlowTester) FlushRawTable ¶
func (t *DataFlowTester) FlushRawTable(rawTableName string)
FlushRawTable migrate table and deletes all records from specified table
func (*DataFlowTester) FlushTabler ¶
func (t *DataFlowTester) FlushTabler(dst schema.Tabler)
FlushTabler migrate table and deletes all records from specified table
func (*DataFlowTester) ImportCsvIntoRawTable ¶
func (t *DataFlowTester) ImportCsvIntoRawTable(csvRelPath string, rawTableName string)
ImportCsvIntoRawTable imports records from specified csv file into target raw table, note that existing data would be deleted first.
func (*DataFlowTester) ImportCsvIntoTabler ¶
func (t *DataFlowTester) ImportCsvIntoTabler(csvRelPath string, dst schema.Tabler)
ImportCsvIntoTabler imports records from specified csv file into target tabler, note that existing data would be deleted first.
func (*DataFlowTester) Subtask ¶
func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})
Subtask executes specified subtasks
func (*DataFlowTester) SubtaskContext ¶ added in v0.13.0
func (t *DataFlowTester) SubtaskContext(taskData interface{}) core.SubTaskContext
SubtaskContext creates a subtask context
func (*DataFlowTester) VerifyTable ¶
func (t *DataFlowTester) VerifyTable(dst schema.Tabler, csvRelPath string, targetFields []string)
VerifyTable reads rows from csv file and compare with records from database one by one. You must specify the Primary Key Fields with `pkFields` so DataFlowTester could select the exact record from database, as well as which fields to compare with by specifying `targetFields` parameter. Leaving `targetFields` empty/nil will compare all fields.
func (*DataFlowTester) VerifyTableWithOptions ¶
func (t *DataFlowTester) VerifyTableWithOptions(dst schema.Tabler, opts TableOptions)
VerifyTableWithOptions extends VerifyTable and allows for more advanced usages using TableOptions
type TableOptions ¶
type TableOptions struct {
// CSVRelPath relative path to the CSV file that contains the seeded data
CSVRelPath string
// TargetFields the fields (columns) to consider for verification. Leave empty to default to all.
TargetFields []string
// IgnoreFields the fields (columns) to ignore/skip.
IgnoreFields []string
// IgnoreTypes similar to IgnoreFields, this will ignore the fields contained in the type. Useful for ignoring embedded
// types and their fields in the target model
IgnoreTypes []interface{}
}
TableOptions FIXME ...
Source Files