gotaskflow

Go-Taskflow

A General-purpose Task-parallel Programming Framework for Go, inspired by taskflow-cpp, with Go's native capabilities and simplicity, suitable for complex dependency management in concurrent tasks.

Feature

• High extensibility: Easily extend the framework to adapt to various specific use cases.

• Native Go's concurrency model: Leverages Go's goroutines to manage concurrent task execution effectively.

• User-friendly programming interface: Simplify complex task dependency management using Go.

• Static\Subflow\Conditional\Cyclic tasking: Define static tasks, condition nodes, nested subflows and cyclic flow to enhance modularity and programmability.

  Static	Subflow	Condition	Cyclic

• Priority Task Schedule: Define tasks' priority, higher priority tasks will be scheduled first.

• Built-in visualization & profiling tools: Generate visual representations of tasks and profile task execution performance using integrated tools, making debugging and optimization easier.

Use Cases

• Data Pipeline: Orchestrate data processing stages that have complex dependencies.

• AI Agent Workflow Automation: Define and run AI Agent automation workflows where tasks have a clear sequence and dependency structure.

• Parallel Graph Tasking: Execute Graph-based tasks concurrently to fully utilize CPU resources.

Example

import latest version: go get -u github.com/noneback/go-taskflow

package main

import (
	"fmt"
	"log"
	"math/rand"
	"os"
	"slices"
	"strconv"
	"sync"

	gtf "github.com/noneback/go-taskflow"
)

// merge sorted src to sorted dest
func mergeInto(dest, src []int) []int {
	size := len(dest) + len(src)
	tmp := make([]int, 0, size)
	i, j := 0, 0
	for i < len(dest) && j < len(src) {
		if dest[i] < src[j] {
			tmp = append(tmp, dest[i])
			i++
		} else {
			tmp = append(tmp, src[j])
			j++
		}
	}

	if i < len(dest) {
		tmp = append(tmp, dest[i:]...)
	} else {
		tmp = append(tmp, src[j:]...)
	}

	return tmp
}
func main() {
	size := 100
	radomArr := make([][]int, 10)
	sortedArr := make([]int, 0, 10*size)
	mutex := &sync.Mutex{}

	for i := 0; i < 10; i++ {
		for j := 0; j < size; j++ {
			radomArr[i] = append(radomArr[i], rand.Int())
		}
	}

	sortTasks := make([]*gtf.Task, 10)
	tf := gtf.NewTaskFlow("merge sort")
	done := tf.NewTask("Done", func() {
		if !slices.IsSorted(sortedArr) {
			log.Fatal("Failed")
		}
		fmt.Println("Sorted")
		fmt.Println(sortedArr[:1000])
	})

	for i := 0; i < 10; i++ {
		sortTasks[i] = tf.NewTask("sort_"+strconv.Itoa(i), func() {
			arr := radomArr[i]
			slices.Sort(arr)
			mutex.Lock()
			defer mutex.Unlock()
			sortedArr = mergeInto(sortedArr, arr)
		})

	}
	done.Succeed(sortTasks...)

	executor := gtf.NewExecutor(1000)

	executor.Run(tf).Wait()

	if err := tf.Dump(os.Stdout); err != nil {
		log.Fatal("V->", err)
	}

	if err := executor.Profile(os.Stdout); err != nil {
		log.Fatal("P->", err)
	}

}

more code examples

Benchmark

We provide a basic benchmark to give a rough estimate of performance. However, most realistic workloads are I/O-bound, and their performance cannot be accurately reflected by the benchmark results. So, don’t take it too seriously.

If you really care about CPU Performance, we strongly recommend taskflow-cpp.

goos: linux
goarch: amd64
pkg: github.com/noneback/go-taskflow/benchmark
cpu: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
BenchmarkC32-4             17964             68105 ns/op            7368 B/op        226 allocs/op
BenchmarkS32-4              5848            195952 ns/op            6907 B/op        255 allocs/op
BenchmarkC6-4              53138             22913 ns/op            1296 B/op         46 allocs/op
BenchmarkC8x8-4             6099            194579 ns/op           16956 B/op        503 allocs/op
PASS
ok      github.com/noneback/go-taskflow/benchmark       5.802s

Understand Condition Task Correctly

Condition Node is special in taskflow-cpp. It not only enrolls in Condition Control but also in Looping.

Our repo keeps almost the same behavior. You should read ConditionTasking to avoid common pitfalls.

Error Handling in go-taskflow

errors in golang are values. It is the user's job to handle it correctly.

Only unrecovered panic needs to be addressed by the framework. Now, if it happens, the whole parent graph will be canceled, leaving the rest tasks undone. This behavior may evolve someday. If you have any good thoughts, feel free to let me know.

If you prefer not to interrupt the whole taskflow when panics occur, you can also handle panics manually while registering tasks. Eg:

tf.NewTask("not interrupt", func() {
	defer func() {
		if r := recover(); r != nil {
			// deal with it.
		}
	}()
	// user functions.
)

How to use visualize taskflow

if err := tf.Dump(os.Stdout); err != nil {
		log.Fatal(err)
}

tf.Dump generates raw strings in dot format, use dot to draw a Graph svg.

How to use profile taskflow

if err :=exector.Profile(os.Stdout);err != nil {
		log.Fatal(err)
}

Profile generates raw strings in flamegraph format, use flamegraph to draw a flamegraph svg.

Stargazer