pipelaner

Pipelaner

Pipelaner is a high-performance and efficient Framework and Agent for creating data pipelines. The core of pipeline descriptions is based on the Configuration As Code concept and the Pkl configuration language by Apple.

Pipelaner manages data streams through three key entities: Generator, Transform and Sink.

📖 Contents

• Core Entities
  • Generator
  • Transform
  • Sink
  • Basic Parameters
• Built-in Pipeline Elements
  • Generators
  • Transforms
  • Sinks
• Scalability
  • Single-Node Deployment
  • Multi-Node Deployment
• Examples
• Support
• License

📌 Core Entities

Generator

The component responsible for creating or retrieving source data for the pipeline. Generators can produce messages, events, or retrieve data from various sources such as files, databases, or APIs.

• Example use case:
  Reading data from a file or receiving events via webhooks.

Transform

The component that processes data within the pipeline. Transforms perform operations such as filtering, aggregation, data transformation, or cleaning to prepare it for further processing.

• Example use case:
  Filtering records based on specific conditions or converting data format from JSON to CSV.

Sink

The final destination for the data stream. Sinks send processed data to a target system, such as a database, API, or message queue.

• Example use case:
  Saving data to PostgreSQL or sending it to a Kafka topic.

Basic Parameters

📦 Built-in Pipeline Elements

Generators

Transforms

Sinks

🌐 Scalability

Single-Node Deployment

For operation on a single host:

Multi-Node Deployment

For distributed data processing across multiple hosts:

For distributed interaction between nodes, you can use:

1. gRPC — via generators and sinks with the parameter sourceName: "pipelaner".
2. Apache Kafka — for reading/writing data via topics.

Example configuration using Kafka:

new Inputs.Kafka {
    ...
    common {
        ...
        topics {
            "kafka-topic"
        }         
    }
}

new Sinks.Kafka {
    ...
    common {
        ...
        topics {
            "kafka-topic"
        }         
    }
}

🚀 Examples

Overview

1. 🌟 Basic Pipeline
   Learn the fundamentals of creating a pipeline with minimal configuration using ready-to-use components.

2. 🛠 Custom Components
   Extend Pipelaner’s functionality by developing your own Generators, Transforms, and Sinks.

Each example includes clear configuration files and explanations to help you get started quickly.

💡 Tip: Use these examples as templates to customize and build your own pipelines efficiently.

🤝 Support

If you have questions, suggestions, or encounter any issues, please create an Issue in the repository.
You can also participate in discussions in the Discussions section.

📜 License

This project is licensed under the Apache 2.0 license.
You are free to use, modify, and distribute the code under the terms of the license.