fluid

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 16, 2021 License: Apache-2.0 Imports: 3 Imported by: 0

README

License CircleCI Build Status codecov Go Report Card

Fluid

English | 简体中文

notification What is NEW!
Mar. 16th, 2021. Fluid v0.5.0 is RELEASED! It provides various new features, such as on-the-fly dataset scale out/in, metadata backup, support Fuse global mode and so on. Please check the CHANGELOG for details.
Nov. 6th, 2020. Fluid v0.4.0 is RELEASED! It provides various features and bugfix, such as Prefetch Dataset automatically before using it. Please check the CHANGELOG for details.
Oct. 1st, 2020. Fluid v0.3.0 is RELEASED! It provides various features and bugfix, such as Data Access Acceleration For Persistent Volume and Hostpath mode in K8s. Please check the CHANGELOG for details.

What is Fluid?

Fluid is an open source Kubernetes-native Distributed Dataset Orchestrator and Accelerator for data-intesive applications, such as big data and AI applications.

Features

  • Native Support for DataSet Abstraction

    Make the abilities needed by data-intensive applictions as navtive-supported functions, to achieve efficient data access and reduce the cost of multidimensional management.

  • Cloud Data Warming up and Accessing Acceleration

    Fluid empowers Distributed Cache Capaicty(Alluixo inside) in Kubernetes with Observability, Portability, Horizontal Scalability

  • Co-Orchestration for Data and Application

    During application scheduling and data placement on cloud, taking both the app's characteristics and data location into consideration, to improve the performance.

  • Support Multiple Namespaces Management

    User can create and manage datasets in multiple namespaces

  • Support Heterogeneous Data Source Management

    Unify the Data access for OSS, HDFS, CEPH and Other underlayer storages

Key Concepts

Dataset: A set of logically related data that will be used by a computing engine, such as Spark for big data and TensorFlow for AI scenarios. The management of dataset has many metrics, has multiple dimensions, such as security, version management and data acceleration. And we hope to start with data acceleration and provide support for the management of data sets.

Runtime: Security, version management and data acceleration, and defines a series of life cycle interfaces. You can implement them.

AlluxioRuntime: From Alluixo, Fluid manages and schedules Alluxio Runtime to achieve dataset visibility, elastic scaling, and data migration. It is an engine which supports data management and caching of datasets.

Prerequisites

  • Kubernetes version > 1.14, and support CSI
  • Golang 1.12+
  • Helm 3

Quick Start

You can follow our Get Started guide to quickly start a testing Kubernetes cluster.

Documentation

You can see our documentation at docs for more in-depth installation and instructions for production:

Qucik Demo

Demo 1: Accelerate Remote File Accessing with Fluid

Demo 2: Machine Learning with Fluid

Demo 3: Accelerate PVC with Fluid

Demo 4: Preload dataset with Fluid

Community

Feel free to reach out if you have any questions. The maintainers of this project are reachable via:

DingTalk:

Contributing

Contributions are welcome and greatly appreciated. See CONTRIBUTING.md for details on submitting patches and the contribution workflow.

Open Srouce License

Fluid is under the Apache 2.0 license. See the LICENSE file for details.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func LogVersion

func LogVersion()

Print version info in log when start

func PrintVersion

func PrintVersion(short bool)

Print version info directly by command

Types

type Version

type Version struct {
	Version      string
	BuildDate    string
	GitCommit    string
	GitTag       string
	GitTreeState string
	GoVersion    string
	Compiler     string
	Platform     string
}

Directories

Path Synopsis
api
v1alpha1
Package v1alpha1 is the v1alpha1 version of the API.
Package v1alpha1 is the v1alpha1 version of the API.
cmd
alluxio command
csi command
dataset command
jindo command
pkg
ddc
ddc/alluxio
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
ddc/base/mock
Package base is a generated GoMock package.
Package base is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL