frames

V3IO Frames

V3IO Frames ("Frames") is a multi-model open-source data-access library, developed by Iguazio, which provides a unified high-performance DataFrame API for working with data in the data store of the Iguazio Data Science Platform ("the platform").

In This Document

• Client Python API Reference
• Contributing
• LICENSE

Client Python API Reference

• Overview
• User Authentication
• Client Constructor
• Common Client Method Parameters
• create Method
• write Method
• read Method
• delete Method
• execute Method

Overview

To use Frames, you first need to import the v3io_frames Python library. For example:

import v3io_frames as v3f

Then, you need to create and initialize an instance of the Client class; see Client Constructor. You can then use the client methods to perform different data operations on the supported backend types:

Client Methods

The Client class features the following methods for supporting basic data operations:

• create — creates a new TSDB table or a stream ("backend data").
• delete — deletes a table or stream or specific table items
• read — reads data from a table or stream into pandas DataFrames.
• write — writes data from pandas DataFrames to a table or stream.
• execute — executes a backend-specific command on a table or stream. Each backend may support multiple commands.

Backend Types

All Frames client methods receive a backend parameter for setting the Frames backend type. Frames supports the following backend types:

• kv — a platform NoSQL (key/value) table.
• stream — a platform data stream.
• tsdb — a time-series database (TSDB).
• csv — a comma-separated-value (CSV) file. This backend type is used only for testing purposes.

Note: Some method parameters are common to all backend types and some are backend-specific, as detailed in this reference.

User Authentication

When creating a Frames client, you must provide valid platform credentials for accessing the backend data, which Frames will use to identify the identity of the user. This can be done by using any of the following alternative methods (documented in order of precedence):

• Provide the authentication credentials in the Client constructor parameters by using either of the following methods:

  • Set the token constructor parameter to a valid platform access key with the required data-access permissions.
  • Set the user and password constructor parameters to the username and password of a platform user with the required data-access permissions.
    

  Note: You can't use both methods concurrently: setting both the token and username and password parameters in the same constructor call will produce an error.

• Set the authentication credentials in environment variables, by using either of the following methods:

  • Set the V3IO_ACCESS_KEY environment variable to a valid platform access key with the required data-access permissions.
  • Set the V3IO_USERNAME and V3IO_PASSWORD environment variables to the username and password of a platform user with the required data-access permissions.
    

  Note:

  • When the client constructor is called with authentication parameters, the authentication-credentials environment variables (if defined) are ignored.
  • When V3IO_ACCESS_KEY is defined, V3IO_USERNAME and V3IO_PASSWORD are ignored.
  • The platform's Jupyter Notebook service automatically defines the V3IO_ACCESS_KEY environment variable and initializes it to a valid access key for the running user of the service.

Client Constructor

All Frames operations are executed via an object of the Client class.

• Syntax
• Parameters and Data Members
• Return Value
• Example

Syntax

Client(address='', container='', user='', password='', token='')

Parameters and Data Members

• address — The address of the Frames service (framesdb).
  When running locally on the platform (for example, from a Jupyter Notebook service), set this parameter to framesd:8081 to use the gRPC (recommended) or to framesd:8080 to use HTTP.
  When connecting to the platform remotely, set this parameter to the API address of Frames platform service of the parent tenant. You can copy this address from the API column of the V3IO Frames service on the Services platform dashboard page.

  • Type: str
  • Requirement: Required

• container — The name of the platform data container that contains the backend data. For example, "bigdata" or "users".

  • Type: str
  • Requirement: Required

• user — The username of a platform user with permissions to access the backend data.

  • Type: str
  • Requirement: Required when neither the token parameter or the authentication environment variables are set. See User Authentication.
    When the user parameter is set, the password parameter must also be set to a matching user password.

• password — A platform password for the user configured in the user parameter.

  • Type: str
  • Requirement: Required when the user parameter is set. See User Authentication.

• token — A valid platform access key that allows access to the backend data. To get this access key, select the user profile icon on any platform dashboard page, select Access Tokens, and copy an existing access key or create a new key.

  • Type: str
  • Requirement: Required when neither the user or password parameters or the authentication environment variables are set. See User Authentication.

Return Value

Returns a new Frames Client data object.

Example

The following example, for local platform execution, creates a Frames client for accessing data in the "users" container by using the authentication credentials of the user "iguazio":

import v3io_frames as v3f
client = v3f.Client("framesd:8081", user="iguazio", password="mypass", container="users")

Common Client Method Parameters

All client methods receive the following common parameters:

• backend — The backend data type for the operation. See the backend-types descriptions in the overview.

  • Type: str
  • Valid Values: "csv" | "kv" | "stream" | "tsdb"
  • Requirement: Required

• table — The relative path to the backend data — A directory in the target platform data container (as configured for the client object) that represents a TSDB or NoSQL table or a data stream. For example, "mytable" or "examples/tsdb/my_metrics".

  • Type: str
  • Requirement: Required unless otherwise specified in the method-specific documentation

Additional method-specific parameters are described for each method.

create Method

Creates a new TSDB table or a stream in a platform data container, according to the specified backend type.

The create method is supported by the tsdb and stream backends, but not by the kv backend, because NoSQL tables in the platform don't need to be created prior to ingestion; when ingesting data into a table that doesn't exist, the table is automatically created.

• Syntax
• Common parameters
• tsdb backend create parameters
• stream backend create parameters

Syntax

create(backend, table, attrs=None)

Common create Parameters

All Frames backends that support the create method support the following common parameters:

• attrs — A dictionary of <argument name>: <value> pairs for passing additional backend-specific parameters (arguments).

  • Type: dict
  • Requirement: Optional
  • Default Value: None

tsdb Backend create Parameters

The following tsdb backend parameters are passed via the attrs parameter of the create method:

• rate — The ingestion rate TSDB's metric-samples, as "[0-9]+/[smh]" (where s = seconds, m = minutes, and h = hours); for example, "1/s" (one sample per minute). The rate should be calculated according to the slowest expected ingestion rate.

  • Type: str
  • Requirement: Required

• aggregates — Default aggregates to calculate in real time during the samples ingestion, as a comma-separated list of supported aggregation functions.

  • Type: str
  • Requirement: Optional

• aggregation-granularity — Aggregation granularity; i.e., a time interval for applying the aggregation functions, if configured in the aggregates parameter.

  • Type: str
  • Requirement: Optional

For detailed information about these parameters, refer to the V3IO TSDB documentation.

Example:

client.create("tsdb", "/mytable", attrs={"rate": "1/m"})

stream Backend create Parameters

The following stream backend parameters are passed via the attrs parameter of the create method:

• shards (Optional) (default: 1) — int — The number of stream shards to create.
• retention_hours (Optional) (default: 24) — int — The stream's retention period, in hours.

For detailed information about these parameters, refer to the platform streams documentation.

Example:

client.create("stream", "/mystream", attrs={"shards": 6})

write Method

Writes data from a DataFrame to a table or stream in a platform data container, according to the specified backend type.

• Syntax
• Common parameters
• tsdb backend write parameters
• kv backend write parameters

Syntax

write(backend, table, dfs, condition='', labels=None, max_in_message=0,
    index_cols=None, partition_keys=None)

• When the value of the iterator parameter is False (default) — returns a single DataFrame.
• When the value of the iterator parameter is True — returns a DataFrames iterator. The returned DataFrames include a "labels" DataFrame attribute with backend-specific data, if applicable; for example, for the stream backend, this attribute holds the sequence number of the last stream record that was read.

Common write Parameters

All Frames backends that support the write method support the following common parameters:

• dfs (Required) — A single DataFrame, a list of DataFrames, or a DataFrames iterator — One or more DataFrames containing the data to write.
• index_cols (Optional) (default: None) — []str — A list of column (attribute) names to be used as index columns for the write operation, regardless of any index-column definitions in the DataFrame. By default, the DataFrame's index columns are used.
  

  Note: The significance and supported number of index columns is backend specific. For example, the kv backend supports only a single index column for the primary-key item attribute, while the tsdb backend supports additional index columns for metric labels.

• labels (Optional) (default: None) — This parameter is currently defined for all backends but is used only for the TSDB backend, therefore it's documented as part of the write method's tsdb backend parameters.
• max_in_message (Optional) (default: 0)
• partition_keys (Optional) (default: None) — []str — [Not supported in this version]

Example:

data = [["tom", 10], ["nick", 15], ["juli", 14]]
df = pd.DataFrame(data, columns = ["name", "age"])
df.set_index("name")
client.write(backend="kv", table="mytable", dfs=df)

tsdb Backend write Parameters

• labels (Optional) (default: None) — dict — A dictionary of <label name>: <label value> pairs that define metric labels to add to all written metric-sample table items. Note that the values of the metric labels must be of type string.

kv Backend write Parameters

• condition (Optional) (default: None) — A platform condition expression that defines conditions for performing the write operation. For detailed information about platform condition expressions, see the platform documentation.

Example:

data = [["tom", 10, "TLV"], ["nick", 15, "Berlin"], ["juli", 14, "NY"]]
df = pd.DataFrame(data, columns = ["name", "age", "city"])
df.set_index("name")
v3c.write(backend="kv", table="mytable", dfs=df, condition="age>14")

read Method

Reads data from a table or stream in a platform data container to a DataFrame, according to the configured backend.

• Syntax
• Common parameters
• tsdb backend read parameters
• kv backend read parameters
• stream backend read parameters
• Return Value

Reads data from a backend.

Syntax

read(backend='', table='', query='', columns=None, filter='', group_by='',
    limit=0, data_format='', row_layout=False, max_in_message=0, marker='',
    iterator=False, **kw)

Common read Parameters

All Frames backends that support the read method support the following common parameters:

• iterator — (Optional) (default: False) — bool — True to return a DataFrames iterator; False to return a single DataFrame.
• filter (Optional) — str — A query filter.
  This parameter can't be used concurrently with the query parameter.
• columns — []str — A list of attributes (columns) to return.
  This parameter can't be used concurrently with the query parameter.
• data_format — str — The data format. [Not supported in this version]
• marker — str — A query marker. [Not supported in this version]
• limit — int — The maximum number of rows to return. [Not supported in this version]
• row_layout (Optional) (default: False) — bool — True to use a row layout; False (default) to use a column layout. [Not supported in this version]

tsdb Backend read Parameters

• start — str — Start (minimum) time for the read operation, as a string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".
  The default start time is <end time> - 1h.
• end — str — End (maximum) time for the read operation, as a string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2018-09-26T14:10:20Z"; "1537971006000"; "now-3h"; "now-7d".
  The default end time is "now".
• step (Optional) — str — For an aggregation query, this parameter specifies the aggregation interval for applying the aggregation functions; by default, the aggregation is applied to all sample data within the requested time range.
  When the query doesn't include aggregates, this parameter specifies an interval for downsampling the raw sample data.
• aggregators (Optional) — str — Aggregation information to return, as a comma-separated list of supported aggregation functions.
• aggregationWindow (Optional) — str — Aggregation interval for applying the aggregation functions, if set in the aggregators or query parameters.
• query (Optional) — str — A query string in SQL format.

  Note: When the query parameter is set, you can either specify the target table within the query string (FROM <table>) or by setting the table parameter of the read method to the table path. When the query string specifies the target table, the value of the table parameter (if set) is ignored.

• group_by (Optional) — str — A group-by query string.
  This parameter can't be used concurrently with the query parameter.
• multi_index (Optional) — bool — True to receive the read results as multi-index DataFrames where the labels are used as index columns in addition to the metric sample-time primary-key attribute; False (default) only the timestamp will function as the index.

For detailed information about these parameters, refer to the V3IO TSDB documentation.

Example:

df = client.read(backend="tsdb", query="select avg(cpu) as cpu, avg(diskio), avg(network)from mytable", start="now-1d", end="now", step="2h")

kv Backend read Parameters

• reset_index — bool — Reset the index. When set to false (default), the DataFrame will have the key column of the v3io kv as the index column. When set to true, the index will be reset to a range index.
• max_in_message — int — The maximum number of rows per message.
• sharding_keys — []string (Experimental) — A list of specific sharding keys to query, for range-scan formatted tables only.
• segments — []int64 [Not supported in this version]
• total_segments — int64 [Not supported in this version]
• sort_key_range_start — str [Not supported in this version]
• sort_key_range_end — str [Not supported in this version]

For detailed information about these parameters, refer to the platform's NoSQL documentation.

Example:

df = client.read(backend="kv", table="mytable", filter="col1>666")

stream Backend read Parameters

• seek — str — Valid values: "time" | "seq"/"sequence" | "latest" | "earliest".
  If the "seq"|"sequence" seek type is set, you need to provide the desired record sequence ID via the sequence parameter.
  If the time seek type is set, you need to provide the desired start time via the start parameter.
• shard_id — str
• sequence — int64 (Optional)

For detailed information about these parameters, refer to the platform streams documentation.

Example:

df = client.read(backend="stream", table="mytable", seek="latest", shard_id="5")

Return Value

• When the value of the iterator parameter is False (default) — returns a single DataFrame.
• When the value of the iterator parameter is True — returns a DataFrames iterator.

Note: The returned DataFrames include a labels DataFrame attribute with backend-specific data, if applicable. For example, for the stream backend, this attribute holds the sequence number of the last stream record that was read.

delete Method

Deletes a table or stream or specific table items from a platform data container, according to the specified backend type.

• Syntax
• tsdb backend delete parameters
• kv backend delete parameters

Syntax

delete(backend, table, filter='', start='', end='')

tsdb Backend delete Parameters

• start — str — Start (minimum) time for the delete operation, as a string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2016-01-02T15:34:26Z"; "1451748866"; "now-90m"; "0".
  The default start time is <end time> - 1h.
• end — str — End (maximum) time for the delete operation, as a string containing an RFC 3339 time, a Unix timestamp in milliseconds, a relative time of the format "now" or "now-[0-9]+[mhd]" (where m = minutes, h = hours, and 'd' = days), or 0 for the earliest time. For example: "2018-09-26T14:10:20Z"; "1537971006000"; "now-3h"; "now-7d".
  The default end time is "now".

Note: When neither the start or end parameters are set, the entire TSDB table is deleted.

For detailed information about these parameters, refer to the V3IO TSDB documentation.

Example:

df = client.delete(backend="tsdb", table="mytable", start="now-1d", end="now-5h")

kv Backend delete Parameters

• filter — str — A platform filter expression that identifies specific items to delete. For detailed information about platform filter expressions, see the platform documentation.

Note: When the filter parameter isn't set, the entire table is deleted.

Example:

df = client.delete(backend="kv", table="mytable", filter="age > 40")

execute Method

Extends the basic CRUD functionality of the other client methods via backend-specific commands.

• Syntax
• Common parameters
• tsdb backend commands
• kv backend commands
• stream backend commands

Syntax

execute(backend, table, command='', args=None)

Common execute Parameters

All Frames backends that support the execute method support the following common parameters:

• args — A dictionary of <argument name>: <value> pairs for passing command-specific parameters (arguments).

  • Type: dict
  • Requirement: Optional
  • Default Value: None

tsdb Backend execute Commands

Currently, no execute commands are available for the tsdb backend.

kv Backend execute Commands

• infer | inferschema — Infers the data schema of a given NoSQL table and creates a schema file for the table.

  Example:

  client.execute(backend="kv", table="mytable", command="infer")
  

stream Backend execute Commands

• put — Adds records to a stream.

  Example:

  client.execute(backend="stream", table="mystream", command="put", args={"data": "this a record", "clientinfo": "some_info", "partition": "partition_key"})
  

Contributing

To contribute to V3IO Frames, you need to be aware of the following:

• Components
• Development
  • Adding and Changing Dependencies
  • Travis CI
• Docker Image
  • Building the Image
  • Running the Image

Components

The following components are required for building Frames code:

• Go server with support for both the gRPC and HTTP protocols
• Go client
• Python client

Development

The core is written in Go. The development is done on the development branch and then released to the master branch.

Before submitting changes, test the code:

• To execute the Go tests, run make test.
• To execute the Python tests, run make test-python.

Adding and Changing Dependencies

• If you add Go dependencies, run make update-go-deps.
• If you add Python dependencies, update clients/py/Pipfile and run make update-py-deps.

Travis CI

Integration tests are run on Travis CI. See .travis.yml for details.

The following environment variables are defined in the Travis settings:

• Docker Container Registry (Quay.io)
  • DOCKER_PASSWORD — a password for pushing images to Quay.io.
  • DOCKER_USERNAME — a username for pushing images to Quay.io.
• Python Package Index (PyPI)
  • V3IO_PYPI_PASSWORD — a password for pushing a new release to PyPi.
  • V3IO_PYPI_USER — a username for pushing a new release to PyPi.
• Iguazio Data Science Platform

  • V3IO_SESSION — a JSON encoded map with session information for running tests. For example:

    '{"url":"45.39.128.5:8081","container":"mitzi","user":"daffy","password":"rabbit season"}'
    

    Note: Make sure to embed the JSON object within single quotes ('{...}').

Docker Image

Building the Image

Use the following command to build the Docker image:

make build-docker

Running the Image

Use the following command to run the Docker image:

docker run \
	-v /path/to/config.yaml:/etc/framesd.yaml \
	quay.io/v3io/frames:unstable

LICENSE

Apache 2