Register Entities

Register Entities allows you to register custom components i.e. custom parsers, channels and processors to be used in the pipelines.

There are six types of entities:

Register Components

Register custom code for using more out of components.

Functions

A rich library of pre-defined functions, and user defined functions.

Variables

Use variables in your pipelines at runtime as per the scope.

Transformations

A transformation rule explains how the value of field in a message should be transformed to be used in predictive model.

Tables

Register Tables functionality enables you to store streaming data coming from different sources (AWS S3, HDFS, JDBC connection) in Tables, which can be queried using SparkSQL processor.

Models

Register Models functionality allows you to register the predictive models that are created using Spark Machine Learning API and are stored on HDFS or a Local file system.

Register Components

StreamAnalytix allows you implement your custom code in the platform to extend functionalities for:

  • Channel: To read from any source.
  • Processor: To perform any operation on data-in-motion.
  • Parser: To parse the incoming data.
  • Function: Custom function to be executed on data-in-motion via the Enricher operator.
  • CEP (Storm only): Custom logic for executing on data over a CEP window.

StreamAnalytix makes custom code implementation very easy by providing: a simple interface to import and use custom components, and features like versioning that allows you to version your custom code.

Steps for developing Custom Code

Steps for registering Custom Code

Functions

Functions enables you to enrich an incoming message with additional data not originally provided by the source.

System Defined Functions: StreamAnalytix provides a rich library of system-defined functions for Date, String, Lookup, Object.

User Defined Functions: StreamAnalytix provides a rich library of pre-defined function, and you can also define your own functions as user defined functions.

Variables

Allows you to use variables in your pipelines at runtime as per the scope.

Transformation

A transformation rule explains how the value of field in a message should be transformed to be used in predictive model.

Transformation variable is associated with a message and can be defined on any field in the message. Once it is defined, it can be used as any other message field, while defining the model.

Tables

Register Tables functionality enables you to store streaming data coming from different sources (AWS S3, HDFS, JDBC connection) in Tables, which can be queried using SparkSQL processor.

Models

Register Models functionality allows you to register the predictive models that are created using Spark Machine Learning API and are stored on HDFS or a Local file system.

Follow below steps to register the model with StreamAnalytix:

Step 1: Train the model on certain dataset and save it at any location on HDFS or local file system.

Step 2: Register a model.