In order to get started with StreamAnalytix, the first step is to setup StreamAnalytix by creating a workspace and adding component connections.
Workspace provides multi-tenancy support as you can access data pipelines and other menus only from your workspace. All the StreamAnalytix users (except Superuser) are allotted a workspace. One user can be allocated to multiple workspaces.
To create a workspace, login as a Superuser and go to Workspace page.
Click on Create New Workspace and assign users to the workspace.
Note: You can create new user from the workspace creation form or select from already create users.
Connections allow you to connect to infrastructure components like RabbitMQ, Elastic search etc.
To add a connection, go to Connections page, and click on Add Connection.
Select a Component Type, and provide connection details. For example, below are the details for a RabbitMQ connection:
Build your First Pipeline
For the Quick Start guide, we will implement a Stock Tick Analyzer to monitor data coming from multiple sources i.e. stock exchanges. The analyzer filters the stock tick data and generate alerts whenever a stock of interest crosses a defined threshold value.
Step 1: Create a Message Group and define the message schema.
Go to Group Configuration and click on Create New Group. Create a message group by providing a name as shown below.
Create a Message to define the message schema by selecting the message type and adding the corresponding fields.
To add fields to the message, go to Field Configuration tab.
Step 2: Configure Alerts to get alerts on the Stock price.
Go to Configuration tab on the Alerts page and Create New Alert.
Step 3: Create a Spark Pipeline and test it against input data.
On the Data Pipeline page, go to Spark tab and click Create New Pipeline.
Drag RabbitMQ Channel, Filter Processor, Alert Processor and RabbitMQ Emitter to the canvas by selecting them from the right-hand panel and connect the components.
Right-click on each component to configure their properties.
Data Pipeline page lists all the pipelines. Click the Start button to start the pipeline.
To monitor a pipeline’s health, click on the View Summary link on the pipeline card.
For system monitoring, go to Spark tab under Monitoring. This allows you to view the summary for our StockTickSparkPipieline with different kinds of parameters.
Jobs: shows the Active, Completed or Skipped stages of the jobs.
Stages: each stage can be further viewed w.r.t Bytes, Records, shuffle in bytes, etc., for all stages.
Executors: details of Active, Completed and Failed tasks of executors.