Pipeline Management

StreamAnalytix provides tools, such as Import/Export, Pipeline Integration, Versioning, Data Lineage, Pipeline Inspect, and Error Mining, to manage an application pipelines at all the stages of the application lifecycle.

Import Export Pipeline

Export Pipeline (Download Pipeline)

Export Pipeline feature enables you to download a pipeline in a file along with all its component configurations such as Message Configuration, Group Configuration, and Alert Configuration and Registered entities.

Import Pipeline (Upload Pipeline)

Import Pipeline feature enables you to import a downloaded pipeline along with its component configurations such as messages, groups, alert definitions and registered entities.​

 

Data Lineage

The Data Lineage feature is capable of capturing the life cycle of data that goes through pipeline components. It enables tracking progress of data at every stage when it flows in the pipeline.  It represents how information flows between various data points. Data Lineage provides complete audit trail of the data from its source to its destination. At a very high level, Data Lineage provides which systems the data interacts with before it reaches the final destination.

It provides visibility into the analytics Pipeline and assists in tracking errors. It helps in step by step debugging of the data flow and regenerating the lost output.

As the volume of the data multiplies in an organization, information about data becomes crucial. Data Lineage approach works like an X-ray for data flow in an organization. It captures information from source to destination of various processes and rules involved and depicts how data is being used.

 

Pipeline Inspect

Pipeline inspection is a mechanism through which the processing components of a pipeline can be investigated. It helps in identifying the functionality of individual components, like what meticulous enrichments or processing is happening with input of component and what is the output of the component.

It gives ease to visualize the input and output of individual components and to compare individual message fields, to a granular level. It helps to build or calibrate the business pipelines by reviewing the individual components.

 

Pipeline Versioning

StreamAnalytix enables you to version your pipelines.

You can create a new version for a pipeline while saving the pipeline after editing it.

 

Error Mining

StreamAnalytix enables you to view errors in your pipelines on the Application Errors tab.

Errors may occur due to various reasons such as failure in any component, inaccurate data or trouble with a component connection.

 

Pipeline Integration

StreamAnalytix platform allows you to integrate two or more pipelines to create a bigger pipeline for complex business use cases.

 

Pipeline Cluster Isolation

Admin users can use Cluster Isolation feature to allocate a dedicated set of machines, out of the machines allocated to the Workspace by the Superuser, to run selected pipelines in the workspace.