Read Data from File

Log Agent reads files data from remote data sources and ingests the data into Kafka or RabbitMQ channels.

Log Agent is installed on a remote machine and has three java processes running internally, LogAgent, LogThread, and Main. LogAgent process reads system logs, whereas LogThread and Main processes read all other types of logs.

Steps to install Log Agent on a machine

  • Extract StreamAnalytix bundle, and configure agent.yaml file located at /StreamAnalytix/conf/yaml.
  • Extract logstash tar located at /StreamAnalytix/lib folder, and configure property logstash.bin.script.location in agent.yaml file.
  • Run the log agent using following commands:

$ sh logAgent.sh install

$ sh logAgent.sh start

Note: Install command is required only for the first time when Log Agent is to be installed, after that you need to run only start command.

Agent Configuration

Kafka

On the Agent Configuration page,  select message broker  as Kafka and provide the path from where to read log files to configure a log agent.

Field

Description

Message

Name of the message to be used in the pipeline

Message Broker

Kafka

Connection Name

Select a connection for the message broker

Topic Name

Message broker topic name

Replication Factor

Number of replications

Partitions

Number of partitions

Directory or File Path

Path from where to read logs. It is the path of directory on the server, for ex:

 On Windows: c:\apache\apache.error.log,

 On UNIX: /var/log/foo.log

Read Log File

Whether log files should be read from the beginning to the end or from end  to the beginning. Beginning and End options specifies reading point of a new file. This option is applicable only for the first read operation. For successive read operations data will be fetched from the end of the last read

Encrypt Message

If Yes, messages are encrypted before sending it to the broker

Apply Filter

Filter messages for processing based on a given criteria

 

RabbitMQ

On the Agent Configuration page,  select message broker  as RabbitMQ and provide the path from where to read log files to configure a log agent.

Field

Description

Message

Name of the message to be used in the pipeline.

Message Broker

Select  Message Broker as RabbitMQ.

Connection Name

Select a connection for the message broker.

Exchange Name

Is the Exchange Name where data is pushed.

Exchange Type

Type of exchange where data is pushed.

Exchange Durable

If RabbitMQ is selected as broker, and if this property is set to True then the exchange will not be deleted if you restart RabbitMQ .

Routing Key

Routing key helps in routing specific data  to specific queue  based on binding using this key.

Queue Name

Name of the queue where data is stored.

Queue Durable

If this property is set to True then the exchange will not be deleted if you restart RabbitMQ

Add Config Queue

Additional RabbitMQ properties can be configured using Add Config Queue.

Directory or File Path

Path from where to read logs. It is the path of directory on the server, for example,  

 On Windows: c:\apache\apache.error.log,

 On UNIX: /var/log/foo.log

Read Log File

Whether log files should be read from the beginning to the end or from end to the beginning. Beginning and End options specifies reading point of a new file. This option is applicable only for the first read operation. For successive read operations data will be fetched from the end of the last read

Encrypt Message

If Yes, messages are encrypted before sending it to the broker

Apply Filter

Filter messages for processing based on a given criteria