ES Standalone MSA

Overview

There are 2 ways to setup an ES Standalone MSA.

Mode 1 - The MSA and Elasticsearch run on the same machine.

Image

Mode 2 - The MSA and Elasticsearch run on separate machines.

Image

For both modes, the configuration will be the same.

1. Configuration

Configuration steps can be started, once your Elasticsearch, ubi-configurator and ubi-elasticsearch RPM are installed in your ES (with Mode 1 or 2 ).

a. Node Configuration

The Elasticsearch configuration is handled by the ubi-elasticsearch package which contains the configuration file : /etc/elasticsearch/elasticsearch.yml.

The ubi-configurator package contains certain required variables used in /etc/elasticsearch/elasticsearch.yml.

Launch the MSA configuration tool with option –-expert and select option “Elasticsearch configuration”.

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
1

cluster configuration

1: general
2: network settings
3: retention policy
4: discovery
5: advanced config: breakers and gateway

0: back
1

general

1: name of the elasticsearch cluster for this node (Cluster_Name)
2: Version (2.4) (ES supported version)
3: ID (integer) of this elasticsearch node (Node_Name)
4: set this node as a master node (true)
5: set this node as a data node (true)
6: set the number of shard (split) of an index (1)
7: set the number of replicas of an index. If the cluster is mono-node, this setting is irrelevant (0)
8: set the directory where Elasticsearch will store data (/opt/elasticsearch/data)
9: activate the daily index management script. This script will create the replica if needed, optimize the index... (true/false) (true)

0: back


Note:

The default cluster name is “ubiqube” and unless you have several clusters in the same network, you can leave it as is.

The node ID ("es_msa" per default) should be unique in the cluster.

For a single node setup, there should be only one shard and the replica should be set to 0.

The directory to store the data can be left at the default value. It is possible to customize it in order to use a dedicated disk to store the data.


b. Update Elasticsearch Configuration File

To update the configuration file in accordance with the previous variable's setting, run this command:

# /opt/ubi-elasticsearch/configure

It will update the /etc/elasticsearch/elasticsearch.yml

c. Starting Elasticsearch Cluster

# service elasticsearch start

# service elasticsearch status

d. Apply the Mapping

Before, make sure that Elasticsearch is running.

# /opt/ubi-elasticsearch/configure

2. Check the Cluster and Node(s) Status

Restart the Elasticsearch service and check the log /var/log/elasticsearch/<CLUSTER_NAME>.log to make sure that Elasticsearch starts properly.

To verify the cluster and node status by command line:

# curl -X GET 'ES_IP:9200/_cluster/health?pretty'

# curl -X GET 'localhost:9200/_nodes?pretty'

Note: With Elasticseach version 2.4 node status can be checked by the kopf plugin from Elasticsearch:

http://<ES_IP>:9200/_plugin/kopf

With Elasticseach version 5.6 node status can be checked by the cerebro application:

http://<ES_IP>:9000

then

http://<ES_IP>:9200

Warning : From ESv5.x if you are using a single node, you will need to uncomment this line " #discovery.type: single-node" in /etc/elasticsearch/elasticsearch.yml

3. MSA Configuration

a. Configuration Option

The MSA configurator allows 2 main options to configure access to Elasticsearch:

  • The indexing endpoint
  • The search endpoint

i. Indexer endpoint configuration

The indexing endpoint is the ES node that will receive the bulk indexing request from the SecEngine

From MSA configuration CLI:

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
2

Web portal (SES and JENTREPRISE)

1: IP of the log search service (ES_IP)

0: back

ii. How to check that indexing is working

Once the SecEngine is configured, in the web portal you need to create and activate a managed device to send syslogs to the SecEngine (gold monitoring and optionally detailed reporting should be activated). After that the data sent to Elasticsearch can be checked in the MSA.

    # tail -f /opt/sms/logs/ElasticSearchBulk.log
    

iii. Search endpoint configuration

The search endpoint is the ES node that receives the search request.

From MSA configuration CLI:

a

Note: When the MSA uses only one ES node to index, store and search for the data simply use the same endpoint IP for each configuration item.

NB: Once MSA configuration is done, the SecEngine should be restarted by service ubi-sms restart.