ES DMSA

Overview

For an Elasticsearch with Distributed MSA, the node master and data status depends on the node role, this has to be decided at design time.

Reminder

  • Data node: this node stores only the data. When it receives a request from a client, it searches data from shards or creates an index.
  • Master node: this node's function is to maintain a cluster and request indexing or searches from data nodes.
  • Client node: the client node takes the search requests from the MSA web portal as well as from the MSA SecEngine event-based notification system. 
    The client node is also used by the SecEngine to index syslogs when the amount of data to index is small.

Configuration 

Node Name and Node ID

Each node is identified by an ID and knows its cluster by the cluster name.

Cluster name and node ID should be configured for each ES node of a cluster.

Make sure that the node ID is unique in the cluster and that the cluster name is the same for each node in the cluster:

  • name of the Elasticsearch cluster for this node (ubiqube)
  • ID of this Elasticsearch node (es_msa)

The Elasticsearch configuration is handled by the ubi-elasticsearch package which contains the configuration file: /etc/elasticsearch/elasticsearch.yml.

The ubi-configurator package contains certain required variables used in /etc/elasticsearch/elasticsearch.yml.

Launch the MSA configuration tool with option –expert and select option “Elasticsearch configuration”.

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

Elasticsearch configuration

1: Cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: Back
1

Cluster configuration

1: General
2: Network settings
3: Retention policy
4: Discovery
5: Advanced config: breakers and gateway

0: Back
1

General

1: Name of the Elasticsearch cluster for this node (Cluster_Name)
2: Version (2.4) (ES supported version)
3: ID (integer) of this Elasticsearch node (Node_Name)

0: back


Note:

The default cluster name is “ubiqube” and, unless you have several clusters in the same network, you can leave it as is.

The node ID ("es_msa" per default) should be unique in the cluster.

The directory to store the data can be left to the default value. It is possible to customize it in order to use a dedicated disk to store the data.


Configure the Node Master and Data Flags

Every node should be configured to be either a master node, a data node or a client node.

The role of a node is defined by the architecture choice.

  master flag value data flag value
master node true false
client node false false
data node false true

To set flags, launch the MSA configuration tool with option –expert and select option “Elasticsearch configuration”.

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
1

cluster configuration

1: general
2: network settings
3: retention policy
4: discovery
5: advanced config: breakers and gateway

0: back
1

general

4: set this node as a master node (true)
5: set this node as a data node (true)

8: set the directory where Elasticsearch will store data (/opt/elasticsearch/data)

0: back

Configure the Number of Shards and Replicas

The number of shards for the indexes should be configured on each Master Node. This number should be equal to the number of data nodes that your cluster will have (3 data node = 3 shards).

The number of replicas can be more than one if the cluster has more than one data node.

To configure the number of shards/replicas from master nodes:

# /opt/configurator/configure --expert --no-reconf

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
1

cluster configuration

1: general
2: network settings
3: retention policy
4: discovery
5: advanced config: breakers and gateway

0: back
1

general


6: set the number of shard (split) of an index (1)
7: set the number of replicas of an index. If the cluster is mono-node, this setting is irrelevant (0)
8: set the directory where Elasticsearch will store data (/opt/elasticsearch/data)

0: back

 

Note: When there is scaling up by adding a data node, an adjustment of the shard's number is required on each master node to keep the same number of data nodes.

Discovery

On each cluster node (including the master nodes) configure the list of master nodes:

 

# /opt/configurator/configure --expert --no-reconf

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
1

cluster configuration

1: general
2: network settings
3: retention policy
4: discovery
5: advanced config: breakers and gateway

0: back
2

network settings

2: Pass an initial list of hosts to perform discovery when new node is started (ES_Master_Node_IP_1,ES_Master_Node_IP_2)

0: back

 

Note: The next step will be to update the configuration file in accordiance with the previous variable's setting. Once all configuration variable are done for each node, run this command:

 

# /opt/ubi-elasticsearch/configure

It will update the /etc/elasticsearch/elasticsearch.yml

Configure the OS network settings

Each nodes should be able to talk to each other on port 9200 and 9300.

 

When the system is cloned the Mac address of the network card changes and it’s necessary to change the configuration files:

  • Remove the networking interface rules files so that it can be regenerated

    # rm -f /etc/udev/rules.d/70-persistent-net.rule

     

  • Restart the VM

    # reboot

     

  • Update your interface configuration files eth0, eth1,..

    # vim /etc/sysconfig/networking/devices/ifcfg-eth0

    Remove the MACADDR entry or update it to the new MACADDR for the interface (listed in this file: /etc/udev/rules.d/70-persistent-net.rules).

    Remove the UUID entry

    Save and exit the file

  • Restart the networking service

    # service network restart

     


Configure the Elasticsearch Service

Make sure that the Elasticsearch service is set to on in chkconfig:

# chkconfig elasticsearch on

Start the service:

# service elasticsearch start

 

Check the Cluster and Node(s) Status

Restart the Elasticsearch service and check the log /var/log/elasticsearch/<CLUSTER_NAME>.log to make sure that Elasticsearch starts properly.

To verify the cluster and node status by command line:

#  curl -X GET 'ES_IP:9200/_cluster/health?pretty'

# curl -X GET 'localhost:9200/_nodes?pretty'

Note:

 With Elasticseach version 2.4 node status can be checked by the kopf plugin from Elasticsearch.

http://<ES_IP>:9200/_plugin/kopf

With Elasticseach version 5.6 node status can be checked by the cerebro application.

http://<ES_IP>:9000


Then:

http://<ES_IP>:9200

MSA Configuration

The MSA configurator exposes 2 main options to configure access to Elasticsearch:

  1. The indexing endpoint

  2. The search endpoint

Configuration Option

Indexer endpoint configuration

The indexing endpoint is the ES node that will receive the bulk indexing request from the SecEngine. It should be a client node with a data flag and a master flag set to false.

On each SecEngine node from MSA configuration CLI, set the IP of the indexing endpoint (it should be the client node):

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
3

SEC Engine

1: Private IP address of the log indexer service (ES_Client_Node_IP)

0: back


With Distributed-MSA, it is enough to use the same endpoint for each SecEngine node.

When scaling up, it's very likely that there will be several data nodes in the cluster in order to handle the growing need for indexing more and more logs.

It is possible (and advised) to leverage the data nodes to run the log indexing processes.

On each SecEngine node, provide the list of data node IPs:

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
3

SEC Engine


2: List of ElasticSearch data nodes IP addresses (ES_Data_Node_IP_1,ES_Data_Node_IP_N)

0: back

 

How to check that indexing is working

Once the SecEngine is configured, in the web portal node, you need to create and activate a managed device to send syslogs to the SecEngine (gold monitoring and optionally detailed reporting should be activated).
After that, the data sent to Elasticsearch can be checked in the MSA.

 # tail -f /opt/sms/logs/ElasticSearchBulk.log

Search endpoint configuration

The search endpoint is the ES node that receives the search request.

It should be a client node with a data flag and a master flag set to false.

On each web portal node, from MSA configuration CLI:

# /opt/configurator/configure --expert

UBIqube SOC Configuration Menu

1: System configuration
2: Web Portal configuration
3: JEntreprise configuration
4: SEC Engine configuration
5: Reports configuration
6: Database configuration
7: Alarms and events notifications
8: SOC Customisation
9: Zero Touch Deployment
10: OSS BSS third party tools integration
11: ElasticSearch configuration

0: save & exit (use CTRL-C to exit without saving)
11

ElasticSearch configuration

1: cluster configuration
2: Web portal (SES and JENTREPRISE)
3: SEC Engine

0: back
2

Web portal (SES and JENTREPRISE)

1: IP of the log search service (ES_Client_Node_IP)

0: back

Once D-MSA configuration is done, each SecEngine should be restarted by service ubi-sms restart.