ES Logging Management

ES Internal Log Configuration - Logging.yml

ES1.7 + 2.4 - /opt/ubi-elasticsearch/resources/templates/etc/elasticsearch/logging.yml (template)

(/etc/elasticsearch/logging.yml)

Base configuration - NO rotation configured (rotation of all ES logs will be managed by logrotate) - simply ES cluster logs being written to .log file

(note: red highlighted line are lines replaced/removed from the previous configuration)

file:
type: dailyRollingFile
type: file
file: ${path.logs}/${cluster.name}.log

maxFileSize: 100000
maxBackupIndex: 3

layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_search_slow_log_file:
type: dailyRollingFile

type: file

file: ${path.logs}/${cluster.name}_index_search_slowlog.log

datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_indexing_slow_log_file:
type: dailyRollingFile

type: file
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

ES5.6 - /etc/elasticsearch/log4j2.properties

Base configuration - NO rotation configured (rotation of all ES logs will be managed by logrotate) - simply ES cluster logs being written to .log file

(note: red highlighted lines are an example of the lines removed from one of the appenders - previous configuration)

--------------------------------------------------------------

status = error

# log action execution errors for easier debugging
logger.action.name = org.elasticsearch.action
logger.action.level = debug

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n

appender.rolling.type = File
appender.rolling.name = file
appender.rolling.fileName = /var/log/elasticsearch/DEV_ES_1622.log
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n

appender.rolling.filePattern = /var/log/elasticsearch/DEV_ES_1622-%d{yyyy-MM-dd}-%i.log
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = /var/log/elasticsearch
appender.rolling.strategy.action.condition.type = IfLastModified
appender.rolling.strategy.action.condition.age = 2D
appender.rolling.strategy.action.PathConditions.type = IfFileName
appender.rolling.strategy.action.PathConditions.glob = *
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 6MB
appender.rolling.strategy.max = 5

rootLogger.level = info

rootLogger.appenderRef.console.ref = console
rootLogger.appenderRef.rolling.ref = file

appender.deprecation_rolling.type = File
appender.deprecation_rolling.name = deprecation_file
appender.deprecation_rolling.fileName = /var/log/elasticsearch/DEV_ES_1622_deprecation.log
appender.deprecation_rolling.layout.type = PatternLayout
appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n

logger.deprecation.name = org.elasticsearch.deprecation
logger.deprecation.level = warn
logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_file
logger.deprecation.additivity = false

appender.index_search_slowlog_rolling.type = File
appender.index_search_slowlog_rolling.name = index_search_slowlog_file
appender.index_search_slowlog_rolling.fileName = /var/log/elasticsearch/DEV_ES_1622_index_search_slowlog.log
appender.index_search_slowlog_rolling.layout.type = PatternLayout
appender.index_search_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n

logger.index_search_slowlog_rolling.name = index.search.slowlog
logger.index_search_slowlog_rolling.level = trace
logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_file
logger.index_search_slowlog_rolling.additivity = false

appender.index_indexing_slowlog_rolling.type = File
appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_file
appender.index_indexing_slowlog_rolling.fileName = /var/log/elasticsearch/DEV_ES_1622_index_indexing_slowlog.log

appender.index_indexing_slowlog_rolling.layout.type = PatternLayout
appender.index_indexing_slowlog_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n

logger.index_indexing_slowlog.name = index.indexing.slowlog.index
logger.index_indexing_slowlog.level = trace
logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_file
logger.index_indexing_slowlog.additivity = false

--------------------------------------------------------------

Logrotate Configuration

(Note: as per cron.daily configuration, logrotate runs daily after 3am, meaning that file is NOT rotated at 00:00, so every new file starts to get logs from 03:XX) 

ES1.7 +2.4 - /opt/ubi-elasticsearch/resources/templates/etc/logrotate.d/elasticsearch (template)

(/etc/logrotate.d/elasticsearch)

(Note 1: size NOT used, since size makes daily useless

note 2: create NOT used, since copytruncate makes create useless

note 3: where $UBI_ES_CLUSTERNAME - configurator variable corresponding to the cluster name

note 4: rotate 30 (default NTT) - means 30 logs maintained - should be adjusted to the number of days logs should be retained)

http://ubibucket/projects/UBI/repos/elasticsearch/browse/resources/templates/etc/logrotate.d/elasticsearch?at=refs%2Fheads%2FBR__R15-3-20


/var/log/elasticsearch/snmp.log {

daily
rotate 30
# size 50M
copytruncate
compress
delaycompress
missingok
notifempty
# create 644 elasticsearch elasticsearch
}

/var/log/elasticsearch/log_retention.log {
daily
rotate 30
# size 50M
copytruncate
compress
delaycompress
missingok
notifempty
# create 644 elasticsearch elasticsearch
}

/var/log/elasticsearch/index_management.log {
daily
rotate 30
# size 50M
copytruncate
compress
delaycompress
missingok
notifempty
# create 644 elasticsearch elasticsearch
}

/var/log/elasticsearch/${UBI_ES_CLUSTERNAME}.log {
daily
rotate 30
copytruncate
compress
delaycompress
missingok
notifempty
}

/var/log/elasticsearch/${UBI_ES_CLUSTERNAME}_index_indexing_slowlog.log {
daily
rotate 30
copytruncate
compress
delaycompress
missingok
notifempty
}

/var/log/elasticsearch/${UBI_ES_CLUSTERNAME}_index_search_slowlog.log {
daily
rotate 30
copytruncate
compress
delaycompress
missingok
notifempty
}

ES5.6 - 1 Additional Files (+ same 5 files than ES1.7)

/var/log/elasticsearch/${UBI_ES_CLUSTERNAME}_deprecation.log {
daily
rotate 2
copytruncate
compress
delaycompress
missingok
notifempty
}

Sample Command to Run/Test Logrotate Manually

  • /usr/sbin/logrotate /etc/logrotate.d/elasticsearch

Additional INFO on Logrotate

  • logrotate default configuration - /etc/logrotate.conf
  • logrotate script running daily in cron - /etc/cron.daily/logrotate
  • cron daily/weekly/monthly configurations - /etc/anacrontab
  • file where is stored info on when last time logrotate run per file - /var/lib/logrotate.status