Question

Debug bad formatted performance data

  • 24 November 2022
  • 4 replies
  • 293 views

Userlevel 1
Badge +6

Hello,

I see frequently ibd files growing on my DB du to bad formatted performance data collected (space added is the major issue).

Searching thoose bad data is really painfull in broker logs.
Is there any other way to identify easily thoose data?

 

Thx


4 replies

Userlevel 4
Badge +13

👋

 

@nicnictout there are a couple of things to take into consideration:

  • confirm that the purge is working / scheduled
    • /etc/cron.d/centstorage
      • /bin/php /usr/share/centreon/cron/centstorage_purge.php
  • adjust your retention parameters
  • look into the perfdata to fix the format
    • confirm the plugin/device produces the right perfdata
    • look for the host/service that might be consistan for the one service/host and fix as needed
      • For example the error bellow corresponds to host id 1234 and service id 98765 
[YYYY-MM-DDTHH:MM:01.005-00:00] [perfdata] [error] invalid perfdata format in service (1234:98765): equal sign not present or misplaced '| FAN1_RPM='

🤞

Userlevel 1
Badge +6

Hello,

 

ok that’s already the workflow I had to identify bad datas, I’ve hope there were a way with maybe a DB query to identify services in trouble more easily.

I’ll developp a logfile template or script in order to generate a report as expected so.

 

Thks for your time @ponchoh 

Userlevel 4
Badge +13

will 😍 to see it out of curiosity, tag me if you do 😎!

Userlevel 1
Badge +6

Hello @ponchoh 

 

sorry for delay, I finnaly write a short script to proceed:

#!/bin/bash
#
# Script to detect bad formatted performance data sent to Centreon
#
broker_log="/var/log/centreon-broker/central-broker-master.log"
API_login=yyyy
API_pwd=xxxx
CMD_CENTREON="/usr/share/centreon/bin/centreon -u ${API_login} -p ${API_pwd} -o SERVICE -a SHOW"
regex=".*invalid perfdata format in service \((.*):(.*)\):.*"

# temp files
tmp_result="/tmp/perf_tmp"
sort_result="/tmp/perf_sort"
perf_result="/tmp/perf_result"

# Cleaning
rm -f /tmp/perf_*

# populate an array with all entris found in broker log
while IFS= read -r line; do
        my_array+=( "$line" )
done < <(grep "invalid perfdata format" $broker_log | sed -E "s/$regex/\1;\2/")

# echo all entries one per line
for i in ${my_array[@]}
do
  echo $i >> $tmp_result
done

# remove duplicates
cat $tmp_result | sort -u > $sort_result

# search services infos with clapi
for line in $(cat $sort_result)
do
  host=$(echo $line | awk -F';' '{print $1}')
  service=$(echo $line | awk -F';' '{print $2}')
  `${CMD_CENTREON} | grep "$host;.*;$service;" >> $perf_result`
done

if [ -a $perf_result ]
then
  cat $perf_result | /usr/bin/mailx -s "Centreon bad Perf Data format" <email>

fi

 

As you see not a breakthrough stuff, but at least it provide an easier and clearer output with host and service name rather than just ids.

Reply