Solved

Pollers and Central not running after upgrade from 21.04 to 21.10

  • 20 April 2022
  • 5 replies
  • 3642 views
quentinrey
Badge +5

Bonjour,

Je viens d’effectuer un upgrade de notre Centreon et des Pollers (1 Central et 2 Pollers).

La montée de version 20.10 à 21.04 s’est passée correctement.

Par contre j’ai un soucis au passage de la 21.04 à la 21.10 (Doc suivie : Montée de version depuis Centreon 21.04 | Centreon Documentation)

Après la mise à jour, l’export de la configuration et la relance des services, mon Central et mes Pollers semblent ne pas communiquer.

ENGLISH
Hello,

I have just upgraded our Centreon and Pollers (1 Central and 2 Pollers).

The upgrade from version 20.10 to 21.04 went well.

On the other hand I have a problem with the upgrade from 21.04 to 21.10

After the update, the export of the configuration and the restart of the services, my Central and my Pollers seem not to communicate.

 

Les versions de leurs centengine respectifs ne semblent pas bons, et je n’arrive pas à savoir pourquoi cela ne communique pas, et je ne sais pas trop quelles logs regarder.

L’état des différents services sur le Central :

ENGLISH
The versions of their respective centengines don't look good, and I can't figure out why it's not communicating, and I'm not sure which logs to look at.

The status of the various services on the Central :

# systemctl status cbd centengine centreontrapd gorgoned
● cbd.service - Centreon Broker watchdog
   Loaded: loaded (/usr/lib/systemd/system/cbd.service; enabled; vendor preset: disabled)
   Active: active (running) since mer. 2022-04-20 18:02:49 CEST; 18min ago
  Process: 14537 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 13477 (cbwd)
   CGroup: /system.slice/cbd.service
           ├─13477 /usr/sbin/cbwd /etc/centreon-broker/watchdog.json
           └─13479 /usr/sbin/cbd /etc/centreon-broker/central-rrd.json

avril 20 18:20:43 Charybde cbwd[13477]: [2022-04-20 18:20:43.931] [core] [info] Copyright 2009-2021 Centreon
avril 20 18:20:43 Charybde cbwd[13477]: [2022-04-20 18:20:43.931] [core] [info] License ASL 2.0 <http://www.apache.org/licenses/LICENSE-2.0>
avril 20 18:20:48 Charybde cbwd[13477]: [2022-04-20T18:20:48.929+02:00] [cbwd] [error] cbd instance with PID 23061 has stopped, attempt ...start it
avril 20 18:20:48 Charybde cbwd[13477]: [2022-04-20 18:20:48.932] [core] [info] Centreon Broker 21.10.0
avril 20 18:20:48 Charybde cbwd[13477]: [2022-04-20 18:20:48.932] [core] [info] Copyright 2009-2021 Centreon
avril 20 18:20:48 Charybde cbwd[13477]: [2022-04-20 18:20:48.932] [core] [info] License ASL 2.0 <http://www.apache.org/licenses/LICENSE-2.0>
avril 20 18:20:53 Charybde cbwd[13477]: [2022-04-20T18:20:53.930+02:00] [cbwd] [error] cbd instance with PID 23112 has stopped, attempt ...start it
avril 20 18:20:53 Charybde cbwd[13477]: [2022-04-20 18:20:53.933] [core] [info] Centreon Broker 21.10.0
avril 20 18:20:53 Charybde cbwd[13477]: [2022-04-20 18:20:53.933] [core] [info] Copyright 2009-2021 Centreon
avril 20 18:20:53 Charybde cbwd[13477]: [2022-04-20 18:20:53.933] [core] [info] License ASL 2.0 <http://www.apache.org/licenses/LICENSE-2.0>

● centengine.service - Centreon Engine
   Loaded: loaded (/usr/lib/systemd/system/centengine.service; enabled; vendor preset: disabled)
   Active: active (running) since mer. 2022-04-20 18:05:05 CEST; 15min ago
  Process: 3819 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 14569 (centengine)
   CGroup: /system.slice/centengine.service
           └─14569 /usr/sbin/centengine /etc/centreon-engine/centengine.cfg

avril 20 18:05:05 Charybde centengine[14569]: [1650470705] [14569] Processing object config file '/etc/centreon-engine/connectors.cfg'
avril 20 18:05:05 Charybde centengine[14569]: [1650470705] [14569] Processing object config file '/etc/centreon-engine/meta_commands.cfg'
avril 20 18:05:05 Charybde centengine[14569]: [1650470705] [14569] Processing object config file '/etc/centreon-engine/meta_timeperiod.cfg'
avril 20 18:05:05 Charybde centengine[14569]: [1650470705] [14569] Processing object config file '/etc/centreon-engine/meta_host.cfg'
avril 20 18:05:05 Charybde centengine[14569]: [1650470705] [14569] Processing object config file '/etc/centreon-engine/meta_services.cfg'
avril 20 18:05:05 Charybde centengine[14569]: [1650470705] [14569] Reading resource file '/etc/centreon-engine/resource.cfg'
avril 20 18:05:05 Charybde centengine[14569]: [2022-04-20 18:05:05.540] [core] [info] Starting the TCP thread pool of 4 threads
avril 20 18:05:05 Charybde centengine[14569]: [2022-04-20 18:05:05.541] [core] [info] protocols: registering protocol ('file' (layers 1-3)
avril 20 18:05:05 Charybde centengine[14569]: [2022-04-20 18:05:05.541] [core] [info] protocols: registering protocol ('compression' (layers 6-6)
avril 20 18:05:05 Charybde centengine[14569]: [2022-04-20 18:05:05.541] [core] [info] protocols: registering protocol ('BBDO' (layers 7-7)

● centreontrapd.service - Centreon Trapd Daemon is a Centreon program that manage traps
   Loaded: loaded (/usr/lib/systemd/system/centreontrapd.service; enabled; vendor preset: disabled)
   Active: active (running) since mer. 2022-04-20 17:51:13 CEST; 29min ago
 Main PID: 5872 (centreontrapd)
   CGroup: /system.slice/centreontrapd.service
           └─5872 /usr/bin/perl /usr/share/centreon/bin/centreontrapd --logfile=/var/log/centreon/centreontrapd.log --severity=error --config=/e...

avril 20 17:51:13 Charybde systemd[1]: Started Centreon Trapd Daemon is a Centreon program that manage traps.

● gorgoned.service - Centreon Gorgone
   Loaded: loaded (/etc/systemd/system/gorgoned.service; enabled; vendor preset: disabled)
   Active: active (running) since mer. 2022-04-20 17:51:19 CEST; 29min ago
 Main PID: 5958 (perl)
   CGroup: /system.slice/gorgoned.service
           ├─5958 /usr/bin/perl /usr/bin/gorgoned --config=/etc/centreon-gorgone/config.yaml --logfile=/var/log/centreon-gorgone/gorgoned.log --...
           ├─5973 gorgone-audit
           ├─5976 gorgone-nodes
           ├─5987 gorgone-dbcleaner
           ├─5994 gorgone-autodiscovery
           ├─5995 gorgone-cron
           ├─6008 gorgone-engine
           ├─6009 gorgone-statistics
           ├─6010 gorgone-action
           ├─6030 gorgone-httpserver
           ├─6032 gorgone-legacycmd
           ├─6061 gorgone-proxy
           ├─6068 gorgone-proxy
           ├─6075 gorgone-proxy
           ├─6082 gorgone-proxy
           └─6089 gorgone-proxy

avril 20 17:51:19 Charybde systemd[1]: Started Centreon Gorgone.
avril 20 18:05:04 Charybde sudo[14558]: centreon-gorgone : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/service centengine restart
Hint: Some lines were ellipsized, use -l to show in full.

 

EDIT : Ajout de logs

Central - /var/log/centreon-broker/central-module-master.log :

[2022-04-20T18:55:52.043+02:00] [tcp] [error] TCP: Connection refused to localhost:5669
[2022-04-20T18:55:52.043+02:00] [core] [error] failover: global error: failover: 'central-module-master-output' cannot connect endpoint.
[2022-04-20T18:56:22.047+02:00] [tcp] [error] TCP: Connection refused to localhost:5669
[2022-04-20T18:56:22.047+02:00] [core] [error] failover: global error: failover: 'central-module-master-output' cannot connect endpoint.
[2022-04-20T18:56:52.051+02:00] [tcp] [error] TCP: Connection refused to localhost:5669
[2022-04-20T18:56:52.051+02:00] [core] [error] failover: global error: failover: 'central-module-master-output' cannot connect endpoint.
[2022-04-20T18:57:22.055+02:00] [tcp] [error] TCP: Connection refused to localhost:5669
[2022-04-20T18:57:22.055+02:00] [core] [error] failover: global error: failover: 'central-module-master-output' cannot connect endpoint.
[2022-04-20T18:57:52.059+02:00] [tcp] [error] TCP: Connection refused to localhost:5669
[2022-04-20T18:57:52.059+02:00] [core] [error] failover: global error: failover: 'central-module-master-output' cannot connect endpoint.

2 Pollers - /var/log/centreon-broker/poller.log

[2022-04-20T18:59:21.488+02:00] [tcp] [error] TCP: Connection refused to [My_Central_IP]:5669
[2022-04-20T18:59:21.488+02:00] [core] [error] failover: global error: failover: 'Central_connection' cannot connect endpoint.
[2022-04-20T18:59:51.493+02:00] [tcp] [error] TCP: Connection refused to [My_Central_IP]:5669
[2022-04-20T18:59:51.493+02:00] [core] [error] failover: global error: failover: 'Central_connection' cannot connect endpoint.
[2022-04-20T19:00:21.498+02:00] [tcp] [error] TCP: Connection refused to [My_Central_IP]:5669
[2022-04-20T19:00:21.498+02:00] [core] [error] failover: global error: failover: 'Central_connection' cannot connect endpoint.
[2022-04-20T19:00:51.503+02:00] [tcp] [error] TCP: Connection refused to [My_Central_IP]:5669
[2022-04-20T19:00:51.503+02:00] [core] [error] failover: global error: failover: 'Central_connection' cannot connect endpoint.
[2022-04-20T19:01:21.508+02:00] [tcp] [error] TCP: Connection refused to [My_Central_IP]:5669
[2022-04-20T19:01:21.508+02:00] [core] [error] failover: global error: failover: 'Central_connection' cannot connect endpoint.

Merci par avance pour votre aide !

icon

Best answer by quentinrey 21 April 2022, 11:01

View original

5 replies

K
Userlevel 5
Badge +16

Hello @quentinrey ,

Thanks you for all the logs.

It seems that the “CBD sql” is not running:

           ├─13477 /usr/sbin/cbwd /etc/centreon-broker/watchdog.json
           └─13479 /usr/sbin/cbd /etc/centreon-broker/central-rrd.json

You should have something like this:

      ├─20916 /usr/sbin/cbwd /etc/centreon-broker/watchdog.json
           ├─20917 /usr/sbin/cbd /etc/centreon-broker/central-broker.json
           └─20918 /usr/sbin/cbd /etc/centreon-broker/central-rrd.json

So could we take a look of the log /var/log/centreon-broker/central-broker-master.log ? which is the log of CBD sql.

Regards,

quentinrey
Badge +5

Hello @quentinrey ,

Thanks you for all the logs.

It seems that the “CBD sql” is not running:

           ├─13477 /usr/sbin/cbwd /etc/centreon-broker/watchdog.json
           └─13479 /usr/sbin/cbd /etc/centreon-broker/central-rrd.json

You should have something like this:

      ├─20916 /usr/sbin/cbwd /etc/centreon-broker/watchdog.json
           ├─20917 /usr/sbin/cbd /etc/centreon-broker/central-broker.json
           └─20918 /usr/sbin/cbd /etc/centreon-broker/central-rrd.json

So could we take a look of the log /var/log/centreon-broker/central-broker-master.log ? which is the log of CBD sql.

Regards,

Here is the log:

[2022-04-21T10:25:13.990+02:00] [core] [info] /var/log/centreon-broker//central-broker-master.log : log started
[2022-04-21T10:25:13.990+02:00] [core] [info] Starting the TCP thread pool of 4 threads
[2022-04-21T10:25:13.990+02:00] [core] [info] protocols: registering protocol ('file' (layers 1-3)
[2022-04-21T10:25:13.990+02:00] [core] [info] protocols: registering protocol ('compression' (layers 6-6)
[2022-04-21T10:25:13.990+02:00] [core] [info] protocols: registering protocol ('BBDO' (layers 7-7)
[2022-04-21T10:25:13.990+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/50-tcp.so'
[2022-04-21T10:25:13.991+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/10-neb.so'
[2022-04-21T10:25:13.992+02:00] [core] [info] NEB: module for Centreon Broker 21.10.0
[2022-04-21T10:25:13.992+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/60-tls.so'
[2022-04-21T10:25:13.996+02:00] [core] [info] protocols: registering protocol ('TLS' (layers 5-5)
[2022-04-21T10:25:13.996+02:00] [core] [info] protocols: registering protocol ('TCP' (layers 1-4)
[2022-04-21T10:25:13.996+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/80-sql.so'
[2022-04-21T10:25:13.996+02:00] [core] [info] modules: attempt to load '/usr/share/centreon/lib/centreon-broker/10-neb.so' which is already loaded
[2022-04-21T10:25:13.996+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/20-storage.so'
[2022-04-21T10:25:13.997+02:00] [core] [info] modules: attempt to load '/usr/share/centreon/lib/centreon-broker/10-neb.so' which is already loaded
[2022-04-21T10:25:13.997+02:00] [core] [info] protocols: registering protocol ('storage' (layers 1-7)
[2022-04-21T10:25:13.997+02:00] [core] [info] protocols: registering protocol ('SQL' (layers 1-7)
[2022-04-21T10:25:13.997+02:00] [core] [info] modules: attempt to load '/usr/share/centreon/lib/centreon-broker/20-storage.so' which is already loaded
[2022-04-21T10:25:13.997+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/15-stats.so'
[2022-04-21T10:25:13.998+02:00] [core] [info] Destroying muxer central-broker-master-input: number of events in the queue: 0
[2022-04-21T10:25:14.000+02:00] [core] [error] Error during cbd exit: core: cannot find event category 'bam'

 

quentinrey
Badge +5

Here is the /var/log/centreon-broker/central-broker-master.log log:

[2022-04-21T10:25:13.990+02:00] [core] [info] /var/log/centreon-broker//central-broker-master.log : log started
[2022-04-21T10:25:13.990+02:00] [core] [info] Starting the TCP thread pool of 4 threads
[2022-04-21T10:25:13.990+02:00] [core] [info] protocols: registering protocol ('file' (layers 1-3)
[2022-04-21T10:25:13.990+02:00] [core] [info] protocols: registering protocol ('compression' (layers 6-6)
[2022-04-21T10:25:13.990+02:00] [core] [info] protocols: registering protocol ('BBDO' (layers 7-7)
[2022-04-21T10:25:13.990+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/50-tcp.so'
[2022-04-21T10:25:13.991+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/10-neb.so'
[2022-04-21T10:25:13.992+02:00] [core] [info] NEB: module for Centreon Broker 21.10.0
[2022-04-21T10:25:13.992+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/60-tls.so'
[2022-04-21T10:25:13.996+02:00] [core] [info] protocols: registering protocol ('TLS' (layers 5-5)
[2022-04-21T10:25:13.996+02:00] [core] [info] protocols: registering protocol ('TCP' (layers 1-4)
[2022-04-21T10:25:13.996+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/80-sql.so'
[2022-04-21T10:25:13.996+02:00] [core] [info] modules: attempt to load '/usr/share/centreon/lib/centreon-broker/10-neb.so' which is already loaded
[2022-04-21T10:25:13.996+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/20-storage.so'
[2022-04-21T10:25:13.997+02:00] [core] [info] modules: attempt to load '/usr/share/centreon/lib/centreon-broker/10-neb.so' which is already loaded
[2022-04-21T10:25:13.997+02:00] [core] [info] protocols: registering protocol ('storage' (layers 1-7)
[2022-04-21T10:25:13.997+02:00] [core] [info] protocols: registering protocol ('SQL' (layers 1-7)
[2022-04-21T10:25:13.997+02:00] [core] [info] modules: attempt to load '/usr/share/centreon/lib/centreon-broker/20-storage.so' which is already loaded
[2022-04-21T10:25:13.997+02:00] [core] [info] modules: attempt to load module '/usr/share/centreon/lib/centreon-broker/15-stats.so'
[2022-04-21T10:25:13.998+02:00] [core] [info] Destroying muxer central-broker-master-input: number of events in the queue: 0
[2022-04-21T10:25:14.000+02:00] [core] [error] Error during cbd exit: core: cannot find event category 'bam'

I tried this :

[root@Charybde ~]# /usr/sbin/cbd -d /etc/centreon-broker/central-broker.json
[2022-04-21 10:40:50.622] [core] [info] Centreon Broker 21.10.0
[2022-04-21 10:40:50.622] [core] [info] Copyright 2009-2021 Centreon
[2022-04-21 10:40:50.622] [core] [info] License ASL 2.0 <http://www.apache.org/licenses/LICENSE-2.0>
Erreur de segmentation

Erreur de segmentation → Segmentation error/faults

quentinrey
Badge +5

Solved the issue, it was a bit tricky.

In the Centreon Broker Configuration, the central-broker-master had some filter on one of the Output :

Just deleted them, then restart the services, and it works!

K
Userlevel 5
Badge +16

Good to know :)

For you information, this behavior is a known issue and it will be fix/release soon.

Regards,
Kriko.

Reply


Terms & ConditionsCookie settings