Partition /var/lib/centreon-broker is full.

Hi @Didier97180 , if this partition is full, you may have issue with Centreon Broker: can insert data into database or can’t write performance data into RRD files.

You must check errors in /var/log/centreon-broker.*.log files to fix issue, then restart “cbd” process and wait to retention analysis.

Hello,

In the files log

[2024-04-07 03:30:59.761] [core] [error] failover: global error: Attempt to read data from peer "Central":5669 on a closing socket
[2024-04-07 03:31:01.768] [core] [error] failover: global error: Attempt to read data from peer "Central":5669 on a closing socket
[2024-04-07 03:31:03.765] [core] [error] failover: global error: Attempt to read data from peer "Central":5669 on a closing socket
[2024-04-07 03:31:05.763] [core] [error] failover: global error: Attempt to read data from peer "Central":5669 on a closing socket
[2024-04-07 03:31:07.763] [core] [error] failover: global error: Attempt to read data from peer "Central":5669 on a closing socket
[2024-04-07 03:31:09.763] [core] [error] failover: global error: Attempt to read data from peer "Central":5669 on a closing socketCentral":5669 on a closing socket

In the files log

Did you try to restart cbd service:

systemctl restart cbd

Hello,

that didn't solve the problem
In addition on the central I can see my poller clearly (because it is on the pooler that there is the problem) but the date of the last update is December 2023

So the issue with full FS is on the poller and data collected by this poller are up-to-date?

Can you give us the content o: cat /var/lib/centreon-engine/<poller_name>-stats.json?

{"asio_version":"1.24.0","endpoint central-module-master-output":{"bbdo_input_ack_limit":1000.0,"bbdo_unacknowledged_events":0.0,"event_processing_speed":2.838709677419355,"failover":null,"last_connection_attempt":1713353909.0,"last_connection_success":1713353909.0,"last_event_at":1713354494.0,"memory_file_path":"/var/lib/centreon-engine//sv36cent2-module.memory.central-module-master-output","name":"central-module-master-output","queue_file_enabled":false,"queue_file_path":"/var/lib/centreon-engine//sv36cent2-module.queue.central-module-master-output","queued_events":798,"read_filters":"all","state":"connected","status":"reading event from multiplexing engine","unacknowledged_events":798,"write_filters":"all"},"module/usr/share/centreon/lib/centreon-broker/10-neb.so":{"name":"/usr/share/centreon/lib/centreon-broker/10-neb.so","size":"857664B","state":"loaded"},"module/usr/share/centreon/lib/centreon-broker/15-stats.so":{"name":"/usr/share/centreon/lib/centreon-broker/15-stats.so","size":"692136B","state":"loaded"},"module/usr/share/centreon/lib/centreon-broker/50-tcp.so":{"name":"/usr/share/centreon/lib/centreon-broker/50-tcp.so","size":"887816B","state":"loaded"},"module/usr/share/centreon/lib/centreon-broker/60-tls.so":{"name":"/usr/share/centreon/lib/centreon-broker/60-tls.so","size":"593848B","state":"loaded"},"mysql manager":{"delay since last check":"0"},"now":"1713354495","pid":1500,"thread_pool":{"latency":"","size":3},"version":"21.10.5"}

From this statistics you don’t have retention, so I don’t understand why your FS is full.

Can you list files present in var/lib/centreon-broker directory?

total 5102780
-rw-rw-r-- 1 centreon-broker centreon-broker      259 25 mai    2023 central-broker-master.memory.central-broker-master-sql
-rw-rw-r-- 1 centreon-broker centreon-broker        8 25 mai    2023 central-broker-master.unprocessed
-rw-rw-r-- 1 centreon-broker centreon-broker        8 25 mai    2023 central-rrd-master.unprocessed
-rw-rw-r-- 1 centreon-broker centreon-broker 99999774 15 avril 16:25 Pooler-Name-broker.queue.forward-to-master
-rw-rw-r-- 1 centreon-broker centreon-broker 99999801 24 déc.  14:52 Pooler-Name-broker.queue.forward-to-master1
-rw-rw-r-- 1 centreon-broker centreon-broker 99999936 14 janv. 12:10 Pooler-Name-broker.queue.forward-to-master10
-rw-rw-r-- 1 centreon-broker centreon-broker 99999958 16 janv. 19:49 Pooler-Name-broker.queue.forward-to-master11
-rw-rw-r-- 1 centreon-broker centreon-broker 99999877 19 janv. 03:28 Pooler-Name-broker.queue.forward-to-master12
-rw-rw-r-- 1 centreon-broker centreon-broker 99999901 21 janv. 11:07 Pooler-Name-broker.queue.forward-to-master13
-rw-rw-r-- 1 centreon-broker centreon-broker 99999922 23 janv. 18:46 Pooler-Name-broker.queue.forward-to-master14
-rw-rw-r-- 1 centreon-broker centreon-broker 99999811 26 janv. 02:25 Pooler-Name-broker.queue.forward-to-master15
-rw-rw-r-- 1 centreon-broker centreon-broker 99999840 28 janv. 10:05 Pooler-Name-broker.queue.forward-to-master16
-rw-rw-r-- 1 centreon-broker centreon-broker 99999889 30 janv. 17:44 Pooler-Name-broker.queue.forward-to-master17
-rw-rw-r-- 1 centreon-broker centreon-broker 99999846  2 févr. 01:24 Pooler-Name-broker.queue.forward-to-master18
-rw-rw-r-- 1 centreon-broker centreon-broker 99999849  4 févr. 09:03 Pooler-Name-broker.queue.forward-to-master19
-rw-rw-r-- 1 centreon-broker centreon-broker 99999859 26 déc.  22:37 Pooler-Name-broker.queue.forward-to-master2
-rw-rw-r-- 1 centreon-broker centreon-broker 99999937  6 févr. 16:43 Pooler-Name-broker.queue.forward-to-master20
-rw-rw-r-- 1 centreon-broker centreon-broker 99999981  9 févr. 00:22 Pooler-Name-broker.queue.forward-to-master21
-rw-rw-r-- 1 centreon-broker centreon-broker 99999610 11 févr. 08:00 Pooler-Name-broker.queue.forward-to-master22
-rw-rw-r-- 1 centreon-broker centreon-broker 99999901 13 févr. 15:39 Pooler-Name-broker.queue.forward-to-master23
-rw-rw-r-- 1 centreon-broker centreon-broker 99999929 15 févr. 23:18 Pooler-Name-broker.queue.forward-to-master24
-rw-rw-r-- 1 centreon-broker centreon-broker 99999730 18 févr. 06:55 Pooler-Name-broker.queue.forward-to-master25
-rw-rw-r-- 1 centreon-broker centreon-broker 99999858 20 févr. 14:33 Pooler-Name-broker.queue.forward-to-master26
-rw-rw-r-- 1 centreon-broker centreon-broker 99999914 22 févr. 22:12 Pooler-Name-broker.queue.forward-to-master27
-rw-rw-r-- 1 centreon-broker centreon-broker 99999977 25 févr. 05:49 Pooler-Name-broker.queue.forward-to-master28
-rw-rw-r-- 1 centreon-broker centreon-broker 99999983 27 févr. 13:26 Pooler-Name-broker.queue.forward-to-master29
-rw-rw-r-- 1 centreon-broker centreon-broker 99999989 29 déc.  06:23 Pooler-Name-broker.queue.forward-to-master3
-rw-rw-r-- 1 centreon-broker centreon-broker 99999939 29 févr. 21:05 Pooler-Name-broker.queue.forward-to-master30
-rw-rw-r-- 1 centreon-broker centreon-broker 99999996  3 mars  04:43 Pooler-Name-broker.queue.forward-to-master31
-rw-rw-r-- 1 centreon-broker centreon-broker 99999990  5 mars  12:20 Pooler-Name-broker.queue.forward-to-master32
-rw-rw-r-- 1 centreon-broker centreon-broker 99999950  7 mars  20:01 Pooler-Name-broker.queue.forward-to-master33
-rw-rw-r-- 1 centreon-broker centreon-broker 99999998 10 mars  03:41 Pooler-Name-broker.queue.forward-to-master34
-rw-rw-r-- 1 centreon-broker centreon-broker 99999809 12 mars  11:19 Pooler-Name-broker.queue.forward-to-master35
-rw-rw-r-- 1 centreon-broker centreon-broker 99999931 14 mars  18:59 Pooler-Name-broker.queue.forward-to-master36
-rw-rw-r-- 1 centreon-broker centreon-broker 99999765 17 mars  02:42 Pooler-Name-broker.queue.forward-to-master37
-rw-rw-r-- 1 centreon-broker centreon-broker 99999896 19 mars  10:23 Pooler-Name-broker.queue.forward-to-master38
-rw-rw-r-- 1 centreon-broker centreon-broker 99999902 21 mars  18:04 Pooler-Name-broker.queue.forward-to-master39
-rw-rw-r-- 1 centreon-broker centreon-broker 99999848 31 déc.  14:06 Pooler-Name-broker.queue.forward-to-master4
-rw-rw-r-- 1 centreon-broker centreon-broker 99999903 24 mars  01:47 Pooler-Name-broker.queue.forward-to-master40
-rw-rw-r-- 1 centreon-broker centreon-broker 99999893 26 mars  09:30 Pooler-Name-broker.queue.forward-to-master41
-rw-rw-r-- 1 centreon-broker centreon-broker 99999939 28 mars  17:06 Pooler-Name-broker.queue.forward-to-master42
-rw-rw-r-- 1 centreon-broker centreon-broker 99999922 31 mars  00:40 Pooler-Name-broker.queue.forward-to-master43
-rw-rw-r-- 1 centreon-broker centreon-broker 99999841  2 avril 09:18 Pooler-Name-broker.queue.forward-to-master44
-rw-rw-r-- 1 centreon-broker centreon-broker 99999827  4 avril 16:51 Pooler-Name-broker.queue.forward-to-master45
-rw-rw-r-- 1 centreon-broker centreon-broker 99999935  7 avril 00:26 Pooler-Name-broker.queue.forward-to-master46
-rw-rw-r-- 1 centreon-broker centreon-broker 99999811  9 avril 08:01 Pooler-Name-broker.queue.forward-to-master47
-rw-rw-r-- 1 centreon-broker centreon-broker 99999841 11 avril 15:36 Pooler-Name-broker.queue.forward-to-master48
-rw-rw-r-- 1 centreon-broker centreon-broker 99999939 13 avril 23:12 Pooler-Name-broker.queue.forward-to-master49
-rw-rw-r-- 1 centreon-broker centreon-broker 99999824  2 janv. 21:49 Pooler-Name-broker.queue.forward-to-master5
-rw-rw-r-- 1 centreon-broker centreon-broker 99999975 16 avril 06:43 Pooler-Name-broker.queue.forward-to-master50
-rw-rw-r-- 1 centreon-broker centreon-broker 59451143 17 avril 15:47 Pooler-Name-broker.queue.forward-to-master51
-rw-rw-r-- 1 centreon-broker centreon-broker 99999805  5 janv. 05:31 Pooler-Name-broker.queue.forward-to-master6
-rw-rw-r-- 1 centreon-broker centreon-broker 99999919  7 janv. 13:11 Pooler-Name-broker.queue.forward-to-master7
-rw-rw-r-- 1 centreon-broker centreon-broker 99999737  9 janv. 20:51 Pooler-Name-broker.queue.forward-to-master8
-rw-rw-r-- 1 centreon-broker centreon-broker 99999882 12 janv. 04:30 Pooler-Name-broker.queue.forward-to-master9
prw-rw-r-- 1 centreon-broker centreon-broker        0 17 avril 13:38 Pooler-Name-broker-stats.json
-rw-rw-r-- 1 centreon-broker centreon-broker        8 17 avril 13:38 Pooler-Name-broker.unprocessed
prw-rw-r-- 1 centreon-broker centreon-broker        0 17 avril 13:38 Pooler-Name-rrd-stats.json
-rw-rw-r-- 1 centreon-broker centreon-broker        8 17 avril 13:38 Pooler-Name-rrd.unprocessed

pooler-name was replaced by my server name

It look like some time, there is a disconnection between your poller and your central server and when files (Pooler-Name-broker.queue.forward-to-master*)are processed after reconnection, they are not deleted.

You can remove them:

rm /var/lib/centreon-broker/Pooler-Name-broker.queue.forward-to-master* -f
systemctl restart centengine

But your poller is using Centreon 21.10 and this version is no more supported.

Please see Product lifecycle policy.

I emptied it but the last update is still from December 2023 on the central

it is planned to migrate the collector

Hello,

I already had a remote server in CentOs7

I reinstalled a new server in Alma 9 and when trying to put it into service I had this problem.

The remote server is "Not Running" only on itself, but appears online on the central server.

Central:

Remote server:
 

The communication works because when I restart the remote I can see the uptime updating on the central.

I checked all my services they are up: CBD, centre-engine, centreon, gorgoned, centreontrapd By checking in /var/log/Centreon-broker with the “poller-broker.log” file I have this error message: [warning] Poller '' with id 0 already known as connected. Replacing it with '' [tcp] [error] TCP: Connection refused to central:5669 [tcp] [error] Error while reading on socket from central: Operation canceled [tcp] [error] TCP: Connection refused to localhost:5670