How to generate a coredump when centengine or cbd crash

The following procedure describes how to configure coredumpctl to retrieve core dumps if a segfault happens.

Generate a core dump during a segfault

Activate core dump generation

Before retrieving a core dump, you must configure the system for it to generate core dumps at segfault.

The first step consists in installing gdb.

On Centos7:

yum install devtoolset-9-gdb

ln -s /opt/rh/devtoolset-9/root/bin/{gdb,gcore,gdb-add-index,gstack,pstack} /usr/bin/

For other distributions, use your OS software manager, for examples:

yum install gdb

or

apt install gdb

Once this step done, you have to change several things in your configuration:

1. Uncomment "DumpCore=yes" in "/etc/systemd/system.conf"
2. Replace "DefaultLimitCORE=" by "DefaultLimitCORE=infinity" in "/etc/systemd/system.conf"
3. Add these lines in "/etc/sysctl.conf":

   kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e 
   fs.suid_dumpable=2
   

4. Add "LimitCORE=infinity" in "/usr/lib/systemd/system/cbd.service" just after the line "User"
5. Add "LimitCORE=infinity" in "/usr/lib/systemd/system/centengine.service" just after the line "User"

Restart systemd to commit the changes:

sysctl -p
systemctl daemon-reexec

After activating core dumps generation, you must wait for a segfault to occur. You will see them in the system's log /var/log/messages.

1. List the core dumps:

   coredumpctl list

2. Retrieve the core dump:

   coredumpctl -o filename.core dump /usr/sbin/centengine

This command will store the last coredump of /usr/sbin/centengine to a file named filename.core.

You can do the same with /usr/sbin/cbd.

Generate a coredump manually

Sometimes, it can be interesting to generate a coredump manually. For example if centreon-engine looks blocked but not crashed.

The procedure for that is to install gdb on the server.

On Centos7: (the command for the installation and the one to update the path)

yum install devtoolset-9-gdb
source /opt/rh/devtoolset-9/enable

On other distributions, use the package manager, for example:

yum install gdb or apt install gdb

Then the generation is made in two steps:

1. Get the PID of the program you want a coredump ; here is an example with centengine:

   ps ax | grep centengine
   2272 ?        Ssl   33:43 /usr/sbin/centengine /etc/centreon-engine/centengine.cfg

    

2. Generate the coredump:

   gcore 2272

   where 2272 is the PID you get at the step 1.

   In the current directory, you should find a new file core.2272 that is the coredump.
    

3. You can now send the coredump to the support of Centreon for more investigation.