How to handle stale process

Hello

I had this issue on a specific case : I was using a poller on a “burstable” vm on azure with 10%cpu limit (model B1), 

this was causing process to get stuck and never ending because I did not have cpu credit to run when I started monitoring more and more services.

is it your case?

if it is, try allocating more resources

if not :

I have never see that behaviour, unless your script is simply not exiting and sending it’s exit code/output text

could you send your engine configuration with the “Freshness” in “check options”, and your “tuning”, maybe something is not correctly set up there.

also your check command line

(does your script exit correctly and with the right exit code when you run the command manually on the poller shell? you can check the exit code with “echo $?” right after your command)

the centreon engine is a scheduler that will run the command and wait for the exit code, if it never arrives it should timeout at somepoint

also it should not wait for the previous execution to finish and should run at each normal check interval

even if the previous one is not exited. (I’m not 100% sure about this)

That reminds me, and you should start there, what is your “normal check interval” on these services (or service template if you use one)

you can check this by looking at this on the resource detail

the “Next Check” is 5min after the last check, and my normal check interval is “5”

is your “next check” correctly scheduled with the “normal check interval” ?

to your question, there is no best way to manage stale process, there shouldn’t be any :-/ 

this will list you the pid of any process named “yourprocessname” running for more than 1h, (3600s)

 ps -C yourprocessname -o pid,etimes,command | awk '(NR>1){if($2>3600) print $1}'

you could kill the list

but if it’s “perl” or “php” or “python” it could have bad consequences.

killing a process launched by centreon engine should result in a “no ouput” Unknown status, I’m not sure if that have an impact on the engine, it should not, the process are exiting on their own normally, having something that kill such process should not cause issue (except the unknown status)

and the crontab is not really an issue on the whole, you can add whatever you want there

Hi @christophe.niel-ACT thank you for your answer

is it your case?

 it might be my case. My resources for this poller are being like this:

I have never see that behaviour, unless your script is simply not exiting and sending it’s exit code/output text

I have check one of my scripts that behave like this and it has an exit line for every condition

the centreon engine is a scheduler that will run the command and wait for the exit code, if it never arrives it should timeout at somepoint

Indeed, it should timeout and return exit code 3, but you can see it remains ok (code 0)

also it should not wait for the previous execution to finish and should run at each normal check interval

looks like it only check for the next execution when I kill the stuck process

ps -C yourprocessname -o pid,etimes,command | awk '(NR>1){if($2>3600) print $1}'

I have managed to write a working script using a similar filter executed by centreon-engine, but I really should find the root cause of this resource performance.

Thank you very much