[MBI] Handle ETL rebuild through web interface

Hello,

problem: 

BI ETL rebuild is quite difficult to use. Not user friendly at all and doesn’t meet the standard level of quality that we should aim at. 

• documentation is fine but difficult to understand because the ETL concepts are not monitoring concepts and people are not familiar with that
• script usage is prone to error (not running it in nohup and losing ssh connexion, running them with wrong arguments...)

solution: 

do better! (easy isn’t it?) 

We now have all the technologies to build a good UX.

1. Put gorgone on the BI server, create a gorgone module called “centreon-bi-etl” since gorgone and etl are all Perl, it can just call the ETL. (could be a good opportunity to refine the ETL. There may have some optimisations to do)
2. Gorgone on the central server is going to send orders to gorgone on the BI server. Since it can only work using async communication, gorgone on the BI server must send errors, logs and end of execution to gorgone on the central server
3. Do a well crafted web interface that will help people build the ETL orders that gorgone must send. It must work in a full async mode, gorgone on the central server must be able to communicate all information to the web interface hours after the user has made the action.

Features: 

This is not a comprehensive list

• web interface must parse results of centreonBIMonitoring --db-content script and compute them to preconfigure the action that the user will have to do. For example [Table mod_bi_hostavailability, last entry: 2020-07-01 00:00:00] means that the web interface needs to guide the user through a host data import, host event rebuild without purge since 2020-07-01 (that means no data bin import, when using importdata.pl, and then use the build dimension script and then the build event script and not the perfdata one)
• web interface must have an easy wizard that clearly displays what are the consequences of the actions. For example, the web interface ask you 
  • do you want to rebuild or apply a new configuration to old data ? (we can even explain what’s the meaning of applying a new configuration to old data)
  • what kind of data do you want to work on ? events, perfdata, both  ? (we can even explain what’s the purpose of those types of rebuild
  • rebuild starting date is and end date is ? Since start and end date are confusing (we never now if the date is included or excluded) we can compute on the fly the meaning of that. For exemple start date = 2022-03-10 means that it will include datas from 2022-03-10 00:00:00 and end date = 2022-03-22 means that it will include datas until 2022-03-22 00:00:00
  • do you want to purge data that are outside of the selected time range ? (no should be ticked by default)
  • verbosity of the action ? (error, debug...)
  • before running this it should : display an overview of what you’ve configured. You should have a clear warning message if purge is ticked to yes
  • ​​​​before running you should have the list of commands that gorgone will run in case you would like to do that manually (could be very useful when trying to rebuild huge amount of data
• when a rebuild is ran through the web interface, the bi-engine cron must be disabled. It must be enabled back when the rebuild is done and a centreonBIMonitoring --db-content must be run automatically. And if things are not up to date, the web interface should compute things for the user as stated in the first bullet point
• If a rebuild fails, the logs must be available through the web interface (see the solution part of this topic). There are numerous reason why it could fail and it can be quite tricky to solve this. In order to help people, it should display the link to the Centreon support website and it must have a button that will copy in the clipboard all the required information for the support. This means: 
  • centreon versions (web + gorgone + bi)
  • the configuration made by the user using the wizard
  • the commands that gorgone must run according to this configuration
  • the logs of the operation
  • the result of a the biMonitoring --db-content command
• it must have a “task history” menu. Meaning that a rebuild run by gorgone should have a kind of task id. When clicking on an already finished task you should be able to see: 
  • centreon versions (web + gorgone + bi)
  • the configuration made by the user using the wizard
  • the commands that gorgone must run according to this configuration
  • the logs of the operation
  • the result of a the biMonitoring --db-content command

----------------------------------------------------

I think that’s all the ideas I have at the moment. Maybe I’ll do a mockup/POC on my free time for the sake of fun and learn more about gorgone.