====== Prometheus monitoring ======

  * Grafana prometheus dashboard: https://grafana.com/grafana/dashboards/1860
  * Grafana prometheus percona mysqld dashboard: https://grafana.com/grafana/dashboards/7362
  * Prometheus rules collection: https://awesome-prometheus-alerts.grep.to/rules.html


===== Agent node_exporter install =====
<code bash>
sudo useradd node_exporter -s /sbin/nologin
mkdir -p /opt/prom-exporters
wget https://github.com/prometheus/node_exporter/releases/download/v*/node_exporter-*.*-amd64.tar.gz
tar xvfz node_exporter-*.*-amd64.tar.gz
sudo mv node_exporter-*.*-amd64/node_exporter /opt/prom-exporters/
rm -rf node_exporter-*.*-amd64
</code>

Create a systemd service file ''/etc/systemd/system/node_exporter.service'':
<code>
[Unit]
Description=Node Exporter

[Service]
User=node_exporter
EnvironmentFile=/etc/sysconfig/node_exporter
ExecStart=/opt/prom-exporters/node_exporter $OPTIONS

[Install]
WantedBy=multi-user.target
</code>

Create a sysconfig file:
<code bash>
sudo mkdir -p /etc/sysconfig
sudo echo "OPTIONS=\"--collector.textfile.directory /var/lib/node_exporter/textfile_collector\"" > /etc/sysconfig/node_exporter
</code>

Reload systemd configuration and configure node_exporter to auto-start on system boot:
<code bash>
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
</code>

Once the Node Exporter is installed, verify that metrics are being exported:
<code bash>
curl http://localhost:9100/metrics
</code>

===== Agent mysqld_exporter install =====

Prepare MySQL user for agent:
<code sql>
GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'mysqld_exporter'@'localhost' identified by 'My_Secure_Password';
GRANT SELECT ON performance_schema.* TO 'mysqld_exporter'@'localhost';
FLUSH PRIVILEGES;
</code>

Create a mysql password file ''/opt/prom-exporters/.mysqld_exporter.cnf'':
<code>
[client]
user=mysqld_exporter
password=My_Secure_Password
</code>

Change file owner and permission:
<code>
chown prometheus. /opt/prom-exporters/.mysqld_exporter.cnf
chmod 600 /opt/prom-exporters/.mysqld_exporter.cnf
</code>

Install mysqld_exporter:
<code bash>
wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.1/mysqld_exporter-0.12.1.linux-amd64.tar.gz
tar zxvf mysqld_exporter-0.12.1.linux-amd64.tar.gz
cd mysqld_exporter-0.12.1.linux-amd64
chown prometheus. mysqld_exporter
mv mysqld_exporter /opt/prom-exporters/
cd ..
rm -rf mysqld_exporter-0.12.1.linux-amd64*
</code>

Create a systemd service file ''/etc/systemd/system/mysqld_exporter.service'':
<code>
[Unit]
Description=Prometheus MySQL Exporter
After=network.target
User=prometheus
Group=prometheus

[Service]
Type=simple
Restart=always
ExecStart=/opt/prom-exporters/mysqld_exporter \
--config.my-cnf /opt/prom-exporters/.mysqld_exporter.cnf \
--collect.global_status \
--collect.info_schema.innodb_metrics \
--collect.auto_increment.columns \
--collect.info_schema.processlist \
--collect.binlog_size \
--collect.info_schema.tablestats \
--collect.global_variables \
--collect.info_schema.query_response_time \
--collect.info_schema.userstats \
--collect.info_schema.tables \
--collect.perf_schema.tablelocks \
--collect.perf_schema.file_events \
--collect.perf_schema.eventswaits \
--collect.perf_schema.indexiowaits \
--collect.perf_schema.tableiowaits \
--collect.slave_status \
--web.listen-address=0.0.0.0:9104

[Install]
WantedBy=multi-user.target
</code>

Reload systemd configuration and configure mysqld_exporter to auto-start on system boot:
<code bash>
sudo systemctl daemon-reload
sudo systemctl enable mysqld_exporter
sudo systemctl start mysqld_exporter
</code>

Once the MySQLd Exporter is installed, verify that metrics are being exported:
<code bash>
curl http://localhost:9104/metrics
</code>

===== Useful links =====
  * About monitoring: https://vc.ru/dev/62715-kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk