Server Monitoring with Grafana, Prometheus and node_exporter

server monitoring is now a mandatory that must be done for every individual / group / company that owns a server, whether on-premise or on-cloud. Especially if the servers that are owned are many. We need a tool or application that can help monitor it all. Therefore, Grafana is here to meet these needs. Quoted from Wikipedia – “Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources”. So with Grafana we can prepare all the needs for server monitoring. Therefore, in this article I will share information about Server Monitoring with Grafana, Prometheus and node_exporter.

Server Monitoring with Grafana (Concept) ¶

Before starting all kinds of things related to installation. I will explain the concept first regarding server monitoring with Grafana, Prometheus and node_exporter. Look at the following picture :

Server Monitoring with Grafana, Prometheus and node_exporter (Explanation of the Concept) ¶

From the picture above, I want to give an example of implementing server monitoring with grafana, prometheus and node_exporter. Suppose we have several servers, like the example above, there are servers a, b, c, d, e and f. We want to monitor all of these servers simultaneously. We need to put in place centralized access to monitor it. in the picture above, the user monitors via server a. Where on server a, Grafana is installed (for interactive visualization), Prometheus (for processing metrics data), and node_exporter (for collecting metrics data). Then what about servers b, c, d, e and f? For other than server a, we only need to install node_exporter, provided that node_exporter can be accessed by prometheus on server a. So that the user simply monitors it by accessing grafana via a browser which will point to server a, where prometheus server a will visualize all the data that node_exporter has obtained from all servers (b, c, d, e and f). Then we just go straight to the installation steps.

Install Prometheus ¶

The first thing we install is prometheus. Prometheus is an open-source application that we will use to monitor the system. Prometheus has tons of measurements we need to monitor servers. Then for the visualization later we will combine it with grafana. For how to install it, please read my previous post on How to Install Prometheus on Linux for Server Monitoring.

Install node_exporter ¶

The second one we install is node_exporter. node_exporter is a daemon that functions to collect metrics / data on the server that we have, which prometheus will process the data. node_exporter will be installed to every node server that we have. as an illustration, node_exporter on each server will send their respective server metrics data and then send it to the Prometheus server for processing. For how to install it, please read my previous post on How to Install node_exporter on Linux for Data Server Monitoring.

Connecting Prometheus with node_exporter ¶

Next, if you have installed promethues and node_exporter we will connect them. Prometheus can access metrics data from other servers that have node_exporter installed by adding it to the prometheus.yml configuration file. Simply add a job for this node_exporter

  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['IP_SERVER:9100']

into the prometheus.yml configuration file after the jobs section for prometheus.

Then, based on the example of the explanation concept that I have explained, more or less the prometheus.yml configuration file will be like this:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['IP_SERVER_A:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['IP_SERVER_A:9100', 'IP_SERVER_B:9100', 'IP_SERVER_C:9100', 'IP_SERVER_D:9100', 'IP_SERVER_E:9100', 'IP_SERVER_F:9100']

If each server has node_exporter installed, we just need to add it to - targets: by giving a comma like the example above. After finishing with the configuration file, restart the prometheus service.

systemctl restart prometheus.

When it’s finished, we try to access Prometheus

http://IP_ADDR_or_DOMAIN:9090

Then we execute one of the queries / expressions, namely “up” to just test whether prometheus can access the metrics of all servers installed with node_exporter. If it is connected, it will be like this:

 

Install Grafana ¶

Install Prometheus already, install node_exporter already, connect prometheus with node_exporter as well. great job! Now we move on to the grafana installation.

Install grafana on CentOS 7 ¶

For CentOS 7, here’s how:

• Create grafana repository

sudo nano /etc/yum.repos.d/grafana.repo

• fill in this

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

• update repository, then install with yum

yum update
yum install grafana

• activate, run and check status grafana

systemctl enable grafana-server
systemctl start grafana-server
systemctl status grafana-server

Install grafana on ubuntu ¶

For Ubuntu, here’s how:

• Add and install the APT repository for grafana

sudo apt-get install -y gnupg2 curl software-properties-common
curl https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"

• Update repo and install grafana

sudo apt-get update
sudo apt-get install grafana

• activate, run and check status grafana

systemctl enable grafana-server
systemctl start grafana-server
systemctl status grafana-server

Allow ports on firewall (optional) ¶

If you manage your port access with a firewall, you need to open port 3000 for grafana access. Here are a few ways to open ports for grafana:

• allow ports on ufw firewall

sudo ufw allow 3000/tcp

• allow ports on iptables

iptables -A INPUT -p tcp --dport 3000 -j ACCEPT
iptables-save | sudo tee /etc/sysconfig/iptables
systemctl restart iptables

Disable registration and anonymous access (optional) ¶

Usually, I disable this feature by going to the grafana configuration file in  /etc/grafana/grafana.ini. Find allow_sign_up under [user] , remove the ; (semicolon) lthen type false into that value. like this :

...
[users]
# disable user signup / registration
allow_sign_up = false
...

Then find enabled under [auth.anonymous] , remove the ; (semicolon) then type false into that value. like this :

...
[auth.anonymous]
# enable anonymous access
enabled = false
...

When it’s finished,, restart grafana and check grafana status

systemctl restart grafana-server
systemctl status grafana-server

Access grafana, then update user and password ¶

After the installation is complete, then we need to update the grafana admin password. Access grafana at http://IP_ADDR_or_DOMAIN:3000 , then login using user: admin and password: admin.

then you will be asked to change the admin password,

If you have successfully changed the password, you will be directed to the grafana homepage

So far we have successfully logged in and accessed Grafana. However, we can’t monitor anything because the dashboard is still empty. Next, let’s connect Prometheus with Grafana so that we can create a server monitoring dashboard.

 

Add prometheus datasource to Grafana ¶

Before creating the dashboard, we need to add Prometheus as the data source that we will use to create the dashboard.

• First of all, we go to the Configuration menu, then click Add data source,

 

• Next, we Select Prometheus as the datasource

 

• Fill in the configuration section as needed, most importantly fill in the prometheus URL with http://IP_ADDR_or_DOMAIN:9090 according to where our prometheus is.

 

• Then click Save & Test to save and test whether the configuration is successful.

 

• If the configuration is successful, a notification will appear Data source is working

At this point we have successfully added the prometheus source data to the grafana. Next we will try to make a simple monitoring dashboard.

 

Create simple monitoring dashboard ¶

I will create a simple dashboard example. Approximately the dashboard display will be like this:

From the GIF animation above, we can see, I created a dashboard to monitor my 3 servers, namely main-server, node-a and node-b. To make it easier to say, we initialize main-server as $server-1, node-a as $server-2 and node-b as $server-3.

 

To shorten the time, you can download the JSON file below which I have created to upload on Grafana to become a dashboard like above. But, before you use it, change the values of $server-1, $server-2 and $server-3 to be the IP address of each of your own servers.

→ Download simple-dashboard

If you have adjusted it, you can accept the following steps :

• Go to Grafana, access menu Dashboard > Manage > Import
• Upload JSON file or Import via panel json , to upload the json file you only need to upload the json file that you downloaded and adjusted the value, then to import via the json panel, you only need to paste the json code that you have adjusted into the form that you have adjusted. provided.
• then click Import

For iImport via panel json the process is more or less like the following GIF animation :

Okay, you have succeeded in making a simple server monitoring dashboard. If there is a chance I will try to explain how the dashboard is made.

 

Alright, it looks like I’ve been explaining Server Monitoring for a while with Grafana, Prometheus and node_exporter. If there is anything you want to ask, please ask in the comments column. Hopefully my writing can be useful, Thank you very much 😀 😎