MC DDOS protection part 3: Monitoring
So now our tunnels are set up, our round-robin SRV records are distributing our players across our VPS's, next it would be nice if we could check the traffic passing across them, and also keep an eye on the system stats for each machine.
In this section we're going to look at the basic structure of Zabbix, the terminology, and adding custom graphs.
Basics
Zabbix is made up of 3 parts:
Server Core
- Receives and processes stats from each server
- Stores the data
- Can send alerts based on certain criteria ("triggers")
Server Web Interface
- Displays graphs and information about the servers
- Can be on a separate machine from the core
Zabbix Agent
- Runs on each of the servers you want to monitor
- Reports the servers status to Zabbix
- Can be customised with extra scripts
Installing zabbix core and web interface
To install Zabbix you will need a webserver and a mysql database. Your webserver can be on another machine to the one you're installing zabbix server on, you just need to tell it where during the configuration.
Zabbix has some good documentation so I won't re-write it, the version I used is here but feel free to use a later release, the only thing that's important is that you don't use the default from the apt repository: it will most likely be < v2.0 and not support interface auto discovery properly.
You will also need to enable access to port 10051 incoming to your zabbix server, either through your firewall or its iptables.
Installing zabbix agent on your servers
As above: add the repository, update, and "apt-get install zabbix-agent".
Few extra notes:
- Make sure you're on a version later than 2.0 with "zabbix-agent --version"
- Change "Server=127.0.0.1" in /etc/zabbix/zabbix_agentd.conf to your zabbix server IP
- Restart the agent with "/etc/init.d/zabbix_agentd restart"
- Allow port 10050 through iptables with " iptables -A INPUT -p tcp --dport 10050 -j ACCEPT"
Host Configuration
Assuming you've now installed Zabbix and aren't screaming at me for not including more detail, you're probably now staring at the web GUI wondering where to start. Here are the basic terms you need to know:
Items
- Measure a specific value on the server, e.g. CPU load
Applications
- Group Items into categories, e.g. "OS Memory" = all memory items.
Discovery Rules
- Auto-generate new items by scanning the system for extra hard drives, interfaces, etc.
Triggers
- Perform an event when an item matches certain criteria, e.g. send an email when CPU load exceeds 100%
Graphs
- Self explanatory, graph an item or set of items on a host.
Screens
- Can be used to create a page with multiple graphs.
Templates
- Group all of the above together into a package that can be easily added to a new host with minimal extra configuration.
Add new host
New hosts can be added via configuration > Hosts > Create host
Give the host a descriptive name, add it to "linux servers" group, and enter it's IP address. You then need to add a template to tell it what to monitor. "Template OS Linux" covers everything we need for our VPS, including an interface discovery script for our GRE tunnels.
Save your new host, and go back to Configuration > Hosts. After a few minutes you should see Status: Monitored and a green Z icon indicating that everything is fine.
If you have a grey icon, you may have not added any templates (hence nothing to monitor.
If there is a red icon then there is most likely an issue with communication to the host, hover over the red icon to see the details, check that your firewall rules are allowing the correct ports, and that the IP's are correct.
Graphing:
If you navigate to Monitoring > Graphs and select your new host. There should be a graph available with a name like "Network traffic on gre1", which if you have any players connected will look something like this:
Using this page when can now switch between each host we've added and view the bandwidth usage on each interface. Great start, but it would be nice if we could combine the information into one place, which we'll do with a custom graph.
Creating a custom graph
Since our MC server host has an interface for each tunnel, we can add each item for each tunnel onto a new custom graph:
Go to Configuration > Hosts and select "Graphs" on the entry for your MC server.
Select "Create Graph" in the top right corner.
You can leave everything default except the name and the Items.
Select "add items" and then select each incoming and outgoing item for your GRE interfaces. You can change any colours you want, and preview your changes in the preview tab.
Once you're happy with the result, save the result and your new graph should appear under Monitoring > Graphs.
Screens
Depending on how many VPS's you have and which interfaces you have selected, your graph may look a bit messy, even with some custom colours assigned to interfaces. Another option is to use a screen to display the data, with each interfaces as a separate graph, but all on the same page.
To do this go to Configuration > Screens and "Create Screen."
Give it a name and decide an initial size, you can change it later.
Save it and it should appear in the list of screens: select your screen and you will be presented with a grid and an option to change the data for each one.
Select the data you want and you're done, you can now view your new screen under Monitoring > Screens:
And we're done for now. Thanks for reading, next up: traffic shaping on our VPS's