Elphel Development Blog

Hardening the Asterisk-based phone system

Last week our phone system was broken into and we’ve got a phone bill for some five hundred dollars for the calls to Gambia. That expense was not terrible, but still that amount is usually enough for many months of the phone service for our small company – the international phone rates in the VoIP era are (for the destinations we use) are really low. The scary thing was that the attack lasted for very short time – just minutes, not hours, so our damage could be significantly higher.

The system we used was just Trixbox CE  – phone system distribution that includes “everything” on a single installation – the CentOS version of GNU/Linux with standard services, Asterisk server, fork of the FreePBX web control panel as well as Flash operator panel. That was a rather basic installation that we did not make ourselves, and no other security measures were implemented in addition to just  changing the default passwords that I did myself – it turned  out to be not so easy task as there are  multiple default passwords in the Trixbox, some appear in many (and even many-many) files and it is not obvious – which occurrences have to be replaced. And if I understand correctly,  using multiple passwords in this case does not really improve the security – breaking just one of the passwords can grant you access to the system.

I do not know how this particular break in happened (far from being a professional in this area), but obviously the intruder eventually got the full access to the system, was able to modify the Asterisk configuration, removed the pin code for outgoing long distance calls and deleted some of the system log files.

After searching the Internet I understood that we got lucky to have such limited damage, so I decided to spend several days on trying to understand the related security risks and ways to mitigate them.

There are multiple resources about protecting Asterisk-base servers, but the solutions are spread over multiple sites. I liked the page Hacking and securing your Asterisk server – it includes instructions how to install and the software that can simulate break-in attempts, helping you to test the security of your server and how to improve it. It has also links to other resources, such as Seven Steps to Better SIP Security with  Asterisk (when using Asterisk with FreePBX or with Trixbox,  the file names where the changes should be made are different). Trying to play it safe, I opted for implementing multiple security measures, including:

1. Improving the asterisk configuration (mostly – understanding and modifying /etc/asterisk/sip_*.conf configuration files)
2. closing all unneeded ports on the server and limiting remote access to the services essential (like sshd, httpd ) only to the listed ip addresses, keeping them to the minimum. Anyway, if I would really need to access the server from a new location, I would always be able to ssh to one of the trusted (listed as such in my current configuration) computers and modify the list of the trusted ones.
3. detect and fight sip attacks coming from the IP addresses that I can not white- or black-list, so our users can connect from other places where IP addresses may be assigned dynamically.

Removing the trojan installed by Fonality (Trixbox only)

If you use Trixbox with the version above certain there likely is a cron job pointing to /var/adm/bin/registry.pl:

crontab -l
18 0 * * * /usr/bin/perl /var/adm/bin/registry.pl > /var/log/crontab.log

Every night that registry.pl program connects to the remote server, reads the list of commands and executes them as root. Even as this service is said to be designed just for spying (collecting information/statistics about the server), it still is a very dangerous security hole. You may disable this functionality by editing crontab with the command
crontab -e

(move the pointer to that line, twice press “D” key to delete the line, then press “Esc” key and enter “:wq” (without quotes). Alternatively you may just delete or rename the target  – /var/adm/bin/registry.pl .

Modifying Asterisk SIP configuration

The Seven Steps to Better SIP Security with  Asterisk discuss modifications to the sip.conf file. When using Asterisk with FreePBX or with Trixbox same modification need to be moved to other files, so they would apply and not interfere with (not be overwritten by) the operation of the web control panel.

One of the files to modify is /etc/asterisk/sip_custom_post.conf – this you can use to specify/overwrite settings automatically generated by there web control panel. Here you can put per-extension rules, that will limit the range of IP addresses from which the  particular extension can register (“deny” and “permit” lines below, 0.0.0.0/0.0.0.0 means “any”). In the example below extension 1236 is allowed to register from a single IP address 192.168.0.12. Next line is “call-limit” – by default in the auto-generated file /etc/asterisk/sip-additional.conf this limit was set to “50”, so somebody who cracked your extension password could theoretically run up to 50 calls simultaneously, and if each was directed to a satellite phone destination, it could cost you hundreds of dollars every minute.
Each extension settings start with the line [<EXTENSION>] followed by (+) – that means that the settings after will apply to <EXTENSION> in addition to (possibly overwriting) those that are specified before (i.e. in /etc/asterisk/sip-additional.conf ).
If needed you may add “[<EXTENSION>] (-)” section – that will delete the following parameter settings.
/etc/asterisk/sip_custom_post.conf :
...
[1236](+)
deny=0.0.0.0/0.0.0.0
permit=192.168.0.12/255.255.255.255
call-limit=1
...


Another setting that you would like to overwrite is “alwaysauthreject=yes” (it is “no” by default). That means that the server would ask for a password even for the non-existent extension (of course, it will reply as “wrong password” for any password),
that makes scanning for the extensions significantly more difficult – see Hacking...  This is a global setting that applies to all extensions. The right place for it is in /etc/asterisk/sip_custom.conf file – just  add (in my case the file was empty) the following line there:
alwaysauthreject=yes


After changes in the configuration files you need to restart the Asterisk server to apply them. You may do it differently, including pressing “Submit” button in the web interface, on the page were you edit some extension, and then pressing 2 more buttons that will appear. More clicks than running “amportal restart”, but you do not need to remember the command name.

Setting up the firewall

Modifying the settings in /etc/asterisk/sip*.conf files increases security of your asterisk server, but intruders can still find a vulnerability in your system and modify the Asterisk configuration files in the way they want them to be, so it is a good idea to limit other access to your system, not just to SIP port.  GNU/Linux kernel provides means to filter Ethernet packets depending on different properties, following the rules set up by iptables command. The filtering itself happens in the kernel, so it is fast in applying rules and deciding the fate of the packets. The simplest rules (and we are interested here only in the INPUT chain of the filter: table (iptables support much more), determine which incoming packets are allowed to reach the server looking at just the source and destination IP addresses (ranges) and ports. Other rules allow to eavesdrop on the packets themselves, looking for the particular patterns in their contents. It is also possible to create rules based on the rate of the incoming packets of certain kind. All these features are useful to protect the server simple rules allow to statically define who is allowed to access the server, content-analysis and rate-limiting rules can be used to mitigate the sip attack targeted to discover your extensions and guess the associated passwords. There are several scripts that are designed to set such firewall available on the Internet, I liked the one written by Gordon Henderson (there are his explanations of the script  here).

When I tried this script on Ubuntu+FreePBX setup it worked OK, but on the CentOS+Trixbox it reported some errors. I found that these errors were related to the older version of iptables used in the Trixbox version, so I made modifications that determine the supported iptables syntax and use it. You may find the modified version of the firewall3 script in Elphel git repository. This repository includes all other files mentioned below, you may download them separately or as a snapshot archive.

To set up the firewall you may copy the firewall3 file to /etc/init.d/firewall3, copy the template allowSites and blockSites files to /etc/network (on Trixbox system there was no directory /etc/network directory so I had to create one:
mkdir /etc/network
cp firewall3 /etc/init.d/
cp allowSites /etc/network/
cp blockSites /etc/network/
chmod 700 /etc/init.d/firewall3
chmod 600 /etc/network/allowSites
chmod 600 /etc/network/blockSites

You may leave blockSites as is (with no IP addresses defined) but you definitely need to edit allowSites file if you plan to have any remote access to the server.
nano /etc/network/allowSites
Just specify several trusted IP addresses (or address ranges), one address/range in a line. Be especially careful if you are connecting to the server remotely over ssh – do not forget to include your ssh client IP so you will not get blocked from the server yourself. When working remotely it may be useful to temporarily set the task of disabling the firewall (/etc/init.d/firewall stop) with cron. If you have the allowSites and blockSites in the default locations (/etc/network) you may be able to use the firewall3 without any modifications, but it is better to read it  at least. One area were modifications may be needed is that in the default version all such services as ssh, http, https are disabled there. They are enabled only for the clients listed within allowSites file, but if you really need broader access you may uncomment the appropriate lines in the script:
342 ####################################################################################################
343 # This firewall is configured so all ssh, web control (including Flash Operator Panel (FOP)
344 # is enabled only from the IP addresses specified in the list, so all the ports needed for these
345 # services are not enabled here. You may uncomment some, if **really** needed.
346 ####################################################################################################
347
348 # Allow WWW & HTTPS traffic
349 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport www -j ACCEPT
350 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport https -j ACCEPT
351
352 # Allow SMTP, IMAP and POP-3 traffic
353 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport smtp -j ACCEPT
354 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport submission -j ACCEPT
355
356 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport pop-3 -j ACCEPT
357 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport imap2 -j ACCEPT
358 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport imap3 -j ACCEPT
359 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport imaps -j ACCEPT
360 # $ipt -A inetIn -p udp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport imap2 -j ACCEPT
361 # $ipt -A inetIn -p udp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport imap3 -j ACCEPT
362 # $ipt -A inetIn -p udp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport imaps -j ACCEPT
363
364 # FOP - flash operator panel
365 # $ipt -A inetIn -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport 4445 -j ACCEPT

Logging the firewall output

The iptables output may already be logged in your system (i.e. in /var/log/messages), but you may enable separate logging of kernel messages of priority “warning” (or higher) in a separate file to keep it smaller, the fail2ban program monitors the log files. To do this you may edit the syslog configuration file:
nano /etc/syslog.conf
and add the following line
kern.warning /var/log/iptables.log
When done, restart the syslog service with
/etc/init.d/syslog restart

Testing the firewall

To test the firewall (IMPORTANT: Make sure that your IP is “white-listed” in /etc/network/allowSites if you are accessing the server remotely !), run
/etc/init.d/firefall3 start
It will list the white-listed and black-listed IPs, as well as the interface it used (eth0, eth1,…). It determines the interface by reading the “route” output, You may overwrite that guess by specifying the interface in the firewall3 body.
There is another option that the script tries to determine automatically that may be needed to set in the script – differences in the iptable syntax for different versions. Older version uses “–hashlimit” parameters, while newer – “–hashlimit-upto” instead. So the script just runs “man iptables” on the server and looks for these parameters. If manual entries are missing from the server, you may uncomment appropriate line in the firewall3 and watch that there are no errors reported when script executes iptable commands.

When the firewall is started, you may run iptables with the predefined set of parameters to see the status:

/etc/init.d/firewall3 status

It will show current iptables rules, number of packets (and bytes) detected by those rules. You may also run
cat /var/log/iptables.log
to see the dropped packet info, make sure data there is what you expect. You may try to ssh to the server from some other IP and make sure the /var/log/iptables.log has “DROPPED: …” lines with the IP of the “intruder”. You may also test the simulated SIP attack using sipvicious

If you need to turn off the firewall, you can do it with

/etc/init.d/firewall3 stop

Starting/stopping the firewall removes all other iptables rules that may be defined, so when used with fail2ban (described below), you need to make sure that fail2ban is started (or restarted) after the firewall3.

Adding fail2ban

Fail2ban is a popular program that can monitor multiple log files generated by different services, detect malicious attempts and block server access from the identified addresses. It can also email you when the suspicious activity (resulted in ban) is detected. I decided to use this program to supplement the pure iptables firewall – iptables are faster to react, kernel immediately sees all the incoming (and not only) Ethernet packets (analyzing logs is slower), but fail2ban can detect other intruder attempts, has more flexibility for regular expression-based filters and has email notifications easy to set up. In the current setup I used the filter for the Asterisk logs and an additional one to detect SIP attack attempts stopped by the firewall3. It detects “SIP: ” strings in the iptables.log and blocks the intruder (with the current settings – for 72 hours).

Fail2ban installation

Fail2ban installation is straightforward, in most many cases it is possible to use repositories of the particular GNU/Linux distribution, there is a detailed manual. There are also instructions how to install and use fail2ban with Asterisk. It seems though that current version of fail2ban reads native asterisk timestamps, you just need to enable asterisk to log warnings to /var/log/asterisk/messages (may be disabled by default), so fail2ban will not have to read the full asterisk log (/var/log/asterisk/full).
nano /etc/asterisk/logger.conf
and uncomment the line (if it is not done already):
messages => notice,warning,error
Then run
asterisk -rx "logger reload" so the new settings will become effective. Next you may copy the filters:
cp etc_fail2ban/filter.d/firewall3.conf /etc/fail2ban/filter.d/
cp etc_fail2ban/filter.d/asterisk.conf /etc/fail2ban/filter.d/
and the jail.local fail2ban configuration file (or merge it with you existent one if you already used fail2ban on the server):
cp etc_fail2ban/filter.d/jail.local /etc/fail2ban/
The filters should be OK without any modifications, but you will need to edit /etc/fail2ban/jail.local configuration file, at the very minimum you’ll have to put there your email for notifications (or disable them). You may also need to enable/disable other services log processing, change the paths to the log files.

Starting fail2ban:

/etc/init.d/fail2ban start
will start fail2ban. If you now run “/etc/init.d/firewall3 status” it will show additional rules related to the fail2ban operation.
cat /var/log/fail2ban.log
displays the fail2ban log, it will include lines about starting/stopping this service, starting “jails” as well as banning/unbanning intruder IP addresses. You should also receive corresponding notifications by email, as I did shortly after enabling fail2ban:
Subject: [Fail2Ban] ASTERISK: banned 41.237.186.49
From: Fail2Ban
To: xxx
Date: Sat, 12 Mar 2011 15:24:17 -0700 (MST)
Hi,
The IP 41.237.186.49 has just been banned by Fail2Ban after
6 attempts against ASTERISK.
Here are more information about 41.237.186.49:
Regards,
Fail2Ban

This particular intrusion attempt was detected by fail2ban from the analysis of the  /var/log/asterisk/messages, not from the firewall3 output (/var/log/iptables.log). There was no SIP attack this time – this guy came to connect to the extension and password he already new from the earlier server break-in (of course, these extensions do not exist anymore), it is the same IP that was used to call to Gambia before. And here is the corresponding excerpt from the /var/log/asterisk/messages file
[2011-03-12 15:23:55] NOTICE[3636] chan_sip.c: Registration from '<sip:117@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:24:16] NOTICE[3636] chan_sip.c: Registration from '<sip:117@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:24:16] NOTICE[3636] chan_sip.c: Registration from '<sip:117@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:24:16] NOTICE[3636] chan_sip.c: Registration from '<sip:109@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:24:17] NOTICE[3636] chan_sip.c: Registration from '<sip:109@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:34:59] NOTICE[3636] chan_sip.c: Registration from '<sip:109@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:34:59] NOTICE[3636] chan_sip.c: Registration from '<sip:109@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:35:19] NOTICE[3636] chan_sip.c: Registration from '<sip:109@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found
[2011-03-12 15:35:20] NOTICE[3636] chan_sip.c: Registration from '<sip:109@xxx.xxx.xxx.xxx:5060>' failed for '41.237.186.49' - No matching peer found

Making firewall3 and fail2ban start at boot time

When the firewall3 and fail2ban are tested being started manually you can make them start automatically at boot time (firewall3 should start before fail2ban). It may be slightly different on different systems, on the standard Trixbox installation (the default init level is 3) I did the following:
ls -al /etc/rc3.dl
Looking at the output:
...
lrwxrwxrwx 1 root root 20 Aug 27 2008 S26lm_sensors -> ../init.d/lm_sensors
lrwxrwxrwx 1 root root 23 Aug 27 2008 S34mDNSResponder -> ../init.d/mDNSResponder
lrwxrwxrwx 1 root root 14 Aug 27 2008 S55sshd -> ../init.d/sshd
lrwxrwxrwx 1 root root 16 Aug 27 2008 S56xinetd -> ../init.d/xinetd
...
I found which Sxx numbers are not used. In my case S39 and S40 were free, so I entered:
cd /etc/rc3.d
ln -s ../init.d/firewall3 S39firewall3
ln -s ../init.d/fail2ban S40fail2ban

That did the job, and “ls -all /etc/rc3.d/” had now these 2 new services:
ls -al /etc/rc3.dl
...
lrwxrwxrwx 1 root root 20 Aug 27 2008 S26lm_sensors -> ../init.d/lm_sensors
lrwxrwxrwx 1 root root 23 Aug 27 2008 S34mDNSResponder -> ../init.d/mDNSResponder
lrwxrwxrwx 1 root root 19 Jan 22 2002 S39firewall3 -> ../init.d/firewall3
lrwxrwxrwx 1 root root 18 Jan 22 2002 S40fail2ban -> ../init.d/fail2ban
lrwxrwxrwx 1 root root 14 Aug 27 2008 S55sshd -> ../init.d/sshd
lrwxrwxrwx 1 root root 16 Aug 27 2008 S56xinetd -> ../init.d/xinetd
...

Testing the protection measures

I would strongly recommend to test your server after you hardened it, following the instructions of the Hacking and securing your Asterisk server. You may want to selectively (and temporarily) undo some of the protection steps, create insecure extension and subject your server to a simulated SIP attack. If “alwaysauthreject=yes” is applied, scanning for extensions with svwar.py will fail, so you can try breaking into test “insecure” extension (with the password, but without IP limitation) using svcrack.py. This is when the firewall3 should kick in (make sure that you run sipvicious from the IP not listed in /etc/network/allowSites). You would be able to watch the attack in the last lines of the output of the
/etc/init.d/firewall3 status
or
service firewall3 status
First column will show number of packets, second – number of bytes for each rule. The last line with DROP (it belongs to the chain “sipAttack”) shows total number of packets dropped as identified to be malicious, the previous line with LOG – reduced number of lines that went to the file /var/log/iptables.log (you may change the LOG rate in the body of the firewall3).

You may also view the log file – it will have the line(s) including prefix “SIP: “, and these lines will include the IP of the “offender” as well as the timestamp of the simulated attack. You may test it both with and without fail2ban running. Without it the attack
will be significantly slown down. In my case it was hundreds times slower (than without the firewall), so it took svcrack more than 10 seconds to just scan only 10 different passwords, less that 1 per second.

If the fail2ban is enabled, it would notice the “SIP: ” patterns in the log, ban the offender IP for a long time (setting of “bantime=259200” in /etc/fail2ban/jail.custom will ban it for 3 days). And if the email server is configured properly (FreePBX
uses it to email voice mails as attachments, so if you receive them you have a working mailserver) and the /etc/fail2ban/jail.custom has the correct destination addresses specified (in the included sample there are “dest=PUT-YOUR-EMAIL-HERE” templates), you should receive email notifications about the detected attack (and action).

You may verify that the server does not allow ssh, http and https access from the unauthorized IP, watching response both on the “offender” side, and on the server (over the ssh from the allowed site) with
service firewall3 status
and
cat /etc/var/iptables.log