Blog index–Rolling🥎blog–Permalink
I have decided to shill Firehol to people who have a remote box that is accessible outside of their LAN but don't quite have firewall or write their own by manually writing iptables commands, most likely in a shell script that they run every time they restart the box, much like me in the past. Perhaps, like me, this person also tried using ufw since that's 'easy' but found it worse than useless since it told lies (told it to block ports and it said it has. I could still poke said ports from another box.) In other words, I'm shilling firehol to a younger me, because I wish I knew about it before.
A firewall can help stop this shit below.
... auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jake rhost=211.104.172.54 postfix/submission/smtpd[2530437]: warning: unknown[211.104.172.54]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 postfix/submission/smtpd[2530437]: lost connection after AUTH from unknown[211.104.172.54] postfix/submission/smtpd[2530437]: disconnect from unknown[211.104.172.54] ehlo=2 starttls=1 auth=0/1 commands=3/4 postfix/submission/smtpd[2530456]: connect from unknown[78.36.74.231] auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jake rhost=78.36.74.231 postfix/submission/smtpd[2530456]: warning: unknown[78.36.74.231]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 postfix/submission/smtpd[2530456]: lost connection after AUTH from unknown[78.36.74.231] postfix/submission/smtpd[2530456]: disconnect from unknown[78.36.74.231] ehlo=2 starttls=1 auth=0/1 commands=3/4 postfix/submission/smtpd[2530432]: connect from unknown[91.73.194.178] auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jake rhost=91.73.194.178 postfix/submission/smtpd[2530432]: warning: unknown[91.73.194.178]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 postfix/submission/smtpd[2530432]: lost connection after AUTH from unknown[91.73.194.178] postfix/submission/smtpd[2530432]: disconnect from unknown[91.73.194.178] ehlo=2 starttls=1 auth=0/1 commands=3/4 postfix/submission/smtpd[2530456]: connect from unknown[60.169.35.245] auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jake rhost=60.169.35.245 postfix/submission/smtpd[2530456]: warning: unknown[60.169.35.245]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 postfix/submission/smtpd[2530456]: lost connection after AUTH from unknown[60.169.35.245] postfix/submission/smtpd[2530456]: disconnect from unknown[60.169.35.245] ehlo=2 starttls=1 auth=0/1 commands=3/4 auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jake rhost=122.165.220.183 postfix/submission/smtpd[2530432]: warning: unknown[122.165.220.183]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 postfix/submission/smtpd[2530432]: lost connection after AUTH from unknown[122.165.220.183] postfix/submission/smtpd[2530432]: disconnect from unknown[122.165.220.183] ehlo=2 starttls=1 auth=0/1 commands=3/4 ...
Lets see here...
If there is a service that uses password authentication, something will try to brute-force/wordlist it[1], perhaps, for a user that may not actually exist. Fail2Ban could get these IP addresses and ban them for a few hours/days/forever, however, trust me when I say this: it doesn't matter because they have an endless supply of IP addresses. With IPv6 coming along this is more true than not. This is not to say that Fail2Ban is worthless, it is very useful but not perfect. What to do in this case? It would be better if they didn't touch the port at all.
Before I dump my firehol config, let me say I am intending to accomplish here:
One could modify each of the config files for their open services (postfix, dovecot, openssh, ...) but I find this to be a bit cumbersome as config files often are not straight forward as they should be (documentation on them tend to be confusing and overly verbose) and the config file syntax rarely matches from service to service leading into research to do just one thing like whitelisting. Even IF a way was found it may not be as flexible as it should be (only listen on certain NICs? Most likely doable. Accept connections from specific range of IP addresses? Not likely. And just forget about port knocking).
Therefore, a system firewall would considered useful here, the services could do their defaults (typically listening on 0.0.0.0:port or [::]:port) and the firewall will take care of the rest.
version 6
FIREHOL_LOG_MODE="NFLOG" # uses ulogd2 instead of syslog (logchecker will pick up syslog unless configured not to)
server_mosh_ports="udp/60000:61000"
client_mosh_ports="default"
#server_postfix_ports="tcp/587 tcp/465 tcp/25" # everything including submission and smtp
server_postfix_ports="tcp/25" # just smtp
client_postfix_ports="default"
server_dovecot_ports="tcp/143 tcp/993"
client_dovecot_ports="default"
server_wg0opts_ports="udp/60100 tcp/60100"
client_wg0opts_ports="default"
extface="eth0"
ipset4 create whitelist hash:net
ipset4 add whitelist "123.123.123.123"
ipset6 create whitelist6 hash:net
ipset6 add whitelist6 "dead::beef"
# catch nmap users and bots trying to do naughty things. feel free to expand/shrink this
for x in tcp/3306 tcp/5432 tcp/22 tcp/23 tcp/1433 tcp/21 tcp/3128 tcp/8080 tcp/5038 tcp/111 udp/111 tcp/5060 udp/5070 tcp/5900 tcp/3389 udp/3389
do
iptrap4 trap src 86400 \
log "TRAP ${x}" \
inface "${extface}" proto ${x/\/*/} dport ${x/*\//} \
src not ipset:whitelist state NEW
iptrap6 trap6 src 86400 \
log "TRAP ${x}" \
inface "${extface}" proto ${x/\/*/} dport ${x/*\//} \
src not ipset:whitelist6 state NEW
done
# port knocking, in case I lock ourself out somehow
iptrap4 knock1 src 30 \
inface "${extface}" proto tcp dport 1234 \
log "KNOCK 1"
iptrap6 knock16 src 30 \
inface "${extface}" proto tcp dport 1234 \
log "KNOCK 1"
iptrap4 knock2 src 30 \
src ipset:knock1 proto tcp dport 2345 \
log "KNOCK 2"
iptrap6 knock26 src 30 \
src ipset:knock16 proto tcp dport 2345 \
log "KNOCK 2"
ipuntrap4 trap src \
src ipset:knock2 proto tcp dport 5678 \
log "UNTRAP"
ipuntrap6 trap6 src \
src ipset:knock26 proto tcp dport 5678 \
log "UNTRAP"
# drop troublesome ip addresses
blacklist4 full log "BLACKLIST TRAP" \
inface "${extface}" src ipset:trap \
except src ipset:whitelist
blacklist6 full log "BLACKLIST TRAP" \
inface "${extface}" src ipset:trap6 \
except src ipset:whitelist6
# allow ipv6 to work
ipv6 interface any v6interop proto icmpv6
client ipv6neigh accept
server ipv6neigh accept
policy return
# wg0 is very trusted
interface wg0 wg_0
policy accept
server all accept
client all accept
# Accept all client traffic on any interface
interface any world
client all accept
server wg0opts accept # wg0 peers need a way to communicate...
server http accept
server https accept
ipv4 ssh accept src "ipset:knock2 ipset:whitelist"
ipv6 ssh accept src "ipset:knock26 ipset:whitelist6"
server mosh accept
server postfix accept
ipv4 server dovecot accept src "ipset:whitelist ipset:knock2"
ipv6 server dovecot accept src "ipset:whitelist6 ipset:knock26"
ipv4 server submission accept src "ipset:whitelist ipset:knock2"
ipv6 server submission accept src "ipset:whitelist ipset:knock26"
server anystateless nolog drop # stop logging drops
# let wireguard on wg0 talk to each other
router wg0_to_wg0 inface wg0 outface wg0
route all accept
# let 'out' into wg0
router eth0_to_wg0 inface eth0 outface wg0
route all accept
# let wg0 'out'
router wg0_to_eth0 inface wg0 outface eth0
masquerade
route all accept
router nolog
server anystateless nolog drop # stop logging drops
I think perhaps one of my favorite firehol feature is that you can read the config and also understand at a glance what it is doing.
Firehol's config, firehol.conf is actually a bash script that firehol compiles into a ton of iptables commands which would do what you expect. Like iptables, first match wins going from top to bottom.
Interface wg0 is my wireguard device, it accepts all connections coming in (server) and allows all connections going out (client), but wireguard does rely on a physical NIC (eth0, wlan0, whatever) which means that interface, whatever it is, needs to accept incoming wireguard connections on the listening port. Then, my peers also go through the firewall for wg0 device which I trust completely so I put server accept all.
If there's a service that needs world access and firehol doesn't know about it, one will have to explicitly create the definition like so:
server_mosh_ports="udp/60000:61000" client_mosh_ports="default"
What client_mosh_ports in this context refers to is what port(s) a client can use in response. I set it at 'default' because that is easy and good enough for me which roughly translates to 'any'.
In my config I utilize ipsets which might not be installed by default. Firehol will create ipsets if they don't already exist. My ipset usage involve tracking port knocking so if I don't have access to my wireguard network but need to access the server I can do this. A neat thing with ipsets is that relevant ports would only be accessible to the users that have port-knocked correctly. The port knocking will expire as well so it won't always be accessible.
Theoretically, a nmap scanner scanning literally random ports and somehow got the two ports (or more if desired) that would put it in the knock2 ipset without also scanning the trap ports would allow it to see that ssh port is open.
That 'for x in ...' consists of tcp/port and udp/port. Firehol 'looks' at these ports (actually the kernel passes what it receives to its firewall, iptables, which is generated by firehol) and IP addresses that touch these end up in a blacklist ipset called trap. All IP addresses in this ipset will be dropped thanks to the helper command blacklist4 and blacklist6 (and if trying to connect to them, as a client, will result with a reject by default).
Blacklisting along with Fail2Ban on active services may provide a rather warm and fuzzy feeling but a dedicated actor will be able to find a way through these. Notably both will be near useless when IPv6 comes along and firehol doesn't really have a dynamic way of 'going up' /16 each time a relevant IP address comes along but even if it does this would just a stop-gap (a way to do this could involve reading from relevant ipsets then updating the blacklist with some script started by cron or systemd-timer[2]). Your server will get port-probed by someone dedicated enough. Blacklisting+Fail2Ban is nice but your main thoughts should be on the firewall itself.
I've made submission, dovecot (IMAP and/or POP3), and ssh ports only accessible via whitelist ipset or port knocking ipset on the 'any' interface. Not much to say about this; it does what it says it'll do. In this context, 'any' can be thought of as a fallback if an interface+rule wasn't matched earlier. So, for wireguard traffic, they match on wg0 interface and the rule 'server all accept'.
It is VERY important to keep track of what is IPv4 or IPv6 or both (by not specifying which) because they cannot be mishmashed; you'll get a nonsense error that doesn't explicitly say something like 'IPv4 used for IPv6 thing'
If you do install Firehol, to begin with (feel free to change later), on the 'any' interface make sure you have put in 'server ssh accept' or another way to connect to your server BEFORE terminating your active ssh connection. By default firehol keeps established connections alive so when it's installed (and later enabled) you don't have to reconnect every time a change is made.
Firehol logs everything including drops. It can create 100mb+ log files in a short time. To save the very limited memory that VPS providers provide, I suggest not logging drops (and maybe use nflog so Firehol doesn't log to syslog which gets picked up by logchecker.) Firehol team thinks that logging drops is useful for diagnostics reasons, and maybe they're right. I'm not a sysadmin professionally and perhaps never will be so what do I know. All I know is my log files by logchecker suddenly got very very big.
Firehol creates a ton of logs. So, so much so I installed ulogd and set FIREHOL_LOG_MODE="NFLOG" which tells firehol to use ulogd and all the logs that I didn't ignore don't go to syslog. And these logs, not random ass drops, might actually be interesting to look at. Probably not though, I cannot actually remember the last time I checked Firehol's logs.
IPv6 doesn't start out working, for reasons I don't know/recall. So consider this a warning that IPv6 is 'special' in Firehol even though it probably shouldn't be, as you can see I have add somethings just to make it work.
Thanks for reading ♥
For something like the ssh daemon, what I do is set public key login (ssh-keygen(1)) as the only login method (disabling password login) which would make it 'safe' for worldwide access because no one can possibly brute-force something that doesn't accept a password. While this is true, openssh is not perfect code! A CVE can be created and considering the ssh daemon tends to be the super user..... So I put the ssh daemon on the 'any' interface part of this config to accept connections if their source IP is found in these ipsets, that way only people who know exactly what ports to knock or is present in the whitelist ipset may access it. (These people would also have a wireguard connection to my box which has
[1] = I recall a time when I rented a VPS and when I checked the log 3 minutes after it was created, it was filled with something trying to login as root.
[2] = What I used to do, and don't recommend, is block entire ASNs. Too much cross-fire on actual human-beans.
I enjoy Firehol's website, which is full of examples for real human beans.
Firehol website