DDOS HELP

Get support for anything JK2 related, find tutorials for in-game and modding topics, and post any suggestions for the site here.
Post Reply
darthboss
Posts: 19
Joined: 09 Nov 2015, 08:17

DDOS HELP

Post by darthboss »

Today all the force jk2 servers in 1.04 got ddosed . I will post a TCPDUMP log and my iptables , I tried everything to prevent the attack but failed.


TCPDUMP

Code: Select all

16:11:03.693981 IP 80.92.170.193.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.693960 IP 141.101.172.150.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.693913 IP 213.109.148.221.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.693967 IP 213.155.215.125.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.693932 IP 193.106.74.94.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.693991 IP 217.9.156.194.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694045 IP 128.0.90.61.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.693975 IP 213.208.182.128.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.693970 IP 213.222.245.217.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694061 IP 109.71.79.19.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694064 IP 79.134.215.194.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694071 IP 130.0.219.206.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694074 IP 91.242.213.7.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694086 IP 213.109.54.182.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694105 IP 212.32.218.119.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694096 IP 79.134.216.144.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694102 IP 193.107.237.192.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694109 IP 213.248.20.245.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694109 IP 109.94.15.129.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694096 IP 95.172.56.68.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694101 IP 79.134.216.96.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694136 IP 195.42.156.170.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694140 IP 109.238.196.129.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694124 IP 109.95.222.246.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694160 IP 109.237.11.96.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694161 IP 213.165.212.222.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694210 IP 213.108.222.168.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694433 IP 79.134.215.162.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694449 IP 185.46.199.64.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694464 IP 109.68.23.188.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694450 IP 79.134.216.137.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694458 IP 185.46.85.216.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694446 IP 79.134.215.25.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694491 IP 79.134.215.234.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694495 IP 80.248.152.252.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694468 IP 185.44.238.66.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694485 IP 91.232.14.216.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694484 IP 80.254.113.126.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694525 IP 213.251.223.2.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694532 IP 80.255.147.163.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694552 IP 109.95.160.175.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694562 IP 91.232.235.26.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694566 IP 80.252.133.138.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694572 IP 134.90.180.205.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694608 IP 195.42.171.158.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694609 IP 195.26.187.234.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694619 IP 109.239.216.130.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694607 IP 80.249.152.12.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694611 IP 195.28.14.165.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694622 IP 144.206.0.195.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694641 IP 79.134.197.117.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694647 IP 130.193.66.82.29070 > 176.28.14.191.28111: UDP, length 15
16:11:03.694641 IP 80.95.44.174.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694656 IP 217.15.18.100.29070 > 176.28.14.191.28111: UDP, length 16
16:11:03.694663 IP 79.134.215.237.29070 > 176.28.14.191.28111: UDP, length 16


IPTABLES

Code: Select all

root@lvps176-28-14-191:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N CHECK1
-N udp-flood
-A INPUT -p udp -m length --length 16 -j DROP
-A INPUT -p udp -m length --length 15 -j DROP
-A INPUT -s 116.31.116.5/32 -j DROP
-A INPUT -s 182.100.67.113/32 -j DROP
-A INPUT -s 213.108.172.121/32 -j DROP
-A INPUT -s 212.220.8.67/32 -j DROP
-A INPUT -p udp -m length --length 1:1024 -m recent --set --name GetStatus --rsource
-A INPUT -p udp -m string --hex-string "|ffffffff676574737461747573|" --algo bm --to 65535 -m recent --update --name DEFAULT --rsource
-A INPUT -p udp -m string --hex-string "|ffffffff676574737461747573|" --algo bm --to 65535 -m recent --update --seconds 1 --hitcount 5 --name GetStatus --rsource -j DROP
-A INPUT -p udp -m length --length 28:32 -j DROP
-A INPUT -p udp -m length --length 15 -j CHECK1
-A INPUT -s MY_IP/32 -p tcp -m tcp --dport 28111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 27015 -j DROP
-A INPUT -p tcp -m tcp --dport 28111 -j DROP
-A OUTPUT -p udp -j udp-flood
-A udp-flood -p udp -m limit --limit 200/sec -j RETURN
-A udp-flood -j LOG --log-prefix "UDP-flood attempt: "
-A udp-flood -j DROP
User avatar
ouned
Administrator
Posts: 596
Joined: 23 Feb 2015, 13:03
Location: Gliese581c

Re: DDOS HELP

Post by ouned »

source ip's seem to be random so you can rarely do anything about it (except for completely blocking getstatus packets but then your server is hidden from the serverlist)
also, use jk2mv which already has some protection (but IP based)

you can try switching your servers port, maybe the attacker doesn't notice quickly
darthboss
Posts: 19
Joined: 09 Nov 2015, 08:17

Re: DDOS HELP

Post by darthboss »

well the attacker ddosed all the servers we went to today , and he did notice when i changed ports and attacked on the new port.
i am using jk2mv latest version.
if anyone here is good with linux , I think it can be blocked by size , block all UDP packets that have the size 15-16.
The tcpdump log i posted is just ddos , i kicked all players including myself then started tcpdump .
i tried using iptables but obviously i failed
User avatar
Caelum
Posts: 8
Joined: 23 Oct 2016, 20:13
Contact:

Re: DDOS HELP

Post by Caelum »

Hey guys,

Daggolin pointed me towards this thread; I usually hang around JK3 places. I run a hosting business over at https://jka.io that kind of specializes in DDoS mitigation. I'm in the process of expanding into other games including JK2 at the moment. Would it help anyone if I offered to host JK2 servers right now?

I may need to make some tweaks to my firewall for JK2 and familiarize myself with JK2MV, but I should be able to mitigate even really big attacks pretty trivially because of how similar JK2 networking is to JK3. If anyone's interested, I'd be happy to give them their first month for free in exchange for bearing with me while I fine tune the firewall for JK2?

For what it's worth, here's a freebie iptables rule I use to filter packets with invalid lengths:

Code: Select all

iptables -t raw -A PREROUTING -p udp -m length ! --length 29:1472 -m comment --comment "UDP: invalid length" -j DROP
But yeah, if anyone is interested in that, I'd be happy to help, and can definitely mitigate pretty much any DDoS thrown at me given some time to tweak the firewall for JK2. I've filtered several hundreds over the years. PM or email me if interested? I have both EU & NA locations, charge €10/month for 32 slots, and should be able to have a JK2 server up and running within 24 hours for anyone who wants one. :)
User avatar
ouned
Administrator
Posts: 596
Joined: 23 Feb 2015, 13:03
Location: Gliese581c

Re: DDOS HELP

Post by ouned »

The problem with iptable rules is that your machine still has to process the packages, it does so in kernel space in case of iptables (I guess), so it's a bit faster then implementing it in userspace but it shouln't make too much of a difference after all.
For actually blocking DDoS attacks you need a real hardware based infrastructure which stands strong before your actual server and filters unwanted stuff. With lot's and lot's of bandwidth.
The hoster OVH offers something like that for all their servers but I don't know what they are doing in a case like that. My guess is they just block the destination port or destination port + packet content tuple
Maybe they can do something more intelligent in case of spoofed ips but in case it's a real DDoS and the UDP packet is correctly formed, how would you filter that?
Last time I had a 1Gbit/s (thats what my hoster told me, I didn't see it) attack targeting my servers the problem was solved by null routing the destination ports.
It's a fight about who has moar bandwidth after all.

https://www.ovh.com/us/anti-ddos/anti-ddos-game.xml
I'd be interested in the technical details how it works but I'm still skeptical it would even help in this case

Caelum: what kind of protection (except for iptable rules) do you have?
User avatar
Caelum
Posts: 8
Joined: 23 Oct 2016, 20:13
Contact:

Re: DDOS HELP

Post by Caelum »

@Ouned what you're saying is not entirely correct. I actually use OVH as my datacenter provider (all game server hosts use a third-party datacenter). Their DDoS filtering takes care of high-bandwidth attacks beautifully (synfloods, DNS reflection, etc), leaving application-layer attacks. Those are usually low on bandwidth (the most I've seen and successfully filtered was 3-4 Gbps) and high in packets per second (the most I've seen & successfully filtered was several million packets per second). OVH does very little for the Quake 3 protocol - getstatus, getinfo, getchallenge, etc floods are not filtered at all, so they won't help against those types of DDoS attacks.

So basically I've got 3 layers:
  • 1) OVH filters high-bandwidth attacks for me, and have several Tbps capacity to do so. That no longer makes it a fight about bandwidth - all the high-bandwidth (L3/L4) attacks are filtered upstream.
  • 2) I filter application-layer attacks OVH doesn't filter (getstatus floods, getinfo floods, getchallenge floods, etc - anything application layer) using an extremely, extremely optimized set of almost a hundred (custom-written) iptables rules. Since application-layer attacks aren't about bandwidth, and I have dedicated hardware, that lets me filter several million packets per second. It turns out iptables scales really well if you bypass conntrack and tweak the kernel. One part of this is filtering invalid packets, but that's not at all the only thing it does.
  • 3) OpenJK (or in JK2's case, JK2MV) does some very basic extra rate limiting and validation, but doesn't play a very big role all in all
Like I said, through this combination, I've filtered several hundreds (!) of attacks on JK3 servers successfully, so I'm pretty confident the same approach would work for JK2 servers given the opportunity.
darthboss
Posts: 19
Joined: 09 Nov 2015, 08:17

Re: DDOS HELP

Post by darthboss »

Caelum , PM me your contact info , steam / skype , so we can talk.
Also OVH isn't good for jk2 , I have already tried , if DDOS starts on the jk2 port , they shut down port for legitimate traffic too.
They cant make the difference between ddos traffic and legit traffic.
User avatar
Caelum
Posts: 8
Joined: 23 Oct 2016, 20:13
Contact:

Re: DDOS HELP

Post by Caelum »

@Darthboss In order of my preference & response speed, my Discord is Caelum#8613, my Steam is caelum_nimmiel, and my Skype is caelumnimmiel. Add any of them. :)

As far as OVH shutting down entire ports goes: I've never experienced that happening on my dedicated hardware even at several million packets per second. Might be a thing with their VPS offering or somesuch?
User avatar
ouned
Administrator
Posts: 596
Joined: 23 Feb 2015, 13:03
Location: Gliese581c

Re: DDOS HELP

Post by ouned »

Caelum wrote:One part of this is filtering invalid packets, but that's not at all the only thing it does.
That's what I'm particularly interested in as I can only imagine two methods for getstatus, getinfo etc.:
- filtering incorrect packages (wrong length etc.)
- filtering based on how many packages come from a particular IP/network
Caelum wrote:leaving application-layer attacks. Those are usually low on bandwidth (the most I've seen and successfully filtered was 3-4 Gbps) and high in packets per second
Your servers connection is faster then 1Gbps? Awesome!
Caelum wrote:Like I said, through this combination, I've filtered several hundreds (!) of attacks on JK3 servers successfully,
Well, darthboss, I guess your best chance is to get a server from Caelum then :D
Caelum wrote:so I'm pretty confident the same approach would work for JK2 servers given the opportunity.
In case you use JK2MV you will feel very familiar, everything is basically the same except for some smaller details.

But just to have it said:
There aren't a lot of people who are going to be interested in JK2 servers.
The problem simply is that there aren't many JK2 players left. If you think JKA has a small community, come to JK2 and you will see what small actually means :D
User avatar
Caelum
Posts: 8
Joined: 23 Oct 2016, 20:13
Contact:

Re: DDOS HELP

Post by Caelum »

That's what I'm particularly interested in as I can only imagine two methods for getstatus, getinfo etc.:
- filtering incorrect packages (wrong length etc.)
- filtering based on how many packages come from a particular IP/network
I don't really want to give away the secrets behind all of my black magic, but to give you an idea, here's some of the other things worth filtering by:
  • IP geolocation. A lot of non-spoofed floods originate from specific countries. Similarly, one might want to filter datacenter-only IP ranges to deflate DDoS attacks.
  • OS fingerprint. Especially effective in many spoofed attacks. For example, Linux is a totally legitimate operating system, but are the 10,000 Linux-based connections to your JK2 server all that likely to be valid during a DDoS?
  • Packet type - i.e. if nothing else works, dropping all of one specific packet type, except from trusted sources.
Just filtering invalid or technically-valid-but-nonhuman packets gets you pretty far once you get into more obscure parts of the UDP protocol though. :)
Your servers connection is faster then 1Gbps? Awesome!
Yup. :)
User avatar
fau
Staff
Posts: 433
Joined: 16 Aug 2015, 01:01
Location: Warsaw / Poland
Contact:

Re: DDOS HELP

Post by fau »

Caelum wrote:OS fingerprint. Especially effective in many spoofed attacks. For example, Linux is a totally legitimate operating system, but are the 10,000 Linux-based connections to your JK2 server all that likely to be valid during a DDoS?
I thought your typical ddos botnet consists mainly of infected windows machines…
User avatar
Caelum
Posts: 8
Joined: 23 Oct 2016, 20:13
Contact:

Re: DDOS HELP

Post by Caelum »

fau wrote: I thought your typical ddos botnet consists mainly of infected windows machines…
It's just an example, and I might obviously be getting a biased view through the types of attacks I see, but the majority of attacks I see come from Unix-based (Linux, *BSD) machines. Often it's people renting VPSes specifically for spoofed DDoS attacks, from BCP 38 incompliant providers.

It's just one example of a filter for one type of attack though. :)
User avatar
fau
Staff
Posts: 433
Joined: 16 Aug 2015, 01:01
Location: Warsaw / Poland
Contact:

Re: DDOS HELP

Post by fau »

Linux IoT devices, come to my mind too now.
User avatar
Caelum
Posts: 8
Joined: 23 Oct 2016, 20:13
Contact:

Re: DDOS HELP

Post by Caelum »

As a quick heads-up, Darthboss' server is now back online at 92.222.234.168:28070. It was DDoSed (unsuccessfully) almost immediately after I got it online, so if it helps anyone, here's a downloadable packet capture of the attack it was being hit by: https://jka.io/ddos.20161024_21h.49m.02s.cap
User avatar
ouned
Administrator
Posts: 596
Joined: 23 Feb 2015, 13:03
Location: Gliese581c

Re: DDOS HELP

Post by ouned »

:!: nice
Post Reply
Created by Matti from StylesFactory.pl and Warlords of Draenor (modified by jk2.info)
Powered by phpBB® Forum Software © phpBB Limited