Today there were a lot of requests from bots. How can I block them in Nginx or Fail2ban?
# tail -f -n 100 /var/log/nginx/access.log
176.28.122.158 - - [16/Sep/2017:16:00:16 +0300] "GET /actor/%D0%A2%D0%B8%D0%BC%D0%BE%D1%82%D0%B8%20%D0%A0%D0%B5%D0%B4%D1%84%D0%BE%D1%80%D0%B4 HTTP/1.1" 200 18298 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:16 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:16 +0300] "GET /actor/%D0%99%D0%BE%D1%85%D0%B0%D0%BD%20%D0%A5%D0%B5%D0%BB%D0%B4%D0%B5%D0%BD%D0%B1%D0%B5%D1%80%D0%B3 HTTP/1.1" 200 18390 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:16 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:16 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:16 +0300] "GET /movie/id569071-zhena-smotritelya-zooparka-the-zookeeper-s-wife HTTP/1.1" 200 33660 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:16 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:17 +0300] "GET /movie/id885658-dzhon-uik-2-john-wick-chapter-two HTTP/1.1" 200 32346 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:17 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:17 +0300] "GET /movie/id885658-dzhon-uik-2-john-wick-chapter-two HTTP/1.1" 200 32346 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:17 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:17 +0300] "GET /year/2017 HTTP/1.1" 200 72389 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:17 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:17 +0300] "GET /country/%D0%A1%D0%A8%D0%90 HTTP/1.1" 200 71408 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:17 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:17 +0300] "GET /genre/%D1%82%D1%80%D0%B8%D0%BB%D0%BB%D0%B5%D1%80 HTTP/1.1" 200 73832 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:17 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:17 +0300] "GET /genre/%D0%B1%D0%BE%D0%B5%D0%B2%D0%B8%D0%BA HTTP/1.1" 200 72251 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:18 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:18 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:18 +0300] "GET /genre/%D0%BA%D1%80%D0%B8%D0%BC%D0%B8%D0%BD%D0%B0%D0%BB HTTP/1.1" 200 62785 "-" "Java/1.6.0_24" "-"
176.28.122.158 - - [16/Sep/2017:16:00:18 +0300] "GET /director/%D0%A7%D0%B0%D0%B4%20%D0%A1%D1%82%D0%B0%D1%85%D0%B5%D0%BB%D1%81%D0%BA%D0%B8 HTTP/1.1" 200 17674 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:18 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:18 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:18 +0300] "GET /actor/%D0%9A%D0%B8%D0%B0%D0%BD%D1%83%20%D0%A0%D0%B8%D0%B2%D0%B7 HTTP/1.1" 200 17408 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:18 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:18 +0300] "GET /actor/%D0%A0%D1%83%D0%B1%D0%B8%20%D0%A0%D0%BE%D1%83%D0%B7 HTTP/1.1" 200 17362 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:18 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:18 +0300] "GET /actor/%D0%98%D1%8D%D0%BD%20%D0%9C%D0%B0%D0%BA%D0%A8%D0%B5%D0%B9%D0%BD HTTP/1.1" 200 17454 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:19 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:19 +0300] "GET /actor/%D0%9A%D0%BE%D0%BC%D0%BC%D0%BE%D0%BD HTTP/1.1" 200 17247 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:19 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:19 +0300] "GET /actor/%D0%A0%D0%B8%D0%BA%D0%BA%D0%B0%D1%80%D0%B4%D0%BE%20%D0%A1%D0%BA%D0%B0%D0%BC%D0%B0%D1%80%D1%87%D0%BE HTTP/1.1" 200 17730 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:19 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:19 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:19 +0300] "GET /movie/id885658-dzhon-uik-2-john-wick-chapter-two HTTP/1.1" 200 32346 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:19 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:19 +0300] "GET /type/%D1%84%D0%B8%D0%BB%D1%8C%D0%BC%D1%8B/2 HTTP/1.1" 200 71649 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:19 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:20 +0300] "GET /type/%D1%84%D0%B8%D0%BB%D1%8C%D0%BC%D1%8B/3 HTTP/1.1" 200 51007 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:20 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:20 +0300] "GET /type/%D1%84%D0%B8%D0%BB%D1%8C%D0%BC%D1%8B/4 HTTP/1.1" 200 18296 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:20 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:20 +0300] "GET /type/%D1%84%D0%B8%D0%BB%D1%8C%D0%BC%D1%8B/5 HTTP/1.1" 200 18296 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:20 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:20 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:20 +0300] "GET /movie/id589290-begushii-po-lezviyu-2049-blade-runner-2049 HTTP/1.1" 200 33391 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:20 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:20 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:21 +0300] "GET /movie/id623250-chernaya-pantera-black-panther HTTP/1.1" 200 32793 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:21 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:21 +0300] "GET /content/8-y-sezon-kultovogo-seriala-igra-prestolov-vyydet-ne-ranshe-2019-goda HTTP/1.1" 200 36418 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:21 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:21 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
176.28.122.158 - - [16/Sep/2017:16:00:21 +0300] "GET /content/vankuver-nodovolen-semkami-filma-dedpul-2 HTTP/1.1" 200 35782 "-" "Java/1.6.0_24" "-"
199.168.139.211 - - [16/Sep/2017:16:00:21 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:21 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:21 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:22 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:22 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:22 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:22 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
199.168.139.211 - - [16/Sep/2017:16:00:22 +0300] "\x03\x00\x00)$\xE0\x00\x00\x00\x00\x00Cookie: mstshash=NCRACK_USER" 400 173 "-" "-" "-"
There is a mention of this bot in 2012 year.
11 Jan 2012 - https://twitter.com/mubix/status/157115321155723264
3 Oct 2012 - https://twitter.com/mubix/status/253705438581903360
Instead of blocking what is a bad method, one should allow what is acceptable good method
You will have to add more methods to this if your app uses methods like
PUT
,PATCH
,DELETE
,OPTIONS
Or you can block the request using approach mentioned in below
https://serverfault.com/questions/772833/fail2ban-regex-to-block-x00-requests