SecOps - The user agent and user agent spoofing

User agents... agents but not so user.

(PT-BR) Check this video to see the tests used to create this post.

One of the information available on web server logs is the User Agent. What are user agents?

User agents are the software's that make requests from the client to the web server. User agent is an information that tells to the server which software the user (or not) is using to access the web page. So whenever you access a web page, the server will know with software you are using to navigate. Here a simple user agent name:

“Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0”

The user agent information is found on web server logs. Usually all web server logs will have user agent on configured. And this is an example of a basic web server log with the user-agent.

10.1.0.10 - - [20/Jul/2018:13:38:10 -0500] "GET /animatedcollapse.js HTTP/1.1" 304 - "http://10.2.0.101/" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" (Get more information about web server logs here.

So, what is the problem of user agents? None. They work and are useful on log analysis.

The point is that they can be faked or crafted during the web request. There are many ways to do it. Changing the real user agent for some other is called "user agent spoofing". This is a common way to avoid basic attack detections.  Here I will show an easy way to do it. It uses Linux wget command. In this example. our Linux machine has the IP 10.1.0.10 and the web server has the IP: 10.2.0.110.

The first wget request:

$wget 10.2.0.110

The web server will show this log and the user agent from the request:

10.1.0.10 - - [18/Jul/2020:10:28:07 -0400] "GET / HTTP/1.1" 200 28067 "-" "Wget/1.20.3 (linux-gnu)"

Let’s force the user agent to change. Our user agent will be: i’m a firefox browser. Check the command and the web server logs bellow.

$wget -U "i'm a firefox browser" 10.2.0.110

10.1.0.10 - - [18/Jul/2020:10:28:34 -0400] "GET / HTTP/1.1" 200 28067 "-" "i'm a firefox browser"

The next scenario we will use a user agent that is common on Firefox browsers. You can find a good list of user agents here.

$wget -U "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" 10.2.0.110

10.1.0.10 - - [18/Jul/2020:10:29:39 -0400] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"

This last user agent is the same of the one that we saw if using the Firefox browser. The difference would be the number of requests. Wget only perform a GET, while Firefox browser would perform a GET Request and try to download all content to show the web page. You can see this on this video.

Some user agents are used by well-known software’s to perform automatically tasks like vulnerability scan. Some user agents from vulnerability scanners are:

-          Nikto: "Mozilla/5.00 (Nikto/2.1.6) (Evasions:None) (Test:Port Check)"

-          OpenVas: "Mozilla/5.0 [en] (X11, U; OpenVAS-VT 11.0.0)"

Check this web site to learn about others vulnerability scanners

The question would be: Why someone is scanning my network? Red Teams, security auditors usually scan the web pages. Also attacker do it too. So, its better to identify. 

What are the most important things about user agent information?

-          If you see a weird user agent, you should investigate. This is true because mostly users will not use weird agents.

-          If you do not see a weird agent, you should look to the log carefully. This can be a benign request or a malicious request with a fake user agent.

Understand the logs during an incident investigation is really important. Web application investigation can use the web server logs and user agent is an important information. Take care about your conclusions when you see the user agent information.

Check this video to see the tests used to create this post.