Some web pages that check the user agent won't work if there is no user agent specified. For example, some old websites require Internet Explorer (IE). If a different browser is used, they display a message that the site must be viewed with IE. This is because the website checks for a user agent. You can set the user agent with curl.
The --user-agent or -A option sets the user agent:
$ curl URL --user-agent "Mozilla/5.0"
Additional headers can be passed with cURL. Use -H "Header" to pass additional headers:
$ curl -H "Host: www.knopper.net" -H "Accept-language: en" URL
There are many different user agent strings across multiple browsers and crawlers on the Web. You can find a list of some of them at http://www.useragentstring.com/pages/useragentstring.php.