Этот URL-адрес также дает ошибку с curl:
curl -i http://www.fox.com/
HTTP/1.1 403 Forbidden
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 260
Cache-Control: max-age=3600
Date: Thu, 06 Dec 2012 20:17:17 GMT
Connection: close
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
You don't have permission to access "http://www.fox.com/" on this server.<P>
Reference #18.2fa2f5cc.1354825037.8323a81
</BODY>
</HTML>
По-видимому, на сайте происходит какое-то странное обнюхивание пользовательского агента. Передача пользовательского агента Firefox работает:
curl -i -A 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/17.0 Firefox/17.0' http://www.fox.com/
HTTP/1.1 200 OK
Server: Apache
Content-Length: 70647
Content-Type: text/html; charset=utf-8
X-FarmName: www.fox.com
X-FarmAddr: 10.96.57.103
Cache-Control: max-age=3600
Date: Thu, 06 Dec 2012 20:22:58 GMT
Connection: keep-alive
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:tp="http://player.theplatform.com/" xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml" >
<head>
<title>FOX Broadcasting Company - FOX Television Shows</title>