Instead of making the assumption that the proxy is at fault for any issue involving a page or part content not displaying in the browser, the following method makes it possible to identify the point of failure and allow the correct line of investigation to be taken. Simply assuming the proxy is at fault with no further investigation will lead in the wrong direction if, as is often the case, the proxy is not at fault and is working correctly within protocols. These steps can be used to find the nature of the problem:
Task One: Identify the exact link or resource (by URL) that is not loading
- Usually a page that fails to load will still have the URL in the browser address bar - make a note of it or take a screenshot of the browser page.
- If the resource not loading is a link you click on the page, hover the mouse on the link first to check the full URL. Take a screenshot or make a note of it.
- If the object not loading is an embedded object on the page, right click on the page - view html source. The objects on the page should be visible in the source by filename or url.
- if you are able to get the page to load properly by bypassing the proxy, right click on the page to view the source code on the correctly loaded page, and look for the resource. This can be difficult with complex web pages, but by searching neighbouring text strings, it's usually possible to find them.
In this example we, will assume we identified a .gif image that does not load, that has the full URL http://www.facebook.com/profile_images/686180626/logo-big_normal.gif
Task Two: Gather diagnostic data
Run the following two diagnostic tools simultaneously, test the site until you see the error, and stop the traces:
Add these lines in local policy (Configuration -> Policy -> Policy Files -> Install Local File from "Text Editor", and add the lines to the end):
client.address=x.x.x.x trace.rules(yes) trace.request(yes)
(where x.x.x.x is the client address)
Download the policy trace via https://IP.IP.IP.IP:8082/Policy
PACKET CAPTURE LOG
It is best to have an unfiltered capture, but if your proxy receives a high volume of traffic during the capture, it might be necessary to prepare a filter Proxy GUI > Console > Maintenance > Service information > pcaps > filter.
In the filter field, enter the following without brackets:
host <clients ip> || host <server/website ip or hostname> || host <ip of the dns server used by the proxy>
Start a packet capture via Management Console > Maintenance > Service information > Packet Capture > Start
Test the internet site(s)
Stop the packet capture Log Management Console > Maintenance > Service information > Packet Capture > Stop
And download the file as *.pcap or *.cap
Task Three: Analyse the policy trace data.
Using a text editor tool, eg Notepad++, open the policy trace and search for the URL string. This will show whether the request reaches the proxy and how it is handled by policy. For eg, you might see something similar to this format:
DNS lookup was unrestricted
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SV1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; InfoPath.3; MS-RTC LM 8)
user: name="DOMAIN\facebook.surfer" realm=IWAEircom
EXCEPTION(custom): Either 'deny' or 'exception' was matched in policy
url.category: Social Networking@Blue Coat
Any exception would explain a policy rule blocking the content. Typically, you would also expect to see the exception displayed on the page, which would give a clear indication there is a policy rule blocking the content.
Task Four: Analyse the packet capture data.
If there is no “EXCEPTION” or “DENY” action being hit, then it is time to analyse the packet capture with the Wireshark tool. The goal is to find the same request that
We identified in step (1) above, and trace it through both client and server side request/response pairs, to find the point of failure. This can be done by applying the following Wireshark filter techniques, although some knowledge of the http protocol is required:
- Take all or part of the URL string, and apply it to the Wireshark filter, eg:
http contains “686180626/logo-big_normal.gif” or tcp contains “686180626/logo-big_normal.gif”
- Search alternatives are to use the string search function (useful when the traffic is not on standard http ports) in Wireshark under Edit > Find packet > Find – String
- A good method to scan all the http requests, depending on how big the capture is, is the filter “http.request.method”
Once the request is found, the goal is to track both the client<>proxy side of the exchange and the proxy<>website side. Since there is no built-in way of associating the two sides with a session table of some kind, it requires some careful analysis:
- Typically the search methods above will reveal the pair of client and “duplicate” proxy requests with no further searching. One can then right click on both of the requests, and follow the sequence of request/response to see whether the website responds. A typical http request/response pair consists of a GET request, and a 200OK response. Failure response codes of 4xx or 5xx generally indicate server side failures, while 3xx responses imply an instruction to carry out a further action, such as serve from cache, or redirect etc. Any number of sites detail this info, such as http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
- If the website is down, or any upstream device prevents the proxy from creating a TCP session with the site, no proxy-side request will be seen. This is more difficult to find in the capture, since the only clue will be TCP SYN packets sent to the website IP. If the capture is not too big, it’s usually possible to see a series of SYN packets sent shortly after the proxy receives the client request. A check with a “whois” site for the destination IP, or pinging the hostname as seen in the client request will confim the destination. If there is no response, the website could be down or not responding, or any upstream network device (such as a Firewall) may be blocking the SYN packet, or the returning SYN,ACK from the website.
- If the proxy does not receive a SYN,ACK packet from the website IP, then logic would direct the investigation to the upstream network. Such question should be asked as: does the firewall receive and forward the SYN packets?; does the Firewall receive and return the SYN,ACK response from the site?
- If it can be seen that the proxy does not attempt to contact the site, rule out a DNS issue. If the capture is unfltered or includes the DNS server(s) IP address(es), the DNS request and response should be visible.
- If this proxy is being used an ADN web gateway, it’s worth ruling out the issue as described here:
KB4771 - "Unknown" content in SYN packet causes some websites to refuse connections to ADN enabled web-gateway proxies.
Other related solutions:
KB4771 - "Unknown" content in SYN packet causes some websites to refuse connections to ADN enabled web-gateway proxies
KB3754 - Some web page load slowly, or a blank page loads when going through the proxy (RFC 1323)
000016755 - www.google-analytics.com is causing web pages to load slowly or sometimes not load at all
FAQ739 - Flash Video does not play when going via a Proxy