Symptomatically, the machine appears to stop forwarding traffic. Yet, existing established connections work. Access to the Management Console and CLI via SSH work as well. However, new TCP connections are not possible and users will be unable to obtain web pages, map CIFs shares (if CIFs is intercepted), or really access any TCP application that requires a new connection setup (e.g. 3-way handshake).
If you experience these symptoms, then the issue could be TCP port exhaustion. TCP port exhaustion occurs when the 4-tuple that defines a TCP connection (source IP (src_ip), source port (src_port), destination IP (dst_ip), destination port (dst_port)) uses every source port available and the src_ip, dst_ip and dst_port remain constant.
It is most common to see this issue when a common server (dst_ip) and a common destination application such as HTTP (dst_port=80) or CIFS (dst_port=445) are accessed from a proxy's singular outgoing interface (src_ip). If reflect client IP is not being used or an explicit connection is used from the client to the proxy , then the proxy will use it's own IP address as the source IP to communicate with a server/application (OCS). This sets up the limitation of 16K source ports to establish a unique 4-tuple, because the source IP, destination IP and destination port will be constants and the only variable is therefore the source port.
On the Proxy SG, the default range of available source ports is 16K, staring at 49152 and ending at 65535. Under normal conditions, it is unlikely that this range will be exhausted since there are limitations on how many incoming client connections can be accepted.
However, under certain abnormal conditions, where TCP connections are not terminated as expected (4-way handshake), then it is possible that 4-tuples will be held up in a timed state before they are released for re-use. For example, If only one side of the TCP connection has closed , then the TCP state will be FIN_WAIT_2. Thus, if the Proxy SG sends a FIN and receives an ACK to that FIN but never receives a FIN from the other side of the connection, then it will hold the 4-tuple in the FIN_WAIT_2 state. There is no specification for how long a connection should be kept in FIN_WAIT_2. Thus, various products will have different timers. On the Proxy SG, the timer is set at 10 minutes by default, before the connections are released back to the pool. At this time, this is not configurable on the Proxy SG.
The other common wait state for connections in TCP is TIME_WAIT. This is the final stage of the TCP state machine and is defined to take 2 times the maximum segment lifetime. It is meant to insure that no packets remain on the segment from the prior connection before releasing the connection for re-use. This is configurable on the Proxy SG.
If many, many connections are held up in either one of these wait states then the possibility of port exhaustion increases greatly because the system must continue to use new ports in the defined range.
Additionally, the Proxy SG uses a randomization algorithm for selection of the next available source port for a connection. This is done as an additional security measure, primarily to defeat session hijacking. Blue Coat Support has determined that in cases where port exhaustion occurs then this algorithm may engender some undesirable side-effects, preventing other applications from establishing connections besides the application affected by the original port exhaustion. This is due to the selection algorithm itself.
On account of this last issue, you may find that many or all of your applications are not able to create new connections.
In order to determine if you are faced with this issue, pull the TCP connection table from the advanced URL https://<proxy_ip:>8082/TCP/Connections. This could take a few minutes to complete as a listing. Filter the output for just the source ip, destination ip and destination port you suspect and then count the connections. If it approaches 16K, you know you have a port exhaustion issue.