Slow performance, high CPU usage and the Event Log reports "TCP in Livelock"


<< Back to Knowledge Search

Solution

Overview

Slow performance, high CPU usage and the Event Log reports "TCP in Livelock"
You experience slow performance, high CPU usage and the Event Log reports "TCP in Livelock"

Cause
Resolution

If you have a deployment where a child ProxySG is forwarding all requests to a parent ProxySG then you may be encountering a forwarding loop.

The child ProxySG may have policy similar to the following:

 forward(parent_proxy) forward.fail_open(no)

This means all requests will be forwarded to the parent ProxySG.

You may encounter a forwarding loop if a parent ProxySG (or upstream client) sends a request to the child ProxySG because it will be immediately forwarded to the parent, which sends it back to the child, which is sent back to the parent and the process repeats. The symptoms are extreme slowness, high CPU and possibly TCP LiveLock messages in the Event Log.

Policy suggestions to prevent this:

  • Only forward requests if they come from IP addresses on the child ProxySG network.
  • Consider creating a policy to Deny connections from any parent proxies.

 

Livelocks will happen when an interface becomes so saturated with packets that the ProxySG is unable to keep up. For the interface to become saturated, it takes more than a high volume of legitimate connections, it usually involves a network loop. For example, if a policy forwards traffic from proxy 'A' to proxy 'B', and proxy 'B' is configured to forward to proxy 'A', that will generate a loop that will most likely cause one of the interface to go in livelock mode until traffic quiets down.

A routing loop can also be the cause of a livelock. This is far more likely to happen when the proxy is inline on the network. If the proxy is installed between two redundant switches and spanning tree is disable, it could create a network loop. Other possible causes could be denial of service attacks (ping floods for example).

To best way to troubleshoot a livelock issue is to take a packet capture and look for symptoms. Here are a few common symptoms

  • Lots of duplicate packets (SYN packets are seen more than one) are a good indication that there is a loop on the network
  • Lots of SYN packets from the same source IP address, on many different destination ports usually indicate that a denial of service attack
  • Lots of http connections from the same source IP that seems to keep authenticating over and over can mean that a workstation is configured to ignore cookies. This would cause the ProxySG to keep authenticating the same connection non-stop and go into a loop that can result in a livelock (as well as a high CPU usage)

 

 

Workaround
Additional Information
Bug Number
InQuira Doc IdKB1209
Attachment

Article Feedback

Hide Properties
First Published      10/01/2014
Last Modified      10/01/2014
Last Published      10/01/2014
Article Audience
Product      ProxySG
Topic      Crash / Restart, Hardware, Networking, Performance, Usability
Article Number      000013553
Summary     
Was this helpful?
Comments:
 
Previous MonthNext Month
SunMonTueWedThuFriSat