The FTP log reader is unloading


<< Back to Knowledge Search

Solution

Overview

The FTP log source, intermittently, closes itself down with this message " Fatal log processing error stopped log source 'bluecoat_reporter_logsourcename"

The FTP Log source, intermittently, issues a checkpoint message that says "Checkpoint while processing log source file'

The FTP Log source, intermittently, has trouble unzipping the access log and displays this message "Unzip log source './SG_main_XX231_0319.log.gz' failed with GZ error -5"

The FTP Log source, intermittently, issues a  "  Failed to post process FTP log source file ''bluecoat_reporter_logsourcename:./SG_main_XX231_0319.log.gz'"

Cause
Resolution

The "GZ error -5" is a unzipp error, which means we no longer have a buffer ready to unzip this file.  Looking at the journal file we see that Immediately after the unzip error, but before the other messages, mentioned above,  we see an unexpected response from the FTP server.   And, this is why we we have no buffer- because the FTP server terminated the connection.

Here's the response:

BCRJ:2010-03-20 01:16:32 (4ba48460) ALW.ERRO.LOGSO
   src/sg_logreader.cpp,4064,FTPGetReply
      worker_thread_00000764(1892),,
   Unexpected response code 426 instead of 226 in reply:
 '426 Connection closed; transfer aborted.'

NOTE: In this interaction, with the FTP server, the Bluecoat Reporter server is acting as a FTP client.

These two numbers, mentioned in this journal entry,  are standard FTP error codes which mean this:

  • 426 Connection closed; transfer aborted.
  • 226 Closing data connection. Requested file action successful (for example, file transfer or file abort).

The reason the log source unloaded was because the remote FTP server terminated the transfer early.  For the purposes of being complete, here is a complete list of all the journal entries for this FTP fault.

NOTE:  Early on, in the below journal , we see the FTP server successfully processing access logs, until it starts to download "SG_main_XX231_0319.log.gz' , which is where the fault occurs. Reporter, as we have seen , was expecting to see a 226 from the server, which would have indicated that it had received the complete file, and was ready to inject this last portion of it into the database. However, because it realized the file transfer had been interrupted, it instructed a log source to shutdown, waiting for human intervention. This is so the Reporter administrator can investigate why we lost the FTP connection, and conduct the necessary remediation.   As you can see below, we checkpoint the file so when we see this file again we will only continue where we stopped, and not inject the same data again. As soon as the log source is started up, we then pull down the entire file, discarding the data we have already processed, and finding, through what we recorded in the checkpoint, where we stopped so we can continue processing the data into the database at the right place.

BCRJ:2010-03-19 23:12:41 (4ba46759) NOR.INFO.LOGSO
   src/sg_logreader.cpp,4887,FTPGetLogFile
      worker_thread_0000111c(4380),,
   Processing FTP log source file 'bluecoat_reporter_db:useclifwp120:./SG_main_YY231_0319.log.gz'
BCRJ:2010-03-19 23:42:59 (4ba46e73) NOR.INFO.LOGSO
   src/sg_logreader.cpp,4887,FTPGetLogFile
      worker_thread_00000494(1172),,
   Processing FTP log source file 'bluecoat_reporter_db:useclifwp120:./SG_main_YY232_0319.log.gz'
BCRJ:2010-03-20 00:15:59 (4ba4762f) NOR.INFO.LOGSO
   src/sg_logreader.cpp,4887,FTPGetLogFile
      worker_thread_00001170(4464),,
   Processing FTP log source file 'bluecoat_reporter_db:useclifwp120:./SG_main_YY233_0319.log.gz'
BCRJ:2010-03-20 00:48:25 (4ba47dc9) NOR.INFO.LOGSO
   src/sg_logreader.cpp,4887,FTPGetLogFile
      worker_thread_00001160(4448),,
   Processing FTP log source file 'bluecoat_reporter_db:useclifwp120:./SG_main_XX231_0319.log.gz'
BCRJ:2010-03-20 01:16:32 (4ba48460) ALW.ERRO.LOGSO
   src/sg_logreader.cpp,2846,DeflateGZStream
      worker_thread_00000698(1688),,
   Unzip log source './SG_main_XX231_0319.log.gz' failed with GZ error -5
BCRJ:2010-03-20 01:16:32 (4ba48460) ALW.ERRO.LOGSO
   src/sg_logreader.cpp,4064,FTPGetReply
      worker_thread_00000764(1892),,
   Unexpected response code 426 instead of 226 in reply:
 '426 Connection closed; transfer aborted.'
BCRJ:2010-03-20 01:16:32 (4ba48460) NOR.INFO.LOGSO
   src/sg_logreader.cpp,5638,FTPCloseLogFile
      worker_thread_00000764(1892),,
   Failed to post process FTP log source file 'bluecoat_reporter_db:useclifwp120:./SG_main_XX231_0319.log.gz'
BCRJ:2010-03-20 01:16:32 (4ba48460) NOR.INFO.LOGSO
   src/sg_logreader.cpp,963,LogReader::WriteCheckpointFile
      worker_thread_00000764(1892),,
   Checkpoint while processing log source file 'bluecoat_reporter_db:useclifwp120:./SG_main_XX231_0319.log.gz'
BCRJ:2010-03-20 01:16:32 (4ba48460) ALW.ERRO.LOGSO
   src/sg_profile.cpp,4188,DatabaseManager::PhasedLogSource
      worker_thread_00000764(1892),,
   Fatal log processing error stopped log source ''bluecoat_reporter_logsourcename"'

In some other cases, you may see a 425, instead of a 426, such as this entry below, which means we can't open a data connection.  

BCRJ:2010-04-10 03:13:11 (4bbf6e37) ALW.ERRO.LOGSO
   src/sg_logreader.cpp,4064,FTPGetReply
      worker_thread_000010c0(4288),,
   Unexpected response code 425 instead of 125,150 in reply:'425 Can't open data connection.'

NOTE1: for a complete list of all FTP error codes, please see this WIKI Page.

NOTE2: For other journal entries and what they mean, please see 000014639

NOTE3: For more information on the GZ -5 error when you use a Bluecoat Reporter Client ( streaming ) connection , please see 000014964

NOTE4: Other FTP and Reporter articles are:

For best practices, see  000014642 , For a list of supported FTP servers, see 00012435 

How to configure reporter to receive access logs via the FTP from the SG  - 000008692

 

Workaround
Additional Information
Bug Number
InQuira Doc IdKB3753
Attachment

Article Feedback

Hide Properties
First Published      10/01/2014
Last Modified      10/30/2014
Last Published      10/30/2014
Article Audience
Topic      Access Logging, Authentication, Crash / Restart, Database, Log Processing, Reporting
Article Number      000013828
Summary     
Was this helpful?
Comments:
 
Previous MonthNext Month
SunMonTueWedThuFriSat