This document will help you configure access logging on your ProxySG, upload the files to an FTP server, and then have Reporter process the logs. The document is meant to help you get access logging and reporting up and running in a relatively short amount of time. For full details on setting up access logging to Reporter, including other options, please see the Blue Coat Systems Reporter 9.x Initial Configuration Guide found at https://bto.bluecoat.com/doc/18443 . NOTE: BlueTouch Online (BTO) credentials are required in order to access this document. If you need assistance with obtaining BTO credentials, please search the knowledge base for "bto credentials" (without quotes) and the first response will direct you on how to obtain BTO credentials.
PREAMBLE: WHY SHOULD I USE FTP?
The ProxySG is able to upload the access logs using various protocols. This document will focus on one specific protocol, and that is the FTP protocol. Why is Blue Coat recommending the FTP protocol for access log uploads? Because it offers the most/best options in case you ever need to restore/re-import your access log data. The direct connection configuration does not keep the access log data in raw format. The data is imported into the Blue Coat Reporter database and the access log file is discarded. With FTP, you can easily create new profiles, recreate profiles, or send data into Blue Coat Technical Support if need be.
Sizing: Make sure that you have sized your Blue Coat Reporter server appropriately. Please see the Blue Coat Reporter 9 sizing guide found at https://bto.bluecoat.com/doc/19582 . You will need a Blue Touch Online user account in order to obtain a copy of the Blue Coat Reporter 9 sizing guide. Because Reporter is resource (disk, CPU, and memory) intensive, for the best performance please consider using real hardware and not virtualized hardware. If your dataset is large, you may also want to consider a 64-bit version of Windows or Linux to install a 64-bit version of Reporter on.
FTP server: Any proprietary or opensource FTP server will do. For simplicity sake this document will use a free opensource FTP server named FileZilla Server. Blue Coat does not implicitly or explicitly promote this free FTP server software. It is merely using it as an example in the configuration of access log to Reporter server setup. Please use your discretion when selecting an FTP server. NOTE: If you are interested in connecting to an external FTP server, or using the direct connect method please see the Blue Coat Systems Reporter 9.x Initial Configuration Guide found at https://bto.bluecoat.com/doc/18443 . BlueTouch Online (BTO) credentials are required to access this document.
Base Reporter OS: The base operating system used for Reporter in the setup article will be a 32-bit Windows system. Other OSes (32-bit Linux or 64-bit Windows/Linux) may be more appropriate, depending on the results of the sizing guide. Again, this base OS was selected for simplicity sake.
The easiest way to set this up is to install the FTP server on the Reporter server. Make sure you have lots of free disk space. Then the FTP server will be setup and configured. Once the FTP server is up and running, the ProxySG will be configured to upload the access logs to the FTP server. The connectivity between the FTP server and the ProxySG will be tested. Lastly, Reporter 9 will be installed on the Windows server.
STEP I - Setting up Filezilla FTP server.
1.) Download the Filezilla FTP server from http://filezilla-project.org/download.php?type=server . NOTE: This link is valid as of the date this document was published. The URLs are subject to change without notice. If the link is dead, please use your favorite search engine to find the Filezilla FTP server.
2.) Install Filezilla FTP server. Accept the application defaults.
3.) Create a directory where you want your access logs to be stored. For this example, the files will be stored in the D:\ftp\proxysg\ directory.
4.) With Filezilla up and running, a Filezilla server window will now open. You will want to click on Edit > Users. Here you will be shown the current users (none) and setup and configure new users.
5.) On the "General" page (box on the left hand side), click on the "Add" button under the Users section on the right hand side. Type in the FTP account name in the pop-up box. In this example, use "proxysg" as the account name. You do not need to make that user a member of a group as the group is optional.
6.) Make sure "Enable Account" is checked under the account settings section. Also put a check mark next to "Password:" and give the newly created proxysg user a password. In this example, the password will be "bluecoat". For security purposes, please make sure that this password is complex.
7.) Click on the "Shared Folders" page. Click on the "Add" button. Walk the filesystem directory tree to D:\ftp\proxysg\ and click on the OK button. For files and directories, give that user all file rights (Read, Write, Delete, Append) and all directory rights (Create, Delete, List, + Subdirs) to D:\ftp\proxysg\ . Make sure that D:\ftp\proxysg\ has a capital H next to it. If not, highlight the directory and click on the "Set as home dir" button to make that is the home directory for that user. The "H" signifies that D:\ftp\proxysg\ is the home directory for that particular user. When the proxysg FTP user logs into the FTP server, the root directory for the proxysg user will be D:\ftp\proxysg\ . That user will not be able to go any higher in the directory tree.
8.) Click on the "OK" button to save the user. NOTE: The "Speed Limits" and "IP Filter" pages are optional and will not be discussed in this article. You can implement them at your own discretion if you so desire. However, Blue Coat recommends that you not implement any speed limits or IP filters until after everything is configured and running correctly.
9.) The Filezilla FTP server should be up and running at this point and your proxysg user ready to go.
STEP II - Setting up access logging on the ProxySG to upload files
1.) Login to the Management Console on your ProxySG. Click on the Configuration tab > Access Logging > General > Default Logging tab. Make sure there is a check mark next to "Enable Access Logging". Click on the the "Apply" button to save your changes. IMPORTANT NOTE: Make note of all the protocols on the left hand of the box and the corresponding access logs they belong to on the right hand side. If you only want to upload your HTTP traffic and FTP traffic, make sure all the other protocols are set to "<None>", otherwise you may have problems with early uploads. Please see 000007708 for full details.
2.) Make note of the names of the access logs that you want to upload. All the other protocols, set them to "<None>".
3.) Click on the "Global Settings" (Management Console > Configuration tab > Access Logging > General > Global Settings) tab. Review the defaults entered into this page. NOTE: Before changing the global early upload limit, please read the "Background" information, including examples, found in 000007708. For now, just leave it at the default settings.
4.) Click on the "Upload Client" tab (Management Console > Configuration tab > Access Logging > Logs > Upload Client tab). This is where the access logs are configured to upload their data to the Filezilla server that was setup in STEP I above. In this example, the HTTP protocol that is goes to the main log file will be configured. For "Log:", make sure that "main" shows up. If you are doing this for a different log file, make sure the log file name you wish to configure shows up there.
5.) For "Upload Client: Client type:", select "FTP Client" and then click on the "Settings" button. Within the "FTP Client settings", enter in the IP address of the FTP server. The default FTP port is 21. The Path is "/" (without quotes). Put in the username, which in this example is "proxysg". Click on the "Change Primary Password" button and enter the password twice. In our example, the password is "bluecoat". Click the "OK" button and then click the "Apply" button. NOTE: If you wish to do secure FTP or FTPS between the ProxySG and the Filezilla FTP server, please see 000007724 for details.
6.) In the "Upload Client" tab, there is a "Save the log file as" setting. To help reduce the amount of disk space used, please select the radio button next to "gzip file". The ProxySG will compress the access log into a .log.gz file name format and upload that to the Filezilla FTP server.
7.) Click on the "Upload Schedule" tab next to the "Upload Client" tab. Make sure the appropriate log is selected. In this example, it is "main". In the middle there are two types of uploads. Select "periodically" as the upload type. At the bottom there is the "Upload the log file". You can choose to upload the access log on a daily basis at a particular time, or you can choose to have it upload every so often. Please see the "Post Installation Thoughts to Consider" at the end of the document for some additional information regarding what to select here.
8.) Test - Now is the time to test. In the "Upload Client" tab, click on the "Test Upload" button. Go to the Filezilla server. You should see some output stating that the user proxysg logged in successfully and that it uploaded a file called main_upload_result to the FTP server.
Troubleshooting: If your test from the ProxySG was unsuccessful, there are several things that you can try to troubleshoot the problem. They are as follows:
* Check/validate the username and password entered in Step 5 above.
* Double check the IP address of the FTP server.
* Make sure the Filezilla server is not blocking FTP traffic from an IP subnet.
* You can use the Filezilla server interface to view what is happening. The interface can also be configured to show the passwords being sent in clear text so you can verify/validate what is being sent to the FTP server.
* Go to a DOS prompt and open an FTP session from your DOS window to the FTP server. Make sure you can login using the credentials and that you can upload a file to your FTP server. If you can't login, check your FTP user credentials on the FTP server. If you can login but you can't upload, check your file system permissions and make sure all file and directory permissions have been given. On the Filezilla server, you can look at the Filezilla server interface.
* From your workstation if you get a long delay (30 - 60 seconds) before receiving an error and you are never able to reach the Filezilla server, then you may have a network problem. A network problem can be as simple as a firewall blocking FTP traffic. Or there isn't any route between your workstation and the FTP server.
* If you have a short delay (1 or 2 seconds) before failure, that indicates that the server is reachable, but the port is not open. Make sure Filezilla is running and something like Windows Firewall is not blocking the port.
* Take a packet capture on the ProxySG (Management Console > Maintenance tab > Service Information > Packet Captures) for a minute or so while you force an access log upload. That way you can see if the ProxySG is communicating with the FTP server or not. If you see multiple (three) SYN requests that have no response, then you probably have some sort of networking issue. If you see SYN > RST three times, then the FTP port is not opened on the remote FTP server, or the wrong FTP port was entered into the access log configuration on the ProxySG.
9.) Repeat steps 4 through 8 above for any other log files that you wish to upload. Make sure that when you setup the log files that you select the appropriate log, such as main, or SSL, or P2P, etc.
BEST PRACTISES FOR FTP UPLOADS:
* Be vigilant in ensuring your access logs are never left on the SG. Monitor your FTP upload/connectivitly to ensure the access logs aren't left on your SG for days, as this will create a backlog of access logs needing to be uploaded to your FTP server.
* Install a syslog tool that monitors the proxy FTP server, possibly using the second interface so you can alert yourself if the main interface goes down.
* Ensure you upload your acess logs at regular intervals. Uploading acess logs of greater than 12 gigs in size is considered burdensome, but uploading acess logs that are 12 K in size is too small. Find an intervall that, on average, uploads a size that is a good fit for your network.
STEP III - Setting up Reporter.
1.) Go to https://bto.bluecoat.com/download/ to download the latest version of Reporter to your system.
2.) Run the Reporter install. Install Reporter onto the hard drive that has the most room (LOTS of free disk space). In our example, that was on the D:\ drive. The install will also ask you for an Admin user name, password, and license file (not mandatory).
3.) From the Reporter server, open the following URL in your Internet Explorer or Firefox Browser: http://127.0.0.1:8081/ . Or you are remote you can login using the IP address of the Reporter server: http://<ip.address.of.reporter.server>:8081/ Login to Reporter using the Admin user that you created in the previous step. You should receive a message that states, "Welcome to Reporter. In order to view reports in Reporter, you must first have a loaded database. After you have created (and/or loaded) your database, you can click on "View Reports" in the top right corner to see the data in the database." Click the OK button to remove the message.
4.) Within Reporter under the Reporter Settings > Data Settings > Databases, click on the "New" button. You will be prompted for a database name. In this example, we will use "proxysg" (without quotes). Click on the "Next" button.
5.) Now you will add the source of your log files.
a.) Click on the "New Log Source" button. A new box will appear. It will ask you if you want to pull data from a local file source or an FTP server source. Since the Filezilla FTP server was installed on the Reporter server, select "Local File Source" and click on the "Next" button.
b.) You will give your log source a name. You can call it proxysg and then click on the "Next" button.
c.) For "Directory Path", browse to the FTP directory. In this example, this is D:\ftp\proxysg\ . Once you browse to your source directory, click on the "OK" button.
d.) For file pattern, leave it a wild card by using * as the wildcard marker. Click on the "Next" button.
e.) You are now asked what to do with the file after processing the log file. You can rename, move, or delete it. In this example, we will use the default of "Rename: Append '.done' to filename". Click on the "Done" button.
6.) You are now back to the Log Sources box. The default polling time is every 10 minutes. You can increase or decrease this interval if necessary. Once the polling interval has been selected, click on the "Next" button.
7.) By default the Reporter server will expire any data that is older than 30 days in the Reporter server. You can increase or decrease the expiration date, and also select when to run the database expiration command. (IMPORTANT LICENSING NOTE: Reporter 9 is licensed based on the number of lines in the Reporter database. Having more data in your Reporter server may cause licensing issues. Additionally Reporter may reach its limit and no longer import access logs into the Reporter database. So if Reporter runs, but all the reports contain old data, check the number of lines in your Reporter database and compare that with the Reporter licensing model to ensure you have not reached that upper limit. If you have reached your limit, expire some data so you can restart the data import. Please see 000012156 for full details regarding Reporter licensing.) Leave the database at defaults and click on the "Next" button.
8.) You are prompted for the location of your database files. If the database location is on the hard drive with the most room, then click on the "Done" button. The Reporter server will go out and start processing any uploaded access logs if there are any logs to be processed.
9.) All done. You can click on "View Reports" in the top right hand corner to start viewing your log data.
POST INSTALLATION THINGS TO CONSIDER:
In STEP II, number 7, it discusses the frequency of uploads to the FTP server. If the ProxySG is configured for frequent uploads, such as every five minutes, then the FTP server will end up with a lot of small files in that incoming FTP server directory. If the proxy is used in a 24x7 environment, there will be 288 files uploaded to the FTP server on a daily basis. Over a month's time, that will result in approximately 8,600 files, and over a year's time, that will result in about 100,000 files uploaded. File system performance and backup performance can suffer greatly with that many files stored in a single directory. If a Reporter database rebuild needs to occur, all those files will need to be renamed, which can be a time consuming process.
Because of the size and number of files that are uploaded to the FTP user's incoming directory, some sort of periodic movement of files from the FTP user's home directory to a separate storage location may be warranted. For example, a job can be scheduled to kick off a batch file that will move the files from the FTP directory where Reporter looks for new files to another directory. That way a minimal number of files will be maintained. Please see KB article 000008295 for futher details.