yorkspace.com

Recently, my friend Rob pointed me at a really cool program for doing statistical analysis on web logs. It’s called Funnel Web by Quest Software. After seeing the sample reports I was very impressed, but initially I had some problems setting it up to use the web logs from my hosting provider (1and1) Here’s how I fixed it.

Running Funnel Web on my 1and1 logs produced an incomplete report. It was not able to detect the log format correctly. It did the best it could, but did not produce any errors about detecting the format. After a little research, I came across this article on the 1and1 website that gives you the Apache log format string.

So, since they are using a custom format, it means you have to setup Funnel Web to recognize this format.

Looking through Quest’s website, you won’t find a direct link to the Funnel Web application. They are no longer supporting it. But, through the direct link above, you can still download it for free.

Since they have stopped supporting the product, this also means that they have taken down all of their support forums and articles regarding the product. However, I was able to retrieve this article out of Google’s cache. Do you ever feel like you’re dumpster diving when digging though their cache?

In the article, it spells out how to create a custom log format using the command line. The same parameters can also be saved in the Funnel Web settings file (fwp). They are shown at the end of this entry. Based on that article, I was able to come up with this format for my 1and1 logs:

# Custom log file: Format
usecustomformat
custom_seperator 32 #0x00000020 ( )
custom_dateformat 8
custom_dataIndex 0 4
custom_dataIndex 2 6
custom_dataIndex 3 1
custom_dataIndex 6 10
custom_dataIndex 10 9
custom_dataIndex 12 8
custom_dataIndex 5 5
custom_dataIndex 7 7


To use this, just open up your settings file (fwp) with a text editor and search for “Custom log file” and you will find where to place this. There is one other important thing to note. Whenever you save your settings with the Funnel Web program, it will strip off the first two lines: “usecustomformat” and “custom_seperator” breaking your format. To get around this, just make all your setting changes with Funnel Web first. Then open the settings file with your text editor and paste in the custom format changes. Finally, use Funnel Web to open your settings file and you’re all set. Just don’t use the program to make changes or you’ll need to repeat the process.

Used to define the separator between fields used in the log file. Separator values (X) must be entered in ASCII format. See the table below for a translation of common separators into ASCII format:

-usecustomformat	This command turns on custom settings format for processing log files.
-custom_seperator X

Field Separator	ASCII Equivalent
comma	44
tab	9
space	32

-custom_dateformat X This field is used to define the Data Format (X)used in the log file. For an explanation of the numbers associated with each date format, see the table below

Date Format	Numeric Value
Fri 07 Mar 2002	1
DD/MMM/YYYY	2
MM/DD/YY	3
DD/MM/YY	4
YYYY-MM-DD	5
Thu Mar 1 10:32:18 2002	6
Unix Time	7
[06/May/2000:14:59:29 -400]	8

-custom_timeformat X This field is used to define the Time Format (X)used in the log file. For an explanation of the numbers associated with each time format, see the table below

Time Format	Numeric Value
HH:MM:SS	1
1:40 PM	2
Unix Time	3

-custom_dataIndex X Z This command is used to define the positions of all the different fields in the log file, where X = the Log field type ansd Z = the position in the log file. In the table below is a translation of the number equivalent of the Log Field Types to be entered in the X position:

Log Field Type	Numeric Value
Date	0
Time	1
Status	2
Visitor	3
Source	4
File	5
Agent	6
Bytes	7
BytesIn	8
Duration	9
Referral	10
UserName	11
VirtualHost	12
Method	13
Protocol	14
Port	15

This entry was posted on Thursday, July 21st, 2005 at 6:06 pm and is filed under Website. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.