Category Archives: awstats

Lines in my Apache Access Log Appear to be Decapitated – missing the beginning of the line

I run a LAMP stack that hosts about 100 virtual hosts. Each virtual Host logs to its own CustomLog file using the "combined" format.

For one of these log files (just the one so far), our AWStats parser is choking on the log file citing bad data. It says that not all the lines are in the "custom log" format.

Create/Update database for config "/etc/awstats/awstats.example.com.conf" by AWStats version 7.0 (build 1.971)
From data in log file "/wwwlogs/example.com.log"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 39179)
AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.
Your log file /wwwlogs/example.com.log must have a bad format or LogFormat parameter setup does not match this format.
Your AWStats LogFormat parameter is:
1
This means each line in your web server log file need to have "combined log format" like this:
111.22.33.44 - - [10/Jan/2001:02:14:14 +0200] "GET / HTTP/1.1" 200 1234 "http://www.fromserver.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
And this is an example of records AWStats found in your log file (the record number 50 in your log):
5%BE%C3%82%C2%A2s-strategic-plan?page=45 HTTP/1.1" 200 12980 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
Setup ('/etc/awstats/awstats.example.com.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).

I looked in the log file and found this is an intermittent problem. MOST of the lines look just fine. But every once in a while, there's a line that looks like it starts somewhere in the middle of the line.

Example:

Good Line

123.123.123.123 - - [19/Jul/2016:13:11:19 -0400] "HEAD /blog/compensation-plans-commercial-lenders HTTP/1.1" 200 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)"

Bad Line

ozilla/5.0 (compatible; Baiduspider/2.0;+http://www.baidu.com/search/spider.html)"

That's the whole line. See how it starts in the middle of the word "Mozilla"? And it doesn't look like it's a continuation from the previous line either - like a random line break got inserted. It's like it's a whole new line that just didn't output the first 100 or so characters of the line.

Here's a different example:

1%C3%82%C2%AC%C3%83%C2%A2%C3%A2%E2%82%AC%C5%BE%C3%82%C2%A2s-strategic-plan?page=60 HTTP/1.1" 200

Here it looks like it's starting in the middle of the requested resource URI.

Does anyone know what might be causing this? Our site is up and running as far as anybody on the front end can tell, but AWStats is completely unable to parse the logs.

Apache version: 2.2.15

Operating system: CentOS 6

PHP Version: 5.5.36

httpd RewriteRule not working as expected

On my server I am running awstats, a script that I can currently access via the following URL:

https://stats.example.com/bin/awstats.pl/?config=global

I am trying to use rewrite rules such that I can just use

https://stats.example.com/global

This is what I have written

RewriteRule ^(.*)$ bin/awstats.pl/?config=$1 [NC,L]

The problem is that anything I try and access (besides the index), will give me a 400, and my apache logs show no errors.

Should this rule be working correctly, do I have a different configuration issue? Or am I missing something? Yes, RewriteEngine is on.

edit

Based on Michael Berkowski's comment I determined that is is infact an issue with resources also being directed to the pl script, I have since modified and am using the following:

RewriteCond             %{REQUEST_FILENAME} !-d
RewriteCond             %{REQUEST_FILENAME} !-f
RewriteRule             ^/([0-9a-z]+\.[0-9a-z]+\.[0-9a-z]+)$    bin/awstats.pl/?config=$1 [NC,L]

I can now load the page again using

https://stats.example.com/bin/awstats.pl/?config=www.example.com

This means that all resources can be loaded correctly, however

https://stats.example.com/www.exmaple.com

will return a 400 ( this does not come from the pl script which will return a 200 and error message if the specified config file can not be found, again, no error messages in the logs.

another edit

In changing [NC,L] to [R=302], I am provided with the correct redirect upon request,

curl -k "https://stats.example.com/a.b.c"
...
<p>The document has moved <a href="https://stats.example.com/bin/awstats.pl/?config=a.b.c">here</a>.</p>
...

The problem is that when I change the redirect back to NC, I am again getting 400s.

yet another edit

here is my entire VirtualHost definition

# Address
ServerName              stats.example.com

# Site Files
DocumentRoot            /data/stats/wwwroot

# Logs
ErrorLog                /logs/httpd/example.com/stats.secure_error.log
CustomLog               /logs/httpd/example.com/stats.secure_access.log combined

# SSL
SSLEngine               on
SSLProtocol             all -SSLv2 -SSLv3
SSLHonorCipherOrder     On
SSLCipherSuite          EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH

# Certificates
SSLCertificateFile      /etc/ssl/certs/example.com/example.com.crt
SSLCertificateKeyFile   /etc/ssl/certs/example.com/example.com.key
SSLCACertificateFile    /etc/ssl/certs/example.com/example.com.ca-bundle

# Rewrite
RewriteCond             %{REQUEST_FILENAME} !-d
RewriteCond             %{REQUEST_FILENAME} !-f
RewriteRule             ^/([0-9a-z]+\.[0-9a-z]+\.[0-9a-z]+)$    bin/awstats.pl/?config=$1 [R=302]

Options                 ExecCGI

AddHandler              cgi-script .cgi .pl

Alias                   /awstatsclasses "/data/stats/wwwroot/classes/"
Alias                   /awstatscss     "/data/stats/wwwroot/css/"
Alias                   /awstatsicons   "/data/stats/wwwroot/icon/"
Alias                   /lib            "/data/stats/lib"

<Directory "/data/stats/wwwroot">
        Options ExecCGI
        AllowOverride None
        Order allow,deny
        Allow from all
</Directory>

SetEnv PERL5LIB /data/stats/lib:/data/stats/plugins

Awstats is not working for one domain but it is working for other domain on same server

I have installed awstats on Amazon ec2 server (Centos). It is working for one domain but not working for other domain. I have tried many times but did not find any solution.

After running below command,

/usr/bin/perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=mydomain.com -update

It is parsing the access log, but not showing any result on browser.

I have implemented this Apache access log.

Please suggest me any solution. Thank you

debug malfunction in logrotate

I need a way to debug and fix a malfunction in log rotation and pre-rotation in debian.

I have a EC2 instance with Debian Jessie and several websites, each one with its log.

I installed and configured AWStats implementing log rotation and statistics update following directions in https://debian-handbook.info/browse/stable/sect.http-web-server.html. everything worked fine for a couple of months.

a few days ago the log rotating job stopped working (the logs in /var/log/apache2/ keep growing bigger and bigger and are not rotated, awstats statistics are not updated).

the only thing I changed was the number of logs to keep in /etc/logrotate.d/apache2.

I tried to debug with the command

sudo logrotate -d /etc/logrotate.conf

but I could not find any sign of problem. here is the output for one of the log files. as you can see, it seems that the log is rotated, compressed and renamed, but nothing happens.

rotating pattern: 
/var/log/apache2/*.log  after 1 days (15 rotations)
empty log files are not rotated, old logs are removed 
considering log /var/log/apache2/access.log   
  log needs rotating
rotating log /var/log/apache2/access.log, log->rotateCount is 15
dateext suffix '-20150922'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/apache2/access.log.15.gz to
/var/log/apache2/access.log.16.gz (rotatecount 15, logstart 1, i 15),

running prerotate script
running script with arg /var/log/apache2/*.log : "
    if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
        run-parts /etc/logrotate.d/httpd-prerotate; \
    fi; \
"
renaming /var/log/apache2/access.log to /var/log/apache2/access.log.1
creating new /var/log/apache2/access.log mode = 0644 uid = 0 gid = 4

syslog is completely silent when I run logrotate

thank you for your help.

AWStats and Stock Ubuntu 14.04 Apache 2.4 Log Format

I have AWStats setup and reporting for my site, but it fails to parse/understand OS and browser detection from the Apache logs. I've not changed any of the Apache config from the stock settings apt installed. I've read you can change the LogFormat directive in AWStats to create a custom parser, but with what I believed was a stock combined log format this seems odd (http://www.internetofficer.com/awstats/log-format/). I'm fine with changing either directive of LogFormat, but I'd like to change the one that is easiest. Are the stock Apache 2.4 log files not compatible with any AWStats included parsers? I've included a sample of my Apache log files below as well.

From awstats.domain.ext.conf

LogFormat=1

From apache.conf file

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Sample apache logs:

70.198.69.128 - - [06/Jul/2015:11:43:54 +0000] "GET /official/calendar/                                                 HTTP/1.1" 200 2195  "-"                                              "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:54 +0000] "GET /static/css/bootstrap.min.css                                       HTTP/1.1" 200 19055 "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:55 +0000] "GET /static/css/custom.css                                              HTTP/1.1" 200 705   "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:55 +0000] "GET /static/css/selectize.bootstrap3.css                                HTTP/1.1" 200 2726  "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:55 +0000] "GET /static/bower_components/fullcalendar/dist/fullcalendar.min.css     HTTP/1.1" 200 3103  "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:55 +0000] "GET /static/bower_components/fullcalendar/dist/fullcalendar.print.css   HTTP/1.1" 200 2137  "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:55 +0000] "GET /static//js/base.js                                                 HTTP/1.1" 200 797   "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:55 +0000] "GET /static/bower_components/jquery/dist/jquery.min.js                  HTTP/1.1" 200 29899 "http://www.example.com/official/calendar/"      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
207.46.13.83  - - [06/Jul/2015:07:38:14 +0000] "GET /                                                                   HTTP/1.1" 301 286   "-"                                              "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
185.49.15.23  - - [06/Jul/2015:07:44:28 +0000] "GET http://testp2.czar.bielawa.pl/testproxy.php                         HTTP/1.1" 301 249   "-"                                              "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/31.0"
70.198.69.128 - - [06/Jul/2015:11:43:53 +0000] "GET /                                                                   HTTP/1.1" 302 302   "-"                                              "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:54 +0000] "GET /                                                                   HTTP/1.1" 302 302   "-"                                              "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:54 +0000] "GET /official/home/                                                     HTTP/1.1" 302 305   "-"                                              "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4"
70.198.69.128 - - [06/Jul/2015:11:43:54 +0000] "GET /official/calendar/                                                 HTTP/1.1" 200 2195  "-"                                              "Mozilla/5.0 (iPhone; CPU iPhone OS 
182.118.53.91 - - [06/Jul/2015:22:36:08 +0000] "GET /                                                                   HTTP/1.1" 301 226   "-"                                              "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36"
173.10.47.105 - - [07/Jul/2015:14:27:43 +0000] "GET /official/calendar/                                                 HTTP/1.1" 200 2197  "-"                                              "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36"

Which is the correct value for apache traffic usage monitoring?

Good day

I am trying to determine the correct value for traffic used in Apache's access log.

My existing log format is:

"%V %T _HOS_EXTENDED_ %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combined

In terms of usage there are three possibilities here:

  • %b = Size of response in bytes, excluding HTTP headers. In CLF
    format, i.e. a '-' rather than a 0 when no bytes are sent.
  • %I = Bytes received, including request and headers, cannot be zero
  • %O = Bytes sent, including headers

Looking at the Awstats source, it appears they do not cover %I and %O, and use %b.

  1. What is the correct value in the access log to determine traffic used by a domain?

Thanks

Addendum:

One wonders if awstats has not adjusted their code to cater for %I and %O (semi-recent Apache change) or it's a legitimate decision.

AWStats on Debian Jessie Forbidden from Browser / Unresponsive from CLI

Installed AWStats with apt-get install awstats. No Complaints.

Installed Apache2 the same way.

Copied /usr/share/doc/awstats/examples/apache.conf to /etc/apache2/conf-available/awstats.conf.

Activated with sudo a2enconf awstats.

Restarted apachec with sudo systemctl restart apache2.service.

Using port 8888 and getting "Forbidden" response in browser at:

infiniteglitch.net:8888/cgi-bin/awstats.pl

Running from command line just gives the help doc.

What step or configuration is missing here, please?

Why does Apache logs and AWStats data not match?

I created a plugin for a Joomla 3.3 website which displays (on the front-end) download numbers for specific RSS feeds. The data is being pulled from Apache logs.

However, when looking in AWStats, the download numbers for the same feeds are higher.

Ideally we want to display the AWStats data.

  1. Why is the Apache logs data and AWStats different? What data are they filtering, if any?
  2. How can I grab AWStats data (which is displayed in AWStats) to display on front-end?

Thank you in advance.

Combining multiple subdomain log files in one AWStats install?

Running a website on a CentOS server with WHM and cPanel, that has a single subdomain. Client would like to have the stats for both the main website and the subdomain all in the same AWStats install (long story short, client doesn't want to pay for the entire site to be rebuilt in a responsive design, so we've got a separate smaller site for mobile devices running in Wordpress).

I have (after researching the matter) tried using logresolve.pl to combine the stats, but there's precious little examples of this working and what I have tried fails to work. The following at least results in no errors, it's just that it's not creating combined logs to build the statistics from.

In awstats.pl:
do 'logresolvemerge.pl /usr/local/apache/domlogs/domain.co.uk /usr/local/apache/domlogs/m.domain.co.uk > /home/domain/public_html/cgi-bin/combinedlogs';

In awstats.conf:
LogFile="/home/domain/public_html/cgi-bin/combinedlogs"

The log locations are correct, I updated the HostAliases list with the subdomain, hunted through the settings for any other possible settings that need be changed.

Am I missing something obvious or am I barking up the wrong tree entirely?