Help with Apache access and error logs

This is getting to be a real problem. And I'm sure its something I've done. But here goes... I have been running a website on an older iMac G4 for a couple years. When I started, it was running on Jaguar. Now it is on Panther. When I was using Jaguar, I ended up with a really nice big long log file from which I was bale to generate really nice reports using some freeware analysers. However, after upgrading to Panther (and a bit of messing around with the config files in Apache trying to get php to work) something strange is happenning with my access and error logs. When I check on them, I often find a Gzip file for both logs along next to the current log file. But when I unzip the log file and check it, it only displays a few days of data. Where did the rest go? I think the Gzip file may be over-writing a previous file or something but I'm not sure. Why is Apache archiving my log files and can it be stopped? I have a HUGH hard drive on this iMac. A 200MB log file is not going to be a problem. Since the Gzip thing started the largest log file I get is less than 2MB. HELP!

Posted on Apr 8, 2005 2:05 PM

Reply
12 replies

Apr 8, 2005 6:06 PM in response to Scott Maiden

The standard install of Mac OS X will periodically rotate log files. Part of this process moves the current log from 'access_log' to access_log.0 and starts a new access_log. The old logs are then compressed and rotated out (by default the last 4 log files are kept, but older ones are deleted).

This is all controlled via the maintenance script /etc/periodic/weekly/500.weekly, specifically:

...
cd /var/log/httpd
for i in access_log error_log; do
if [ -f "${i}" ]; then
echo -n " $i"
if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
if [ -f "${i}" ]; then mv -f "${i}" "${i}.0" && if [ -x /usr/bin/gzip ]; then gzip -9 "${i}.0"; fi; fi
touch "${i}" && chmod 640 "${i}" && chown root:admin "${i}"
fi
done


If you don't want this to happen, either set your own cron task to move the log files as necessary or edit this script to do what you want.

Why is Apache archiving my log files and can it be stopped?

I have a HUGH hard drive on this iMac. A 200MB log file is not going to be a problem

Everything's relative I suppose.
I run some large servers that generate that amount of logs in less than a week so one perpetual log file is not an option.
It's also more efficient for the system to maintain smaller log files. You'll also find that parsing 10 small log files is quicker than parsing one big file, especially if you're looking for a specific time entry. All my logs rotate at least daily and some hourly) so at worst I have one day's worth of logs to walk through, and I know exactly which log file to look at.

At the end of the day it's your machine and you should configure it in whatever way makes sense for you. If you don't like the defaults that Apple have set (and neither you nor I do, for different reasons) feel free to change it.

Apr 11, 2005 3:52 PM in response to Community User

Thanks Camelot! This is starting to make sense to me now. However when I came in and checked my logs this morning I found the same thing had happened. The access log had been archive on Saturday. However, in the httpd folder, there where again only two logs; the active log and the archive. The archive was named access_log.0.gz and was created on Saturday. Before I left on Friday, there were two logs as well, except the access_log.0.gz was made on April 2 (last Saturday). Today, THAT log seems to have been over-written by the new archive. It doesn't seem like the server is rotating )or at least keeping) my logs properly. Unless, of course, I should be looking somewhere else for these "rotated" logs. They are not in the httpd folder. Here's the cron script, maybe someone can find out what is happening to the older archived logs. Or, maybe show me how to set the cron up to only archive once a month. Thanks for your help! This is great!

#!/bin/sh -
#
# @(#)weekly 8.2 (Berkeley) 1/2/94
#

PATH=/bin:/sbin:/usr/sbin:/usr/bin:/usr/libexec
export PATH

host=`hostname -s`

if [ -x /usr/libexec/locate.updatedb ]; then
echo ""
echo "Rebuilding locate database:"
locdb=/var/db/locate.database
touch "${locdb}"; chown nobody "${locdb}"; chmod 644 "${locdb}"
echo /usr/libexec/locate.updatedb | nice -5 su -fm nobody 2>&1 | sed -e '/Permission denied/d'
chmod 444 "${locdb}"
fi

if [ -x /usr/libexec/makewhatis.local ]; then
echo ""
echo "Rebuilding whatis database:"
if [ -d /usr/X11R6/man ]; then
MANPATH=${MANPATH:-/usr/share/man:/usr/X11R6/man:/usr/local/man}
else
MANPATH=${MANPATH:-/usr/share/man:/usr/local/man}
fi

makewhatis.local "${MANPATH}"
fi

echo ""
echo -n "Rotating log files:"
cd /var/log
for i in ftp.log lookupd.log lpr.log mail.log netinfo.log hwmond.log ipfw.log; do
if [ -f "${i}" ]; then
echo -n " $i"
if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
if [ -f "${i}" ]; then mv -f "${i}" "${i}.0" && if [ -x /usr/bin/gzip ]; then gzip -9 "${i}.0"; fi; fi
touch "${i}" && chmod 640 "${i}" && chown root:admin "${i}"
fi
done
if [ -f /var/run/syslog.pid ]; then kill -HUP $(cat /var/run/syslog.pid | head -1); fi
echo ""

cd /var/log/httpd
for i in access_log error_log; do
if [ -f "${i}" ]; then
echo -n " $i"
if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi
if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi
if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi
if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi
if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi
if [ -f "${i}" ]; then mv -f "${i}" "${i}.0" && if [ -x /usr/bin/gzip ]; then gzip -9 "${i}.0"; fi; fi
touch "${i}" && chmod 640 "${i}" && chown root:admin "${i}"
fi
done
if [ -f /var/run/httpd.pid ]; then /usr/sbin/apachectl restart; fi
echo ""

if [ -f /etc/weekly.local ]; then
echo ""
echo "Running weekly.local:"
sh /etc/weekly.local
fi

Apr 11, 2005 6:57 PM in response to Community User

I'm in the exact same boat as the original poster but don't feel too confident about editing cron files. Is there any hand holding resources that you can point out?

I'm looking to keep the log info generated by an Xserve running OS X.3.7. According to Server Admin it's set to archive every 7 days to '/var/log/httpd/access_log' but the problem is that once it gets to that 7th day it will take the current log, gzip it, and erase the last gzip file. I don't mind it archiving the log as a gzip file, but I really don't want the old gzip file deleted. I'd like to be able to keep the last 20-30 gzip files at the very least.

Also is there a way to get the cron script to copy a gzipped file to a different directory - one that I can access with AFP instead of SSH (as FTP will not be turned on for this machine)?

Any help or pointers would be greatly appreciated.

Apr 12, 2005 1:20 PM in response to Landis

Hi Landis,
I am the original poster with the same problem you have. I also posted this discussion in another area and got a little more response. However, as you have already found out, dealing with OSX's Apache server and the Cron daemon require a bit of UNIX chops. These I do not have either. But some good people here have helped shed light on the subject. Here's what I'm going to do:

The script (file) that's cuasing the "trouble" is the weekly.500 cron instruction. It is located here /etc/periodic/week. Now, as you may be aware, there are hidden folders on your hard drive containing UNIX code. These are accessed by your Mac at various times. Your Mac has another name, it's root. Only root has the ability to change the documents inside these hidden folders. This is what we need to do in order to fix the error. We need to modify the weekly.500 file and in my case, the monthly.500 file. See, I would rather the system archive log files monthy. Weekly is too fast for me, too much housekeeping. So, basically, we need to "comment out" some of the instructions in the weekly.500 file (so the system will ignore it) and then move that same bit of code to the monthly.500 file. Now the system will perform the archive once a month. Sounds a little scary but it's not that hard, I'll walk you through it if you need. You will need to do the following first:

1. get a UNIX text editor program, TextEdit or Word won't work. I use BBEdit Lite for Mac OSX you can get it here for free:

http://www.versiontracker.com/dyn/moreinfo/macosx/604

2. next, you need to enable the root user for your computer. This will allow you to "become" the root for a moment in order to make the changes. This is tricky as well. I'm sure there's a quick way in the terminal to do this, but here's how I do it:

a. find a program named "NetInfo Manager" in your Utilites folder inside your Applications folder, Open this program.

b. Under "Security" in th menu bar, select "Authenticate" and enter your admin password.

c. Under the same menu selection, select "Enable Root User", you will then be prompted to enter a new password for the root. Conventional wisdom says don't use the same one as you admin account but it will still work if you do. Just don't forget it. Write it on a sticky and place it under your mac.

d. close NetInfo, then open your System Prefs, go to System: Accounts

e. select the "login Options" button in the lower left corner, Set "Display Login Window as LIST OF USERS, then make sure fast user switching is enabled.

f. close system prefs and log out, at the login window, you'll have fields for a user name and password. Your user name will be "root" and whatever password you choose earlier. You will now be logged in as root.

From there, we will simply replace two files after making backup copies of the origianl. Then log back out and into your user account. After disabling the root user and restarting our web server it will be all over.

Read this over, maybe somewho knows an easier way. If you game to try it, I write back more specifically on how to edit the files we need to change.

Common! Screw up your computer it's fun, just joking. But be careful, while logged in as root you certanly can screw things up if your not careful!

Apr 12, 2005 1:38 PM in response to Scott Maiden

Thanks Scott for the response and the info.

This is a real webserver running a real website, my site is not the only one, and the server is not owned by me. I do not have physical access to it, and even if I did I don't think it's got a monitor attached. Everything I do will have to be done remotely via SSH and must be accomplished by an admin user. I can guarantee that they won't let me login as the root user on that server. I'd guess that there is the possibility of using the 'sudo' command however.

I've seen the cron files that you've talked about. I can open them and view them via 'pico'. I just don't know why they are doing what they are doing as it looks to me like it should be archiving at least the last 5 copies of the access_log and it's not.

Perhaps if you could show me what lines should be changed and why - that would be what I'm after.

Thanks!

May 2, 2005 1:32 PM in response to Scott Maiden

I have yet to see this question answered anywhere. I, too have a Panther Server 10.3.9 , and despite my 500.weekly looking exactly like everyone else's (the default), I do NOT have 5 access_log files per virtual host,
named...

vhost accesslog.0.log
vhost accesslog.1.log
vhost accesslog.2.log
vhost accesslog.3.log
vhost accesslog.4.log

Instead I have...

vhost accesslog.1114560000
vhost accesslog.1114560000.gz

So, the questions still stand: 1) WHY aren't the 5 logs beging created per the defualt config in 500.weekly, and 2) HOW do you edit it to stop deleting the old logs?

Jun 9, 2005 6:09 PM in response to Douglas Ward6

I have the same situation with my logs.
Instead of:

circle6media com_accesslog
circle6media com_accesslog.0.gz
circle6media com_accesslog.1.gz

I get:

2 Jun 23:03 circle6media com_accesslog
8 Jun 09:20 circle6media com_accesslog.1117670400.gz
9 Jun 13:22 circle6media com_accesslog.1118275200

Those dates are right outta the terminal...

I'm guessing I did something wrong when I switched to weekly archival recently. I don't understand the long-number suffix. Must be some sort of time stamp. Problem with that though, is that when using log analysis software, or even the google sitemaps_gen beta, I can't predict what the log will be named.

Anyone have any advice?

Thanks

Jun 10, 2005 1:27 AM in response to Eric Hassler

We roll our log files daily for our web server and use funnelweb for our web stats. To process the logs, we specify a wildcard in the config file, e.g.

/docs/logs/access_log.*

Another possible alternative is to write a script that outputs the content of all the log files and pipes them into the web stats process. If you've no idea where to start with that, post a message in the Terminal forum.

Regards,
Chris.

Sep 5, 2005 4:32 PM in response to Scott Maiden

I believe the key here is "Cleaning web server log files".

This phrase is found in the "daily" text file (which is a cron script).
In order to keep your web log files forever, comment out 6 lines in the "/etc/daily" file:

Add a "#" at the beginning of the following lines to deactivate a specific if-then statement related to "Cleaning web server log files".

Like so:

#if [ -d /var/log/httpd ]; then
# echo ""
# echo -n "Cleaning web server log files:"
# cd /var/log/httpd && \
# find . -type f -name ' _log' -mtime +7 -exec rm -f -- {} \; >/dev/null 2>&1;
#fi

One way to do this is by using the following 3 commands from the "Terminal" utility:

(1) Change to the directory where "daily" resides
cd /etc

(2) Copy the "daily" file to a backup file
sudo cp daily daily.BAK

(3) Edit the file to keep all web logs. Add the pound sign to the scipt's lines if what you see looks like those listed above.

sudo pico daily

About the parts of command number (3) above:

"sudo"
requests your admin password for temporary "root" permissions.

"pico"
is a simple text editor with its few options listed at bottom of window. The options use a "Control Key" plus another key. To see Where is the "Cleaning" statement, use Control + W and type "Clean" (no quotes). If you do not see a line containing "Cleaning web server log files:"
then I have screwed up by assuming your server is like mine and I humbly beg your pardon and please ignore my suggestion with no harm done.

"daily"
is the file to edit (unless you [or Apple] has changed default Server scripting options.

MORE:
If you need to access your server remotely with Terminal use the "ssh" command.
Like this:
ssh "YourUserID"@"YourServer.Host.Name" (your info without quotes)

After connecting with the server, proceed with the 3 commands listed above.

- g -

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Help with Apache access and error logs

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.