python - Need some ideas on how to code my log parser -

i have vps that's hosting multiple virtual hosts. each host has it's own access.log , error.log. currently, there's no log rotation setup, though, may change.

basically, want parse these logs monitor bandwidth , collect stats.

my idea write parser , save information small sqlite database. script run every 5 minutes , use python's seek , tell methods open log files last parsed locations. prevents me parsing 10gb log file every 5 minutes when need new information sitting @ end of (no log rotation, remember?).

after thought, realised i'm doing taking information log files , putting them database... moving data 1 location :/

so how else can this? want able like:

python logparse.py --show=bandwidth --between-dates=25,05|30,05 --vhost=test.com 

this open log file test.com , show me bandwidth used specified 5 days.

now, question is, how prevent myself parsing 10gb worth of data when want 5 days worth of data?

if use idea of saving log data database every 5 minutes, save unix timestamp of dates , pull out data between them. easy. i'd prefer parse log file directly.

unless create different log files each day, have no way other parse on request whole log.

i still use database hold log data, desired time-unit resolution (eg. hold bandwidth @ day / hour interval). advantage in using database can make range queries, 1 give in example, , fast. whenever have old data don't need more can delete database save space.

also, don't need parse whole file each time. monitor writes file of pyinotify whenever line written update counters in database. or can store last position in file whenever read , read position next time. careful when file truncated.

to sum up:

• hold data in database @ day resolution (eg. bandwith each day)
• use pyinotify monitor writes log file don't read whole file on , on again

if don't want code own solution, take @ webalizer, awstats or pick tool this list.

edit:

weblog expert looks promising. take of 1 of reports.