Each time I get a complaint that one of my web servers is running slow, the first thing I do is look at the Apache httpd logs.  For anyone familiar with Apache HTTP Server logs, they are not particularly useful when trying to analyze huge amounts logging information.  Trying to figure out what client(s) may or may not be misbehaving usually requires some log analysis tool whether commercial or open-source.  As it is, the http server logs alone do not tell a complete story of what is going on.  Additional data is necessary from other network systems to piece together a more complete picture of what is going on in order to make a meaningful analysis.  These other systems provide data in various logging formats.  Expensive log aggregation software can be purchased to solve this problem, but, why not try to fix the real problem? - all of our logs are in different formats.  Let us homogenized them with linked data!

Here, I attempt to go after some low-hanging fruit by focusing on the Apache HTTP Server logs by configuring Apache to generate it's logs in RDF Turtle and start the process of homogenizing and linking my system and network data to get the bigger picture.

Custom Log Formats in Apache httpd

Apache httpd has the ability to generate logs in multiple custom formats using the "LogFormat" command.  It is with this command that we can get the Apache HTTP Server to serve it's logs in RDF Turtle. I make use of the following ontologies/vocabularies in doing so:

HTTP Vocabulary in RDF 1.0 (rdfs)
Time Ontology in OWL (rdf/xml)

prefix http: <http://www.w3.org/2011/http>
prefix time: <http://www.w3.org/2006/time>

The third ontology is custom-made to describe the Apache-specific data elements:
prefix : <http://www.ebremer.com/apache>  (default prefix in config file to keep things compact)

Rather than worrying about creating URIs for each log event, blank nodes are used instead.  The following modifications are needed to the /etc/httpd/conf/httpd.conf file:

LogFormat "[\
a http:Request;\
:referrer \"%{Referer}i\";\
:useragent \"%{User-agent}i\";\
:remotehost \"%h\";\
:remotehostip \"%a\";\
:remoteuser \"%u\";\
time:inXSDDateTime \"%{%Y-%m-%dT%T%z}t\";\
:canonicalservername \"%v\";\
:querystring \"%q\";\
:numKArequests %k;\
http:httpVersion \"%H\";\
http:methodName \"%m\";\
:port %p;\
:requestsize %I;\
http:absoluteURI <http://%{Host}i%U>;\
http:absolutePath \"%U\";\
http:authority <http://%{Host}i:%p>\
] http:resp [\
http:statusCodeValue %>s;\
:requesttime %D;\
:responsesizenh %B;\
:responsesize %O;\
:connectionstatus \"%X\"\
] ." logRDF
CustomLog "/var/log/rdflog" logRDF

With these changes made, simply restart the httpd server and your server will now generate it's logs in RDF Turtle!  Some of these elements may not be so interesting like "port" if port 80 is the only one being listened on.  The lines can be removed to reduce the number of triples per log event.  The RDF generated can be loaded into any triple store where you can perform multiple SPARQL queries to analyze the logging information.  You can even combine it with other data to paint an even bigger picture.  A geolocation dataset for IPs would enabled analysis to locate requests to their originating countries and cities.  I am currently in the process of doing this now and this data set will be posted.  But wait, there's more!  Why bother saving the data to disk when we can send it directly to the triple store?

Enter the SPARQL 1.1 Graph Store HTTP Protocol
Part of the new SPARQL 1.1 recommendations is the SPARQL 1.1 Graph Store HTTP Protocol.  We can take advantage of this with a new program that I wrote that will take the output of Apache and send it to the triple store, in my case, Virtuoso, an open-source RDF quad store which supports this new graph store protocol.  Mine runs on a Amazon Cloud EC2 instance.  Since the data flow is into Amazon's cloud, network transfer costs are minimal.  Local servers can be used. OpenLink Software's Virtuoso also has a commercially supported version.  Other triple/quad stores can be used if they support the Graph Store Protocol.  Logging output from Apache is redirected to the new program using Apache HTTP Server's "piped logging".  This is enabled by modifying the CustomLog line (or adding a new line - you can have multiple log channels) in the following way:

CustomLog "|java -cp /opt/PipeLogger/PipeLogger PipeLogger > /opt/PipeLogger/PipeLogger.log" logRDF

The new program, named PipeLogger, is written in java and utilitizes Apache Jena and Jetty.  PipeLogger reads the piped data and then loads it into a Jena Model.  PipeLogger is responsible for adding the RDF prefixes to the data sent to the RDF triplestore.  Configuration parameters let PipeLogger know how many triples to load into memory before flushing it to Virtuoso.  In case the triplestore is unavailable, for whatever reason, PipeLogger will buffer the data in memory and then eventually, save the RDF to disk until the triplestore becomes available again.  Multiple web sites, whether hosted on the same server and/or multiple servers in different locations, can all send their Apache HTTP Server log data back to a central triple store as RDF.  Different web sites and servers can send to different named graphs.  PipeLogger will be freely available as open-source after initial testing is complete.

Analyzing the logging data with SPARQL

Looking at various Apache HTTP Server log analysis programs listed here, I have begun to construct various SPARQL queries to emulate the same types of reports.

Query #1 - This SPARQL query will list pages and rank them by the number of visitors but with the addition of the error response codes.  In this case, I'm curious to what people are looking for and what they are not finding when the response code is not 200 in addition to the successful connections.

prefix httpd: <http://www.ebremer.com/httpd/>
prefix http: <http://www.w3.org/2011/http#>
prefix time: <http://www.w3.org/2006/time/>
select ?uri ?statusCodeValue count(distinct ?ip) as ?visitors
          where {?s http:absolutePath ?uri .
                 ?s httpd:remotehostip ?ip .
                 ?s http:resp ?r .
                 ?r http:statusCodeValue ?statusCodeValue .
                 filter (!regex(?uri, "^.*\\.(jpg|png|gif|js|css|ico|txt)$", "i"))
}
order by ?statusCodeValue desc(?visitors)

Query #2 - This SPARQL query lists the referrers ranked by the number of visitors.  Self-referencing referrers are removed with the filter command.

prefix httpd: <http://www.ebremer.com/httpd/>
prefix http: <http://www.w3.org/2011/http#>
prefix time: <http://www.w3.org/2006/time/>
select ?referrer count(distinct ?ip) as ?visitors
          where {?s httpd:referrer ?referrer .
                 ?s httpd:remotehostip ?ip .
                 filter(!regex(?referrer,"ebremer.com"))
}
order by desc(?visitors)

Results for both sample query #1 and #2 are at the end of this post.  The queries were executed against a week's worth of logs for my website and totalled 601,795 triples.  At some point, it will be interesting to visualize this RDF data graphically.  Yes you guessed it, I will be using my 3D RDF visualizating system for this.  Stay tuned for the PipeLogger open source and testing post and further development on RDF logging.

Query #1 Results

Query #2 Results