SQUID and CEF

Document created by konrad.kaczkowski on Mar 10, 2015Last modified by konrad.kaczkowski on Mar 10, 2015
Version 2Show Document
  • View in full screen mode

CEF format for Squid

 

#Standard format from documentation:

logformat arcsight_standard_agent %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt

 

#CEF definition:

logformat CEF CEF:0|Squid|Squid|3.1|%Ss:%Sh|%rm %>Hs:%<Hs|Unknown|src=%>a shost=%>A sport=%>p dst=%<A srcUserName=%un userName_from_auth=%ul userName_from_ident=%ul userName_from_external_acl_helper=%ue cn1Label=Responce Time cn1=%tr squidRequestStatus=%Ss squidHierarhyStatus=%Sh statusCode=%>Hs statusCodeFromNextHop=%<Hs fileType=%mt requestMethod=%rm request=%ru aclTag=%et aclTagReturned=%ea in=%st requestProtocol=HTTP/%rv cs4Label=Referer cs4=%{Referer}>h requestClientApplication=%{User-Agent}>h

 

access_log /opt/squid/arcsight_standard_agent.log arcsight_standard_agent

access_log /opt/squid/CEF.log CEF

 

Log file is forwarded by syslog-ng daemon - so date definition is not required.

 

Sample output:

 

Standard SmartConenctor definition:

 

1421834729.600     77 10.0.0.83 TCP_MISS/200 1477 GET http://idm.hit.gemius.pl/_1421834729126/redot.js? - DIRECT/94.23.211.118 application/x-javascript

1421834731.006    366 10.0.0.92 TCP_MISS/200 173298 GET http://178.216.202.224/mstream/zubryonline.stream/index.m3u8? - DIRECT/178.216.202.224 application/vnd.apple.mpegurl

 

CEF definition

 

CEF:0|Squid|Squid|3.1|TCP_MISS:DIRECT|GET 200:200|Unknown|src=10.0.0.64 shost=bc-ksg-nb-03-kh.domain.com sport=51922 dst=64.233.166.149 srcUserName=- userName_from_auth=- userName_from_ident=- userName_from_external_acl_helper=- cn1Label=Responce Time cn1=216 squidRequestStatus=TCP_MISS squidHierarhyStatus=DIRECT statusCode=200 statusCodeFromNextHop=200 fileType=image/jpeg requestMethod=GET request=http://s0.2mdn.net/dynamic/2/1247219/ws2-media2.tchibo-content.de/newmedia/art_img/MAIN-IMPORTED/40a0f47b1b09d9e2/.jpg_1421827345612_.jpg aclTag=- aclTagReturned=- in=111779 requestProtocol=HTTP/1.1 cs4Label=Referer cs4=http://s0.2mdn.net/ads/richmedia/studio/pv2/35440134/20150120060456283/360x200_tchibo_rem_promo_parent.swf requestClientApplication=Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20rv:31.0)%20Gecko/20100101%20Firefox/31.0

CEF:0|Squid|Squid|3.1|TCP_MISS:DIRECT|GET 200:200|Unknown|src=10.0.0.64 shost=bc-ksg-nb-03-kh.domain.com sport=51944 dst=178.250.0.100 srcUserName=- userName_from_auth=- userName_from_ident=- userName_from_external_acl_helper=- cn1Label=Responce Time cn1=168 squidRequestStatus=TCP_MISS squidHierarhyStatus=DIRECT statusCode=200 statusCodeFromNextHop=200 fileType=text/javascript requestMethod=GET request=http://rtax.criteo.com/delivery/rta/rta.js? aclTag=- aclTagReturned=- in=14580 requestProtocol=HTTP/1.1 cs4Label=Referer cs4=http://s.csr.onet.pl/external/iframe.html?v=20130313 requestClientApplication=Mozilla/5.0%20(Windows%20NT%206.1;%20WOW64;%20rv:31.0)%20Gecko/20100101%20Firefox/31.0

 

 

The only thing to update is UserAgent - Squid replace space with %20

 

 

Syslog-ng definition

 

 

source file_SQUID_log {

        file("/opt/squid/CEF.log" follow_freq(1) );

};

 

destination d_10.0.1.33 {

        tcp(10.0.1.33 port(514));

};

 

filter SQUID_filter_out {

        not match("cache_object://localhost/info")

        and not match("Built local-host outside:");

};

log { source(file_SQUID_log); filter(SQUID_filter_out); destination(d_10.0.1.33); };

 

 

Difference between standard format and CEF

 

CEF dataConfiguration ParameterStandard SmartConnector
src%>aX
shost%>A
sport%>p
dst%<AX
srcUserName%unX
userName_from_auth%ul
userName_from_ident%ul
userName_from_external_acl_helper%ue
cn1LabelResponce Time
cn1%trX
squidRequestStatus%SsX
squidHierarhyStatus%ShX
statusCode%>HsX
statusCodeFromNextHop%<Hs
fileType%mtX
requestMethod%rmX
request%ruX
aclTag%et
aclTagReturned%ea
bytesIn%stX
requestProtocolHTTP/%rv
cs4LabelReferer
cs4%{Referer}>h
requestClientApplication%{User-Agent}>h

Attachments

    Outcomes