Army of Loggers - Peering Loggers for >1bn EPS search speeds

Document created by on Sep 30, 2014
Version 1Show Document
  • View in full screen mode

For those who were lucky enough to make it to Protect 2014, you may have seen my "Army of Loggers" turbo talk. This demonstrated how peering multiple small Logger 6.0 instances can provide vastly faster search speeds for a relatively small increment in licensing cost.


This video shows the net result of that peering. This was run on standard AWS m3.xlarge instances (4 CPU, 15GB RAM, 12kIOPS SSD, shared infrastructure), on EBS optimised. The data was a demo set of 30,000 CEF events, replayed over and over, for a total of around 8-9bn events online, with a few 'needle' unique events sown in to search for.


Note: Results did vary - peering 9 Loggers varied between 1-9x faster, depending on the search. YMMV.


This video was at the end of the presentation; this version has narration, whereas the embedded presentation copy didn't.