Finding method in the madness: the challenges of the automatic classification of log messages
von Péter Gyöngyösi (BalaBit IT Security)
System logging is a particularly important function in current computing infrastructures, used in several fields from tracking down errors over alerting on security-related events to providing an audit trail of events for later use. One of the most critical problems is the real-time and postponed processing of the huge amount of data that is generated, which is especially difficult in the case of logs generated as plain unstructured text messages. Although various system logging techniques producing and handling structured data are available to make log processing easier, using them proves to be too complicated in several cases and the most widespread form of logs remains the plain text message.
The first step towards effectively handling this vast amount of data it is to be able to recognize events in the logs and to be able to classify them. In this presentation we will give an overview of the various approaches to this. We will cover solutions based on manually maintained patterns, heuristics of various complexity and advanced algorithms based on statistical clustering algorithms. We will discuss the algorithmic background of these approaches, analyze their effectiveness and the possible caveats and name a few open source and commercial solutions available on the market that follow the given approach.
We will share real-life experience about these solutions and describe use cases at organizations we've seen. We will discuss the advantages and drawbacks of them and cover topics like the required initial setup and integration time, maintenance needs and false alert ratio. We will investigate performance and scalability issues. We will analyze which solutions can be used real-time and which can only be used in post-hoc situations.
The presentation will give you a good overview of the various solutions available to create structured information from unstructured log-related data and help you make a better decision by learning about the possibilities and the experiences of others walking in your shoes.
Ãœber den Autor Péter Gyöngyösi:
