Daytime Project

Anomalous Logline Detection Tool

Physical systems and machines nowadays have a lot of components that are running software, and thus, are generating logs for monitoring and debugging purposes. The volume of logs that can be generated can become large quickly, which limits the feasibility of manually inspecting log files for anomalous behavior. The help detect anomalous behavior in logs, Thunderbyte.AI created a domain knowledge agnostic tool that is capable of detection individual log lines that show anomalous behavior.


The designed tool works as an extension on your current log line storage, and currently offers an easy integration, but not limited to, with Elasticsearch. The tool makes use of unsupervised machine learning algorithms and pre-processors to group similar loglines together. Every cluster of similar loglines can be labeled as containing normal behavior loglines or error indicative loglines by domain experts and/or system administrators. The tool is able to trigger warnings when it finds loglines that are similar (or very similar) to loglines that previously have been marked as indicative for system errors, or can trigger warnings when it encounters lines that are considered outliers, and thus a system should not produce under normal operating conditions.


Within the DayTime project, this tool has been developed to be used for the MRI log use-case. For which the tool is used as an aid to monitor new software releases of MRI machines, in which it is capable of successfully detection anomalous loglines that are indicative of a system failure.




Contact Information: |