Understanding Log Analytics at Scale

Using log analytics to generate insight and value is challenging. The volume of log data generated all over an enterprise is staggeringly large, and the relationships among individual pieces of log data are complex. There is no simple way of determining what is important or unimportant when the logs are first collected, and conventional data analytics are ill suited to handle the variety, velocity, and volume of log data.

This report examines emerging opportunities for deriving value from log data, as well as the associated challenges and some approaches for meeting those challenges. It investigates the mechanics of log analytics and places them in the context of specific use cases, before turning to the tools that enable organizations to fulfil those use cases. The report next outlines key architectural considerations for data storage to support the demands of log analytics. It concludes with guidance for architects to consider when planning and designing their own solutions to drive the full value out of log data, culminating in best practices.