Despite substantial investments in tooling, 74% of companies are struggling to achieve true observability, according to a new report from LogDNA and the Harris Poll.

Two-thirds of survey respondents report spending $100,000 or more annually on observability tools, with 38% spending $300,000 or more annually. In many cases, companies are using more than four different tools, yet many are dissatisfied, citing issues with usability (66%), barriers to collaboration across teams (67%), and challenges with routing security events (58%). Other pain points include ingesting data into a standard format (32%) and routing it into multiple tools for different use cases (30%). More than half of respondents indicate that they would like to replace the tools they are using.

It’s clear that organizations need to drive more value from their observability data, particularly log data, but they are struggling to find tools that make it easy for multiple stakeholders to extract actionable insights. On average, more than three teams require access to this data — including development, IT operations, site reliability engineering, and security — but survey results indicate that existing tools fail to support multiple use cases. As a result, companies are spending more time trying to resolve issues.

Survey results also show that organizations are struggling to control the costs associated with skyrocketing volumes of machine data. Nine out of 10 companies indicate that they rely on log data for tracking the performance of applications and identifying threats and anomalous activity. However, in an effort to reduce costs, 57% limit the amount of log data they ingest or store, and 55% limit the amount they route to their SIEM. In many cases, they do not have access to the data they need, which slows down troubleshooting, debugging, and incident response efforts, and increases security risk.

Still, 85% of participants said they believe true observability is attainable. Innovation in the space can bring exceptional value as new technology emerges to make observability data more accessible for multiple use cases, more actionable for teams, and less expensive to ingest, store, and manage.

The report findings are based on a survey by The Harris Poll of more than 200 senior engineering professionals who are responsible for observability and log data management at companies across the United States.

Read the full report by LogDNA.