Come on, you scuzzy data, be in there. Come on.

I wrote a few weeks back about the theme of Big Data and organizations such as EMC’s GreenPlum and Apache’s Hadoop ushering the way for the application of large data in cloud environments. It reminds me about the debate years ago on whether we would ever see a paperless society, which we realized sadly isn’t going to happen. Data just seems to compound and we cannot begin to imagine how the rates of data usage and creation will increase.

But in order to embrace big data, we need to figure out the barriers to adoption. David Asprey, a regular on the cloud expo circuit, notes that there are 2 key elements missing in the log management space right now: real scalability and security.
David recently wrote a blog on Trend Micro that illustrate these 2 main barriers to adoption of big data.

“The 1st is real scalability, which means thinking beyond what data centers can do. That inevitably leads to ambient cloud models for log management. Splunk has done an amazing job of pioneering an ambient cloud model with the way they created an eventual consistency model which allows you to make a query to get a “good enough” answer quickly, or a perfect answer in more time. They can do this because the data is spread all over the place but it is controlled centrally, which is a hallmark of ambient cloud architecture. Plus, ambient cloud providers are valued higher than IaaS cloud vendors. That sucks for us infrastructure guys.“

And the second:

“The 2nd thing is security. Log data is next to useless if it is not nonrepudiatable. (is that even a word?) Basically, all the log data in the world is not useful as evidence unless you can prove that nobody changed it. That’s why I’m a believer in what Mark Searle, the original Addamark founder, is doing at Kinamik. His experience founding 2 early log management companies has led him to focus on the emerging problem of security for log management. It’s very meta. His 1st start up a decade ago ended up focusing the other way around – on using log management for security.”

I absolutely believe in the 2nd key issue being security. I’ve said before that the problem with cloud is abstraction and lack of visibility. Part of this relates to ensuring that you can provide audit-worthy information on all aspects of your cloud infrastructure, but because cloud environments consist of flexible moving parts that can be spread across multiple geographic regions, it is very difficult to prove if something was moved, since many of today’s SIEM tools cannot provide this visibility. Traditional network based log management solutions are designed to track information that happens to the physical network, but with cloud, many of the key events happen through WAN or web-based entries and thus may not be seen by these tools.

We need to start looking at ways to address these issues in a more consistent way. Part of this might be through the use of cloud-based logging tools (Splunk and Loggly are great examples of this, as well as Sumo Logic which comes from the folks behind ArcSight and are funded by the likes of Shlomo Kramer of Imperva), or it might be through the evolution of security tools to protect log management (instead of log management for security). The Telecom Working Group, an arm of the Cloud Security Alliance, is focused on driving innovation and best practices for log management for carrier grade environments. This is a key step towards solving these barriers to cloud adoption, and will help to increase awareness around the security gaps in cloud and virtualization.

To read more on Dave Asprey’s blog, visit

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s