You are here


How Big Data is Changing Your Monitoring

Every 60 seconds around 100 thousand tweets are sent, 3,600 instagram photos uploaded and countless YouTube videos watched. Each and every day we use more data than you could possibly imagine whether this is for social media posts, recording climate change or tripping a sensor. Each time an email alert is created for a server fault data is created - data is everywhere.

Big data is being generated by everything around us at all times. Every digital process and social media exchange produces it. Systems, sensors and mobile devices transmit it. In simple terms, big-data is so large it is difficult to process using traditional databases and software techniques. To make matters more complicated big data also encompasses tools and processes making it even harder to extract value from.

This is the key to large data sets, making sense of them. In monitoring terms you often need to know more than the bare essentials as it potentially concerns a host of technologies used in conjunction with one another to solve a problem. Your IT systems and infrastructure generate data every second of the day, monitoring helps to detect bottle necks, identify root-causes and is always on (like your data).  

The Big Data Bang

This wealth of data is changing the way people within organizations work together as insights from big data can enable better business decisions and increase consumer engagement resulting in new ways of providing information.

Macy's Inc. was one of the first to effectively utilise this data with their use of real-time pricing. Macy's adjusts the prices of around 73 million items for their customers in near-real time, cleverly configured based on demand and inventory to react to local competition.

The continued growth of multimedia, enterprise, social media and the internet of things will result in exponential growth in data for the foreseeable future. As your big data application and tool usage beings to grow so too will the strain on your repositories. Your monitoring software needs to cope with this additional abundance as well.

Monitoring now underpins the way organizations adopt new technologies and frameworks to accommodate growing network volume, velocity and variety. For the instance of Macy's, their monitoring was agile enough to tune and optimize the performance of their big data applications ensuring their strategy was successful.

The Evidence

Exhibit A) Tesco PLC  
Tesco, the largest grocery chain in the United Kingdom and third largest retailer worldwide, collected 70 million refrigerator related data points to keep tabs on each ones performance.

By collecting real-time data they aim to save over $23 million a year in cooling costs alone by optimizing temperatures in stores. This worked by sensors taking readings every 3 seconds before the data was processed in real time and actioned via process maps.

Key to the success of this model is the monitoring equipment overseeing the process giving a pivotal view of the chain. If one link in the chain (or one sensor) fails the rest is flawed. Monitoring also helps justify the expense of this technology by measuring performance improvements.

Exhibit B) The Daily Mail
One of the world's largest online publishers utilises real-time big data to promote their content. The Mail track everything from unique users and page views to social shares and engagements to prioritise what is shown on their homepage. This process requires numerous technologies talking to one another at the same time to make sense of the data. Monitoring underpins this method keeping tabs on analytics, content management systems, servers and more.

Big Data Means Monitoring Does More

If you don't have the right monitoring software you can spend countless hours sifting through log files and error reports searching for the root cause of a problem. With big data this needle in hay stack scenario can come to light as traditional database monitoring tools offer only a simplistic view of health metrics and alerts. This is great for simple processes but it leaves a large hole if you are trying to determine the cause of poor performance and downtime for example.

Monitoring tools are of course required to do more, this might include monitoring and measuring database wait time which can help to calculate the correlation between resources and wait time. This in-turn provides greater clarity for SysAdmins to resolve problems, find and fix database performance issues and rectify broken queries. However that's really just the tip of the iceberg.

Big data may be where everything comes from, but monitoring is how everything can make sense of it all.

Future of Big Data depends on future of monitoring

Big data has changed the way your monitoring needs to work. If your company is using big data technology the IT department has a responsibility to support a cohesive performance monitoring strategy for any degradation that it may cause the business. As new technologies emerge and big data arrives from multiple sources at greater velocity, volume and variety, keeping a keen eye on things can only be a wise choice.

How big are we talking?

Since you've been reading there have been approximately:

500,000 Tweets sent
173,610 Brand and organization likes on Facebook
720 hours of YouTube video uploaded
2,855 NEW websites created

Get unified insight into your IT operations with Opsview Monitor

webteam's picture
by Opsview Team,
Opsview is passionately focused on monitoring that enables DevOps teams to deliver smarter business services, faster.

More like this

Mar 20, 2017
By Freya Ballan Whitfield, Digital Marketing Executive

An insight into CeBIT Conference 2017, a summary of the highlights and what to look out for. 

Mar 18, 2016
By Opsview Team, Administrator

A comprehensive guide on how to prepare your company and its IT for growth by means of office expansion. 

Apr 15, 2016
By Opsview Team, Administrator

There may be initial challenges in convincing your manager to switch from a free tool such as Nagios to a more costly investment. It is your job...