What is Big Data?

Data has always been an important asset in every industry. Since the early days of the information age, business intelligence and descriptive statistics have been used as the standard tools for extracting information and make important decisions from all kinds of collected data. However, as the cost of collecting, storing, and processing data has been dropping exponentially, the amount and the diversity of the data has reached the point where traditional approaches are no longer feasible. In fact, the term Big Data is often used to refer to any data that requires new techniques and tools in order for it to be processed and analyzed.

A more formal definition of Big Data was introduced by Gartner in 2012, in which the well-known 3Vs – Volume, Velocity, and Variety - were used to characterize Big Data. Since then, the 3Vs model has been expanded to several other characteristics, including a fourth V for Veracity and more recently a fifth for Value. Without going into the details of each V, and resisting the temptation to look for a sixth V, we could also look at Big Data from the point of view of the new set of technologies that are helping to solve the challenges in collecting, managing, and analyzing Big Data. These technologies include cloud computing and cluster computing (with Hadoop MapReduce as the most well-known example) for data storage and manipulation, and machine learning for data analysis.

What value does Big Data bring?

The value of Big Data comes from two main use cases: as a source of analytics, and as an enabler of new products and services. In the first use case, big data analytics is used to improve an existing business model by revealing insights from data which was previously too costly to store or process. Amazon’s recommendation system, USPS’s preventive maintenance system, or Walmart’s demand forecasting system are all very good examples. These companies track, collect, and store all available data, from customer transactions to social data, from GPS trails to geographical and meteorological data, then combine them together and use big data analytics to produce high value actionable insights. This would not be possible without the new big data technologies such as cluster computing and machine learning.

In the second use case, big data technologies open up completely new business models and introduce new products and services. Many recent so-called unicorn startups, such as Airbnb, Uber, or Snapdeal, are founded on and enabled by big data analytics. Their products have unique features thanks to the new technologies that they are using. Without following the big data approach, their products would not be able to compete against all the traditional business models.

Big data at eSmart Systems

Since the early days of the company, big data has always been considered as our core value. Big data analytics and machine learning are involved in many of our products.  They are used to forecast electricity demand at substation level, segment customers based on their power consumption patterns, and implement demand response strategies. We have also started to use machine learning to automate the analysis of power lines imaging surveys autonomously conducted by drones with the aim of assisting failure detection and preventive maintenance.