By Ms. Palak Gupta

In today’s scenario when data is a very important asset in the organizations, there are multiple techniques that the organisations are seeking now in order to not only collect data from multiple sources but also to store it and analyse it to perform analytics on it so as to get better insights and better understanding of the data which imparts better knowledge management and also supports management in taking good decisions varying from short term to long term. Today the data is not only coming from various structured sources like we have various text, image and multi-media enabled data for which we are using various database solutions in the form of Oracle, SQL server or MySQL but a lot of data today is coming in the form of semi structured and unstructured because now we are exposed to more sources of online platforms rather than offline platforms. These online platforms actually generate data in multiple forms that are totally different from the conventional data types that are used in basic database solutions so here the term big data comes into play.

Big Data

So big data is similar to small data but it is bigger in size. When we say data is bigger in size it means it is characterized by volume, velocity and variety i.e., data that is generated from digital and online platforms is immense in volume, continuously generating online and it is of variety of formats. So the three Vs of big data are volume, velocity and variety but apart from this we also associate other big data characteristics in the form of viability, value and veracity.  In case of big data, the data generated is heterogeneous and of diverse dimensionality as it is generated from autonomous sources having distributed and decentralized control and having complex and evolving relationships. The data structure of big data is basically combination of all structured, semi-structured and unstructured data that is generated continuously from various online platforms along with other types of data like sensor generated data, chip data, circuit data, CC camera generated data, data from various social media platforms where we keep posting and on these posts we get lots of likes, share, comments, tweets etc. which all account for big data data types. The big data sources are users, application systems, sensors, social network profiles, social influencers, mapreduce results, cloud apps, activity generated data, legacy documents, data warehouse appliances, network and in-stream monitoring technologies and live streaming of audio and video etc. 

Tools for Big Data

To work on big data there are different types of tools and technologies that are deployed like distributed server is used for processing, data is stored in distributed storage, the programming model generally followed is of MapReduce and Massive Parallel Processing (MPP), data is stored in schema free data bases that is allowed for lots of analytics and semantic processing. So, for these we use different categories of softwares to process not only storage level but also analytical level big data like the most common one is Hadoop along with Amazon EC2, Amazon S3, Hadoop distributed file system, mongo database, Cassandra etc. For Big Data Analytics we need to examine large amount of data and extract appropriate information by identifying hidden patterns and unknown correlations as it gives competitive advantage, better business decisions and helps the organisations in leading to effective marketing, customer satisfaction, and increase revenue. Big Data Analytics has diverse applications in the areas of smart healthcare, multi channel sales, homeland security, trading, analytics, telecom, traffic control, search quality and manufacturing. Big data is widely used for protection and prevention in areas of revenue protection, site integrity and uptime. It is used in digital marketing optimisation in the form of web analytics and doing a lot of golden path analysis. Social networking and relationship analysis is also done through Big Data Analytics by evaluating influencer marketing, outsourcing, and attrition prediction. It is also used in machine generated data analytics, data retention and data exploration and discovery by identifying new data-driven products and new markets.

Figure-Applications of Big Data Analytics

Challenges

Big data has lots of challenges for the management with respect to voluminous data, decision making, change management, clash of technology as market is full of options for normal file system, distributed file system, Hadoop, Amazon platform, R, Google, Oracle etc. There is also shortage of skill set in the area of Big Data Analytics so people need to progress themselves in the areas of data science and analytical tools to fill this gap. There are a lot of challenges in big data related to its storage complexity, utilisation gap,  privacy, security, real-time analysis, management etc. but of course big data is not only for today but it is the future and we cannot say no to big data management so we should prepare ourselves and our corporate for handling this big data. Despite of so many technologies available for big data management still there are lots of challenges that are dealt and will be dealt in the future also because the level of data that is generated online along with offline sources is going to increase day by day so better improvement in the existing technologies are also required to make up with this big data and it’s ever increasing demand in the future.

Benefits

Despite all this, big data has lots of benefits to provide to the corporates in form of real time analytics, storing penta bytes and even more data in data warehouses where it has the ability to make better decisions and take meaningful actions at the right time fast forward to the present and technologies like Hadoop that enables scalability and flexibility for data storage before we actually know how to process the data. Technologies like MapReduce, Hive and Impala will be enabling to run queries without changing the data structures underneath thus supporting not only structured but also all forms of semi-structured and unstructured data.

Ms. Palak Gupta

Assistant Professor

Jims Kalkaji