BIG DATA, transforming the future?

Introduction

Big Data is a term coined in 1980 by American sociologist Dr. Charles Tilly in his article “The old-new social history and the new old social history. Charles Tilly in his article “The old-new social history and the new old social history “* and although the context for the use of the term Big Data in this article was different at the time it was published,  in 1997 two NASA researchers Michael Cox and David Ellsworth published the article “Application-controlled demand paging for out-of-core visualization “* where the term Big Data was used for the first time within the current technological context and has been gaining a lot of space and notoriety in recent years including recently boosted by the COVID pandemic that motivated many people to work from home. It is a much-discussed technological topic at the moment and involves large and complex volumes of data produced daily in milliseconds by capturing and sharing information around the world. The whole process encompasses large corporations, social networks, and, more recently, the application of Artificial Intelligence associated with the growing use of smartphones with easy access to the Internet in many countries (50 billion connected devices by 2020), facilitating the sending and receiving of information at very high speed. This large volume of data in various formats is stored in structured, semi-structured, and unstructured combinations, often with levels of complexity that make it difficult to process by databases or traditional applications, resulting in the need for the use and improvement of specific technologies so that this information can be adequately processed.

Key Features of Big Data and SAP Technologies

According to Anurag Agrahari and Dharmaji Rao (2017), we can consider that Big Data is characterized by 7 Vs: Volume, Velocity, Variety, Value, Variability, Visualization, and Veracity. Moreover, all the data collected during the acquisition phase needs to be stored in different locations due to the large volume of information captured, and this data gets organized in structural models configured in rows and columns or unstructured models that are usually data in video format, images, comments, emails or sounds, common in social networks but that cannot be organized in rows and columns. These types of storage represent a great challenge for information technology professionals, as they are often difficult to access or retrieve data that lack the necessary components for identification and interpretation. However, this information shapes the future of business and attracts the interest of large companies seeking to base their strategic decisions within the context of Big Data to guide their plans and services. 

Considering the way the collected data is stored, along with the great strategic appeal it provides, it is also necessary to understand the challenges the corporate world faces in finding professionals qualified to handle this type of information. Given that Big Data goes beyond the traditional Information Technology storage concept, the future of business depends entirely on the willingness of companies to provide the right skills to market professionals within a new Information Technology context. In this new context, SAP professionals in particular play an important role within consultancies or working directly with clients, compiling large amounts of data and providing information and data analysis mechanisms through new tools.

Among the new SAP tools available, BW/4HANA is considered the core of the SAP big data platform and consists of a data warehouse package based on SAP HANA. It is an on-premise data warehouse layer of SAP’s Business Technology Platform (BTP) that SAP consultants use to consolidate data from across the enterprise and provide a more consistent and simplified view as a source of new ideas in real-time. Its in-memory data architecture has a columnar, distributed format to handle large data sets. This format, allocated entirely in memory, allows the SAP professional to perform additional complex calculations, perform data-intensive functions, and operations on content available directly in the database with the use of a rich SQL layer, and patented indexing without the need for aggregations that can be time-consuming and costly.

SAP HANA can also host data from other information sources such as Hadoop, extending the processing range. Hadoop provides flexible and spacious storage for data objects, regardless of their structure and that are too large to fit into memory, requiring a pre-processing step before being analyzed. By connecting SAP HANA to Hadoop the SAP consultant can run jobs on Hadoop, load the information into HANA, and provide a final, reliable analysis to his client.

SAP BusinessObjects BI can bring Hadoop and database sources together to serve information to business applications. All this technology works together to bring big data into the enterprise.

In addition, a new processing technology called “Intelligent Enterprise” where artificial intelligence is used to automate systems through “machine learning” using Big Data…