1, 2, 3…. Xn  – this is a very simple representation of the exponential growth in services and APIs with which an organization must contend. To store and process this data, most organizations start with a basic architecture that meets their requirements for their initial stages. Once they reach the tipping point where the operational complexity and cost of ownership rises exponentially, or if they have experienced service-level agreement (SLA) violations, organizations will look for a more robust solution.  

Today’s business landscape demands that organizations meet the constant demand for new digital services ‒ fast, while offering high-scalability and low-latency. Conventional IT architectures that integrate data via request-based APIs, struggle to effectively support the increased load from digital applications. Systems of Record (SoR) were centralized to reduce operational costs and to simplify maintenance. This results in performance bottlenecks and also creates a single point of failure for the services that are dependent upon the SoRs. These factors lead to a poor customer experience and impede innovation since the organization cannot build new digital services at scale. 

Introducing the Digital Integration Hub

Enter the Digital Integration Hub (DIH), offering event-based data integration which is optimized to power digital applications. The DIH decouples digital applications from the SoR, aggregating data from multiple sources into an ultra-low-latency, high-performance data fabric. This replication of the appropriate SoR data reduces the latency of the API services layer, dramatically increasing throughput and scalability. Offloading expensive API-generated workloads from the SoRs improves API performance and enables high availability.  It also reduces the risk of introducing new services and enables an end-to-end microservice approach. The real-time, event-driven data delivery characteristic of a DIH optimally combines data integration and data management technologies, data and analytics governance, and related architecture design and services. 

At the tipping point – DIY or Out of the Box 

For organizations that know that their current operations need a more powerful system, they have two main choices in how to implement a DIH – Do It Yourself (DIY) or Out of the Box (OOB). 

On your own – a DIY DIH 

Here’s a look at some of the components and integrations involved in setting up a DIH architecture, incorporating multiple Systems of Record (SoRs). These can be any combination of legacy relational databases, semi-structured NoSQL databases; reporting and analytics applications, in addition to CI/CD, message queues, replication, microservices, ETLs, CDCs, and monitoring services, all of which required integration, orchestration, maintenance, and monitoring. 

Some of the major challenges include: 

• Cache synchronization in OLTP systems   
• Difficulties in replicating SoRs while maintaining consistency between all services
• Maintaining SoR isolation levels and building atomic transactions 
• Ensuring high availability, and scalability of services
• Issues in changing schema on the fly without downtime
• Difficulty in error handling  

Yes, DIY can work, but it requires a very lengthy system integration effort that may take months or even longer. Upgrades and maintenance are a significant effort with so many dependencies, often resulting in extended downtime. This sketch shows a rough outline of the main components in the system, and how easy it is for the system to become a crazy maze of connections, instead of a well-oiled machine that runs efficiently 24/7. 

To the rescue – DIH, Out of the Box

In contrast to DIY, an out-of-the-box Digital Integration Hub (DIH) solution such as Smart DIH enables the rapid launch of digital services, even if they rely on complex integrations. Organizations can implement a single platform that integrates cloud-native and legacy systems, enabling teams to focus development efforts towards delivering a steady flow of high-performance digital services, instead of spending time on data integration efforts. 

GigaSpaces Smart DIH utilizes a microservices architecture and a unified API layer, to accelerate innovation and the rapid building and introduction of new digital applications and services. Not to be confused with a simple cache which keeps the most frequently used data, the DIH keeps all the relevant data and pushes it to the service as soon as it is changed on the SoR, in a manner similar to push cache mechanisms. With Smart DIH, enterprises can rapidly launch new cloud-native, digital services even if they rely on legacy systems, focusing development efforts on delivering a steady flow of high-performance digital services, instead of executing complex SoR integrations.

Let’s dive into the major priorities for organizations as to how to provide frictionless customer experiences, starting with ensuring High Availability and shortening Time to Market, and how an out-of-the-box DIH can offer a significant competitive advantage. 

Enabling Efficient Creation of Data-Driven Services with a Data Layer 

High availability of digital services and the data that feeds them is not optional – enterprises must meet strict SLAs and deliver excellent customer experiences to be competitive. However, most current architectures are not designed to optimize these processes, because legacy monolith systems, by definition, are hard to operate. Those who took the recommended microservice path are struggling with data synchronization issues, often requiring workarounds, business logic redundancy and limited automatic service discovery. 

The original data structures may have been optimized for the use by core legacy applications, but may not be optimized at the level required by modern operational or digital applications. As a result, complex manipulations may be required between multiple layers of cache and databases, and between multiple data stores. These intricate connections lower performance and limit elasticity. 

In the past, a common technique was to write some of the business logic on the actual databases, i.e. stored procedures. While this approach worked well when the number of services was limited, this is not feasible at scale. Running business logic on the SoR has become an anti-pattern, as SoRs are designed and optimized almost exclusively to store data.

To overcome this, modern architectures such as DIH were created to enable online business processes in a scalable manner.  Smart DIH provides a single, unified, high performance data store – “one layer to rule them all” – directly above the SoR, instead of a separate database and cache, ensuring automatic end-to-end data synchronization from the SoR via the DIH to the services. This unified data layer becomes the ‘single source of truth’, reducing errors and eliminating duplication. Validation and cleansing policies are applied upstream as part of the data pipeline configuration (in the data integration layer), reducing the burden from service developers to address quality issues downstream, at the data microservice level. The embedded service management layer enables high availability, eliminating any single point of failure, also providing redundancy and automatic self healing, scaling to meet SLAs. 

Faster time to market by shortening development cycles 

Delivering digital services involves complex integrations because different databases keep data in different formats. In a DIY architecture, integrating relational and non-relational data from systems such as the financial, logistic, HR, retail, and sales, among many other types,  requires manual conversions and integrations, leading to inconsistency, downtime, slow delivery, and error prone data. This results in slower development of new digital services, and time-consuming maintenance for the existing services.

In contrast, Smart DIH offers automatic metadata management across disparate databases, with automatic schema management based on SoR schema: including discovery and mapping. Automatic data type conversion and customization is included, in addition to automatic metadata integration with BI tools, data governance tools and client tools. Since SQL remains the most important tool for querying any data model, Smart DIH is fully compatible with relational data models and enables queries of all the data using SQL queries. With an out-of the-box DIH, data is accurate and new data is integrated automatically despite the original format, almost eliminating manual efforts and vastly improving accuracy.

Ensuring always-on services

Customers and users continue to demand new and improved user experience, and developers continue to create and improve their apps. Certain activities, such as introducing new services pose a risk by introducing new code into the heart of the organization’s data, potentially impacting multiple services. Some of these new services may require a full data load, which requires suspending the real-time stream of online data change events from the SoR. This may mean downtime for live digital services that may last hours or alternatively  running on stale data, which is not an option for mission-critical apps.

When adding new services and conducting a full data load, developers must proceed slowly and carefully to mitigate any risk while introducing new services, usually requiring input from DBAs and service developers, all of which slows the pace of innovation. 

In DIY systems data synchronization between the SoR and the services is challenging, because each service needs to manage its own data. Data management in DIY systems is also challenging on the ‘macro’ level, since no coherent view of data across different SoRs is available. 

Seamless digital service deployment with the addition of new digital services

The Smart DIH architecture decouples the digital services from the SoRs, enabling the introduction of new tables and their initial load in parallel to live Change Data Capture streaming. This design also ensures that existing digital apps are not impacted. These factors minimize downtime, which, combined with the automatic management of service’s lifecycle and its associated data, ensures always on services.

Seamless digital service deployment during schema changes 

A full data load is usually required after events such as SoR data corruption, ad hoc data load use cases, or adding new tables. For a DIY environment this almost always results in service downtime, something all businesses want to avoid. Here the Smart DIH advantage is clear – it supports ingestion from multiple Change Data Capture streams, and provides ultra-fast data loads leveraging proprietary algorithms and parallelism to write to memory. Data event streams can continue during data load, so that services can continue operating. Since the entire process is streamlined, few experts are required, vastly reducing the number of professionals required in DIY environments. 

Seamless digital service deployment during the addition of SoRs into the DIH 

So much easier – Smart DIH includes an embedded CDC with end-to-end data synchronization from the SoR to the service, and offers a single point-in-time view across multiple SoRs, enabling a unified, holistic view of the data, leading to more effective management and decisions. Since each backup node is automatically synced, the system provides automatic failover and service redundancy.

Delivering ultra-low latency and near-linear scalability

Since most services rely at least in part on data from legacy sources, it’s likely that certain data is still located in monolithic disk-based databases, such as Oracle DB and SQL Server. Unfortunately, these databases are not built to achieve the low latency or high performance required to power demanding digital applications. Moving this data to modern data stores is far too complex and risky for most organizations to consider. One solution is to add a cache on top of a relational DB; this will improve output but cannot provide ultra-high performance since data still has to travel over the network, resulting in latency. Lastly, data stores with monolithic or shared resources are difficult to scale.

Achieving high performance and near linear scalability is possible with the memory first, shared nothing distributed architecture that embeds Smart DIH and utilizes in-memory computing. Latency is almost eliminated by co-locating the data with logic, which provides an order of magnitude of speed above sending data over the network.

Ensuring real-time data consistency for transactions across all digital channels

In DIY configurations ensuring data integrity across all data store layers is highly complex, involving manual management of the state of each transaction across all products. While relational databases offer full consistency, most NoSQL databases only provide Eventual Consistency, which can result in gaps in data freshness and accuracy.

Smart DIH is a proven, ACID compliant, fully transactional platform offering data freshness control – the ability to identify stale data, for example when a pipe or stream is down for a period of time, and notify the operations and admin teams, as well the digital app. The DIH also provides automatic data error resolution with conflict resolution logic that doesn’t break the consistency of the  transactions.  

Fully compatible with Relational Data Models 

Most enterprises have many types of data that is stored in various formats, and calling this data requires complex manipulations for developers. Organizations typically have data housed in relational databases, but their newer services usually require data available in NoSQL or NewSQL formats which don’t always support SQL, requiring specific expertise to make this system work. Implementing a relational model on top of a non-relational datastore is no simple task, resulting in data duplication which limits data integrity. 

Here a DIH shines, with full support for both relational data models and for documents, including their nested collections. Developers can utilize simple and common APIs, leveraging the organization’s skill sets of SQL and client SDKs. In particular, Smart DIH provides automatically optimized SQL to work on a distributed or partitioned environment. 

Smart DIH – Ensuring always-on services

Organizations today must offer optimal customer experiences and always-on services, which requires an ultra-high performing platform to ensure efficient operations. Building this type of platform from scratch is possible, but is risky and unproductive, since it involves many moving parts that don’t know how to communicate without serious manual intervention. Scaling these environments is unworkable, and data integrity is at risk. 

To achieve true efficiency and always-on services, an out-of-the-box DIH, like Smart DIH, offers faster time to market for new digital services that are created in shorter development cycles, with minimal service downtime. Smart DIH offers extreme in-memory performance, elasticity and scale, business policy-driven tiered storage, low code microservices creation, no-code data source integration, and real-time data consistency.