Databricks: Architecting our way out of the container hype cycle

This is a Q&A features David Meyer in his role as SVP of products at Databricks – a data lakehouse provider.

Download this free guide

The pandemic’s e-commerce boom

The pandemic appears to have solidified e-commerce's ascendancy against the highstreet. Coronavirus has accelerated technology adoption in many sectors, and people have been forced to stay at home, increasing their online shopping habits in a bid to avoid visiting shops.

• Corporate E-mail Address:
• • I agree to TechTarget’s Terms of Use, Privacy Policy, and the transfer of my information to the United States for processing to provide me with relevant information as described in our Privacy Policy.
• • I agree to my information being processed by TechTarget and its Partners to contact me via phone, email, or other means regarding information relevant to my professional interests. I may unsubscribe at any time.

Computer Weekly: What do enterprises need to think about when it comes to architecting, developing, deploying and maintaining software containers?  

David Meyer: Firstly, every business needs to focus on the overall outcome they want to achieve. Figure that out and then decide whether or not a microservice is better for you than a monolith. If you have a unit of work that you need to scale out on a lot of machines… then put it in a container, but don’t just follow the hype.

Many tend to think microservices are better than monolith architecture but that’s not necessarily true, it all depends on what you’re trying to achieve. Software is complicated, and containers don’t necessarily simplify the system design. If you’re choosing to wrap your system in a container, make sure the pros outweigh the cons for your organisation. 

CW: Some say that containers will never easily fit legacy architecture systems that fail to exhibit a natural affinity (and support) for API connections. Container ecosystems live and breathe the API dream, so connecting legacy non-API-compliant systems to them will always be tough – right?  

DW: That’s right! Let’s break it down. All containers have an API, the software within the container has a way to interact with it and these two need to be compatible within your system design for containers to make sense… and that’s tough. 

CW: Where the above exists, enterprise organisations may be forced into a predicament where they are running containers, but forced to create parallel systems at some level to work with older legacy systems that can’t be migrated successfully to run alongside cloud-native technologies – right?  

DW: That’s not necessarily a predicament – every enterprise is heterogeneous… and they need to use containers only when it makes sense for them. In the container hype cycle, the world was led to believe that containers made software development easy. While they make some things easy, they lead to other complexities.

For example, how to re-architect entire systems which are in production within containers. Every SaaS company that runs workloads at scale will operate with containers because it’s easier. If you’re a developer working on your own project, one that does not need scaling, it might just be more complex to use containers. There are benefits at scale, but ask yourself if you need to do it. 

CW: Containers are not some golden passport to web-scale scalability and infinite elasticity after all… we’ll just leave that comment there for dramatic effect – right?  

DW: Containers don’t make you scale. If you’re scaling, then containers make that process easier – look at what suits your business more broadly. 

CW: At the end of the day, surely we also need to realise that we can’t get away from the fact that container development, management & monitoring, augmentation & enhancement… leading to eventual retirement (or repurposing) is a tough job and not every firm will have the right skills in place in-house to perform these tasks. Should we all have gone to container university (aka containiversity) first? 

DW: Containers are about efficiency, re-use and convention, but we shouldn’t conflate that with ‘it’s simple to run these systems at scale’ because it’s actually very hard. Instead of thinking of how to run your own containers at scale, think of how to encapsulate your work in the containers and use vendors that can run it at scale.

You should create a vendor ecosystem that’s integrated, where vendors can provide containers-based architectures. 

Then, send someone in your enterprise to container university to learn the tricks of the trade, and, make sure your vendors have a degree from container university before you engage them.