There’s a reason the technology industry uses a pipeline metaphor to describe how data moves over the Internet and through applications and hardware. Like a plumbing system, a data pipeline has limited capacity. Old pipes and too much traffic can cause slowdowns and clogs.
Right now, many businesses’ data infrastructure is like an aging plumbing system with leaky, rusty pipes hidden inside the walls. As long as the water continues to run, we don’t see a problem–until a flood breaks through the drywall.
We need to shore up and update our infrastructure to prepare it for the incoming influx of data – at least 175 Zettabytes over the next five years, generated by both humans and machines. To provide some perspective, it would take 1.8 billion years to download that amount of data at the average current Internet connection speed.
And it’s not just a matter of delivering data–it all needs to be organized, sorted, and analyzed to support the insights that are going to drive innovation for the decades to come. An updated data pipeline is the foundation for the nonstop data demands of the future. “There’s the old world of data at rest, and then there’s this new world of data in motion,” says Dell Technologies CTO John Roese, describing it as “telemetry information flowing in from the real world, through the IT infrastructure into AI engines, providing insights, training models, providing feedback systems, automating the world.”
These are the essential components required to support the next-generation data pipeline.
Artificial Intelligence and Machine Learning
Artificial intelligence and machine learning are the heart of the intelligent data pipeline that is required in the Zettabyte world. Easy access to data will not be enough – it must arrive in a form that is relevant and actionable. As Unconventional Ventures founder Theodora Lau puts it:
“Imagine a water system that takes water from different sources and delivers clean drinkable water to the end customers.”
Intelligent infrastructure–with its ability to take data from disparate sources with varying permissions and inconsistent formats and classify, tag, and route it to the appropriate end-users automatically–is key to keeping data running smoothly and continuously. Smart data pipelines also reduce latency, which increases their efficiency. Building on the plumbing comparison, you don’t want water going through creaky old pipes or standing in the system for a long time–it can get stale or rusty. AI and ML act on, augment and enhance the data as it moves continually throughout the system without backups or bottlenecks.
Edge Computing
In the Zettabyte world, more computing will be done at the edge–meaning at the site of smart and connected devices, rather than shuffling data back and forth to the cloud for processing. Going back to our plumbing metaphor, if you think of the cloud as the main municipal line to which all the water flows, the edge is the kitchen and bathroom pipes connected to it. Often main lines get gummed up with debris from all the water they’re collecting, which can cause slowdowns and clogs for the entire system. By bypassing the “mainline” of the cloud, we minimize stress and traffic on the entire data pipeline and keep it clearer, increasing capacity and speed across the whole system.
This diffused approach may seem counterintuitive given the focus on the benefits of cloud centralization, but in fact, edge computing complements the cloud by supporting faster and larger data pipelines–which in turn helps drive more innovation with how data is gathered, analyzed, and activated.
5G
If data is the water in our metaphor, 5G is the physical pipes it travels through. Imagine a big old house with a plumbing system that was originally designed for a single family. The narrow pipes are adequate for its needs–but then the house is divided into condos and there are multiple families living there. The pipes can’t keep up with the demands on the system–one person running a dishwasher means that there’s no water pressure when someone else goes to take a shower.
5G is the upgrade that will support the increased volume and speed of data in the Zettabyte world. While the rollout of this potentially transformative technology is ongoing, we can already foresee the benefits of its implementation. Its capacity to handle data connectivity across an expanding number of devices while also accelerating transmission rates and reducing delays, in tandem with the other elements mentioned here, will help drive greater automation and enhanced functionalities for technologies ranging from self-driving cars to ultra-high-resonance medical scanning.
A Data Pipeline for 2025 and Beyond
Failing to lay the groundwork for the Zettabyte world will mean losing out on the opportunities an expanded capacity for data transmission and analysis offers. A secure and smart data pipeline can support potentially unparalleled levels of technological progress that could drive breakthroughs across industries for decades to come and bring with it economic prosperity, a higher quality of life, and innovations we can’t begin to fathom.
Explore the challenges and promise of preparing for the next five years and the future to come: Download The Zettabyte World: Securing Our Data-Rich Future.