What Is a Data Pipeline?

You may have seen the iconic episode of "I Love Lucy" where Lucy and Ethel get jobs wrapping chocolates in a candy factory. The high-speed conveyor belt starts up and the ladies are immediately out of their depth. By the end of the scene, they are stuffing their hats, pockets, and mouths full of chocolates, while an ever-lengthening procession of unwrapped confections continues to escape their station. It's hilarious. It's also the perfect analog for understanding the significance of the modern data pipeline.

The efficient flow of data from one location to the other - from a SaaS application to a data warehouse, for example - is one of the most critical operations in today's data-driven enterprise. After all, useful analysis cannot begin until the data becomes available. Data flow can be precarious, because there are so many things that can go wrong during the transportation from one system to another: data can become corrupted, it can hit bottlenecks (causing latency), or data sources may conflict and/or generate duplicates. As the complexity of the requirements grows and the number of data sources multiplies, these problems increase in scale and impact.

The Data Pipeline: Built for Efficiency

Enter the data pipeline, software that eliminates many manual steps from the process and enables a smooth, automated flow of data from one station to the next. It starts by defining what, where, and how data is collected. It automates the processes involved in extracting, transforming, combining, validating, and loading data for further analysis and visualization. It provides end-to-end velocity by eliminating errors and combatting bottlenecks or latency. It can process multiple data streams at once. In short, it is an absolute necessity for today's data-driven enterprise.

A data pipeline views all data as streaming data and it allows for flexible schemas. Regardless of whether it comes from static sources (like a flat-file database) or from real-time sources (such as online retail transactions), the data pipeline divides each data stream into smaller chunks that it processes in parallel, conferring extra computing power.

The data pipeline does not require the ultimate destination to be a data warehouse. It can route data into another application, such as a visualization tool or Salesforce. Think of it as the ultimate assembly line (if chocolate was data, imagine how relaxed Lucy and Ethel would have been!).

How Is a Data Pipeline Different From ETL?

You may commonly hear the terms ETL and data pipeline used interchangeably. ETL stands for Extract, Transform, and Load. ETL systems extract data from one system, transform the data and load the data into a database or data warehouse. Legacy ETL pipelines typically run in batches, meaning that the data is moved in one large chunk at a specific time to the target system. Typically, this occurs in regular scheduled intervals; for example, you might configure the batches to run at 12:30 a.m. every day when the system traffic is low.

By contrast, "data pipeline" is a broader term that encompasses ETL as a subset. It refers to a system for moving data from one system to another. The data may or may not be transformed, and it may be processed in real-time (or streaming) instead of batches. When the data is streamed, it is processed in a continuous flow which is useful for data that needs constant updating, such as a data from a sensor monitoring traffic. In addition, the data may not be loaded to a database or data warehouse. It might be loaded to any number of targets, such as an AWS bucket or a data lake, or it might even trigger a webhook on another system to kick off a specific business process.

Who Needs a Data Pipeline?

While a data pipeline is not a necessity for every business, this technology is especially helpful for those that:

• Generate, rely on, or store large amounts or multiple sources of data.
• Maintain siloed data sources.
• Require real-time or highly sophisticated data analysis.
• Store data in the cloud.

As you scan the list above, most of the companies you interface with on a daily basis — and probably your own — would benefit from a data pipeline.

Types of Data Pipeline Solutions

There are a number of different data pipeline solutions available, and each is well-suited to different purposes. For example, you might want to use cloud-native tools if you are attempting to migrate your data to the cloud.

The following list shows the most popular types of pipelines available. Note that these systems are not mutually exclusive. You might have a data pipeline that is optimized for both cloud and real-time, for example.

• Batch. Batch processing is most useful for when you want to move large volumes of data at a regular interval, and you do not need to move data in real-time. For example, it might be useful for integrating your Marketing data into a larger system for analysis.
• Real-time. These tools are optimized to process data in real time. Real-time is useful when you are processing data from a streaming source, such as the data from financial markets or telemetry from connected devices.
• Cloud native. These tools are optimized to work with cloud-based data, such as data from AWS buckets. These tools are hosted in the cloud, allowing you to save money on infrastructure and expert resources because you can rely on the infrastructure and expertise of the vendor hosting your pipeline.
• Open source. These tools are most useful when you need a low-cost alternative to a commercial vendor and you have the expertise to develop or extend the tool for your purposes. Open source tools are often cheaper than their commercial counterparts, but require expertise to use the functionality because the underlying technology is publicly available and meant to be modified or extended by users.

Taking the First Step

Okay, so you're convinced that your company needs a data pipeline. How do you get started?

You could hire a team to build and maintain your own data pipeline in-house. Here's what it entails:

• Developing a way to monitor for incoming data (whether file-based, streaming, or something else).
• Connecting to and transforming data from each source to match the format and schema of its destination.
• Moving the data to the the target database/data warehouse.
• Adding and deleting fields and altering the schema as company requirements change.
• Making an ongoing, permanent commitment to maintaining and improving the data pipeline.

Count on the process being costly, both in terms of resources and time. You'll need experienced (and thus expensive) personnel, either hired or trained and pulled away from other high-value projects and programs. It could take months to build, incurring significant opportunity cost. Lastly, it can be difficult to scale these types of solutions because you need to add hardware and people, which may be out of budget.

A simpler, more cost-effective solution is to invest in a robust data pipeline. Here's why:

• You get immediate, out-of-the-box value, saving you the lead time involved in building an in-house solution.
• You don't have to pull resources from existing projects or products to build or maintain your data pipeline.
• If or when problems arise, you have someone you can trust to fix the issue, rather than having to pull resources off of other projects or failing to meet an SLA.
• It gives you an opportunity to cleanse and enrich your data on the fly.
• It enables real-time, secure analysis of data, even from multiple sources simultaneously by storing the data in a cloud data warehouse.
• You can visualize data in motion.
• You get peace of mind from enterprise-grade security and a 100% SOC 2 Type II, HIPAA, and GDPR compliant solution.
• Schema changes and new data sources are easily incorporated.
• Built in error handling means data won't be lost if loading fails.