概要
The ongoing realization of just how data-dependent the enterprise world has become is reflected in organizations' budgets. IDC projected that spending on data and analytics solutions reached $215.7 billion in 2021—a 10.1% jump from the previous year.
Data engineering doesn't always draw as much attention as its counterpart, data science, which is among the biggest data trends of the past decade. But engineering is every bit as important to overall enterprise data management as data science. In fact, the two are effectively joined at the hip.
What is data engineering?
Data engineering is a discipline focused on the implementation and operation of data pipelines and infrastructure within an organization. It ensures that enterprise data flows efficiently, is properly integrated and governed, and is up to date throughout the data ecosystem. Data engineering also optimizes data for operational and analytical purposes.
The professionals who ply this trade are called data engineers. They work closely with data architects: Those on the architecture side of things design database systems, data models, and other major components of the ecosystem. Meanwhile, engineers make sure that these elements—and the various technologies that support them—are functioning properly, so that data always gets to where it needs to be.
Without the processes of data engineering, many of the advances cited as major data science trends wouldn't be possible. Data engineering serves as the foundation upon which critical data science and analysis projects can be built.
Critical structures of data engineering
Discussing the purpose and function of data engineering must begin by taking a closer look at what a data engineer implements and operates to ensure proper data flow within an enterprise. Arguably, the most important of these structures or systems is the data pipeline, as it forms the connective tissue between sources of data, its ultimate destinations, and whatever stops it makes along the way.
To illustrate how data pipelines work, let's go through a hypothetical one step by step.
- Data enters an organization's pipeline after it is ingested and extracted from data sources. These sources can be just about anything—databases, software as a service (SaaS) applications, household devices connected to the Internet of Things, websites, industrial sensors, and so on.
- Ingestion of data can occur on a real-time basis—in which case it can also be called streaming data—or in batches, depending on the nature of the data's sources and their ability or inability to retain the data they generate.
- The data ingestion stage is then followed by transformation and loading, as part of the greater extract, transform, and load (ETL) process. Specific ETL sequences will vary depending on what the data is and where it's ultimately headed. But in virtually all instances, data from various sources will have to be transformed—often into different formats, so that it can be compatible with its destination application or database—and loaded into the appropriate storage repository.
- Sometimes, extract, load, and transform (ELT) operations will take the place of ETL, especially if a data lake is the storage repository in question.
- Key data storage architectures that will come into play as data passes through the pipeline include data lakes, data marts, and data warehouses. These all have specific purposes. The data lake holds massive volumes of structured or unstructured raw data, whereas the data warehouse is for structured relational data. Data marts are small, dedicated storage structures for specific subjects—e.g., sales data. A data lakehouse combines the storage capacity of a data lake with the structured processing of a data warehouse.
- The pipeline ends when the data reaches the end user and application that needs it. This could be anything from an analyst from the data team looking for key performance indicators (KPIs) of a department's productivity, to a consumer looking up a restaurant's location on a mobile app.
Essential data engineering processes
Data engineering ensures that data's journey through an organization's data pipeline is as seamless and unobstructed as possible. And once the pipeline has been set up, many of its operations are automated. But while it would be fantastic for all involved if data's path along the pipeline was always perfect and free of obstacles or any other issues, this simply isn't always the reality—largely because so many different types of data are being ingested, from a multitude of disparate sources.
As a result, data engineers conduct a variety of intervening operations—or implement scripts to automate them when certain issues are detected—to improve the integrity and quality of data while it is in the pipeline.
Some of these processes fall under the umbrella heading of data normalization, such as the following:
Data modeling
The creation of a data model is often associated with the work of data architects, but it's important to data engineering as well. A data engineer working with data in a pipeline must ensure that the data conforms to a specific model before it heads to its destination. Modeling helps ensure that data is properly formatted so it can be accessed and ultimately leveraged by business users, analysts, and applications.
Data cleansing
As the name of this process suggests, data cleansing or cleaning involves getting rid of errors and redundancies that often crop up when handling large data sets. For example, data deduplication is all about eliminating redundancies from a given piece of data, so that only the unique data is stored. Other data cleansing tasks involve correcting mislabeled data, identifying any missing values and filling them in whenever possible, adjusting values of a field to fit within an appropriate range, and eliminating any irrelevant outlying or corrupted data.
Data accessibility
Data accessibility involves making sure that all data, including the most valuable data, is accessible to all business users and analysts who need it. Data engineers can do this by grouping data based on a specific metric and routing it to a simple reporting interface—as is useful for business analytics teams—or ensuring data can be accessed through the use of an easy-to-understand query language.
Data integration
This is one of the ultimate goals of data engineering and a key end result of ETL operations. Data integration ensures that regardless of data's origins or formats, it is consistently delivered and made accessible to business users. The more data is integrated, the more reliable and reusable the insights derived from that data will be.
Other aspects of data engineering focus on maintaining the areas of the pipeline essential for storage and processing.
Data warehousing
Data warehouses are often put together by data architects, but the responsibility for their ongoing maintenance—and that of related structures—lies with the data engineering staff.
Engineers, for example, are who put together data marts. Sometimes these are inside the data warehouse, sometimes outside, and other times they're created using warehouse and non-warehouse systems simultaneously. The goal of the data mart is to make department-specific data quickly and efficiently accessible. Data engineering teams also oversee data lakes—both cloud-based data lakes created with low-cost object storage, and those made with on-premises Hadoop systems. Database management system (DBMS) platforms are invaluable to these efforts.
OLAP
The traditional transaction-optimized database—based on the online transactional processing (OLTP) method—is limited in ways that may be unsuitable for the modern enterprise. Those limitations not only led to the development and realization of concepts like data warehouses, data lakes, and data marts, but also the creation of a new method for data analysis: online analytical processing (OLAP).
In OLAP, data sets are represented in multidimensional cube-like structures—OLAP cubes—that aggregate a large number of important metrics. If, for example, you are reporting on the efficacy of a maintenance department in an industrial facility, you might include dimensions like technicians' names, years of experience, field of expertise, relevant certifications, and rates of success or failure.
Programming
Every aspect of computer engineering is based on careful use of the right programming languages. For data engineering, Python has become the most popular type of code. Other languages used by data engineering teams include structured query language (SQL), R, Scala, Java, and various specialized code types for NoSQL databases.
Data engineers: Roles, responsibilities, and skills
The success of an enterprise's data ecosystem depends largely on those who are given the responsibility to manage data engineering tasks—data engineers.
Data engineering skills are somewhat similar to the skill sets of data scientists and analysts. But the main difference is that engineers focus more on nuts-and-bolts tasks: creating data storage solutions, maintaining pipelines, and overseeing ETL and data integration processes. Data scientists need to understand the basics of these processes but don't have to be experts. Similarly, data engineers should get the gist of machine learning (ML) and its role in modern data initiatives, but being experts in it is the responsibility of data scientists.
The specifics of a data engineering role can vary between organizations. Some engineers focus on the discipline broadly—creating data pipelines and platforms, supervising integrations, and making sure that data flows remain stable within on-premises and cloud environments. Others have more precisely defined responsibilities:
- Data quality engineers focus specifically on quality. Tasks on their plate include cleansing databases of bad data and working on solutions to prevent or mitigate data quality issues.
- Some data engineers are dedicated to implementing and monitoring single aspects of data architecture, such as data pipelines or databases. They will likely work closely with the data architects and modelers who contribute to the planning of such designs, as well as other general-focus engineers.
- Database administrators—not to be confused with database-centric engineers—focus on the day-to-day management of database functions.
- Software developers and engineers contribute to the field of data engineering by creating and implementing tools needed for pipeline management and operationalization, data warehousing, and other responsibilities.
Because data engineers don't necessarily all do the same work, their skill sets won't all be perfectly identical. But skills such as proficiency in all programming languages involved in database development are a must for all data engineers. Experience with DBMS software, ETL and ELT platforms, and other data integration tools is also essential.
Furthermore, while expertise in machine learning isn't critical for data engineers, those in these roles should be able to deploy ML algorithms in contexts where they're most common, such as on-premises or cloud-based data lakes.
How engineers can support cloud analytics
It has become abundantly clear that the future of data analytics lies in the cloud. Many of the tools most often used by data engineers to keep data architecture running smoothly—so that data analytics operations offer as much value to businesses as possible—are either based in the cloud or can readily be deployed there. Examples include open-source integration tools like Apache Spark and Kafka, object storage platforms like Amazon S3 or Azure Blob, and data pipeline management solutions such as Airflow.
Teradata Vantage can be a critical tool in any data engineer's box. While designed to be intuitive, and thus as valuable to the business user as it is to the data team, Vantage also offers unparalleled multidimensional scalability, unlimited data ingestion, and advanced data workload management. Moreover, it's compatible with the "big three" cloud platforms—AWS, Microsoft Azure, and Google Cloud—as well as the critical data storage, management, and integration tools named above.
To learn more about Vantage and Teradata, take a look at Gartner's 2021 Magic Quadrant and Critical Capabilities reports, which name Teradata a leader in cloud DBMS and Vantage the highest-ranking platform of its kind across all four analytics use cases.
Learn more about Vantage