Data engineering in the cloud
Everyday challenges facing data engineers are similar to the ones facing data scientists. Processing data in its various forms is the center of attention for both specializations. The difference lies in what they aim to achieve: data scientists use data to generate insights and build models, while data engineers ensure that the data is available, consistent, and of high quality. In the data engineering context, however, we focus more on production-grade data pipelines, ETL/ELT workflows, and data platforms that need to be robust, reliable, and efficient.
When it comes to the cloud, Python has proven itself to be good enough to incentivize cloud platform providers to use it for implementing and controlling their services. If we look at the biggest players - namely Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure - they all accommodate Python users in their solutions to a number of problems.
Cloud platforms support Python across a much broader range of data engineering workloads than serverless functions alone. Engineers can use Python to write functions that run as part of event-driven workflows, using services such as AWS Lambda, Google Cloud Run functions, and Azure Functions. They can also use it to develop and run data transformations, notebooks, distributed processing jobs, and production workflows within managed data platforms.
Python is now deeply integrated into environments such as Databricks, Microsoft Fabric, and Snowflake through Snowpark. These platforms allow data teams to build and execute Python-based workloads closer to the data, while relying on managed compute, collaborative notebooks, and cloud-native orchestration instead of maintaining the underlying infrastructure themselves.
Python SDKs also make it possible to manage cloud resources, retrieve data, and trigger jobs programmatically across AWS, Google Cloud, and Microsoft Azure, allowing cloud operations to become part of repeatable data workflows.
Distributed processing is particularly useful for ETL workloads that exceed the memory or processing capacity of a single machine. Frameworks such as Apache Spark divide computations across multiple worker nodes, allowing data engineers to process large datasets in parallel. However, the right approach depends on data volumes, workload characteristics, and whether processing can be optimized or executed within an existing data platform.
While Apache Spark is now the go-to technology for data parallelism, a Python wrapper for the Spark engine called PySpark is supported by AWS Elastic MapReduce (EMR), Dataproc for GCP, and HDInsight for Azure.
As far as controlling and managing the resources in the cloud is concerned, appropriate Application Programming Interfaces (APIs) are exposed for each platform.
Cloud providers offer Python SDKs and client libraries for accessing their services programmatically. These include Boto3 for AWS, Google Cloud's Python client libraries, and the Azure SDK for Python. Data engineers use them to retrieve data, manage cloud resources, and trigger processing jobs directly from Python code.
Python is used in cloud data engineering to build and maintain pipelines that retrieve, transform, and deliver data for analytics, reporting, and AI. Its SDKs and integrations allow engineers to connect cloud services and automate processing across different environments.
Python is particularly useful when pipelines require custom API integrations, file processing, automation, or distributed transformations with PySpark. However, not every workload needs custom Python code. SQL, particularly when used with frameworks like dbt that introduce modularity, version control, testing, and documentation, is often better suited to transformations performed directly in a data warehouse, while managed ingestion services can handle data movement without custom extraction scripts. The choice depends on the costs, workload, existing infrastructure, and maintenance requirements.
Designing and maintaining these pipelines requires more than choosing the right programming language. Modern data engineering services combine cloud architecture, orchestration, data quality, and scalable pipeline design to ensure reliable data delivery across the organization.
Focus on data ingestion with Python
Business data may come from various sources of different natures, including databases (both SQL and noSQL), flat files (e.g. CSVs), other files used by companies (e.g. spreadsheets), external systems, APIs, and web documents.
The popularity of Python as a programming language results in an abundance of libraries and modules, including those used for accessing the data, for example SQLAlchemy for some SQL databases, Scrapy, Beautiful Soup, or Requests for data with web origins, among others.
Pandas is widely used for loading, exploring, and transforming structured data in Python. It supports formats such as CSV, JSON, Excel, SQL, and Apache Parquet, making it useful when data engineers need to work with files, database exports, and other structured data sources.
Pandas also supports columnar file formats such as Apache Parquet, which can improve storage efficiency and query performance in analytical workloads.
In modern data lakehouses, Parquet is often used alongside open table formats such as Apache Iceberg, Delta Lake, and Apache Hudi. These add table-level capabilities such as transactional updates, schema evolution, and versioned data access. Python integrates with these environments through libraries and processing frameworks, including PySpark.
Pandas offers a broad set of functions for cleaning, joining, aggregating, and transforming data. For performance-intensive workloads, data engineers can also consider Polars, which supports parallel execution and lazy query optimization, or DuckDB, which executes analytical SQL queries directly against files and local datasets. The choice depends on data size, transformation requirements, and the tools already used in the pipeline.
AWS SDK for pandas (awswrangler), formerly known as AWS Data Wrangler, simplifies data operations across AWS services such as Amazon S3, Athena, Glue, and Redshift. It can also be deployed as an AWS Lambda layer, allowing serverless functions to use the library without packaging it into each function deployment.
The choice also depends on where and how data is processed. Executing transformations directly in the database can reduce unnecessary data movement and operational complexity, while Python-based frameworks such as PySpark support distributed processing when the workload requires it. In practice, well-designed data platforms combine both approaches, selecting tools based on performance, maintainability, scalability, and cost.
Parallel computing with PySpark
Apache Spark is an open-source engine for distributed data processing. It runs computations across multiple machines, making it suitable for workloads that require more processing capacity than a single machine can provide. PySpark is its Python API, allowing data engineers to work with Spark without writing Scala or Java.
PySpark supports Spark SQL, DataFrames, Structured Streaming, and MLlib for machine learning. These capabilities allow engineers to build batch and streaming data pipelines, although performance still depends on how data is partitioned, transformed, and processed.
PySpark workloads can run on managed platforms such as Amazon EMR, Google Cloud Dataproc, Microsoft Fabric, and Databricks. These services provide the computing infrastructure needed to run distributed data processing jobs without requiring teams to manage every component themselves.
In addition, you can connect a Jupyter Notebook to facilitate the development of the distributed processing Python code, for example with natively supported EMR Notebooks in AWS.
PySpark is therefore a powerful tool for transforming and aggregating huge volumes of data, making it ready for consumption by end users, such as business analysts, or by further components, for instance by involving machine learning.
For extracting and loading data, engineers can use tools such as dlt (data load tool), an open-source Python library that helps build data ingestion pipelines. It simplifies moving data from APIs, databases, and files into data warehouses and other destinations, while supporting incremental loading and schema changes.
Job scheduling with Apache Airflow
Cloud providers offer managed orchestration services, reducing the infrastructure work needed to deploy and operate data pipelines. For teams using Apache Airflow, examples include Amazon Managed Workflows for Apache Airflow (MWAA) and Google Cloud Composer.
Apache Airflow is a widely used platform for developing, scheduling, and monitoring data workflows. Engineers define workflows in Python, specify task dependencies, and integrate processing jobs across different services. Airflow 3 introduced a dedicated Task SDK that provides a stable interface for defining workflows and tasks.
Managed offerings such as Amazon Managed Workflows for Apache Airflow and Google Cloud Composer reduce the infrastructure management required to operate Airflow in cloud environments.
Airflow is not the only option, however. Python-based orchestrators such as Prefect and Dagster offer different approaches to pipeline development, observability, asset management, and local testing. The right choice depends on factors such as the existing cloud ecosystem, deployment model, workflow complexity, team experience, and operational requirements rather than on a single universally best tool.
Regardless of the platform, Python remains valuable because it allows orchestration logic, data processing code, integrations, and automation to work within the same broader engineering ecosystem built around technologies such as dbt, Apache Iceberg, lakehouses, and AI-ready data platforms. Beyond writing transformation code, engineers increasingly use Python to automate testing, monitor data quality, orchestrate cloud services, and support machine learning pipelines.
Although the examples in this article focus on ELT, orchestration, and distributed processing, Python's role in data engineering extends across the entire data lifecycle. The summary below highlights where it fits into today's modern data platforms:
Final thoughts on Python for data engineering
Python is one of the foundational technologies of modern data engineering. Its versatility, extensive ecosystem, and broad support across cloud and data platforms make it a strong choice for building and maintaining data pipelines, integrating systems, and automating complex workflows.
Its strength also lies in how well it integrates with the broader data ecosystem. Python can work alongside SQL-based transformations, distributed processing engines, managed cloud services, and specialized data tools, allowing engineers to choose the most effective approach for each workload without sacrificing flexibility.
However, building a production-ready data platform requires more than choosing the right language. Reliability, scalability, and maintainability also depend on sound architecture, data quality controls, observability, security, governance, and operational practices. Python provides a powerful foundation, while these complementary capabilities help turn individual pipelines into a dependable data platform.
