PySpark offers two fundamental data structures for distributed data processing: Resilient Distributed Datasets (RDDs) and DataFrames. Both provide mechanisms for manipulating and analyzing data, but they differ in structure and operations. Here’s a breakdown of transformations, actions, and execution:
1. RDDs (Resilient Distributed Datasets):
- Represent an immutable collection of data objects partitioned across a cluster of machines.
- Offer fault tolerance: if a worker node fails, the data can be recomputed from other nodes.
Transformations:
- Operations that create a new RDD from an existing one. They are lazy (don’t trigger computation until an action is called). Common transformations include:
map(func)
: Applies a function to each element in the RDD.filter(func)
: Creates a new RDD containing only elements that satisfy a condition defined by the function.flatMap(func)
: Similar tomap
but allows a function to return multiple elements for each input element.join(otherRDD, func)
: Joins two RDDs based on a specified function.groupBy(func)
: Groups elements together based on the output of the function.
Actions:
- Operations that return a result or perform a side effect on the RDD. They trigger the actual computation on the cluster. Common actions include:
collect()
: Gathers all elements of the RDD to the driver program (potentially large for big datasets).count()
: Returns the total number of elements in the RDD.first()
: Returns the first element in the RDD.foreach(func)
: Applies a function to each element in the RDD on the worker nodes.saveAsTextFile(path)
: Saves the RDD as a text file.
Execution:
- Transformations are applied lazily, building a lineage of operations.
- When an action is called, the lineage is traversed, and the actual computation happens on the cluster in stages.
2. DataFrames:
- Built on top of RDDs, providing a more structured and SQL-like interface for data manipulation.
- Offer columnar data storage for efficient querying.
Transformations:
- Similar to RDD transformations but operate on DataFrames with column-level operations. Common transformations include:
select(col1, col2)
: Selects specific columns.filter(condition)
: Filters rows based on a condition.join(otherDF, condition)
: Joins two DataFrames based on a specified condition.groupBy(col)
: Groups rows by a specific column.orderBy(col, ascending=True)
: Orders rows by a column in ascending or descending order.
Actions:
- Similar to RDD actions but work on DataFrames. Common actions include:
collect()
: Gathers all rows of the DataFrame to the driver program.count()
: Returns the total number of rows in the DataFrame.show()
: Displays a limited number of rows in the DataFrame.write.csv(path)
: Saves the DataFrame as a CSV file.
Execution:
- Transformations are generally optimized for DataFrames, leading to potentially faster execution compared to RDDs.
- Similar to RDDs, actions trigger the computation on the cluster.
In summary:
- Use RDDs for low-level data manipulation and when you need maximum control over data processing.
- Use DataFrames for more structured data, leveraging SQL-like operations and potentially better performance for common data analysis tasks.
Contents
RDD Transformations and Actions
Transformations
Transformations create a new RDD from an existing one. They are lazy and are not executed until an action is called.
Transformation | Description | Example |
---|---|---|
map | Applies a function to each element | rdd.map(lambda x: x * 2) |
filter | Filters elements based on a predicate | rdd.filter(lambda x: x > 10) |
flatMap | Similar to map but each input item can be mapped to 0 or more output items | rdd.flatMap(lambda x: x.split(" ")) |
union | Returns a new RDD containing the union of elements | rdd1.union(rdd2) |
distinct | Returns a new RDD with distinct elements | rdd.distinct() |
groupByKey | Groups the values for each key in the RDD | rdd.groupByKey() |
reduceByKey | Merges values for each key using an associative function | rdd.reduceByKey(lambda x, y: x + y) |
sortBy | Sorts the RDD by the given key function | rdd.sortBy(lambda x: x) |
join | Joins two RDDs by their keys | rdd1.join(rdd2) |
Actions
Actions trigger the execution of transformations and return a result.
Action | Description | Example |
---|---|---|
collect | Returns all elements as a list | rdd.collect() |
count | Returns the number of elements in the RDD | rdd.count() |
first | Returns the first element | rdd.first() |
take | Returns the first n elements | rdd.take(5) |
reduce | Aggregates elements using an associative function | rdd.reduce(lambda x, y: x + y) |
saveAsTextFile | Saves the RDD as a text file | rdd.saveAsTextFile("path/to/file") |
countByKey | Returns the count of each key | rdd.countByKey() |
DataFrame Transformations and Actions
Transformations
DataFrame transformations are lazy and return a new DataFrame.
Transformation | Description | Example |
---|---|---|
select | Selects a set of columns | df.select("name", "age") |
filter /where | Filters rows based on a condition | df.filter(df['age'] > 21) |
groupBy | Groups rows by a column | df.groupBy("age").count() |
agg | Performs aggregation | df.groupBy("age").agg({"salary": "avg"}) |
join | Joins two DataFrames | df1.join(df2, "id") |
withColumn | Adds or replaces a column | df.withColumn("new_col", df['existing_col'] * 2) |
drop | Drops a column | df.drop("column_name") |
distinct | Returns distinct rows | df.distinct() |
orderBy | Sorts rows by a column | df.orderBy("age", ascending=False) |
Actions
DataFrame actions trigger the execution of transformations and return a result.
Action | Description | Example |
---|---|---|
show | Displays the top rows of the DataFrame | df.show(5) |
collect | Returns all rows as a list of Row objects | df.collect() |
count | Returns the number of rows | df.count() |
first | Returns the first row | df.first() |
take | Returns the first n rows | df.take(5) |
write | Writes the DataFrame to storage | df.write.csv("path/to/file") |
describe | Computes basic statistics | df.describe().show() |
toPandas | Converts the DataFrame to a Pandas DataFrame | df.toPandas() |
Execution Operations
Execution operations involve how Spark processes and schedules the tasks.
Operation | Description |
---|---|
Lazy Evaluation | Transformations are recorded and optimized but not executed until an action is called. |
DAG Scheduler | Converts the logical plan into a physical execution plan and breaks it into stages. |
Task Scheduler | Schedules tasks within each stage to run on different nodes in the cluster. |
Caching/Persistence | Stores intermediate results in memory or disk to speed up future computations. |
Shuffle | Redistributes data across the cluster, often needed by operations like groupByKey and join . |
Leave a Reply