1.how to use SQL expressions inside the filter or where functions?using the expr function from the pyspark.sql.functions module, which allows you to pass SQL expressions as strings.

Here is an example demonstrating how to use SQL expressions inside the filter or where functions:

from pyspark.sql import SparkSession
from pyspark.sql.functions import expr

# Initialize Spark session
spark = SparkSession.builder 
    .appName("SQL Expression Example") 
    .getOrCreate()

# Sample DataFrame
data = [
    (1, "Alice", 25, 2000),
    (2, "Bob", 30, 3000),
    (3, "Charlie", 35, 4000)
]

columns = ["id", "name", "age", "salary"]
df = spark.createDataFrame(data, columns)

# Using SQL expression in filter
filtered_df = df.filter(expr("age > 30"))

filtered_df.show()

Detailed Example

Let’s go through a more detailed example with multiple conditions and transformations using SQL expressions.

from pyspark.sql import SparkSession
from pyspark.sql.functions import expr

# Initialize Spark session
spark = SparkSession.builder 
    .appName("SQL Expression Example") 
    .getOrCreate()

# Sample DataFrame
data = [
    (1, "Alice", 25, 2000),
    (2, "Bob", 30, 3000),
    (3, "Charlie", 35, 4000),
    (4, "David", 40, 5000)
]

columns = ["id", "name", "age", "salary"]
df = spark.createDataFrame(data, columns)

# Filter using SQL expression
filtered_df = df.filter(expr("age > 30 AND salary > 3000"))

# Select specific columns using SQL expression
selected_df = filtered_df.selectExpr("id", "name", "salary * 1.1 AS adjusted_salary")

# Adding a new column using SQL expression
with_new_column_df = selected_df.withColumn("salary_category", expr("CASE WHEN adjusted_salary > 4000 THEN 'High' ELSE 'Medium' END"))

with_new_column_df.show()

Output

+---+-------+---------------+---------------+
| id|   name|adjusted_salary|salary_category|
+---+-------+---------------+---------------+
|  3|Charlie|         4400.0|           High|
|  4|  David|         5500.0|           High|
+---+-------+---------------+---------------+

---

2. select function syntax variations and examples:

The select function in PySpark is used to select a subset of columns or expressions from a DataFrame. It can be used in several ways to achieve different results.

Basic Syntax

The basic syntax for the select function is:

DataFrame.select(*cols)
Here, *cols can be a list of column names, Column objects, or expressions.

Selecting Specific Columns

df.select('column1', 'column2').show()
This will select and display the 'column1' and 'column2' columns.

Selecting Columns with Expressions

from pyspark.sql.functions import col, expr
df.select(col('column1'), expr('column2 + 1').alias('column2_plus_one')).show()

This will select ‘column1’ and an expression ‘column2 + 1’ as ‘column2_plus_one’.

Using Column Objects

Example:

from pyspark.sql.functions import col
df.select(col('column1'), col('column2')).show()
This will select 'column1' and 'column2' using Column objects.

Renaming Columns

df.select(col('column1').alias('new_column1')).show()
This will select 'column1' and rename it to 'new_column1'.

Selecting All Columns

df.select('*').show()
This will select all columns in the DataFrame.

Using SQL Expressions

df.selectExpr('column1', 'column2 + 1 as column2_plus_one').show()

This allows you to use SQL expressions directly.

---

 

Head to Next

Pages: 1 2 3 4 5 6 7 8 9 10 11 12