by Team AHT | Jan 9, 2025 | How To
This Post is Collection of Handy Tricks and Snippets. Passing Parameters in Automation of Scripts using Python Python provides several ways to pass parameters in automation of scripts, mimicking SAS macro variables, macro modules, and macro scripting. Here are some...
by Team AHT | Jan 7, 2025 | Pyspark, Python
dbname.table_name want to save dbname and table_name in seperate variable and then to pass them as parameters in pyspark/python script # String containing dbname and table_name full_table_name = “my_database.my_table” # Split into dbname and table_name...
by Team AHT | Jan 4, 2025 | SQL
What is Indexing? Indexing is a data structure technique that allows the database to quickly locate and access specific data. It’s similar to the index at the back of a book, which helps you find specific pages quickly. How Indexing Works Index Creation: The...
by Team AHT | Dec 28, 2024 | SQL
Spark SQL Operators Cheatsheet 1. Arithmetic Operators OperatorSyntaxDescriptionExample+a + bAdds two valuesSELECT 5 + 3;-a – bSubtracts one value from anotherSELECT 5 – 3;*a * bMultiplies two valuesSELECT 5 * 3;/a / bDivides one value by anotherSELECT 6 /...
by Team AHT | Dec 28, 2024 | How To
Syntax Rules for Pseudocode Natural Language: Use simple and clear natural language to describe steps. Keywords: Use standard control flow keywords such as: IF, ELSE, ENDIF FOR, WHILE, ENDWHILE FUNCTION, CALL INPUT, OUTPUT Indentation: Indent blocks within loops or...
by Team AHT | Dec 8, 2024 | Pyspark
A quick reference for date manipulation in PySpark:– FunctionDescriptionWorks OnExample (Spark SQL)Example (DataFrame API)to_dateConverts string to date.StringTO_DATE(‘2024-01-15’, ‘yyyy-MM-dd’)to_date(col(“date_str”),...
by Team AHT | Dec 5, 2024 | Pyspark
Window functions in PySpark allow you to perform operations on a subset of your data using a “window” that defines a range of rows. These functions are similar to SQL window functions and are useful for tasks like ranking, cumulative sums, and moving...
by Team AHT | Nov 25, 2024 | SQL
For Better understanding on Spark SQL windows Function and Best Usecases do refer our post Window functions in Oracle Pl/Sql and Hive explained and compared with examples. Window functions in Spark SQL are powerful tools that allow you to perform calculations across a...
by Team AHT | Nov 16, 2024 | Pyspark
PySpark Architecture Cheat Sheet 1. Core Components of PySpark ComponentDescriptionKey FeaturesSpark CoreThe foundational Spark component for scheduling, memory management, and fault tolerance.Task scheduling, data partitioning, RDD APIs.Spark SQLEnables interaction...
by Team AHT | Nov 7, 2024 | News
Yup! Scientists find a ‘Unique’ Black Hole that is hungier than ever in the Universe! Scientists have observed a fascinating phenomenon involving a supermassive black hole, AT2022dsb, which appears to be devouring a star in a “tidal disruption event”...
by Team AHT | Nov 7, 2024 | SQL
Here’s an enhanced Spark SQL cheatsheet with additional details, covering join types, union types, and set operations like EXCEPT and INTERSECT, along with options for table management (DDL operations like UPDATE, INSERT, DELETE, etc.). This comprehensive sheet...
by Team AHT | Nov 7, 2024 | SQL
Here’s a categorized Spark SQL function reference, which organizes common Spark SQL functions by functionality. This can help with selecting the right function based on the operation you want to perform. 1. Aggregate Functions FunctionDescriptionExampleavg()Calculates...
by Team AHT | Nov 6, 2024 | SQL
CRUD stands for Create, Read, Update, and Delete. It’s a set of basic operations that are essential for managing data in a database or any persistent storage system. It refers to the four basic functions that any persistent storage application needs to perform....
by Team AHT | Nov 2, 2024 | Tutorials
Comparative overview of partitions, bucketing, segmentation, and broadcasting in PySpark, Spark SQL, and Hive QL in tabular form, along with examples Here’s a comparative overview of partitions, bucketing, segmentation, and broadcasting in PySpark, Spark SQL,...
by Team AHT | Nov 1, 2024 | SQL
Data Definition Language (DDL) – to define and modify the structure of a database. Data Manipulation Language (DML) – to access, manipulate, and modify data in a database. Data Control Language (DCL) – to control user access to the data in the database...
by Team AHT | Oct 27, 2024 | Python
The pandas Series is a one-dimensional array-like data structure that can store data of any type, including integers, floats, strings, or even Python objects. Each element in a Series is associated with a unique index label, making it easy to perform data retrieval...
by Team AHT | Oct 24, 2024 | Python
This tutorial covers a wide range of pandas operations and advanced concepts with examples that are practical and useful in real-world scenarios. The key topics include: Creating DataFrames, Series from various sources. Checking and changing data types. Looping...
by Team AHT | Oct 22, 2024 | Pyspark
I have divided a pyspark big script in many steps –by using steps1=”’ some codes”’ till steps7, i want to execute all these steps one after another and also if needed some steps can be not be executed. if any steps fails then then next...
by Team AHT | Oct 22, 2024 | Pyspark
How to code in Pyspark a Complete ETL job using only Pyspark sql api not dataframe specific API? Here’s an example of a complete ETL (Extract, Transform, Load) job using PySpark SQL API: from pyspark.sql import SparkSession # Create SparkSession spark =...
by Team AHT | Oct 21, 2024 | Pyspark
PySpark supports various control statements to manage the flow of your Spark applications. PySpark supports using Python’s if-else-elif statements, but with limitations. Supported Usage Conditional statements within PySpark scripts. Controlling flow of Spark...
