Writing Tests for SimEng Python Code conversion concepts 101

Name: Writing Tests for SimEng Python Code conversion concepts 101
Rating: 4.0 (9 reviews)

Interview Prep: Write unit test for prevailing code in Python Pyspark SQL and configure tests in Pycharm

Last updated 11/2024

English

What you'll learn

6 sections • 18 lectures • 31m total length

Introduction1:35
What you will learn in the Course2:41
Learn to write unit tests for sim engines, using Excel power queries for joins (including left joins) and group bys, and set up PyCharm testing with debugging and virtual environments.
Learning Points and Summary2:32
Apply groupby and join on small data to validate results in Python or Spark SQL, and debug Spark sessions while managing Hadoop and Hive views in a local virtual environment.

Intro to diff type of code0:46
Explore different types of code, including back end, full stack, UI based, and functional code, and unit test old code in the full stack framework with task and workflow code.
Intro to Schemas1:34
Update output files0:41
Workflow unit tests2:37
Task based unit tests2:23
Learn task based unit tests where workflows orchestrate input and reference tasks, keeping spark session inside functions, converting csv to pandas to spark data frames, and comparing frames.

Python to excel intro1:21
Learn to translate Python test workflows into Excel by mimicking input handling, csv joins and group by operations, enabling non-programmers to test code using Excel and Power BI tools.
Power BI Tools4:05
Notebook in Pandas to Mimic Pyspark code0:47
Use a pandas notebook to mimic PySpark code and recreate data outputs, enabling independent verification and testing of SimEng Python code conversions for big data workflows.

Interview Prep: Writing Tests for SimEng - Code conversion concepts

Write unit test for prevailing code in Python Pyspark SQL and configure tests in Pycharm

What you will learn:

Topics:

In Excel power queries we use mostly group by and join on small data to check our results and share with seniors.
We can create a simpler notebook of the same logic of the code
A lot of type the errors are due to Local Spark sessions which can be created in many many ways, but if we want to create spark sql views which uses hive then we have to install a hadoop and spard by downloading them.
Debugging and creating break points and then using step into / step over to generate the output and match or save it with excel
Making sure we mimic spark from older tests because otherwise we can get error in Jenkins build