Writing Tests for SimEng Python Code conversion concepts 101

Name: Writing Tests for SimEng Python Code conversion concepts 101
Rating: 4.0 (9 reviews)

Interview Prep: Write unit test for prevailing code in Python Pyspark SQL and configure tests in Pycharm

Last updated 11/2024

English

What you'll learn

6 sections • 18 lectures • 31m total length

Introduction1:35
Learn to write unit tests for a simulation engine using Python, PySpark, Excel, and Jupyter Notebook, including test data, decorators, local Hive setup for PySpark views, and release process basics.
What you will learn in the Course2:41
Learn to write unit tests for sim engines, using Excel power queries for joins (including left joins) and group bys, and set up PyCharm testing with debugging and virtual environments.
Learning Points and Summary2:32
Apply groupby and join on small data to validate results in Python or Spark SQL, and debug Spark sessions while managing Hadoop and Hive views in a local virtual environment.

Intro to Pycharm Settings0:49
Explore PyCharm settings to run tests and grid-based notebooks, compare PyCharm with visual code, and learn to set up, use, and run tests in this course.
Pycharm Settings1:13
Configure PyCharm for Python testing with pytest, set up source and test folders, ignore the virtual environment, and manage installs via pip, setup.py, or setup.sh, including git bash on Windows.
Debug for break points1:07
Disable coverage during debugging to ensure breakpoints work, as they do not operate together. Configure the Python interpreter and working/interpreter directories and choose py test or no set test.

Intro to diff type of code0:46
Explore different types of code, including back end, full stack, UI based, and functional code, and unit test old code in the full stack framework with task and workflow code.
Intro to Schemas1:34
Explore switching between pandas and spark schemas in the sim engine by uploading csvs with the pandas schema, converting to spark for tests, and back as needed.
Update output files0:41
Update output files by swapping the expected test output when changes occur, using csv checkpoints or breakpoints to view and download the data frame during backend and full stack runs.
Workflow unit tests2:37
Develop workflow unit tests by mimicking similar code, handling inputs, reading CSV into pandas, converting to spark dataframe, and ensuring spark calls stay inside functions.
Task based unit tests2:23
Learn task based unit tests where workflows orchestrate input and reference tasks, keeping spark session inside functions, converting csv to pandas to spark data frames, and comparing frames.

Intro setting and fixing venv0:57
Explore setting up and fixing virtual environments (venv) across full-stack and backend, using git bash, pip install requirements, parallel folders, imports, and temporary files.
Full stack venv vs backend venv2:12
Compare full stack venvs and backend venvs, highlighting setup complexity, folder requirements, and common errors, with guidance to recreate or update environments to avoid issues.
Windows Env variables for Full stack Framework2:25
Explore Windows local environment variables like Hadoop home, Java home, and Spark home that power full stack frameworks and Hive metastore, and how backend switches impact testing caches.

Python to excel intro1:21
Learn to translate Python test workflows into Excel by mimicking input handling, csv joins and group by operations, enabling non-programmers to test code using Excel and Power BI tools.
Power BI Tools4:05
Learn to use excel power queries and Power BI tools to mimic Python and Spark code with CSV inputs, handling blanks, nulls, and zeros, performing left joins and group by.
Notebook in Pandas to Mimic Pyspark code0:47
Use a pandas notebook to mimic PySpark code and recreate data outputs, enabling independent verification and testing of SimEng Python code conversions for big data workflows.

Interview Prep: Writing Tests for SimEng - Code conversion concepts

Write unit test for prevailing code in Python Pyspark SQL and configure tests in Pycharm

What you will learn:

Topics:

In Excel power queries we use mostly group by and join on small data to check our results and share with seniors.
We can create a simpler notebook of the same logic of the code
A lot of type the errors are due to Local Spark sessions which can be created in many many ways, but if we want to create spark sql views which uses hive then we have to install a hadoop and spard by downloading them.
Debugging and creating break points and then using step into / step over to generate the output and match or save it with excel
Making sure we mimic spark from older tests because otherwise we can get error in Jenkins build