Full Stack Computational Framework Big Data Sims Refresher

learn to manage permissions for dev and prod users by enforcing unix groups, oracle, hive, hdfs, and spark access, with the front end as a connector and back-end enforcement.

Review and verify access through the permission portal within a full stack computational framework for big data simulations, ensuring secure, authorized data handling.

Maintain both PyCharm and VSCode, integrate git Copilot, and enable pytest in PyCharm while VSCode auto-detects the virtual environment and setup.cfg and v folder for seamless testing with AI tools.

create the local venv by running make in the repository with setup.py and setup.cfg, ensuring the correct Python path and managing constraints, unit tests, and per-repo venv creation.

Learn local spark debugging by setting up the environment, using breakpoints to identify empty data frames, and isolating complex steps like casting and lag value comparisons to fix errors.

Automate full stack input workflows with YML-styled testing in an introductory module, bridging big data sims concepts with a practical refresher.

Analyze internal libraries and version misalignments between the simulation and full stack repos. Create a branch from the simulation repo, build, and adjust pip requirements to resolve the build issue.

Convert notebook logic to full stack code by validating column renaming, operations, filtering, casting, and datetime conversions within the task, and compare results side by side with the notebook.

Learn how to manage casting from string to integer and date in local Hive PySpark data frames, diagnose failures, and use multi-column strategies to preserve granularity.