Big Data Infra Admin 101: Ticket based learning Remote Roles

Run a Python notebook to connect via ODBC, fetch the file list, and store it in the C drive temp folder; sort small files, identify users, and inform them.

Explains how small hive and non hive files are created, and flags troublesome sizes; shows removing old non hive parquet and csv files improves data hygiene.

Identify small files created by automated full stack ui runs, appearing as blocks in reports, and apply purge workflows to remove zero-kb files in alpha, development, and uat environments.

Learn remedies for small files using workflow purges and python scripts to compactify hive tables and folders, plus manual HDFS folder compactification and common run-time errors.

Analyze csv report of small files and partitions in the temp folder. Use string search to locate the position and numbers, and apply Excel analytics to find Mahesh.

Explore purge workflows for deleting old full stack simulation runs across multiple environments, using a CSV schema to locate runs by name, version, and date within the UI front end.

This lecture outlines checkpoints for small files, highlighting trouble zones from 10 kb to 200 mb, and covers running python scripts in the correct virtual environment for hive data.

Explore small file ticket management by identifying root causes with Python scripts and dashboards, then remediate using compactification tools on tables, folders, or CSVs, guided by human outreach.

Identify the small file issue through exploratory steps using Python scripts to query databases and email owners, then apply remedial compactification tools to hive tables, folders, and csv files.

Explore hive vs non-hive small files, comparing non-hive data from pocket and csv with hive tables, across sizes from 10 kb to 200 mb, noting automation-driven hive prevalence.

Examine the checkpoint on small file repartition scripts, detailing Python scripts that run in a virtual environment, manual execution, and workflows to purge them within the full stack framework.

Learn to use Python scripts and UI purge workflows to manage big data infra, comparing manual runs with the full-stack UI framework and generating detailed reports from local notebooks.

Sorts old data files by size and six-month cutoffs, distinguishes hive versus non-hive files, and notes UI-driven hive data in full-stack workflows.

Identify and remove old files across dev and full stack lanes after exploratory analysis with Python and Jupyter, deleting runs less than one month old using HDFS rmr commands.

Learn how archiving data remains a manual process, using CSVs and JIRA tickets to categorize data by development, alpha, beta, UAT, product, customer, and analytics with year mappings.

We examine different front end dashboards—Python dashboard, Tableau, SharePoint, and Hue—and explain roles: exploratory analysis in Python, permissions in Tableau, official releases in SharePoint, and Hive data access via Hue.

Explore how dashboards power infrastructure work by combining Python and Tableau, address internal access and permission issues, and use SharePoint for official documents in a Hadoop-driven environment.

Describe official release documents and research documents across client and our systems, update version numbers and year, use LaTeX and SharePoint, edit wiki pages and Google Docs.

Activate the correct virtual environment on the driver node and connect via ssh to run commands. Manage spark permissions for hive tables by selecting the appropriate environment.

Compare local and grid virtual environments in Windows for managing different repos and local variables. Use the grid virtual environment for infra and admin work.

Develop well-structured notes from wiki pages, with tables of contents, headings, hyperlinks, images, logs, build error logs, and shell script code snippets to help teams understand progress.

Analyze checkpoints, numbers in big data infra, focusing on small files from one KB to one MB, total files 1 to 2 million, 7 to 200 TB, aging, compactification timing.

Craft high quality notes from Wikipedia and read emails to stay on track. Manage scripts and notebooks across local and grid environments using odbc and environment variables.