Data Engineering - ETL, Web Scraping ,Big Data,SQL,Power BI
What you'll learn
- Install Visual Studio
- Install SQL Server and attach sample databases
- Install SQL Server Data Tools and Templates Designers
- Create new SQL SERVER Integration Services Project
- Implement ETL Process
- Test SSIS Package
- Interact with database using SQL
- Perform various database operations with SQL including CRUD
- Extract data from a website using web scraping
- Understand concepts of big data
- Connect to multiple data sources with Power BI
- Clean and transform data with Power BI
- Create visual representation of data
- Data Modelling with Power BI
Requirements
- Computer with internet access required.
- Some software installation required which are covered in the course.
Description
A common problem that organizations face is how to gathering data from multiple sources, in multiple formats, and move it to one or more data stores. The destination may not be the same type of data store as the source, and often the format is different, or the data needs to be shaped or cleaned before loading it into its final destination.
Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources, transform the data according to business rules, and load it into a destination data store.
SQL Server Integration Services (SSIS) is a useful and powerful Business Intelligence Tool . It is best suited to work with SQL Server Database . It is added to SQL Server Database when you install SQL Server Data Tools (SSDT)which adds the Business Intelligence Templates to Visual studio that is used to create Integration projects.
SSIS can be used for:
Data Integration
Data Transformation
Providing solutions to complex Business problems
Updating data warehouses
Cleaning data
Mining data
Managing SQL Server objects and data
Extracting data from a variety of sources
Loading data into one or several destinations
Web scraping is the process of automatically downloading a web page's data and extracting specific information from it. The extracted information can be stored in a database or as various file types.
Web scraping software tools may access the World Wide Web directly using the Hypertext Transfer Protocol, or through a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying, in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.
Scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when you view the page). to fetch pages for later processing. Once fetched, then extraction can take place. The content of a page may be parsed, searched, reformatted, its data copied into a spreadsheet, and so on. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else. An example would be to find and copy names and phone numbers, or companies and their URLs, to a list (contact scraping).
Big data can be characterised as data that has high volume, high variety and high velocity. Data includes numbers, text, images, audio, video, or any other kind of information you might store on your computer. Volume, velocity, and variety are sometimes called "the 3 V's of big data."
What kind of datasets are considered big data?
Examples includes social media network analysing their members' data to learn more about them and connect them with content and advertising relevant to their interests, or search engines looking at the relationship between queries and results to give better answers to users' questions.
SQL is a standard language for accessing and manipulating databases.
SQL stands for Structured Query Language
What Can SQL do?
SQL can execute queries against a database
SQL can retrieve data from a database
SQL can insert records in a database
SQL can update records in a database
SQL can delete records from a database
SQL can create new databases
SQL can create new tables in a database
SQL can create stored procedures in a database
SQL can create views in a database
SQL can set permissions on tables, procedures, and views
Power BI is a business analytics solution that lets you visualize your data and share insights across your organization, or embed them in your app or website. Connect to hundreds of data sources and bring your data to life with live dashboards and reports.
Discover how to quickly glean insights from your data using Power BI. This formidable set of business analytics tools—which includes the Power BI service, Power BI Desktop, and Power BI Mobile—can help you more effectively create and share impactful visualizations with others in your organization.
In this beginners course you will learn how to get started with this powerful toolset. We will cover topics like connecting to and transforming web based data sources. You will learn how to publish and share your reports and visuals on the Power BI service.
Who this course is for:
- Beginners who want to learn how to interact with various types of data.
Instructor
Welcome to the vibrant learning hub of Bluelime Learning Solutions! Based in the UK, we are dedicated to crafting high-quality, easily digestible eLearning experiences. With a robust following of over 776,000 students and backed by more than 43,000 reviews, we take pride in delivering educational content that not only enlightens but also empowers.
At Bluelime Learning Solutions, our approach is simple: clear, direct, and engaging. We specialize in 100% video-based courses filled with hands-on examples that provide practical skills applicable in real-world scenarios. Our courses are designed to translate complex concepts into easy-to-understand lessons that anyone, regardless of their background, can grasp and apply.
Our extensive experience includes collaborating on projects with Fortune 500 companies, equipping us with the insights needed to understand the essential skills that today’s employers value most. At Bluelime, we’re not just about teaching; we’re about equipping you with the tools and knowledge to excel in your career and personal growth endeavors.
Join us on this journey to unlock your potential and master new skills in an easy, enjoyable, and efficient way with Bluelime Learning Solutions, where learning is made simple and effective.