Data Engineering and Data Integration Tools

A warm welcome to the Data Engineering and Data Integration Tools course by Uplatz.

Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) concepts are core of any datawarehousing initiative. Data ingestion, integration, and processing form a critical task for consolidating the data silos across the departments in an organisation and ultimately to build a robust and flexible datawarehouse for enterprise reporting & analytics.

One of such tools is Talend. Talend is an ETL tool/software for Data Integration. It delivers software resolutions for data groundwork, data quality, data integration, application integration, data management and big data.

There exist separate products of these different solutions in Talend. Big Data products and data integration are broadly used in Talend. Data integration and data management solutions are offered by Talend, as an open source platform. Big data integration is a specialty of Talend. Other features provided by Talend are related to cloud, big data, enterprise application integration, master data management and data quality. It also provides a unified repository to store and reuse the Metadata.

Talend is one of the finest tools for cloud computing and big data integration. The most common invention of Talend Studio is data integration and big data. Talend can smoothly arrange big data integration with graphical tools and wizards. This permits the group to generate a condition to easily work with Apache Hadoop, Spark, and NoSQL databases for cloud. Talend data integration software tool has an open, accessible architecture. It permits quicker response to business needs. The tool contracts to modify and arrange data integration jobs faster than hand coding. Talend integration cloud tool offers connectivity, built-in data quality, and native code generation. Talend is protected cloud integration platform which allows IT and business users to connect shared both could and on-premise. It solves the power of cloud design job as it can manage, monitor, and control in the cloud.

Uplatz provides this end-to-end course on this leading Data Integration and ETL tool called Talend. With many organizations using Talend as their leading data warehousing and data integration software, there are huge career prospects by learning and mastering Talend. If you wish to become an ETL Architect or a Data Integration Engineer, then Talend course can be a complete game changer.

Talend - Course Curriculum

1. Role of Open Source ETL Technologies in Big Data

• Overview on: TOS (Talend Open Studio) for Data Integration

• ETL concepts

• Data warehousing concepts

2. Talend

• Why Talend?

• Features

• Advantages

• Talend Installation/System Requirements

• GUI layout (designer)

• Understanding it's Basic Features

• Comparison with other market leader tools in ETL domain

• Important areas in Talend Architecture: Project

• Workspace

• Job

• Metadata

• Propagation

• Linking components

3. Talend: Read & Write various Types of Source/Target System

• Data Source Connection

• File as Source

• Create meta data

• Database as source

• Create metadata

• Using MySQL database (create tables, Insert, Update Data from Talend)

• Read and write into excel files, into multiple tabs

• View data

• How to capture log and navigate around basic errors

• Role of tLogrow and how it makes developers life easy

4. Talend: How to Transform Your Business: Basic

• Using Advanced components like: tMap, tJoin, tFilter, tSortRow, tAggregateRow, tReplicate, tSplit, Lookup, tRowGenerator

5. Talend: How to Transform Your Business: Advanced 1

• Trigger (types) and Row Types

• Context Variables (parameterization)

• Functions (basic to advanced functions to transform business rules such as string, date, mathematical etc.)

• Accessing job level / component level information within the job

6. Talend: How to Transform Your Business: Advanced 2

• Type Casting (convert data types among source-target platforms)

• Looping components (like tLoop, tFor)

• tFileList

• tRunJob

• How to schedule and run talend DI jobs externally (not in GUI)

7. Working with Hierarchical File Structures

• Read and Write an XML file, configure the schema and XPath expression to parse an XML file

• Read and Write a JSON file, configure the schema and JSONPath expression to parse a JSON file

• Read and write delimited, fixed width files.

8. Context Variables and Global Variables

• Create context/global variables

• Use context/global variables in the configuration of Talend components

• Load context variables from a flow

9. Best practices

• Working with databases and implementing data warehousing concepts

• Working with files (excel, delimited, JSON, XML etc.)

10. Orchestration and Controlling Execution Flow

• Files - Use components to list, archive, and delete files from a directory

• Database – Controlling Commit and Rollback

  • COMMIT at end of job/ every x number of rows

  • Rollback on error

11. Shared DB connection across jobs and subjobs

• Use triggers to connect components and subJobs

• Orchestrate several jobs in master jobs.

• Handling Errors

  • Kill a Job on a component error

  • Implement a specific Job execution path on a component error

  • Configure the log level in the console