Data Warehouse Development Process
- 4 hours on-demand video
- 1 downloadable resource
- Full lifetime access
- Access on mobile and TV
- Certificate of Completion
Get your team access to 4,000+ top Udemy courses anytime, anywhere.Try Udemy for Business
- Understand various stages in Data Warehouse development process
- Various processes like Waterfall model, V model and Agile methods
- Specific aspects of Data Warehouse development process
- Importance of the various phases and the practicality of each phase
- Overview of various issues and Project Management issues to be considered in the Data warehouse and Business Intelligence projects
- Basics of Data Warehouse Concepts
- Terms and terminologies used in a Data Warehouse and Business Intelligence projects
Data is the new asset for the enterprises. And, Data Warehouse store the data for better insights and knowledge using Business Intelligence. Development of an Enterprise Data Warehouse has more challenges compared to any other software projects because of the
- Challenges with data structures
- The way data is evaluated for it's quality
- Complex business rules/validations
- Different development methods (various SDLC models like Water Fall model, V model, Agile Model, Incremental model, Iterative model)
- Regulatory requirements for various domains like finance, telecom, insurance, Retail and IME
- Compliance from third party governing bodies
- Extracting data for various visualization purposes
In this course, we talk about the specific aspects of the Data Warehouse Development process taking real time practical situations and how to handle them better using best practices for sustainable, scalable and robust implementations.
- Freshers/Engineering Graduates who are looking for placements
- Software Engineers from different technology background who want to explore the Data Warehouse and Business Intelligence development process
- Software Engineers who are already part of any Data Warehouse and Business Intelligence Projects
- Project Mangers
- Non IT professionals who like to learn how data is handled in enterprises
- Technology experts and Team Leaders
- Architects and Data Modelers
- Data Scientists and Big Data Experts who want to understand the practical Data Warehouse Process
- Database Administrators who want to explore the DWH/ETL/BI areas
- Mainframe developers who want to switch their carrier into the Data Warehouse stream
Any requirement which specifies what the system should do is a functional requirement.
-Transaction corrections, adjustments and cancellations
-Legal or Regulatory Requirements
Any requirement which specifies how the system performs a certain function/activity is considered as Non Functional Requirement.
- Data Integrity
In this lecture, we start with the basic understanding of what is a process in general for a physical product and the process for a software development.
The Waterfall Model was first Process Model to be introduced. It is also referred to as a linear-sequential life cycle model. It is very simple to understand and use. In a waterfall model, each phase must be completed fully before the next phase can begin.
There are multiple versions of the stages which are defined for the Waterfall model. All of them follow the baseline structure of Requirements,Analysis, Development, Testing and Maintenance.
In this course, we have used the below stages to split the various phases to understand it more better,
- System Requirements
- Software Requirements
- Program Design
Subsequent to receiving the system design documents (both technical and functional), the work is shared into various modules and the real coding is commenced. The system/application is developed into small coding units/modules and these units are later integrated in the subsequent phase. Every unit is tested for its functionality.
The modules that are divided into units in the design phase are integrated into a complete system and tested for proper coordination among modules and system behaves as per the specifications. Once the testing is completed, the software product is delivered to the customer for UAT (User Acceptance Tests). And, once the tests are complete, the code will be deployed to production.
It is a never ending phase. Once the system is running in production environment, problems come up. The issues that are related to the system are solved only after deployment of the system. The problems arise from time to time and need to be solved; hence this phase is referred as maintenance.
And, any new regulatory requirements and new policies with in the organization also triggers more enhancements and changes to the application.
Advantages of waterfall model:
- Simple and easy to understand and use.
- Easy to manage due to the rigidity of the model – each phase has specific deliverables and a review process.
- Phases are processed and completed one at a time.
- Works well for smaller projects where requirements are very well understood.
Disadvantages of waterfall model:
- Once an application is in the testing stage, it is very difficult to go back and change something that was not well-thought out in the concept stage.
- No working software is produced until late during the life cycle.
- High amounts of risk and uncertainty.
- Not a good model for complex and object-oriented projects.
- Poor model for long and ongoing projects.
- Not suitable for the projects where requirements are at a moderate to high risk of changing
V- model is also called as the Verification and Validation model. Just like the waterfall model, the V-Shaped life cycle is a sequential path of execution of processes. Each phase must be completed before the next phase begins. Testing of the product is planned in parallel with a corresponding phase of development in V-model.
Following are the Verification phases in V-Model:
- Business Requirement Analysis
- System Design:
- Architectural Design
- Module Design
Advantages of V-model:
- Simple and easy to use.
- Testing activities like planning, test designing happens well before coding. This saves a lot of time. Hence higher chance of success over the waterfall model.
- Proactive defect tracking – that is defects are found at early stage.
- Avoids the downward flow of the defects.
- Works well for small projects where requirements are easily understood.
Disadvantages of V-model:
- Very rigid and least flexible.
- Software is developed during the implementation phase, so no early prototypes of the software are produced.
- If any changes happen in midway, then the test documents along with requirement documents has to be updated.
An iterative life cycle model will not be started with a full specification of requirements. Instead, development begins by specifying and implementing just part of the software, which can then be reviewed in order to identify further requirements. This process is then repeated, producing a new version of the software for each cycle of the model.
In incremental model the whole requirement is divided into various builds. Multiple development cycles take place here, making the life cycle a “multi-waterfall” cycle. Cycles are divided up into smaller, more easily managed modules. Incremental model is a type of software development model like V-model, Agile model etc.
Agile development model is a combination of both Iterative model and Incremental model. Software is developed in incremental, rapid cycles. This results in small incremental releases with each release building on previous functionality. Each release is thoroughly tested to ensure software quality is maintained. It is used for time critical applications.
Advantages of Agile model:
- Customer satisfaction by rapid, continuous delivery of useful software.
- People and interactions are emphasized rather than process and tools. Customers, developers and testers constantly interact with each other.
- Working software is delivered frequently (weeks rather than months).
- Face-to-face conversation is the best form of communication.
- Close, daily cooperation between business people and developers.
- Continuous attention to technical excellence and good design.
- Regular adaptation to changing circumstances.
- Even late changes in requirements are welcomed
Disadvantages of Agile model:
- In case of some software deliverable, especially the large ones, it is difficult to assess the effort required at the beginning of the software development life cycle.
- There is lack of emphasis on necessary designing and documentation.
- The project can easily get taken off track if the customer representative is not clear what final outcome that they want.
- Only senior programmers are capable of taking the kind of decisions required during the development process. Hence it has no place for newbie programmers, unless combined with experienced resources.
Unlike the waterfall model in agile model very limited planning is required to get started with the project. Agile assumes that the end users’ needs are ever changing in a dynamic business and IT world. Changes can be discussed and features can be newly effected or removed based on feedback. This effectively gives the customer the finished system they want or need.
The Waterfall, V and the Agile model are not the only one's available for projects. These are the commonly used model for the Data warehouse implementations. There are other SDLC models which can be used for projects and those are.
- RAD model
- Spiral model
- Big Bang model
- Prototype model
From this session onwards, we will start to see that practical implmemetaion steps for a Datawarehouse Development project. Below are the various phases which we will discuss about.
- Project Planning
- Operations and Maintenance