Data Warehouse Development Process

In this lecture, we talk about the different categories of Data warehouse Projects and how those will be used in the course with real time examples. 

In this section, we define the terms we used in this project. 

• Ad-Hoc Query
• Business Intelligence
• CRM
• Database
• DataMart
• Data Mining
• Data Visualization
• Data Warehouse
• DBMS/RDBMS
• NOSQL
• Dimension Table
• Fact Table

This is part 2 of defining the terms and Keywords.

• Granularity
• Operational System
• Partitioning
• Replication
• Reconciliation
• MDM/Reference Data

Additional information on the Data Warehouse and Business Intelligence glossary. 

Any requirement which specifies what the system should do is a functional requirement. 

-Business Rules
-Transaction corrections, adjustments and cancellations
-Authentication
-Authorization levels
-Audit Tracking
-External Interfaces
-Reporting Requirements
-Historical Data
-Legal or Regulatory  Requirements

Any requirement which specifies how the system performs a certain function/activity is considered as Non Functional Requirement. 

• Performance 
• Scalability
• Capacity
• Availability
• Reliability
• Recoverability
• Maintainability
• Serviceability
• Security
• Regulatory
• Manageability
• Environmental
• Data Integrity
• Usability
• Interoperability

In this lecture, we start with the basic understanding of what is a process in general for a physical product and the process for a software development.

Before we deep dive into the methodologies, lets start with the commonly used processes for an Enterprise Data Warehouse set up. 

The Waterfall Model was first Process Model to be introduced. It is also referred to as a linear-sequential life cycle model.  It is very simple to understand and use.  In a waterfall model, each phase must be completed fully before the next phase can begin.

There are multiple versions of the stages which are defined for the Waterfall model. All of them follow the baseline structure of Requirements,Analysis, Development, Testing and Maintenance. 

In this course, we have used the below stages to split the various phases to understand it more better, 

• System Requirements
• Software Requirements
• Analysis
• Program Design
• Coding
• Testing
• Operations

This phase is focused on possible requirements of the system for the development are captured. Requirements are gathered subsequent to the end user consultation.

Prior to beginning the actual coding, it's important to understand what actions are to be taken at what phase and what they should like.

The requirement specifications are studied in detail in this phase and the design of the system is prepared. The design specifications are the base for the implementation and unit testing model phase.

Subsequent to receiving the system design documents (both technical and functional), the work is shared into various modules and the real coding is commenced. The system/application is developed into small coding units/modules and these units are later integrated in the subsequent phase. Every unit is tested for its functionality.

The modules that are divided into units in the design phase are integrated into a complete system and tested for proper coordination among modules and system behaves as per the specifications. Once the testing is completed, the software product is delivered to the customer for UAT (User Acceptance Tests). And, once the tests are complete, the code will be deployed to production. 

It is a never ending phase. Once the system is running in production environment, problems come up. The issues that are related to the system are solved only after deployment of the system. The problems arise from time to time and need to be solved; hence this phase is referred as maintenance.

And, any new regulatory requirements and new policies with in the organization also triggers more enhancements and changes to the application. 

Following the POC  (Proof of Concept) model for better results. This is not limited to only the waterfall model but for any implementation to know the complexity of the requirements and what we are getting into. 

Advantages of waterfall model:

• Simple and easy to understand and use.
• Easy to manage due to the rigidity of the model – each phase has specific deliverables and a review process.
• Phases are processed and completed one at a time.
• Works well for smaller projects where requirements are very well understood.

Disadvantages of waterfall model:

• Once an application is in the testing stage, it is very difficult to go back and change something that was not well-thought out in the concept stage.
• No working software is produced until late during the life cycle.
• High amounts of risk and uncertainty.
• Not a good model for complex and object-oriented projects.
• Poor model for long and ongoing projects.
• Not suitable for the projects where requirements are at a moderate to high risk of changing

V- model is also called as the Verification and Validation model. Just like the waterfall model, the V-Shaped life cycle is a sequential path of execution of processes. Each phase must be completed before the next phase begins. Testing of the product is planned in parallel with a corresponding phase of development in V-model.

Following are the Verification phases in V-Model:

• Business Requirement Analysis
• System Design:
• Architectural Design
• Module Design
• Coding

Following are the Validation phases in V-Model:

• Unit Testing
• Integration Testing
• System Testing
• Acceptance Testing

Advantages of V-model:

• Simple and easy to use.
• Testing activities like planning, test designing happens well before coding. This saves a lot of time. Hence higher chance of success over the waterfall model.
• Proactive defect tracking – that is defects are found at early stage.
• Avoids the downward flow of the defects.
• Works well for small projects where requirements are easily understood.

Disadvantages of V-model:

• Very rigid and least flexible.
• Software is developed during the implementation phase, so no early prototypes of the software are produced.
• If any changes happen in midway, then the test documents along with requirement documents has to be updated.

An iterative life cycle model will not be started with a full specification of requirements. Instead, development begins by specifying and implementing just part of the software, which can then be reviewed in order to identify further requirements. This process is then repeated, producing a new version of the software for each cycle of the model.

In incremental model the whole requirement is divided into various builds. Multiple development cycles take place here, making the life cycle a “multi-waterfall” cycle.  Cycles are divided up into smaller, more easily managed modules. Incremental model is a type of software development model like V-model, Agile model etc.

Agile development model is a combination of both Iterative model and Incremental model. Software is developed in incremental, rapid cycles. This results in small incremental releases with each release building on previous functionality. Each release is thoroughly tested to ensure software quality is maintained. It is used for time critical applications.  

Advantages of Agile model:

• Customer satisfaction by rapid, continuous delivery of useful software.
• People and interactions are emphasized rather than process and tools. Customers, developers and testers constantly interact with each other.
• Working software is delivered frequently (weeks rather than months).
• Face-to-face conversation is the best form of communication.
• Close, daily cooperation between business people and developers.
• Continuous attention to technical excellence and good design.
• Regular adaptation to changing circumstances.
• Even late changes in requirements are welcomed

Disadvantages of Agile model:

• In case of some software deliverable, especially the large ones, it is difficult to assess the effort required at the beginning of the software development life cycle.
• There is lack of emphasis on necessary designing and documentation.
• The project can easily get taken off track if the customer representative is not clear what final outcome that they want.
• Only senior programmers are capable of taking the kind of decisions required during the development process. Hence it has no place for newbie programmers, unless combined with experienced resources.

Unlike the waterfall model in agile model very limited planning is required to get started with the project. Agile assumes that the end users’ needs are ever changing in a dynamic business and IT world. Changes can be discussed and features can be newly effected or removed based on feedback. This effectively gives the customer the finished system they want or need.

The Waterfall, V and the Agile model are not the only one's available for projects. These are the commonly used model for the Data warehouse implementations. There are other SDLC models which can be used for projects and those are. 

• RAD model
• Spiral model
• Big Bang model
• Prototype model

In this lecture, we discuss the real time scenarios which will help us decide on which model to choose for the Data Warehouse Development Process. 

From this session onwards, we will start to see that practical implmemetaion steps for a Datawarehouse Development  project. Below are the various phases which we will discuss about. 

• Project Planning
• Requirements
• Design
• Development/Testing
• Deploy
• Operations and Maintenance