SQL, NoSQL, Big Data and Hadoop

Explore the relational model by examining tables and rows, and learn how primary keys, null values, and constraints such as not null, unique, and foreign keys ensure data integrity.

Explore sql, a declarative language grounded in relational algebra for interacting with relational databases, covering projection, selection, join, and other operators, plus ddl, dml, and ansi standard procedural extensions.

Model the Movielens dataset in a relational database by creating users, movies, ratings, tags, and genres tables, establishing master-detail relationships and normalization for efficient reads and writes.

Explore normalization versus denormalization in the Movielens relational model, optimizing data integrity and reducing redundancy while balancing transactional and analytical use cases.

MySQL is an open source relational database management system with a pluggable storage engine architecture and InnoDB as the default engine, widely used in web hosting.

Load movielens data into a MySQL database using the Movielens MySQL dump, implement the ERD, and explore DDL, DML, and normalization with primary and foreign keys.

Explore how indexes accelerate read queries and prevent full table scans, while write costs rise with maintenance. Learn primary, unique, non-unique, geospatial, and full-text indexes, plus explain usage.

Contrast OLTP and OLAP systems to justify data warehousing, implement ETL to integrate data, emphasize denormalization, and model a fact table with dimension tables for analytics.

Learn to write aggregation and window analytical queries on a data warehouse, leveraging keys, group by, over, inline views, and ranking to reveal distributions and top results.

Maintain a time-ordered record of events with transaction and commit logs. Use write-ahead logs and log shipping to support crash recovery and durable replication to keep a standby replica synced.

Relational databases dominate the market with a flexible ad hoc query model using select and where clauses; learn sql topics from oltp to olap, normalization, and stored procedures.

Explore how distributed databases enable horizontal scaling, replication, and fault tolerance through the replication factor, primaries and replicas, and load balancing across a cluster.

Explore Brewer's cap theorem for distributed databases, detailing network partition tolerance, consistency, and availability, and how two guarantees are chosen under partitioning.

Explore how databases balance consistency, availability, and partition tolerance through BASE, soft states, and eventual consistency, contrasting ACID for transactions with NoSQL flexibility.

Classify distributed databases beyond relational models by data model, data variability, and operational capability. Explore NoSQL types: key-value, document, wide-column, graph, and specialized systems like time series and search engines.

Explore key value stores, the simplest, high-performance databases with a key-value model, no SQL, and no joins, ideal for distributed in-memory caching and fast lookups using Redis.

Explore Redis, an in-memory, open-source key-value NoSQL database written in C. Leverage master-slave replication, sentinel failover, and automatic sharding, as it scales horizontally while offering Lua scripting and language support.

Install Redis quickly with Docker, or build Redis from source to learn the process. Download the source, compile with make, install jemalloc and line noise, then start Redis and benchmark.

Explain big O notation and its role in assessing time complexity for key-value store operations, outlining constant, linear, and logarithmic cases and worst-case behavior with mixtures like n log n.

Discover how Redis uses binary-safe keys and strings to store diverse data structures, manage expiration, and leverage commands like set, get, del, and type.

Explore Redis data structures: sets and sorted sets. Learn how sets ensure no duplicates and constant-time operations, while sorted sets attach scores for ordered retrieval, range queries, and rankings.

Explore Redis geo and Hyperloglog data structures for geospatial indexing and distinct counting, using geo add, distance, and radius queries, and Hyperloglog merge for weekly unique counts.

Master Redis pub/sub for asynchronous, real-time messaging between loosely coupled systems using channels, and apply atomic Redis transactions with multi and execute to ensure consistent updates.

Model Movielens in Redis by exploring two setups: a cache between a primary database and application, or a Redis primary store, using strict access patterns and sets, hashes, and transactions.

Demonstrates using Redis as a primary database with a Java Maven app, loading Movielens data via Jedis, and modeling data with lists, hashes, sets, and sorted sets.

Explore the key value model with Redis, Riak, and Memcache for caching and publish/subscribe flows, while Redis data structures deliver fast performance yet don’t support analytical queries.

Explore document-oriented databases that store records as schema-less documents in json, xml, or yaml formats. Learn CRUD, indexing, projection, and aggregation with MongoDB, CouchDB, and Elasticsearch.

MongoDB is the most popular document oriented NoSQL database, free and open source, scalable and schemaless, stores data as BSON, and supports ad hoc queries, indexing, and geospatial capabilities.

Install MongoDB in your learning environment using Docker, a VM, or apt-get on Ubuntu; then launch the mongo shell to connect to a default, unauthenticated server.

Model the Movielens dataset in MongoDB, converting the relational design to a document oriented structure of databases, collections, and documents, with cap collections, sharding, replica sets, and write concerns.

Model movielens in mongodb by balancing normalization (object referencing) and denormalization (object embedding), embedding occupation and age group in user and representing genres as an array.

Implement Movielens migration from MySQL to MongoDB using a Java Maven project; migrate movies and ratings, build documents on the fly with no ddl upfront, and compare normalization versus denormalization.

Learn MongoDB CRUD with the mongo shell on the movielens dataset, covering insert variants, read with projection and pagination, updates with $set, deletes, and complex queries.

Discover how indexes boost read performance in MongoDB, explain reveals the query plan, and explore index types—from primary keys to full text and geospatial—and shard key tradeoffs.

Explore how MongoDB uses aggregation frameworks and MapReduce to analyze data by year and genre, detailing map and reduce functions in JavaScript and saving results to a collection.

Explore the MongoDB aggregation framework and its pipeline stages: project, unwind, group, match, and lookup, to analyze data, compare denormalized vs normalized schemas, and note performance versus mapreduce.

Model a MongoDB data model from a MySQL schema using Spark to process UK motor vehicle testing records, then reproduce MySQL aggregations with MongoDB's aggregation framework and study index impact.

Wraps up MongoDB by highlighting capped collections, geospatial support, aggregation framework, and schemaless data with ad hoc indexing, and notes when not to use it.

Explore how search engines store data as documents with full text indexes and inverted indexes, using tokenizing, stemming, and relevance scores to deliver results, powered by Lucene, Elasticsearch, and Solr.

Explore Elasticsearch basics: indexes, mappings, documents, and cluster concepts (nodes, shards, replicas) plus analyzers that use character filters, tokenization, and token filters to build the index.

Model movielens data in elasticsearch by denormalizing documents within a single index, using multi-field types and copy_to, while noting no cross-index joins.

Master CRUD operations in Elasticsearch by using rest endpoints to create, read, update, and delete documents and indexes, perform bulk indexing, and verify results with search and count endpoints.

explore writing queries to search and aggregate data in Elasticsearch using the query dsl, including term level and full text queries, and boolean and compound constructs.

Learn how to build intuition for Elasticsearch aggregation queries and write your own aggregations using bucket, matrix, and pipeline types with nested sub-aggregations.

Explore the elastic stack (ELK) and learn how Elasticsearch stores data, Logstash collects and enriches it in real time, and Kibana visualizes insights.

Load a CSV of UFO sightings into Elasticsearch via bulk upload, define geo point and date mappings, then build Kibana visualizations and a dashboard.

Explore search engines as document-oriented databases with powerful full-text search and analytics, using Elasticsearch for information retrieval, monitoring, NLP integration, and real-time dashboards.

Define column oriented databases, their schemaless and sparse data models, and the role of column families and row keys. Examine Bigtable, Cassandra, and HBase, plus Phoenix SQL on bass.

Explore how HBase, a Hadoop-based wide-column database, enables super fast lookup and range queries over massive data sets on HDFS, inspired by Google Bigtable.

Explore the HBase architecture, including masters, region servers, and Zookeeper coordination, plus data flow with MemStore, HLogs, and HFiles on Hdfs, and region splitting.

Choose from Docker, VM sandbox, cloud, or local cluster to run a Hadoop-based base server, configure Ambari, and install and run HBase across distributions.

Zookeeper acts as a centralized repository and service for shared configuration, coordination, leader election, synchronization, and quorum-based writes, enabling stable, highly available distributed systems such as HBase, Kafka, and Hadoop.

Model Movielens data in HBase as a multi dimensional key value store using row keys and composite genre keys, with column families for movies, users, and ratings.

Explore performing CRUD operations in HBase, including creating namespaces and tables, loading data with batch imports, using put/get/scan, and understanding schema design and performance considerations.

Explore Apache Phoenix, an sql layer on HBase delivering acid transactions and a relational model with tables, views, indexes, and udfs for sql access within Hadoop.

Explore how Apache Phoenix enables read access to HBase base tables by creating Phoenix views over Movielens data, enabling aggregation and analytical queries as if querying a relational table.

Explore a real-time analytics demonstration for GeoLife GPS trajectories, streaming data into HBase via Spark and Kafka, querying with Phoenix, and caching with Redis, using JDBC and a Swing UI.

Explore wide column stores inspired by Google's Bigtable, including HBase on Hadoop and Cosmos DB, emphasizing large-scale throughput, sparse column data, and timestamps.

Explore what time series data is and why time series databases exist, examine popular databases such as InfluxDB and Prometheus, and install and write data to a time series system.

Explore Influxdb, an open source time series database optimized for fast, scalable storage and retrieval with a sql-like language, rest api, and schema-less design.

Install the influxdb server from a binary, bootstrap and start it, connect via the CLI to perform read and write operations, and prepare the stack—telegraph, chronograph, capacitor.

Explore the influxdb data model by defining databases as namespaces, measurements as tables, and retention policies; distinguish fields and tags, and understand series, points, and timestamps in utc.

Explore univariate and multivariate time series in InfluxDB by writing data with Python, using line protocol, creating databases, and applying continuous queries to derive moving averages, then visualize with chronograph.

Explore the tick stack, an end-to-end time series platform for collection, storage, analysis, visualization, and monitoring. Discover telegraf, influxdb, chronograph, and capacitor with plugins for input, output, aggregation, and processing.

Demonstrate monitoring a Redis server and its host using the tick stack components—Telegraf, InfluxDB, Chronograph, and Capacitor—with dashboards, alerts, and benchmarking tools to optimize resource use.

Explore why time series databases are needed for end-to-end scalable storage and management of time series data, including collection, storage, visualization, transformation, abnormality detection, and notification.

Explore how graph databases model relationships with vertices, edges, and graph traversal, compare native versus non-native storage, and examine graph algorithms and popular systems like Neo4j and JanusGraph.

Explore core graph modeling concepts, including nodes and relationships, directed vs undirected, connectedness, cycles, weighted edges, property and labeled graphs, parallel edges, density, and monopartite, bipartite, n-partite, and multigraph structures.

Remodel the Movielens dataset as a graph using an actor-action approach with user and movie nodes and ratings or tags as edges; move genres to a genre node for analytics.

Explore Neo4j, an open source acid-compliant, linearly scalable graph database with native graph storage and replication. Learn its core and read-replica architecture, bolt connectivity, cypher queries, and enterprise edition clustering.

Set up a Neo4j learning environment via Docker sandbox or standalone installation, explore graph data with Bloom, and practice real-time movie-review recommendations using Cypher and CRUD queries.

Master Cypher, the declarative graph query language based on pattern matching, for traversing and manipulating graphs in Neo4j using nodes, relationships, labels, properties, and core clauses.

Import movielens data into Neo4j via a Cypher script, define user, movie, and genre labels, load CSV files, and build a ratings graph with genres moved to relationships.

Demonstrates building a Spring Boot web service backed by a Neo4j graph database for Movielens, exposing CRUD endpoints and movie recommendations tested via Swagger UI.

Explore how graph analytics analyzes nodes and relationships, using pathfinding, community detection, propagation, centrality, and similarity algorithms to reveal insights in a graph database.

Graph databases treat relationships as a first-class component and enable graph algorithms, knowledge graphs, and graph-based search, with use cases from fraud detection to social network analysis, for interconnected data.

Discover how Hadoop provides distributed storage and processing of big data on a cluster of commodity hardware, delivering scalability, redundancy, and fault tolerance.

Explore how HDFS achieves distributed storage with the NameNode and DataNodes, storing file blocks, managing replicas and metadata, and ensuring data integrity with checksums and heartbeats.

Explore how Hadoop’s yarn handles cluster resource management and job scheduling, detailing resource and node managers, application masters, containers, and data localization for efficient processing.

Learn to inspect a Hadoop cluster with Ambari, view HDFS and YARN components, and explore NameNode, DataNode, and secondary NameNode on IBM Analytics Engine.

Explore map and reduce in the MapReduce paradigm and see how Hadoop reads inputs, writes outputs, and orchestrates the shuffle phase, combiners, and reducers for big data.

Explore MapReduce concepts through hands-on examples with the Movielens dataset: counting movies by genre and generating rating summaries, using mappers, reducers, combiners, and custom data types.

Discover how Apache Pig simplifies big data processing on Hadoop with Pig Latin, loaders, transformations, actions, and UDFs, and learn deployment via grunt shell, batch mode, or Java integration.

Explore pig latin exercises using MapReduce and Tez execution modes, load true csv with piggy bank's csv loader, and count genres by aggregation.

Explore Apache Spark, a general-purpose distributed computing framework for big data processing, supporting Java, Python, Scala, and R, with a driver–executor architecture, in-memory RDDs, lazy transformations, and DAG-based execution.

Demonstrates spark usage across scala repl, python repl, and r repl, with spark context and spark session, RDD and spark SQL data frames, joins, and caching.

Explore data serialization and storage formats such as sequence files, Avro, Parquet, and ORC, comparing row- and column-oriented layouts, compression, schema evolution, language compatibility, and read–write performance.

Explore how Hadoop evolved from a storage and processing system to an ecosystem of open source projects for data ingestion, processing, streaming, graph analytics, workflow automation, and security.