Node JS: Advanced Concepts

Mitigate event loop performance issues by running Node in cluster mode, launching multiple server copies, and exploring worker threads with Libuv's thread pool; cluster mode is recommended, worker threads experimental.

Set up a tiny express app in index.js, define a get route, and listen on port 3000; initialize npm, install express, and run node index.js to test locally.

Explore cluster mode and worker threads to boost an express app's performance. See how a CPU-intensive doWork loop blocks the event loop, delaying other requests, and how clustering mitigates this.

Explore how node clustering uses a cluster manager to spawn worker processes, improving request handling by running multiple node instances on a single computer.

Require the cluster module, check cluster.isMaster, fork workers, and run the express server in the worker, demonstrating master vs. worker behavior in a tiny express app.

Learn how to use cluster.fork to run multiple Node.js processes, improving parallel request handling and response times for slow routes.

Explore how to benchmark node server performance using Apache Benchmark, measuring requests per second and latency to reveal diminishing returns from excessive clustering.

Refactor the index.js to replace the doWork function with PBKDF2 hashing to simulate real work, measure performance, and simplify benchmarking by setting each cluster child to a single thread.

Explore how Node.js clustering affects performance by benchmarking with AB, revealing diminishing returns when increasing the number of children beyond CPU cores.

Explore how PM2 provides cluster management for Node.js apps, then install the PM2 CLI globally with npm install -g pm2 and review the PM2 docs and GitHub page for setup.

Explore pm2 cluster mode for node apps, auto restarts, and scalable instances. Learn -i 0 for auto core-based workers, inspect with pm2 list, show, and monit, and manage in production.

Introduce web worker threads as an experimental option in Node.js, compare them with cluster mode for heavy lifting, and show how to install the web worker threads package.

Explore offloading heavy work from the Node.js event loop with the web-worker-threads library in an Express app. Use postMessage and onmessage for communication and note closure limitations.

Learn how to create and benchmark worker threads in node js, using postmessage and onmessage to return results to express, and measure concurrency across cores with ab.

Explore practical NodeJS development by cloning a prebuilt application with basic routing and authentication, then add advanced features to build a capable app without boilerplate.

Walk through a starter express boilerplate with index.js, middleware, mongoose models, and passport setup for Google OAuth, plus a React front end and blog routes.

Run npm run dev to start the express backend and react frontend, open localhost:3000, and login with Google OAuth after updating the MongoDB URI and Google keys in config/dev.js.

Create a new MongoDB instance with in-lab, configure a sandbox blog_dev database and user, then update MongoURI and test Google login to enable writes.

Explore how the Express server exposes authentication and blog routes, including Google OAuth login, logout, and creating user blogs. See how the React frontend consumes these routes to display blogs.

Discover how caching boosts read performance in a Node.js express app using MongoDB, and explore how MongoDB indexes, id and title queries, and potential collection scans impact query speed.

Learn how a cache server sits between Mongoose and MongoDB to store and reuse query results, speeding reads while writes go to MongoDB and clear the cache.

Explore Redis as an in-memory caching server for fast reads and writes, connect with node-redis, and install Redis locally on Mac or Windows.

Install and run Redis on macOS using Homebrew, configure it to start automatically, and verify the installation with redis-cli ping.

Explore how Redis stores data as a key-value store, using node Redis library to set and get values, handle async callbacks, and prepare for caching in your application.

Explore nested hashes in Redis, using hset and hget to manage multi-level keys like Spanish and German translations, and see how nested objects support caching in a Redis-based workflow.

Redis stores values as strings, so plain objects become object object. Stringify objects with JSON.stringify before storing and parse with JSON.parse when retrieving, to support caching in the project.

Cache the api/blogs route with Redis by storing each user's blog list under their user id key, starting small before extending to the whole app.

Learn to promisify a Redis cache lookup in a node.js blogs route, using util.promisify and async/await to check and store cached results before querying MongoDB.

Query the Redis cache for a cached blogs list and return it if present; otherwise fetch from Mongo, send the result, and update Redis with the blogs as JSON.

The lecture spotlights caching issues in a Redis-backed setup: no cache invalidation, bloated request handlers, and user-id keyed data, and it outlines a plan to refactor toward reusable, multi-resource caching.

Override Mongoose's exec to intercept queries, check Redis first, and reuse cached results. Create robust cache keys from query options and collection name, with expiration.

Patch mongoose's exec function by monkey patching query.prototype.exec to intercept queries, generate a unique cache key, and serve results from Redis before MongoDB.

Override a mongoose function to insert custom logic, then restore the /api/blogs route to its original implementation using Blog.find_user(req.user.id) and res.send blogs, and prepare for a reusable cache in cache.js.

Implement caching logic in the exec function and build a unique, consistent Redis cache key by combining the get query object with the collection name.

Combine the getQuery result with MongooseCollection.name using object.assign to create a safe cache key, preventing mutation of the original query and preparing for Redis serialization.

You set up a Redis client in Node.js, require Redis and util, define the Redis URL, create the client, and promisify the get function to prepare for key usage.

Implement a Redis cache: check for a key, retrieve with client.get, stringify the key once, and use async/await to run the query and store the result.

Navigate how exec returns a mongoose document, convert the result to JSON for Redis caching, and ensure cache hits return parsed JSON instead of querying MongoDB.

Hydrate models from redis by converting cached JSON into mongoose model instances with this.model and new, handling both single objects and post arrays, ensuring exec returns proper documents.

Learn to hydrate both single and array-based cache records by parsing json, using a ternary check, mapping with new model instances, and validating in the browser.

Implement a toggleable caching system in Mongoose by adding a chainable .cache method and useCache flag, enabling selective caching for queries and testing with blog post list.

Explore automatic cache expiration in Redis, and learn to programmatically expire cache regions with the EX setting, seconds, and future sets, while understanding that changes are not retroactive.

Learn to force cache invalidation in Redis with per-user nested hashes, enabling selective clearing when a user creates a post. The approach refreshes the affected data from MongoDB for updates.

Refactor caches in redis to use a dynamic top level hash key via an options key, enabling nested hashes for blog post or user ids in node js advanced concepts.

Implement a clear hash function in cache.js that deletes a Redis hash using client.del with a stringified key, and call it from the blog post route after creation.

Automate cache clearing by adding a middleware that clears the user's cache after a blog post is created, wiring it to the blog route and awaiting next to run handler.

We compare unit and integration testing and outline a plan to implement an integration testing pipeline with Jest and a headless browser to test React and Express apps.

Explore building a testing pipeline with Jest and a headless Chromium browser to interact with your app, assert rendered content, and tackle authentication and fresh data concerns in browser tests.

Create a new npm script test to run the Jest test suite, forming a simple testing pipeline. Use Puppeteer to launch a Chromium instance for browser-driven tests.

Create a tests folder with a header.test.js to verify the header elements, such as the logo and logged-in buttons like My Blogs and Logout, using jest to run the tests.

Learn how to launch a Chromium instance with puppeteer in tests, create a browser and page, use async/await, and run non-headless to observe the browser in action.

Learn to automate end-to-end tests with Puppeteer by launching Chromium, navigating to Local Host 3000, selecting DOM elements, and asserting header content to verify the app boots correctly.

Extract the header text by selecting the logo anchor with a CSS selector a.brand-logo and retrieving its innerHTML via page.$eval, then assert it equals blogster.

Discover how puppeteer serializes functions to text, sends them from the Node test runner to a separate Chromium process, and evaluates results back in Node.

Refactor test setup into a before each hook to enforce dry tests, prevent repeating browser and page initialization, manage variable scope, and ensure tests run asynchronously with automatic browser cleanup.

Automatically close Chromium instances after each test by adding an afterEach that calls await browser.close() in an async function, ensuring clean teardown of the test suite.

Automate the header login test to trigger the OAuth flow by clicking the login link, then verify redirection to accounts.google.com using Puppeteer page.url.

Learn to verify by asserting that a captured URL contains accounts.google.com using Jest's toMatch with a regular expression, ensuring a button correctly initiates the OAuth flow.

Demonstrate the challenges of automating Google OAuth in tests, showing header content changing after sign in and the need for a login method that bypasses OAuth in CI.

Develop strategies to handle authentication in automated testing for Node JS: Advanced Concepts, prioritizing fabricating a session over code changes and ensuring a universal OAuth solution.

Explore how node.js uses passport and cookie session to implement Google OAuth, with auth/google and auth/google/callback, code exchange, and cookie-based user sessions stored in MongoDB.

Explore how browser authentication works in a Node.js app, including Google login flow, session cookies (session and session.sig), and how passport reads req.session to identify users in MongoDB.

Review the authentication and session flow with cookie-session and passport, clarifying how req.user is populated, using alternate diagrams, and discuss testing by forging a session cookie.

Explore how session strings are signed with a cookie signing key using keygrip to produce a verifiable session signature that prevents tampering in a passport-based login flow.

Generate a fake session and signature in automated tests by creating a session object with a user id, converting it to a base64 cookie session string, and signing with keygrip.

Learn to set cookies on a running Chromium instance using puppeteer's setCookie, navigate to localhost:3000, and simulate login by applying session and session.sig cookies to bypass OAuth in tests.

Use puppeteer waitFor and dom selectors to verify the logout button text appears after sign in, extracting it with $eval from the anchor href 'auth/logout'.

Refactor authentication by centralizing test data in factory functions. Create a session factory that returns a session and signature, and a user factory that saves a new user to MongoDB.

Create a session factory in the test suite, refactor to top-level requires, and return an object with session and signature from a user. Prepare to wire a user factory later.

Explore assembling a session factory, wiring it into a header test, and refining with a user factory to generate session and sig for test cookies.

Create a user factory using Mongoose to generate and save a user to MongoDB for tests, and explain how Jest isolates tests and why we must require the User model.

Create a global setup file for Jest that runs once, requiring the user model and connecting mongoose to the project mongoURI via keys, and configure setupTestFrameworkScriptFile to load tests/setup.js.

Wire up the user factory to the authentication test and session factory, await the saved user from mongoose, sign in, and verify logout while refactoring toward a reusable test factory.

Explore refactoring authentication logic by adding a login method to the Page class and discuss a more clever ES2015 approach, with Puppeteer page context as the focus.

Evaluate extending a page class versus wrapping it with a custom page to add login and cookie logic, while ensuring easy access to the underlying Puppeteer page.

Learn how javascript proxies (es2015) manage access to multiple targets, enabling a single proxy to delegate to either the Page class or a customPage and simplify method calls.

Explore ES2015 proxies in action by building a proxy to broker access to multiple objects, using a get trap to delegate property access to greetings and moreGreetings.

Explore how ES2015 proxies merge a custom page with the underlying page to form a super page for testing, using a static build method on the custom page.

Combine puppeteer page and browser access into a single CustomPage via a proxy, enabling a streamlined, reusable interface for browser automation in tests.

Refactor puppeteer tests by wrapping browser and page logic in a single Page class, using Page.build to instantiate and run tests against localhost 3000.

Show how to set function lookup priority across custom page, browser, and page in Puppeteer, and compare two approaches to ensure close uses the browser's close function during tests.

Refactor authentication logic into a custom page class, add an async login function, and call page.login in the authentication test to streamline tests with puppeteer and the page object.

Create a page object with a get contents of function to fetch selector content, replacing Puppeteer dollar sign eval and inner html logic, and refactor tests to use the helper.

Learn to automate blog creation flow in a Node.js test suite using Puppeteer, including OAuth login, navigating to /blogs, and verifying the new blog form appears.

Automate login and navigation to the blogs page for blog creation tests using a login helper. Ensure after login you land on the /blogs route before proceeding.

Click the red floating button using a .btn-floating anchor, navigate to the blog creation form, verify the label blog title, and explore async test timeout issues in jest.

Increase jest timeout from five seconds to 30 seconds by configuring setup.js, enabling tests that launch chromium and navigate to the blog creation form.

Explore common test setup strategies for blog post creation, covering login states, validation errors, navigation to confirmation, and reusing shared setup across integration tests.

Use nested describe statements to group tests and share setup with before each, creating clear initial conditions like being logged in to view the blog create form.

Learn to assert validation errors by nesting describes for when logged in and using invalid inputs, simulate form submission, and verify two error messages appear for required fields.

Learn how to test valid form inputs in Puppeteer within Node JS advanced concepts, navigate to review screen, and verify the confirmation text before saving to the blog index page.

Write an end-to-end test for blog creation using Puppeteer, clicking the save button, waiting for AJAX navigation, and asserting the new card title and paragraph appear on the blog list.

Test that unauthenticated users cannot create or view blog posts by securing API interaction between the browser and Express server, using chromium browser tests to simulate sign-in and sign-out flows.

Explore how to test direct API requests from a browser using fetch to a blog route, validating success and 401 errors in a Chromium environment.

Use puppeteer’s page.evaluate to run a function inside a Chromium instance, executing a backend API request from a user-like environment and awaiting the result before returning to tests.

Demonstrates asserting page responses using puppeteer, employing page.evaluate to run a fetch in a not-logged-in scenario, return JSON, and verify the error 'you must log in'.

Enforce get restrictions by requiring sign-in to access /api/blogs, returning a 401 error when unauthenticated and retrieving the blog posts when signed in.

Learn to finalize a get test by returning the fetch result, evaluating the api/blogs request with page.evaluate, and asserting you must log in.

Master super advanced test helpers in Node.js by building flexible get and post request executors, refactoring to page.js, and executing multiple secure-route checks in parallel.

Explore continuous integration by using a CI server to clone the project, run automated tests (Jest and Puppeteer), and merge changes into the remote repository with subsequent deployment.

Learn the essentials of continuous integration with GitHub, including prerequisites, Travis CI setup via a Travis YAML file, and how pushes trigger builds and notifications.

Explore YAML as a simple json-like format with key-value pairs, nesting, and arrays, map it to JSON, and configure travis.yml for CI in Node JS.

Create a Travis yaml file for the project and specify node_js version 8 on trusty. Enable MongoDB and Redis services and set CI environment variables for ci-specific keys and urls.

Learn to configure Travis CI for a node project by setting NODE_ENV to ci, caching node_modules (including nested client/node_modules), and running npm install in the install stage to speed builds.

Build the production React app by running npm install and npm run build in the client directory, producing client/build for a Create React App with Express.

Configure travis script to launch the server in background with nohup and npm run start, pause for boot, then run tests with npm run test.

Update the express API to connect to ci-provided MongoDB at 127.0.0.1:27017 and Redis at localhost:6379 within Travis CI, using docs for guidance.

Set up Travis CI with new keys, environment-specific Redis and MongoDB URIs, and ensure the Express API serves built React files on port 3000 in CI and production.

Prepare your node tests for continuous integration by making puppeteer headless and enabling no-sandbox, and ensure test URLs use http prefixes for travis.

Push your server code to GitHub, link the repository to Travis CI, and set a new origin to your NodeCI project so CI automatically runs tests on changes.

Set up Travis CI to monitor your GitHub repository, authorize Travis CI on GitHub, connect the nodeci repo, and enable automatic clones and builds when you push new code.

Push local changes to GitHub to trigger Travis CI builds and view job logs. Configure environment variables from the Travis file and install Node modules.

Push code to GitHub to trigger Travis CI, watch the build pass as tests run with Jest and npm run test exiting zero, and enable automatic CI for future commits.

Learn how to add image upload to a blog post by integrating a file picker on the client side, sending images to the backend, and displaying them with each post.

Explore big issues around image upload, including optimal storage locations, upload mechanisms, and linking images to blog posts, with MongoDB storage evaluated and found costly.

Examine why storing uploaded images on the server's local hard drive is problematic for scalable multi-server deployments, and learn about the chosen alternative for image uploads.

Discover external image storage with an Express API, using Amazon S3 as the chosen solution, and learn how Azure or Google Cloud Storage could also be used.

Learn how to securely upload images to Amazon S3 by transmitting files from the browser through an Express API, with constraints for sign-in, post association, and image extensions.

Evaluate the common two-hop image upload pattern through an Express API and temporary storage to S3, highlighting CPU and memory costs and a direct-to-S3 alternative.

Learn a scalable upload flow where the client uploads directly to S3 via a pre-signed URL obtained from the server, reducing server load and then creating a blog post.

Demonstrate how presigned URLs grant temporary, one-time access to upload an image to a locked Amazon S3 bucket, enabling secure client uploads.

Explore how presigned URLs securely enable one-time file uploads to a specific S3 bucket, with expiration and encoded file name and type to prevent tampering.

Explore adding an image picker to the blog post review screen in a node js course, using a file input that accepts images and validates image file selection.

Add a file input and onFileChange to store the selected file in state, using event.target.files as a file list to prepare a pre-sign URL request to the Express API.

Select a single image file, extract it from event.target.files, and fetch a pre-signed URL before uploading the image and creating the blog post on the backend API.

Refactor submit blog function to upload image first via a pre-signed URL and only create the blog post in MongoDB if upload succeeds, outlining the pre-sign URL to S3 workflow.

Learn how to create an S3 bucket, generate IAM credentials limited to that bucket, and securely implement presigned URLs for image uploads in a Node.js backend.

Create an s3 bucket through the aws console by choosing a unique bucket name and region near you. Then use iam to create credentials granting access only to that bucket.

Create an iam policy granting access to the blog bucket and attach it to a new api user, limiting keys to that bucket.

Create an IAM user with programmatic access, attach the S3 blog bucket policy, and generate API keys for the Express API; store keys in dev.js and add to gitignore.

Install the aws sdk npm module, create an upload routes file, and wire an api/upload route to generate a pre-signed url from the s3 bucket for image uploads.

Configure the AWS SDK to generate a pre-signed url via a new /upload route, wiring access keys and an S3 client to serve the url to a React app.

Learn to generate a pre-signed URL for uploading images to an S3 bucket using getSignedUrl with the putObject operation, detailing bucket, content type, and a unique key.

Implement a route that calls S3.getSignedUrl with putObject, specifies bucket and key, builds the key from the user id and a uuid, enforces login, and returns the key and url.

Test the pre-signed URL feature by retrieving a key and URL in the browser, using the user ID and a random file name for uploads to the S3 bucket.

Learn how to fetch a presigned URL from the backend and upload an image to AWS S3 using axios.put, including setting the correct content-type and handling preflight errors.

Learn to diagnose and resolve CORS errors when uploading images to an S3 bucket using pre-signed URLs, by configuring allowed origins and put requests from localhost:3000.

Upload an image to amazon s3, verify it appears in the user folder on the s3 dashboard, then enable access and attach the image url to the blog post.

Configure S3 bucket policies to publicly expose images, using the policy generator to enable GetObject for all objects under your bucket ARN, then test.

Tie s3 uploads to blogs by passing the image url with blog data, updating the blog model and routes to store the image url beside title and content.

Test uploading an image and saving its file name with the blog post, then verify the image path is stored and prepended with the domain when displaying the blog.

Implement a render image function in BlogShow.js to display a blog image when an image URL exists, prepending the S3 bucket domain.