AI Vision Systems for Self-Driving Cars in Production on AWS

Compare vision transformers and CNNs, highlighting when to use global-context self-attention versus local feature detection, with guidance for large data sets, segmentation, and fast mobile inference.

Examine how attention in vision transformers links every token to every other, causing quadratic n squared complexity, and note patches as a solution amid CNNs' faster efficiency.

Explore how vision transformers turn image patches into embeddings via a linear projection, with CLS tokens and learned positional encodings, trained jointly with the model for image classification.

Demonstrates how vision transformers use self-attention to compute patch-to-patch scores with query-key dot products, with gradients from the mlp head flowing to patch embeddings, including the cls token.

Learn how a vision transformer uses a CLS token to aggregate image patches through multi-head attention, computing queries, keys, and values and forming a global representation for classification.

Explain how sam's image encoder uses a vision transformer to produce image tokens and a feature embedding, with the prompt encoder turning clicks or auto prompts into sparse tokens.

Explore SAM auto prompt mode, where learned object finder queries guide segmentation without user input, using prompt self-attention and cross-attention with vision transformer features to output masks.

Create patch embeddings from an input image using a vision transformer, add learned positional encodings, and apply single-head self-attention within the transformer encoder.

Project patch tokens to queries, keys, and values to compute attention scores for vision transformers in sam, apply stabilized softmax, and blend with values to form context-aware, residual-connected representations.

Sam encodes images with a vision transformer into embeddings, then uses prompt tokens and two-way attention to generate mask candidates and IoU scores in manual or auto prompt modes.

Explain prompt self-attention in auto prompt mode, covering query, key, value vectors, attention scores, scaling, and how updated prompt embeddings enable communication for subsequent image cross-attention with the Vit encoder.

Explore prompt image cross attention, where updated prompt embeddings query image token edges and colors to attend to relevant image regions, producing richer features for segmentation.

Explore image to prompt cross-attention, updating image tokens with prompt-conditioned evidence. See how queries from image tokens align with updated prompt keys and values to highlight objects and suppress background.

Watch an optional video that demonstrates the Sam architecture using prompt self-attention and image-to-prompt cross attention to build mask maps from patch features, with a toy upsampling example.

Describe how the IOU token and tiny head score and select the best mask, then deduplicate overlaps. Compare toy and real Sam architectures, including multi-head attention and upsampling differences.

Create a SageMaker AI domain in the London region using the quick setup; organizations may tailor roles, encryption, and MLflow, while the process may take time.

Install and configure libraries to run the segment anything model in SageMaker, including pip installs and loading a Facebook Research model from S3.

Stop instances and servers to avoid charges, save your work, shut down all, delete all workspaces, and close this window.

Download Meta's Sam model, upload the 2.4GB ViT H variant to an S3 bucket, and prepare to load it later in a Jupyter notebook via SageMaker AI.

Update the IAM role to include S3 access by attaching the Amazon S3 full access policy, consider the least privilege principle, then rerun the cell to apply changes.

Import essential libraries for a computer vision workflow, including numpy, torch, matplotlib, PIL, cv2, and sam_model_registry and sam_automatic_mask_generator.

Learn how to integrate AWS Rekognition with the Sam model to produce labels and segmentation masks, using bounding boxes and overlap checks, with S3 bucket setup for inputs and outputs.

Explain why we should avoid double rgb conversion in helper functions, keep code in rgb land for consistency, and discuss productionizing by converting once to improve efficiency.

Learn to implement a Rekognition-based detection pipeline with min confidence threshold and filtering by target labels, convert percentage bounding boxes to pixels, deduplicate overlaps, and return unique detections.

Initialize the Sam model from S3, download heavy weights, and load it on CPU or GPU with memory-efficient settings, then configure the automatic mask generator parameters for robust segmentation.

Implement the main processing function to extract cars and pedestrians with segmentation masks from images, using a recognition step and Sam segmentation, with CPU and GPU handling.

Execute the main processing function to find the best recognition and SAM matches, merge labels with precise masks, and return the image, masks, and objects; then run the processing loop.

Run the main processing cell to trigger the pipeline, download the image from S3, detect objects with AWS Rekognition, and generate Sam masks. Print results with confidence and mask area.

Visualize the raw recognition detections with bounding boxes and labels using matplotlib, detailing how to plot, style, and annotate detected objects.

Visualize all SAM masks by overlaying random color masks on the image, sorting by area, and applying alpha blending to reveal masks clearly.

Visualize match quality by computing the intersection over union (IOU) between Rekognition and Sam bounding boxes, converting x y w h to coordinates, and assessing overlap to gauge alignment.

Visualize IOU scores for matched results by plotting a matplotlib bar chart that colors bars by score and labels each bar with the object and score, showing recognition vs Sam.

Refine image visualizations by cleaning up plots, removing excess numbers and whitespace with tight layout, then save the image and prepare a new cell for future visualizations.

Visualize segmentation masks without bounding boxes by overlaying colored, transparent masks and bold labels on the original image to show final segmented objects.

Save segmentation results to S3 by creating a save_results_s3 function, generate a timestamped file name, and upload a metadata json with detections and labels.

Save a matplotlib generated image with segmentation masks, bounding boxes, and labels to a memory buffer, then upload the PNG to S3 using a timestamped, annotated key.

Save each object's binary mask as a separate file on S3 by looping through results, converting boolean masks to grayscale, buffering, and uploading with image-specific names for AI model training.

Move the if block outside the function to enable invocation, and only run the next function when the matched results length is greater than zero.

Learn how to add a GPU-enabled notebook server on SageMaker, upload and test images from a bucket, and navigate AWS quotas to request higher notebook instance limits.

Test the full sam+vision transformers pipeline by loading the model, running recognition and segmentation, visualizing masks and bounding boxes, and saving outputs to your bucket while tuning confidence thresholds.

Optimize memory in a sam+vision transformers workflow by clearing heavy variables, forcing garbage collection, and caching models, then download small images, monitor gpu memory, and save segmentation masks to s3.

Separate Sam loading into its own cell, batch-infer over an s3 folder, and use manual prompt mode with bounding boxes to mask only targeted regions.