The Ultimate AI Pipeline for XR Development

Welcome Letter: The Ultimate AI Pipeline for XR Development

Dear Creator,

Welcome to The Ultimate AI Pipeline for XR Development—I’m thrilled to have you here!

I’m Dominique Wu, your instructor and creative co-pilot on this journey. With over a decade of experience in Product Design and XR development, I’ve had the honor of working with visionary companies like Meta and Walmart’s Store No. 8, as well as leading projects, speaking at events, and organizing hackathons across the XR universe.

I’ve spoken at top industry gatherings like AWE and XR Women, hosted over 500 speaker events, led the ShapesXR Hackathon, and authored the book Spatial Design: Breaking the 2D Paradigm. Today, I’m proud to serve as an AI Fellow in the California Community Colleges Chancellor’s Office and teach XR as an adjunct professor at Cañada College.

But this course? This is where things get real.

What You’ll Learn

This is a hands-on, beginner-friendly course. You don’t need prior coding or Unity experience—we’ll build your skills step-by-step as we create a powerful mixed-reality storytelling game together.

Inspired by Edgar Allan Poe’s haunting poem The Raven, you’ll design, develop, and publish your own immersive mini-game from scratch. Along the way, you’ll harness the power of today’s most exciting AI tools:

• ChatGPT – for brainstorming, vibe-coding, and generating scripts

• Trellis + Blender – for 3D modeling and scene-building

• Anything World – to breathe life into your characters with animation

• ElevenLabs – to craft voiceovers and atmospheric sound

• Suno – to compose original, emotionally resonant music

We’ll bring it all together inside Unity, using Meta Building Blocks to develop your experience for Meta Quest 3. You’ll code simple interactivity with OnTriggerEnter, manage your UI, and export your final build to SideQuest, where your friends, family, and future fans can step into your creation.

What You’ll Walk Away With

By the end of this course, you’ll have:

• A fully published XR game inspired by classic literature

• A deep understanding of an end-to-end AI-powered XR pipeline

• A polished portfolio project ready for employers, grants, or indie game festivals

• And most importantly, the confidence to keep building.

  
  

Let’s Begin

XR isn’t just about technology. It’s about imagination. Storytelling. Empathy. And now, it’s your turn to craft experiences that blur the line between real and unreal.

I’m so excited to be on this journey with you.
Ready to dive in and bring your story to life? Let’s get started!

With creative energy,
Dominique Wu

Instructor | Founder of XReality Pro | AI Fellow | Author

In this video, I share my process for brainstorming game ideas using ChatGPT. I discuss how I transform classic literature, like "The Reven," into immersive game concepts, focusing on game mechanics and emotional styles. I encourage you to read sources and take notes to enhance your creativity. Please take a moment to brainstorm your game ideas and share them with me!

Key points include:

• Initial Reading: Importance of understanding the source material before brainstorming.

• Using ChatGPT: How to effectively utilize ChatGPT for generating game concepts, mechanics, and emotional styles.

• Game Concept: The idea of creating a psychological suspense game that explores themes of obsession and emotional depth.

• Visual and Audio Elements: Suggestions for visual styles and soundscapes that enhance the gaming experience.

• Game Mechanics: Discussion on interactive elements and player engagement strategies.

Assumptions

• The reader has a basic understanding of game development concepts.

• Familiarity with Edgar Allan Poe's works, particularly "The Raven."

• Understanding of how AI tools like ChatGPT can assist in creative processes.

Testing Strategy

• Review the brainstorming session for clarity and coherence.

• Validate the game concept against established game design principles.

• Test the emotional impact of the proposed game mechanics through user feedback sessions.

• Ensure that the visual and audio elements align with the intended emotional experience.

In this tutorial, I will walk you through the process of using ChatGpt 4o to create a 3D character design, specifically focusing on Edgar Allan Poe. I will explain how to generate images with transparent backgrounds and the importance of using references for better results. I also encourage you to create T-poses from different angles for your character, which will be useful for animation later on. Please follow along and share your creations with me!

Key highlights include:

• Step-by-step instructions on how to create a character design with T-pose from multiple angles (front, side, back).

• Emphasis on using transparent backgrounds for better integration into Trellis.

• Recommendations for providing reference images to improve AI-generated outputs.

Assumptions

• Users have access to ChangePT, either through a Plus account or a free version.

• Users are familiar with basic image editing and AI tools.

• Users have a basic understanding of character design principles.

Testing Strategy

• Verify that the tutorial successfully guides users through the character design process.

• Ensure that generated images maintain a transparent background as specified.

• Test the integration of generated images into Trellis to confirm professional quality.

In this video, I will walk you through using Trellis 3D for converting 3D models and highlight its advantages over Mashy. I emphasize that Trellis is free to start and easy to install locally, making it a great option for our projects. I also provide links in the description for downloading the necessary files and running the software.

Key highlights include:

• Integration of Trellis for converting 2D images to 3D models.

• Support for multiple image uploads to enhance model generation.

• Local execution of the model generation process for improved performance and privacy.

• Detailed instructions for installation and usage are included in the documentation.

Assumptions

• Users have basic knowledge of 3D modeling and software installation.

• Users are familiar with using local applications and have the necessary permissions to install software on their machines.

• Users may choose between Trellis and other software like Mashy, based on their preferences and requirements.

Testing Strategy

• Verify the installation process of Trellis on different operating systems (especially Mac).

• Test the functionality of uploading multiple images and generating a 3D model.

• Ensure that the generated models can be opened and edited in Blender without issues.

• Validate the performance of the local execution compared to online alternatives.

In this video, I’m excited to show you how to create 3D animations using Anything World. It’s a free platform that allows you to animate various objects, and I’ll guide you through the process step by step. I also share my preference for using Trellis to generate 3D models. Please make sure to follow along and try out the features I demonstrate!

In this video, I walk you through the process of creating a 3D environment using ChatGPT and Trellis, followed by cleaning up the file in Blender. I emphasize the importance of refining your models and provide tips on how to effectively use the tools. I also encourage you to revisit the previous tutorial if you need a refresher on downloading Trellis. Please make sure to follow the steps I outline for importing and cleaning up your models in Blender.

Summary

In this video, I’m excited to show you how to use ElevenLabs for voiceover work. I’ll walk you through its various functions, including text-to-speech, voice changing, and creating audiobooks. I believe this tool is fantastic for generating engaging audio content. Please take a look at the different voice options and features, and let me know what you think!

In this video, I’m excited to show you how to use Gen.AI to generate background music. I walk you through the process of using Suno.com, where you can create both custom music with lyrics or instrumental tracks. I share my experience with different styles and the emotions they evoke, particularly focusing on a sad and classy atmosphere. I encourage you to experiment with your own ideas and remember to toggle the settings for instrumental tracks. Let’s get creative!

In this video, I walk you through the process of setting up your Oculus Quest and Unity. I cover the necessary requirements, including hardware specifications and software installations, to ensure a smooth development experience. I also share important links and resources for downloading the MetaQuest apps. Please make sure to follow the steps closely, as I highlight key actions needed for successful setup.

This video introduces the setup process for integrating Meta Quest into Unity, enabling XR development. Key features include:

• Creation of a new Unity project using Universal Render Pipeline.

• Installation of Meta XR Core SDK and Interaction SDK for immersive experiences.

• Configuration of XR Plugin Management for Android platform compatibility.

• Implementation of hand interaction features using the Meta SDK.

Assumptions

• The user has Unity installed (version 2022.3.15f1 or later).

• The user has access to the Meta Quest device and the necessary SDKs.

• The user is familiar with basic Unity operations and project management.

Testing Strategy

• Verify the installation of Meta XR Core SDK and Interaction SDK in Unity.

• Test the functionality of hand interactions within a sample scene.

• Ensure that the project builds successfully for the Android platform and runs on the Meta Quest device.

• Validate that all interactions work as expected without errors.

Overview

This video introduces a mixed reality experience utilizing Meta Building Blocks. Key features include:

• Integration of Meta SDK for building mixed reality applications.

• Implementation of camera rig and pass-through functionality to visualize the real world.

• Ability to interact with virtual objects using real hand gestures.

• Set up a grab interaction with a cube object in the mixed reality environment.

Assumptions

• The user has access to a compatible headset and the Meta SDK.

• The development environment is set up for Android builds.

• Users are familiar with basic Unity functionalities and the concept of mixed reality.

Testing Strategy

• Build and run the application on a compatible Android device.

• Verify that the camera rig displays the real-world environment correctly.

• Test the hand tracking functionality to ensure users can grab and interact with the cube object.

• Check for any errors related to camera settings and ensure that the experimental features are enabled.

In this video, I walk you through the process of creating a level design and implementing simple animations in Unity. I cover essential topics like exporting 3D models using GLB and GLTF formats, setting up CameraRig, and managing animations effectively. I also share some tips on using AI tools for faster development. Please make sure to follow along and test the settings I demonstrate.

Overview

This video implements the following features related to importing and animating 3D models in Unity:

• Added functionality to import GLB/GLTF files using the Unity GLTFast package.

  • Created a structured scene setup for XR development, including a CameraRig and empty game objects for organization.

• Implemented simple animations for 3D models, including idle, flying, and landing states for a raven model.

• Integrated keyboard controls to toggle between different animations using C# scripts generated with ChatGPT.

Assumptions

• The Unity GLTFast package is installed and configured correctly in the project.

• The 3D models are prepared and available in the specified folders.

• A basic understanding of Unity's animation system and C# scripting is assumed for further modifications.

Testing Strategy

• Verify that GLB/GLTF files can be imported without errors.

• Test the animation functionality by pressing designated keys (I for idle, F for flying, L for landing) to ensure animations play correctly.

• Check that the scene setup is organized and that all game objects are functioning as intended in XR mode.

In this video, I will show you how to create a new audio management system for a Unity project, focusing on background music and voiceover integration. Key features include:

• Background Music: Plays a looping background track at the start of the scene.

• Sound Effects: Implements a knocking sound effect that triggers at a specific time.

• Voiceover Playback: Sequentially plays a series of voiceover clips with pauses in between to enhance narrative delivery.

• UI Integration: Displays a user interface element that appears after the knocking sound, enhancing user interaction.

Assumptions

• The audio files for background music, sound effects, and voiceovers are available and correctly formatted.

• The Unity project is set up with the necessary audio components and UI elements.

• The developer has a basic understanding of Unity and C# scripting.

Testing Strategy

• Test the scene to ensure the background music plays on start and loops correctly.

• Verify that the knocking sound plays after a specified delay.

• Check that the UI element appears after the knocking sound.

• Ensure that voiceover clips play sequentially with the correct pauses in between.

• Adjust audio priorities to ensure clarity between background music, sound effects, and voiceovers

In this video, I walk you through the process of creating simple interactions in Mixed-Reality. I cover key steps like setting up colliders, triggering animations, and managing sound effects. I also share some tips on how to adjust the positioning of elements for better gameplay. Please make sure to follow along and try implementing these techniques in your projects!

In this video, you’ll learn how to successfully publish your app on SideQuest—from building your APK in Unity to navigating the submission and review process. We'll cover the technical steps, common pitfalls, and strategic tips to get your app in front of users smoothly.

Key Highlights:

• How to configure your build settings and player profile in Unity

• The importance of setting the correct company name, product name, and versioning

• Step-by-step guide to build and export your APK file

• How to upload and submit your app to SideQuest

• Common issues and how to troubleshoot submission errors

Assumptions:

• You have a basic understanding of Unity and APK development

• You’re comfortable navigating build settings and player configurations

• You have access to SideQuest and understand its role in distributing VR apps

Testing Strategy:

• Ensure the APK builds without player-setting-related errors

• Double-check metadata (company name, product name, version) before export

• Test the APK on your Meta Quest headset for functionality and performance

• Submit to SideQuest and monitor the process for feedback or required updates

You’ll also learn how to navigate the SideQuest submission and review system:

Submission Breakdown:

• Create an account on SideQuest to upload your app

• Fill out your app’s details, upload the APK, and attach media like screenshots or trailers

• Submit for review—typically takes about 7 days

• Respond to feedback if any changes are requested

• Use the SideQuest dashboard to monitor downloads and user engagement