SAP Gen AI Advance Training

SAP Generative AI introduces a new era of intelligent enterprise applications by combining Large Language Models (LLMs), SAP Business Technology Platform (BTP), and SAP’s trusted data foundation. The Building Blocks of Agentic AI represent the core components required to design, orchestrate, and deploy autonomous AI agents capable of performing tasks, making decisions, and collaborating across SAP systems.

In this module, learners explore how agentic workflows are created using SAP’s Gen AI capabilities—ranging from prompt engineering, embeddings, vector retrieval, and orchestration to SAP AI Core, SAP AI Launchpad, Joule, and SAP Build Process Automation. By mastering these building blocks, learners can build AI-driven agents that understand business context, execute multi-step tasks, automate decisions, and enhance user productivity across finance, procurement, supply chain, HR, and more.

This module empowers SAP professionals to design end-to-end AI agents that can:

Understand natural-language instructions

Retrieve relevant business data

Act autonomously through workflows

Integrate with SAP applications and external APIs

Provide accurate business recommendations and solutions

By the end of this topic, learners will have a strong foundation to build scalable, responsible, and enterprise-ready agentic AI solutions within the SAP ecosystem.

SAP’s Agentic AI Frameworks provide the architectural foundation for building autonomous, task-oriented AI agents within the SAP ecosystem. These frameworks combine the power of Large Language Models (LLMs) with SAP’s enterprise-grade tools, business context, and process automation capabilities. Their purpose is to enable agents that can think, reason, decide, and act across SAP applications.

1. Foundation Layer: SAP Gen AI + LLMs

At the core of SAP’s agentic architecture is the integration of advanced LLMs with SAP’s semantic and business data models.
This layer includes:

SAP’s Generative AI Hub (model access and governance)

SAP Joule foundational capabilities

LLMs from SAP, OpenAI, and other providers

Embedding and vector services

This layer enables natural-language understanding, reasoning, and contextual decision-making.

2. Orchestration Layer: SAP AI Core & AI Launchpad

This layer provides the runtime environment for complex agent workflows.
Key responsibilities:

Orchestrating multi-step tasks

Managing pipelines

Triggering events

Deploying custom AI services

Monitoring models and workloads

SAP AI Core is where the agent’s logic runs, while AI Launchpad controls lifecycle, governance, and monitoring.

3. Action Layer: SAP Build Apps & SAP Build Process Automation

Once an agent decides what to do, it needs to execute actions.
SAP’s agentic AI uses:

SAP Build Process Automation (BPA) → automate workflows

SAP Build Apps → create user interfaces and micro-apps

API Business Hub → trigger actions in SAP systems

This layer enables agents to perform tasks such as:

Creating purchase orders

Approving workflows

Extracting financial data

Updating master data

Executing custom processes

4. Knowledge Layer: Vector Databases & SAP Business Data

Agents need context and memory to make smart decisions.
SAP provides:

Vector stores for embeddings

Access to SAP business objects

Pre-trained business knowledge models

Document intelligence for unstructured data

This allows the agent to retrieve relevant information before taking action.

5. Interaction Layer: SAP Joule & Conversational Agents

This is where the user interacts with the agent.
Capabilities include:

Conversational prompting

Role-based copilots

Domain-specific AI assistants

Business insight generation

Joule acts as the AI interface, sending user intents to the underlying agentic framework.

6. Governance Layer: Security, Trust, and Monitoring

SAP ensures enterprise-grade governance with:

Responsible AI guardrails

Data privacy controls

Prompt and action monitoring

Access policies

Logging & audit trails

This allows AI agents to operate safely within enterprise boundaries.

In summary

SAP’s Agentic AI Framework combines LLMs + business data + automation + governance to create AI agents that can:

Understand natural language
Retrieve SAP data
Reason with business rules
Execute multi-step tasks
Integrate with SAP and non-SAP systems
Deliver safe, auditable outcomes

1. LangChain – Building Blocks for LLM Applications

LangChain is a framework designed to integrate LLMs with tools, data sources, and workflows.
It helps developers create basic to advanced AI applications using:

Prompt templates

Chains

Memory

Tools & APIs

Vector stores

Where it fits in SAP Gen AI?
LangChain is useful for:

Creating simple SAP chatbot workflows

Connecting SAP data sources (OData, CDS, HANA DB)

Building retrieval systems for SAP business processes

Rapid prototyping of Gen AI apps in SAP BTP

Analogy: LangChain = Lego blocks to assemble AI workflows.

2. LangGraph – Agentic AI Framework for Multi-Step Autonomous Workflows

LangGraph is built on LangChain but takes it one big step ahead.
It enables Agentic AI, meaning agents that can:

Plan

Think

Make decisions

Call tools automatically

Loop, retry, correct themselves

Handle complex SAP workflows end-to-end

Where it fits in SAP Gen AI?
LangGraph is used for:

Building SAP AI Agents that can execute business tasks

Multi-step SAP automation (Purchase Order creation, Ticket resolution, Sales workflows)

Autonomous agents that interact with SAP systems via APIs, BAPIs, RFC, OData

Creating SAP Copilot-like assistants

Analogy:
LangGraph = An intelligent manager that uses LangChain blocks to solve tasks autonomously.

In SAP Generative AI Context

Use LangChain when you want assistive AI.

Use LangGraph when you want autonomous SAP agents that behave like mini-digital employees.

Unlock the power of SAP Generative AI by learning how to set up everything from scratch—no prior experience required. This course guides you step-by-step through creating your SAP BTP Free Tier Account, configuring essential services, and enabling the latest SAP Generative AI Hub, AI Services, Build Apps, and Build Process Automation.

Whether you are a beginner exploring SAP BTP or a professional upgrading your skills for 2025 technologies, this setup module gives you a solid foundation to start building intelligent, enterprise-grade AI solutions.

What You’ll Learn

How to create and activate your SAP BTP Free Tier Account
Global Account, Subaccount & Entitlements setup
Enabling SAP Generative AI Hub & AI Foundation services
Setting up Destinations, Roles, and Trust Configuration
Activating SAP Build Apps and Build Process Automation
Connecting SAP systems for end-to-end Generative AI scenarios
Troubleshooting common setup and configuration issues

Why This Setup Module Is Important

Before you explore advanced AI models, automations, or agentic workflows, you must have a perfectly configured SAP BTP environment. This course eliminates confusion and ensures you follow the correct steps, with no errors or mismatches in entitlements or regions.

Who Is This For?

SAP Consultants & Developers

Beginners entering SAP BTP & Generative AI

Functional and Technical professionals (ABAP, UI5, MM, SD, FICO)

Students preparing for SAP BTP, AI, or automation jobs

The future of enterprise technology is intelligent, and SAP is leading this transformation with a powerful, unified AI Strategy designed for real business impact. This course gives you a deep, practical understanding of SAP Generative AI, SAP’s end-to-end AI framework, and how SAP is embedding AI across applications, business processes, and developer tools.

You will learn how SAP integrates Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), automation, agentic AI, and business context to deliver trusted, responsible, and enterprise-ready AI. From SAP BTP to S/4HANA, SuccessFactors, Ariba, and SAP Build, this course explains how every SAP solution is evolving with next-generation AI capabilities.

Key Highlights

SAP’s AI Vision & Strategy

Understand SAP’s unified AI roadmap for 2025 and beyond

How SAP embeds AI across ERP, HR, procurement, supply chain & CX

Role of SAP BTP as the foundation of enterprise AI

SAP Generative AI Foundation

Generative AI Hub architecture

Model access (LLMs, embeddings, fine-tuning options)

RAG, vector engines, and knowledge source integration

Business-Ready AI Innovation

SAP Joule – the digital copilot across SAP products

AI use cases across S/4HANA, SuccessFactors, Ariba, Sales Cloud & othersAI-driven automations and insights in real-time business processes

Responsible & Trusted AI

SAP’s approach to ethics, security, and governance

Ensuring transparency, accuracy & data protection in enterprise AI

AI for Developers & Consultants

Low-code/no-code AI with SAP Build

Agentic AI frameworks on SAP BTP

Building custom GenAI extensions for real enterprise scenarios

Who Is This Course For?

SAP Consultants (Functional + Technical)

Developers, Architects & Automation Engineers

Business Analysts, Decision Makers & IT leaders

Students preparing for next-gen SAP AI roles

Why This Course Matters

AI is no longer optional—it is the core capability that will define the next decade of enterprise systems. SAP’s AI Strategy ensures organizations can innovate rapidly while maintaining trust, security, and scalability. This course prepares you to be part of this transformation.

The Generative AI Hub is an enterprise-grade platform within SAP’s AI Foundation (part of SAP Business Technology Platform – SAP BTP) that provides centralized, secure access and management of generative AI capabilities — especially large language models (LLMs). It lets developers and organizations experiment with, orchestrate, control, and build AI-powered business solutions using cutting-edge generative models while maintaining governance, compliance, and data security. SAP+1

Core Purpose

Unified access to LLMs
It provides a single interface to access a broad range of generative AI models (from SAP-hosted models or third-party providers like Azure OpenAI, AWS Bedrock, open-source models, etc.), simplifying integration into enterprise apps and workflows. SAP Community+1

Enterprise AI development & orchestration
Allows experimentation, prompt engineering, model selection, and orchestration across multiple models — all from one platform. You can compare responses, refine prompts, and embed AI into custom or existing SAP business applications. SAP Community

Governance, compliance & responsible AI
Built-in governance controls (like prompt management, content filtering, and data protection) ensure secure, compliant use of generative AI for enterprise scenarios, safeguarding sensitive corporate data. SAP Learning

Key Capabilities

1. Flexible Model Access
   Choose and switch between multiple generative AI and foundation models from different providers to optimize performance and cost. SAP Learning

2. Prompt Engineering Tools
   Provides tools like a playground and prompt editor — useful for testing models and refining prompts before embedding them into business solutions. SAP Community

3. Scalable and Custom AI Solutions
   Combine AI models with your enterprise data and integrate them into SAP applications or custom extensions, enabling tailored, context-aware AI capabilities. SAP

4. Trusted Data Usage
   SAP ensures data privacy and control — customer data isn’t used to train external models, and extensive security policies protect all AI interaction

To set up SAP Generative AI with SAP AI Core on SAP BTP, first enable the required services in your BTP subaccount—SAP AI Core, AI Launchpad, and the Generative AI Hub (part of AI Foundation)—in a supported region such as US East or Europe. Create service instances for AI Core and AI Launchpad, then generate their service keys to obtain the client ID, secret, and URLs needed for secure access. Configure a destination in BTP (Connectivity → Destinations) linking AI Launchpad to AI Core using OAuth2 credentials.

Next, create an instance of the Generative AI Hub to access SAP and third-party LLMs, and use its Playground for prompt testing.

Once everything is connected, you can call generative models through AI Core APIs or integrate them into SAP CAPM, UI5, Build Apps, or automation workflows to build intelligent, enterprise-ready AI applications.

To set up SAP Generative AI with SAP AI Launchpad, first enable both “AI Launchpad” and “AI Core” services in your SAP BTP subaccount. Create a service instance for AI Launchpad and generate a service key. Then create a service instance for SAP AI Core and generate its service key as well.

Next, go to BTP Cockpit → Connectivity → Destinations and create a new destination that connects AI Launchpad to AI Core using the AI Core URL, OAuth2 Client Credentials, Client ID, and Secret from the AI Core service key. Once the destination is saved and the connection succeeds, launch the AI Launchpad subscription from the BTP Cockpit.

Inside Launchpad, you will now be able to manage models, monitor workloads, configure deployments, and connect to the Generative AI Hub to orchestrate LLM-based solutions across your SAP landscape.

SAP Generative AI — Deploy Foundation Model

SAP Generative AI — Deploy Foundation Model refers to the capability within SAP’s AI Foundation and Generative AI Hub that enables enterprises to deploy, manage, and run advanced pre-trained foundation models (such as large language models from OpenAI, Anthropic, Google, AWS, or other providers) on SAP’s enterprise-ready infrastructure.

At its core, this feature lets businesses select a foundation model from the SAP Generative AI Hub’s model library and deploy it as a reusable service endpoint that can be integrated into applications, workflows, and business processes. Once deployed, these models are accessible via a secure API and can perform generative AI tasks such as text generation, summarization, translation, content creation, and more. SAP Community

Key highlights:

Enterprise-grade deployment: Models are hosted and managed through SAP AI Core and Generative AI Hub, ensuring scalability, governance, and security suitable for business use. SAP

Unified access: A deployment becomes an accessible endpoint that applications can call for inference (e.g., generate responses to prompts or perform AI tasks) using SAP tools and SDKs. sap.github.io

Model flexibility: You can deploy a variety of foundation models, including those from major cloud providers or open-source alternatives, with version control and configuration options. SAP Help Portal

Integration support: Deployed models can be consumed via SAP SDKs (e.g., @sap-ai-sdk/foundation-models) and through SAP AI Launchpad workflows. sap.github.io

Business extension: Once deployed, these models can be embedded in SAP applications or custom business solutions to automate processes, generate insights, and enhance user experiences.

Why These Matter (Real-Time Impact)

Faster decisions: AI summaries and insights reduce time spent sifting through data. PIKON SAP Consulting International
Operational efficiency: Automation of routine tasks (documents, responses, routing) frees up employees for higher-value work. AST Consulting
Enhanced user experience: Real-time prompts and suggestions improve both internal and customer interactions. SAP
Lower costs: AI co-pilots reduce manual work and errors, improving quality and throughput.

SAP Generative AI Development Tools Overview

SAP’s suite of generative AI development tools enables developers to build, extend, and automate business applications with generative AI that is trusted, context-aware, and enterprise-ready. These tools are typically part of SAP Business AI on the SAP Business Technology Platform (BTP) and can be used within SAP Build environments such as SAP Build Code and related studios. SAP+1

Core Capabilities for AI Development

1. SAP Build Code with Generative AI

This is SAP’s modern development environment where generative AI is embedded directly into the development workflow.

Developers use natural language prompts to generate data models, services, business logic, and sample data.

The tool enhances productivity by simplifying and accelerating creation of full-stack business applications.

AI assistance supports code generation across languages (like Java, JavaScript/TypeScript via CAP) and ABAP in cloud contexts. SAP

2. Joule for Developers

Joule is SAP’s generative AI copilot for developers that:

Suggests and completes code, explains existing code, and generates tests.

Offers chat-based interaction to answer technical questions directly in the IDE.

Reduces development effort and speeds up iteration cycles within SAP Build and ABAP environments. SAP Help Portal

3. Joule Studio (Agent & Skill Builder)

A low-code/no-code environment (inside SAP Build) that lets developers and business users create custom AI agents and Joule skills to automate workflows or complex business tasks:

Drag-and-drop capability for planning, reasoning, and workflow automation.

Combines SAP context and business logic with AI reasoning, enabling solutions that operate autonomously across systems. SAP News Center

4. Generative AI Hub (part of SAP AI Core / AI Foundation)

The central AI lifecycle platform for enterprise-scale development:

Provides secure, governed access to multiple foundational and generative AI models (SAP’s own and third-party).

Offers prompt management, prompt optimization, orchestration tools, and model exploration.

Ensures compliance, security, and efficient scaling of AI solutions in business environments. SAP+1

Developer Experience & Integration

Together, these elements enable:

AI-assisted application development — write less boilerplate and more business logic. SAP
Context-aware generation — models grounded in SAP business data produce more accurate outputs. SAP Learning
Multi-language support — ABAP, Java/JavaScript, CAP, and low-code development all benefit. SAP
Enterprise-grade governance — security, compliance, and scaling built into SAP’s AI development stack. SAP

In Summary

SAP’s generative AI development environment (often used through SAP Build Code and associated modules) provides a unified, AI-powered development platform that accelerates building and extending SAP business applications with:

Natural language prompts for code and logic generation.

Integrated AI copilots (Joule) and agent builders (Joule Studio).

Centralized model access and governance via generative AI hub.

Support for both low-code and pro-code scenarios.

This makes it easier for developers — from technical ABAP experts to citizen developers — to innovate rapidly while maintaining enterprise-grade reliability and security.

Common Challenges in SAP Generative AI using LLMs

Data Privacy & Security

SAP systems handle sensitive business data (finance, HR, procurement).

Sending data to LLMs can risk data leakage if not properly governed.

Strict compliance requirements (GDPR, SOC, ISO) must be met.

SAP response: Secure model access via SAP AI Core & Generative AI Hub, data masking, and enterprise-grade governance.

Hallucinations & Incorrect Outputs

LLMs may generate confident but incorrect answers.

This is risky for:

Financial postings

Business decisions

Automated workflows

Mitigation:
Use grounding with SAP business context, retrieval-augmented generation (RAG), and validation layers.

Lack of Business Context

Generic LLMs don’t understand:

SAP data models

Business rules

Industry-specific processes

Solution:
Context injection via SAP data sources, CDS views, APIs, and SAP-specific prompts.

Integration Complexity

Integrating LLMs with:

SAP S/4HANA

BTP services

ABAP & CAP applications
can be complex.

SAP approach:
Pre-built integrations through SAP BTP, Joule, and AI APIs.

Prompt Engineering Challenges

Poor prompts lead to:

Unreliable responses

Inconsistent results

Maintaining prompts at scale is difficult.

Mitigation:
Prompt versioning, testing, and optimization using Generative AI Hub.

Cost Management

LLM usage can be expensive due to:

Token consumption

High-frequency API calls

Hard to predict costs in production.

Control:
Usage monitoring, throttling, and cost governance via SAP AI services.

Latency & Performance

Real-time SAP applications need low latency.

LLM responses can be slow, especially with complex prompts.

Solution:
Caching responses, async processing, and optimized model selection.

Model Selection & Dependency

Choosing the right model (SAP / third-party) is challenging.

Risk of vendor lock-in.

SAP advantage:
Multi-model access through Generative AI Hub (open & flexible).

Explainability & Trust

Business users need to understand:

Why a result was generated

How decisions were derived

LLMs are often black boxes.

Mitigation:
Human-in-the-loop workflows and transparent AI design.

Change Management & Skill Gap

Developers & business users need new skills:

Prompt engineering

AI ethics

AI operations (AIOps)

Need:
Training, enablement, and structured AI adoption strategies.

Summary (Quick Slide-Ready Points)

Data security & compliance

Hallucinations & accuracy risks

Lack of SAP business context

Integration complexity

Prompt engineering challenges

Cost & performance management

Explainability & user trust

In SAP Generative AI context, RAG stands for Retrieval-Augmented Generation. It’s a powerful approach for improving AI responses, especially in enterprise scenarios like SAP. Here’s a clear breakdown:

RAG (Retrieval-Augmented Generation) in Generative AI

1. Concept:

   Traditional generative AI models (like GPT) generate answers purely from their trained knowledge.

   RAG enhances this by first retrieving relevant information from external sources (databases, documents, SAP systems) and then generating the response using that context.

2. How it works in SAP:

   Example: You ask the AI, “Show me last month’s purchase orders over $10,000.”

   Step 1 – Retrieval: The system searches SAP data (like tables, reports, documents) to find relevant records.

   Step 2 – Generation: The AI uses the retrieved data to generate a coherent, accurate answer in natural language.

3. Benefits:

   Accuracy: Answers are based on real-time enterprise data.

   Context-aware: Can include company-specific knowledge not in the base AI model.

   Efficiency: Reduces errors from AI hallucinations.

4. Use cases in SAP:

   Automating report summaries from SAP modules (Finance, Procurement, HR).

   Chatbots that answer employee or customer queries using SAP data.

   Generating business insights from historical SAP datasets.

Here’s a real-time SAP Generative AI RAG use case demo concept for ATS Bot that you can use for presentation or social media posts:

Demo: ATS Bot with SAP Generative AI + RAG

Scenario:
An employee wants to know the status of their expense claims in SAP.

Step 1 – Query by User:

“ATS Bot, show me all my pending expense claims over $500 for December 2025.”

Step 2 – RAG in Action:

1. Retrieval: ATS Bot searches the SAP system’s Finance & Expense module for relevant data (user ID, amount > $500, date filter).

2. Augmentation: It collects all matching entries, including claim status, approver, and date.

3. Generation: The bot generates a natural language response using the retrieved data.

Step 3 – AI Response:

“You have 3 pending expense claims over $500 for December 2025.
Claim #4578 – $750 – Pending with Manager Approval
Claim #4591 – $620 – Pending with Finance
Claim #4603 – $510 – Pending with Manager Approval”

Benefits:

Real-time access to enterprise data.

Personalized responses for each employee.

Reduces manual search in SAP.

Demonstrates RAG-powered Generative AI in action.

SAP CAP LLM Plugin — Overview

The CAP LLM Plugin is a developer extension for SAP Cloud Application Programming Model (CAP) that helps you integrate generative AI (large language models) into your CAP-based applications. It lets you connect your CAP services with LLMs and vector search capabilities while protecting sensitive data and simplifying AI workflows.

What It Does

1. AI Integration for CAP Apps

Provides APIs and tools to call large language models (LLMs) for tasks like chat completions, summarization, query responses, etc.

It connects to LLMs via the SAP Generative AI Hub and SAP AI Core, which host and manage the models you use.

2. Data Anonymization

Ensures confidential or sensitive business data isn’t exposed directly to the AI models.

You can annotate entities/fields in your CAP data model (e.g., with @anonymize) and the plugin automatically masks/minimizes sensitive values before they’re sent to the LLM, then reinserts original values into results.

3. Embeddings & Vector Search

Generates vector embeddings (numeric representations of text) via SAP AI Core for content like documents, policies, messages, etc.

Stores these in SAP HANA Cloud Vector Engine and performs similarity searches to find relevant content—key for RAG (Retrieval-Augmented Generation) scenarios.

4. RAG (Retrieval-Augmented Generation) Support

Simplifies RAG by automating the cycle: generate embeddings → run similarity search → append top matches → ask the LLM.

This improves answers from the LLM by grounding them in your own business content.

Developer Benefits

Rapid AI-enabled development
Integrate LLM services into CAP apps without building all AI logic from scratch.

Seamless access to AI models
Work with models curated in SAP Generative AI Hub, using standardized methods for embedding and chat.

Enterprise-grade data privacy
Built-in anonymization ensures data protection and compliance when calling AI models.

Focus on business logic
Abstracts AI plumbing so you can focus on business value instead of low-level AI integration.

How It Fits in the SAP AI Ecosystem

The plugin acts as a bridge between:

CAP applications (your business logic),

SAP Generative AI Hub / SAP AI Core (where LLMs and embeddings run), and

SAP HANA Cloud & Vector Engine (where data is stored and searched).

This setup lets you build context-aware AI features in SAP apps — like intelligent chat assistants, policy answer bots, automated summaries, and compliance helpers — all while managing corporate data responsibly and efficiently.

Quick Summary (Key Features)

FeatureWhat It ProvidesGenerative AI AccessChat completions, generative responses from LLMsEmbeddingsVector representations of text for semantic searchSimilarity SearchFinds most relevant info using HANA Cloud Vector EngineRAG SupportCombines embeddings + similarity search + LLMsData PrivacyAnonymizes sensitive data automatically

This architecture allows CAP applications to consume large language models in a governed, scalable, and compliant way, while protecting business data.

Architecture Layers (Explained Clearly)

CAP Application Layer

Built using SAP Cloud Application Programming Model (CAP) in Node.js or Java

Contains:

CDS data models

Services & business logic

APIs consumed by Fiori / SAP Build Apps

This is where GenAI use cases are triggered (chat, Q&A, summarization, insights).

CAP acts as the business orchestration layer.

CAP LLM Plugin (Key Enabler)

A framework extension that connects CAP apps with GenAI services

Responsibilities:

Calling LLMs (chat/completion)

Generating embeddings

Performing vector similarity search

Supporting Retrieval Augmented Generation (RAG)

Automatic data anonymization

It removes the complexity of AI integration for CAP developers.

SAP Generative AI Hub

Central gateway for accessing enterprise-approved LLMs

Provides:

Model abstraction (OpenAI, Azure OpenAI, SAP models, etc.)

Governance and compliance

No customer data used for model training

Ensures responsible and controlled AI usage.

SAP AI Core

Execution and runtime layer for AI workloads

Handles:

Prompt execution

Embedding generation

Scaling, monitoring, and lifecycle management

Fully managed on SAP BTP

AI Core runs the GenAI logic securely and at scale.

SAP HANA Cloud + Vector Engine

Stores:

Business data

Documents and knowledge content

Vector embeddings

Enables:

Semantic search

Context retrieval for RAG scenarios

Provides grounding data to improve LLM accuracy.

Retrieval Augmented Generation (RAG) – Interview Favorite

RAG Flow Explanation

1. User asks a question in the CAP application

2. CAP LLM Plugin converts the query into embeddings

3. HANA Vector Engine finds relevant documents

4. Retrieved context is added to the prompt

5. LLM generates a context-aware, accurate response

RAG minimizes hallucinations and improves trust.

Security & Data Privacy

CDS annotations for data anonymization

Role-based access using BTP XSUAA

Secure API communication

No direct exposure of sensitive data to LLMs

This architecture is designed for enterprise compliance.

End-to-End Flow (Short & Crisp)

User → CAP App → CAP LLM Plugin → Generative AI Hub → AI Core → LLM
Context retrieved from HANA Cloud Vector Engine

Typical Business Use Cases

Intelligent SAP copilots

Policy & document Q&A bots

Automated summaries and insights

Context-aware recommendations

Customer and employee assistants

Why SAP Uses This Architecture (Key Benefits)

Enterprise-grade security

Governance and compliance

Scalable GenAI on BTP

Clean separation of business logic and AI logic

Faster GenAI adoption for SAP developers

SAP Generative AI – Real-Time CAP Application with RAG

What Is It? (One-Line)

A real-time SAP CAP application with GenAI and RAG allows users to ask natural-language questions and receive accurate, business-grounded answers by combining LLMs with enterprise data stored in SAP HANA Cloud.

Architecture Components (Real-Time RAG)

CAP Application (Node.js / Java)

Handles:

User requests (chat, Q&A, search)

Business rules

API orchestration

Exposed via:

SAP Fiori UI

SAP Build Apps

External REST APIs

Acts as the entry point for real-time queries.

CAP LLM Plugin

Core GenAI integration layer for CAP

Provides:

Chat completions

Embedding generation

Vector similarity search

Prompt orchestration

Data anonymization

Eliminates direct interaction with AI APIs.

SAP HANA Cloud + Vector Engine

Stores:

Business documents (policies, SOPs, manuals)

Real-time transactional insights

Vector embeddings

Performs:

Semantic similarity search

Supplies context for RAG in milliseconds.

SAP Generative AI Hub

Secure access layer for LLMs

Features:

Model abstraction

Governance & compliance

Enterprise-safe AI usage

Ensures data is not used to train models.

SAP AI Core

Executes AI workloads

Handles:

Prompt execution

Embedding creation

Runtime scaling Ensures real-time performance.

Real-Time RAG Flow (Step-by-Step)

Live Execution Flow

1. User submits a query in real time

2. CAP app sends the request to CAP LLM Plugin

3. Query is converted into vector embeddings

4. HANA Vector Engine retrieves relevant content

5. Retrieved context is appended to the prompt

6. LLM generates a grounded response

7. Response is returned to the user instantly

Entire flow happens in real time.

Why RAG Is Critical in SAP

Prevents hallucinations

Uses live enterprise data

Improves accuracy and trust

Keeps business data private

Security & Compliance

CDS-based anonymization

Role-based access (XSUAA)Secure BTP communication

Enterprise data never leaves SAP control

Real-World Use Cases

SAP Copilot for business users

Finance & HR policy Q&A

Supply chain insights

Customer service knowledge bots

Developer documentation assistants

Example Real-Time Scenario

A finance user asks:
“What is the approval limit for urgent purchases?”

The CAP app retrieves the policy from HANA, adds it as context, and the LLM generates a precise, policy-compliant answer.

Benefits of CAP + GenAI RAG

Real-time responses

Context-aware answers

Enterprise-grade security

Scalable on SAP BTP

Faster decision-making

Interview-Ready Summary

A real-time SAP CAP GenAI RAG system combines CAP, the CAP LLM Plugin, SAP AI Core, Generative AI Hub, and HANA Cloud Vector Engine to deliver accurate, secure, and context-aware AI responses grounded in enterprise data.

SAP Generative AI Orchestration Service in SAP Generative AI Hub (Gen AI Hub) is a service that helps developers design, manage, and execute workflows for generative AI applications by combining multiple AI models, prompts, and processing steps into a single pipeline.

Description

The Generative AI Orchestration Service in the SAP Generative AI Hub provides a unified framework to orchestrate multiple generative AI models and processing modules such as prompt templates, filtering, grounding, and data masking. It runs on SAP AI Core and allows developers to build enterprise AI solutions using a consistent API while switching between different foundation models without changing the application code.

This service enables organizations to integrate LLMs into business applications by creating pipelines that process input prompts, call AI models, and apply additional steps such as security, compliance, and data enrichment before delivering the final response.

Key Features

1. Unified Model Access

   Access multiple foundation models (SAP or partner models) using a single API.

2. Prompt Templating

   Create reusable prompt templates with dynamic placeholders for user inputs.

3. Content Filtering

   Filter harmful or restricted content from prompts and responses.

4. Data Masking

   Protect sensitive information by anonymizing or pseudonymizing data before sending it to AI models.

5. Grounding with Business Data

   Combine AI models with external or enterprise data sources to generate more accurate responses.

6. Pipeline-Based Processing

   Build AI workflows where multiple steps (templating → LLM → filtering → output transformation) run sequentially.

7. Multimodal Support

   Supports text and image inputs for advanced AI use cases like visual question answering.

Example Use Cases

AI chat assistants for SAP applications

Automated document summarization

Business report generation

Customer support automation

AI agents integrated with SAP S/4HANA data

Simple Definition (For Interviews)

SAP Generative AI Orchestration Service is a service in the SAP Generative AI Hub that allows developers to create AI pipelines by combining prompts, LLM models, and processing modules such as filtering, data masking, and grounding to build secure and scalable enterprise AI applications.

SAP Generative AI Data Masking in Gen AI Hub

Data Masking in SAP Generative AI Hub (Gen AI Hub) is a security feature in the Orchestration Service that protects sensitive data before sending it to Large Language Models (LLMs). It automatically detects and hides confidential information such as names, emails, phone numbers, or IDs in prompts and data used by AI models.

Definition

SAP Generative AI Data Masking is a capability in the Gen AI Hub orchestration pipeline that anonymizes or pseudonymizes sensitive data before it is processed by generative AI models, ensuring privacy and compliance when using enterprise data.

Why Data Masking is Needed

When SAP applications send prompts to AI models, they may contain sensitive business or personal data (PII).
Data masking ensures that:

Sensitive data is not exposed to external AI models

Enterprise security and compliance policies are maintained

AI models receive safe and sanitized data for processing

For example:

Original Prompt

Summarize the customer complaint from John Smith, email john@gmail.com

After Data Masking

Summarize the customer complaint from MASKED_PERSON, email MASKED_EMAIL

The LLM only sees the masked values, protecting real data.

Types of Data Masking in SAP Gen AI Hub

1. Anonymization

Sensitive data is replaced permanently with placeholders.

Original data cannot be recovered.

John Smith → MASKED_PERSON

2. Pseudonymization

Data is replaced with temporary placeholders.

The system can restore the original data in the final response.

John Smith → PERSON_123

This is useful when AI responses must include the original data later.

What Data Can Be Masked

The masking module can detect and hide different types of PII entities, such as:

Person names

Email addresses

Phone numbers

Organization names

IDs or account numbers

Addresses

These are identified automatically by SAP Data Privacy Integration service.

Where Data Masking Fits in the Gen AI Pipeline

Typical Gen AI Hub Orchestration Flow:

1. Prompt Templating – Create structured prompts

2. Data Masking – Hide sensitive data

3. Content Filtering – Remove unsafe content

4. LLM Processing – Send prompt to AI model

5. Response Handling – Return safe output

This pipeline ensures secure enterprise AI usage in SAP systems.

SAP Generative AI Document Grounding in Gen AI Hub

Document Grounding in SAP Generative AI Hub is a capability that allows generative AI models to use external documents or enterprise data as context while generating responses. This helps the AI provide accurate, relevant, and business-specific answers instead of relying only on its general training knowledge.

Definition

Document Grounding is a process in the SAP Gen AI Hub Orchestration Service where the AI model retrieves information from uploaded documents or enterprise knowledge sources and uses that content to generate reliable responses.

In simple words:
AI answers questions based on your business documents.

Why Document Grounding is Important

LLMs normally generate answers from their pre-trained knowledge, which may be:

Outdated

Generic

Sometimes inaccurate (hallucinations)

Document grounding solves this by allowing the AI to reference real company documents before generating the response.

Benefits include:

More accurate responses

Context-aware answers

Reduced AI hallucinations

Secure use of enterprise knowledge

How Document Grounding Works

Typical Gen AI Hub Grounding Flow:

User asks a question
System searches relevant documents (PDF, text, knowledge base)
Relevant content is extracted
Content is added to the prompt as context
LLM generates a response based on that document

Example:

User Query

What is the leave policy for employees?

Grounded Context

Company HR policy document: Employees are entitled to 20 annual leave days per year.

AI Response

According to the company HR policy, employees receive 20 annual leave days per year.

Technologies Used in Document Grounding

In SAP Gen AI Hub, grounding may use:

Vector embeddings

Similarity search

Document retrieval systems

Prompt augmentation

This approach is commonly called Retrieval Augmented Generation (RAG).

Use Cases in SAP

Document grounding can be used in many SAP business scenarios:

HR Policy Chatbots

Financial report analysis

Supply chain documentation queries

Knowledge management assistants

SAP support documentation search

Example:
An employee can ask an AI assistant:

“What is our travel reimbursement policy?”

The AI will search company policy documents and answer based on them.

Document Grounding in Orchestration Pipeline

Typical Gen AI Hub Pipeline

1. Prompt Template

2. Document Grounding (Context Retrieval)

3. Data Masking

4. Content Filtering

5. LLM Model Execution

6. Response Generation