Blog 108: Production grade RAG architecture, for Indian Banks, using Google Cloud Platform

Traditional Large Language Models (LLMs without RAG) may hallucinate.

Retrieval-Augmented Generation (RAG) makes AI answers (from LLMs) more accurate, fixes false information, and connects language models to private or fresh data.

We outline a high-level overview of a Production grade RAG architecture, for Indian banks, using Google Cloud Platform (GCP). Note: GCP services are in bold.

Step 1: Data ingestion & PII scrubbing

1. Set up a hybrid (on-premises and public cloud) environment pipeline (Cloud Interconnect or IPsec VPN) from the bank’s on-premises Core Banking System (CBS) to Google Cloud Platform (GCP).

2. Ingest raw PDFs (documents you want the AI system to refer to) into a Google Cloud Storage Bucket. Have Customer Managed Encryption Keys (CMEK) via Cloud Key Management Service (KMS) enabled.

   1. Ensure all financial data and customer PII reside within Indian geography.

3. Pass all text in the PDFs through the Google Cloud Data Loss Prevention (DLP) to sanitize or tokenise sensitive attributes.

   1. Sensitive attributes include Aadhaar numbers (12 digits), PAN cards (10 alphanumeric), Indian mobile numbers, credit card numbers, UPI IDs.

   2. Cloud DLP is both a managed service and an API that you interact with. Google rebranded the broader service as Sensitive Data Protection.

   3. Cloud DLP has built in detectors to find most of the above attributes out of the box. However, it does not have a built-in detector for UPI IDs, which means we will need to create a custom regex pattern for UPI ID.

Important Note: Never let unmasked raw PII hit the vector database.

Important Note: Consider using Pseudonymization (Tokenisation) rather than outright deletion. This replaces names with dummy tokens (like User_1), allowing the search engine to retain the structural context of the question.

Step 2: Advanced document processing & layout-aware chunking

1. Use Google Cloud Document AI Layout Parser to extract document hierarchy: headers, paragraphs, and tables as structural objects.

2. Table processing strategy: Financial tables are converted into explicit Markdown format or flat JSON arrays before chunking.

3. Metadata headers (e.g., document name, section header etc.) are added to every chunk so context isn’t lost when retrieved independently.

4. Chunking strategy: Google executes the layout-aware chunking. However, you control the rules by passing settings to Google. Understand parent-child chunking concepts.

Important Note: Avoid using the standard fixed character (e.g., 500 character) chunking; as it breaks financial tables.

Step 3: Dual-indexing (hybrid search architecture)

1. Use AlloyDB AI (an advanced extension of AlloyDB for PostgreSQL, engineered to integrate high-performance vector processing directly within the enterprise-grade relational database).

   1. The platform supports automated embedding generation. Operational data (text, images, and video) is converted into vector embeddings using Agent Platform models, such as text-embedding-005.

   2. Build a hybrid search architecture to handle both broad semantic questions (‘what are the home loan options?’) and specific lexical lookup (‘what is the penalty rate under clause 4.2.a?’)

Important Note: AlloyDB AI uses the pgvector extension to add vector AI capabilities directly into a standard relational database. It supports the popular HNSW (Hierarchical Navigable Small Worlds) index, as well as Scalable Nearest Neighbours (ScaNN) index. ScaNN: State-of-the-art vector search (semantic search) algorithm by Google.

Step 4: Contextual retrieval & re-ranking

When a user asks a question:

1. Query rewriting: Use an LLM (e.g., Gemini 3.5 Flash) to rewrite colloquial queries or Hinglish phrases into clean formal queries.

2. Execute hybrid search simultaneously: Run vector similarity search (semantic search) AND exact keyword matching (specific lexical lookup) simultaneously, in AlloyDB AI.

   1. Top 20 results merged via Reciprocal Rank Fusion (RRF).

3. Cross-Encoder re-ranking:

   1. Cross-Encoder re-ranking is typically done in your application backend (e.g., Python using LangChain or LlamaIndex) rather than directly inside AlloyDB AI.

      1. Cross-Encoder is an AI search tool that reads a search query and a text document at the same time to score how well they match.

   2. Get a definitive top 3 to 5 context blocks

   3. Two ways to implement the re-ranking step:

      1. Using a dedicated re-ranker model (GCP recommends this)

      2. Using a Gemini Flash prompt (LLM-as-judge)

Step 5: Ensure grounded generation with hallucination guardrails

1. Add a System Prompt: Enforce strict adherence with provided PDFs.

   1. "You are a compliance assistant for an Indian bank. Answer the question ONLY using the provided context chunks. If the answer is not contained within the context, explicitly state 'Information unavailable in authoritative documents.' For every claim, provide inline references using [Doc_ID, Page_Number]."

2. Validation Layer: Pass the LLM response to a lightweight post-processing check to ensure that facts (such as interest rates, dates, numbers) in the response exist word-for-word in the retrieved source text.

Important Note: Have Security Audit & Logging to ensure that every single answer generated by Gemini can be tracked, verified, and audited by compliance teams.

You can customise this production grade RAG architecture (using Google Cloud Platform) based on your requirements.

Jump to blog 100 to refer to the overall product management mind map.

I wish you the best for your journey. 😊