Part 4: Spring AI Building a RAG Pipeline

A Spring AI RAG pipeline lets an LLM answer using your documents instead of just its training data. RAG — retrieval-augmented generation — is three moves: take the user’s question, find the most relevant chunks from your data (using embeddings and a vector store), and send those chunks plus the question to the model. The model answers grounded in what you gave it. Spring AI’s ChatClient and document/vector abstractions make this surprisingly little code. This is Part 4 of the Spring AI series and it ties the previous parts together.

Why RAG instead of fine-tuning?

Before writing code, know why you’re here. RAG keeps the model fixed and injects fresh context at query time, so you update knowledge by re-indexing documents — no retraining. That makes it ideal for changing internal docs, FAQs, and product data. (For the full trade-off, see To RAG or to Fine-Tune?.) Fine-tuning changes behavior and style; RAG changes what the model knows right now. Most production apps start with RAG.

1. Assemble the pipeline

You need three pieces:

• A vector store preloaded with embedded documents (from Part 3 on embeddings and vector stores).
• A ChatClient to call the model (from Part 2 on chat completions).
• A prompt with a placeholder for the retrieved context.

The flow at runtime is always the same shape: embed the query → similarity search → stuff the top chunks into the prompt → generate.

2. Retrieve and prompt

@Service
public class RagService {

    private final VectorStore vectorStore;
    private final ChatClient chatClient;

    public RagService(VectorStore vectorStore, ChatClient.Builder chatBuilder) {
        this.vectorStore = vectorStore;
        this.chatClient = chatBuilder.build();
    }

    public String ask(String question) {
        var similar = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(5)
        );
        String context = similar.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n\n"));

        return chatClient.prompt()
            .user(u -> u.text("""
                Answer based only on the following context. If the answer is not in the context, say so.

                Context:
                {context}

                Question: {question}
                """)
                .param("context", context)
                .param("question", question))
            .call()
            .content();
    }
}

Version heads-up [needs source]: the SearchRequest API has shifted across Spring AI releases (the fluent SearchRequest.query(...).withTopK(...) vs. a SearchRequest.builder()...build() style). Match whatever your dependency version exposes. Spring AI also offers a built-in QuestionAnswerAdvisor that does retrieve-and-augment for you — handy once you outgrow the manual version above.

3. What you get

The user asks a question; you search the vector store, concatenate the top chunks into context, and pass both into the prompt. The reply is grounded in your documents instead of the model’s general training. From here you can tune topK, add metadata filters (e.g. only search a given tenant or document set), or move the prompt into one of Spring AI’s resource-based templates for consistency across services.

The instruction “answer based only on the context, and say so if it’s not there” matters more than it looks — it’s your main lever against hallucination. Without it, the model happily fills gaps from training data.

4. Preloading the store (ingestion)

A pipeline is only as good as what’s in the store. Ingestion is a one-time (or scheduled) job:

1. Read files with a document reader (PDF, text, Markdown, etc.).
2. Split them into chunks with a text splitter — chunks that are too big dilute relevance and blow your context budget; too small lose meaning. A few hundred tokens with slight overlap is a sane starting point.
3. Embed and store — vectorStore.add(documents) uses the configured EmbeddingModel to vectorize and persist each chunk.

Run that once at startup or via an admin job, and every question afterward flows through the clean retrieve → augment → generate sequence.

Common pitfalls

• Bad chunking. This is the single biggest quality lever. If answers feel vague, fix chunk size and overlap before touching the model.
• Retrieving too much. A huge topK floods the prompt, costs more tokens, and can lower answer quality. Start small (3–5) and measure.
• No grounding instruction. Always tell the model to answer only from context — otherwise RAG and hallucination coexist.
• Embedding/model mismatch on re-index. If you change the embedding model, you must re-embed everything; old and new vectors aren’t comparable.
• Ignoring metadata. Storing source/tenant/date as metadata lets you filter and cite — skip it and you can’t tell users where an answer came from.

FAQ

Key takeaway: A Spring AI RAG pipeline is retrieve → augment → generate over three beans — EmbeddingModel, VectorStore, ChatClient. Quality lives in ingestion (chunking) and a strict grounding instruction, not in clever model settings.

This wraps the Spring AI series: Intro → Chat Completions → Embeddings & Vector Stores → RAG.