Retrieval, embeddings, and vector search for teams that need grounded answers instead of noise

RAG: why the model alone is not enough in enterprise contexts

A language model, however powerful, has two structural limitations that make it unusable in many business contexts: its knowledge stops at the training cutoff date, and it knows nothing about your data.

RAG, Retrieval Augmented Generation, solves both problems.

Instead of relying solely on the model's memory, the system retrieves the most relevant documents at query time and provides them as context.

The model responds based on this specific information, not on statistical generalizations.

The result is a response that can cite sources, that stays current without retraining anything, and that does not invent data because it has it right there.

This is why RAG has become the reference pattern for enterprise applications that need to answer on technical documentation, internal procedures, regulations, or corporate knowledge bases.

But RAG is not a plugin you install: it is an architecture that must be designed.

The quality of retrieval determines the quality of the response far more than the model chosen.

And retrieval depends on how documents were indexed, chunked, and how well the search system understands the user's query.

Chunking, embedding, and retrieval: the choices that determine quality

The three fundamental steps of a RAG pipeline are document chunking, embedding generation, and retrieval.

Each introduces trade-offs that directly impact final quality.

Chunking: how a document is split determines what enters the context. Chunks that are too short lose sentence context; chunks that are too long consume tokens and dilute relevance. Semantic chunking, which respects paragraph or section boundaries, produces better results than fixed-window chunking but requires more preprocessing work.

Embedding: the choice of embedding model affects the quality of semantic search. OpenAI models like text-embedding-3-small and text-embedding-3-large are good defaults for Italian and English; for highly specialized domains it may be worth evaluating models fine-tuned on your own corpus.

Retrieval: pure vector search finds chunks semantically similar to the query, but can miss documents relevant on specific keywords. Hybrid search, combining vector search and full-text BM25, produces more robust results and is used by default in Azure AI Search. Reranking adds a further classification step to improve precision.

RAG with Semantic Kernel in .NET: the practical architecture

Semantic Kernel in .NET provides the abstractions needed to build a RAG pipeline without coupling the code to a specific provider.

The design allows switching from Qdrant to Azure AI Search by changing configuration, not application code.

The typical pipeline is structured in four phases: indexing (preprocessing + chunking + embedding + storage in the vector store), retrieval (query embedding + similarity search + optional filters), augmentation (injection of relevant chunks into the prompt), and generation (model call with the enriched prompt).

In .NET the critical point is the async management of the entire flow: calls to embeddings, the vector store, and the model are all I/O-bound operations that benefit from correct async/await and, where possible, parallelization.

A common mistake is waiting sequentially for operations that could run in parallel, multiplying latency.

The other critical point is testability: Semantic Kernel's interfaces allow replacing real components with mocks for tests, but this requires a conscious design of the composition root to avoid non-injectable dependencies.

When RAG is not enough and which alternatives to consider

RAG is powerful but not universal.

There are scenarios where the pattern is not the right answer.

When questions require reasoning on structured data, an LLM that generates SQL or calls tools that query the database directly produces better results.

RAG on a textual dump of structured data is a solution that works in prototypes and degrades in production.

When the document base is very large and heterogeneous, retrieval quality tends to degrade because relevant documents get lost in the mass.

In these cases Graph RAG, which builds a graph of relationships between concepts instead of indexing flat text, can significantly improve recall on complex queries.

When latency is a tight constraint, a RAG pipeline with retrieval, reranking, and generation may be too slow for certain use contexts.

The options then are pre-caching responses on frequent queries, reducing retrieval steps, or using faster models even at the cost of quality.

The choice of AI architecture is never final: it must be reassessed as you understand real usage patterns and the limits that emerge in production.