How to Build a Simple RAG Pipeline with Open-Source Tools
LangChain vs. LlamaIndex: A Head-to-Head Comparison
Introduction
Retrieval-Augmented Generation (RAG) is the most practical way to make large language models useful for real-world tasks. Instead of relying solely on a model's training data—which goes stale the moment it's published—RAG pulls relevant information from your own documents at query time. The model reads those retrieved chunks and generates an answer grounded in what it actually found. This simple idea solves two problems at once: it keeps answers current, and it dramatically reduces the hallucinations that plague pure LLM generation.
The architecture was introduced by Facebook AI Research in 2020, when Lewis et al. showed that RAG models outperformed parametric-only models on open-domain QA tasks, achieving state-of-the-art results on Natural Questions and TriviaQA. Since then, the approach has exploded in popularity. A 2023 survey by Gao et al. counted just 7 RAG-related papers in 2020—by 2023, that number exceeded 100.
The open-source ecosystem has kept pace. You no longer need enterprise software or expensive APIs to build a working RAG pipeline. Two frameworks dominate the space: LangChain and LlamaIndex. Both are free, both are actively maintained, and both can run entirely on your laptop. However, they take different philosophical approaches, and the choice between them shapes how you'll build, debug, and extend your system.
This article compares the two head-to-head across the components that actually matter: data ingestion, chunking, embedding, retrieval, generation, and evaluation. We'll also walk through building the same simple pipeline in both frameworks so you can see the code side by side.
Understanding RAG Pipelines
Before comparing frameworks, you need to understand what a RAG pipeline actually does. The architecture breaks down into six components:
1. Document Ingestion. Raw files—PDFs, Markdown, HTML, plain text—are loaded into the system. This step is more complex than it sounds because different formats require different parsers. A PDF with tables and images doesn't load the same way as a clean Markdown file.
2. Chunking. Documents are too long to embed as single vectors. They get split into smaller pieces, typically 200–1000 tokens. The chunk size and overlap strategy directly affect retrieval quality. Too large, and chunks contain irrelevant noise; too small, and they lack context.
3. Embedding. Each chunk is passed through an embedding model that converts text into a high-dimensional vector. The all-MiniLM-L6-v2 sentence transformer—downloaded over 50 million times from Hugging Face—is a common starting point. These vectors capture semantic meaning, so similar texts produce similar vectors.
4. Vector Storage. Embeddings go into a vector database like Chroma, FAISS, or Weaviate. These databases support efficient similarity search, typically using cosine similarity or dot product.
5. Retrieval. When a user asks a question, the query is embedded with the same model, and the vector database returns the most similar chunks. More advanced pipelines use hybrid search—combining vector similarity with keyword matching (BM25)—and re-ranking to improve results.
6. Generation. The retrieved chunks are inserted into a prompt as context, and an LLM generates an answer based on that context. The model is instructed to answer only from the provided information, which grounds the output in your documents.
The key terminology you'll encounter: embeddings (vector representations of text), vector databases (storage and search for embeddings), chunking (splitting documents), hybrid search (combining semantic and keyword retrieval), and re-ranking (reordering retrieved chunks by relevance).
The Contenders: LangChain vs. LlamaIndex
Overview of LangChain
LangChain launched in late 2022 and quickly became the most popular framework for LLM applications. Its philosophy is broad: it's not just for RAG. LangChain provides tools for agents, chains, memory, tool use, and multi-step workflows. It aims to be a general-purpose application framework for LLMs.
The ecosystem is massive. LangChain has integrations with hundreds of models, vector stores, and external tools. If you need to connect an LLM to a SQL database, a Slack channel, or a web search API, LangChain probably has an integration for it.
Overview of LlamaIndex
LlamaIndex (formerly GPT Index) started with a narrower focus: data indexing and retrieval. Its core problem is connecting LLMs to your data, period. The framework excels at loading documents, building indices, and retrieving relevant context. It has grown to support agents and workflows, but data-centric operations remain its strength.
LlamaIndex has been downloaded over 1 million times per month from PyPI, and its documentation is arguably the best in the open-source LLM space.
Why These Two?
These frameworks dominate because they solve the hardest part of RAG—gluing together many moving parts—without requiring you to write everything from scratch. Both are open-source, both have active communities, and both abstract away the boilerplate of building a pipeline. The question is which abstraction you want.
Head-to-Head Comparison: LangChain vs. LlamaIndex
Ease of Use: Learning Curve and Documentation
LangChain has a steeper learning curve. Its API surface is enormous, and the framework has undergone significant changes between versions. You'll frequently encounter deprecated methods or tutorials that no longer work. The documentation is comprehensive but sprawling; finding the exact page you need can feel like navigating a maze.
LlamaIndex is more approachable for RAG specifically. The core concepts—Document, Node, Index, Retriever—map directly to pipeline components. The documentation includes clear "getting started" guides and detailed examples for each data source. LlamaIndex also provides high-level VectorStoreIndex classes that let you build a working pipeline in five lines of code.
Verdict: LlamaIndex wins for beginners. LangChain's complexity is justified by its broader scope, but it's a heavier lift for simple RAG.
Flexibility and Extensibility
LangChain is extremely flexible. You can customize every component, swap out models and retrievers, and build complex chains with branching logic. This flexibility is essential for production systems that need custom behavior, but it also means you'll spend more time wiring things together.
LlamaIndex is flexible within its domain. You can customize retrievers, node parsers, and indices, but the framework assumes you're building a data-centric application. If your project grows beyond RAG into general agent territory, you'll hit limits.
Verdict: LangChain for broad flexibility, LlamaIndex for focused data workflows.
Data Ingestion: Supported Formats and Loaders
LangChain has a massive collection of document loaders—over 100 at last count. PDFs, DOCX, HTML, Markdown, JSON, CSV, YouTube transcripts, Notion pages, Google Drive, and more. The Unstructured integration handles messy formats with table extraction and OCR.
LlamaIndex ships with readers for common formats plus LlamaHub, a registry of data connectors. It covers PDF, Markdown, HTML, DOCX, CSV, and a growing list of SaaS integrations. The PDF reader is solid, though it relies on external libraries like pypdf or pdfplumber for parsing.
Verdict: LangChain has more loaders and better support for messy documents. LlamaIndex covers the essentials well.
Chunking Strategies
LangChain offers multiple text splitters: RecursiveCharacterTextSplitter (the default), CharacterTextSplitter, TokenTextSplitter, and MarkdownHeaderTextSplitter for structure-aware chunking. You can also write custom splitters by subclassing the base class.
LlamaIndex provides SentenceSplitter (the default), TokenTextSplitter, and SemanticSplitterNodeParser for embedding-based chunking. The SentenceSplitter is well-tuned for most documents, and the semantic splitter is a nice option when you want chunks that follow topical boundaries.
Verdict: Both are capable. LlamaIndex's defaults are slightly better out of the box; LangChain gives you more manual control.
Embedding Integration
LangChain supports virtually every embedding model provider: OpenAI, Cohere, Hugging Face, Ollama, and more. The HuggingFaceEmbeddings class lets you plug in any sentence-transformers model by name.
LlamaIndex has similar coverage. The HuggingFaceEmbedding class accepts any model ID, and there's built-in support for OpenAI, Cohere, and local models via Ollama.
Verdict: Effectively a tie. Both make it trivial to swap embedding models.
Vector Database Support
LangChain integrates with over 50 vector stores: Chroma, FAISS, Pinecone, Weaviate, Qdrant, Milvus, and more. The Chroma integration is the most common starting point for tutorials and small projects.
LlamaIndex supports the same major databases—Chroma, FAISS, Weaviate, Qdrant, Pinecone—plus a few niche options. The integration pattern is similar: create a vector store, pass it to an index.
Verdict: Tie. Both cover the major options and make swapping easy.
Retrieval Capabilities
LangChain offers the MultiQueryRetriever, which generates multiple query variations, and EnsembleRetriever, which combines vector and BM25 retrieval. Re-ranking is available through integration with CrossEncoder models.
LlamaIndex has strong retrieval features built in: VectorIndexRetriever, BM25Retriever, and a QueryFusionRetriever for hybrid search. LlamaIndex also supports node postprocessors for re-ranking, including SentenceTransformerRerank.
Verdict: LlamaIndex edges out LangChain here because hybrid search and re-ranking are more straightforward to configure.
Integration with LLMs
LangChain supports every major LLM provider plus local models through Ollama, vLLM, and Hugging Face pipelines. The ChatOllama class is a simple way to use Llama 2, Mistral, or Zephyr locally.
LlamaIndex has similar coverage, including Ollama, HuggingFaceLLM, and vLLM integrations. The Settings object centralizes LLM configuration.
Verdict: Tie. Both handle local and API-based models equally well.
Evaluation Tooling
LangChain provides LangSmith for tracing and evaluation, though the full product is commercial. There's also an Evaluator framework in the open-source library, but it's less polished than the commercial offering.
LlamaIndex integrates directly with RAGAS, an open-source library for RAG evaluation. Metrics include faithfulness, answer relevance, and context precision. RAGAS works with both frameworks, but LlamaIndex's integration is tighter.
Verdict: LlamaIndex wins for open-source evaluation. LangChain's best tooling lives behind a paywall.
Community and Ecosystem
LangChain has the largest community. More tutorials, more Stack Overflow answers, more GitHub issues, more third-party integrations. This matters when you're stuck on an obscure bug at 2 AM.
LlamaIndex has a smaller but highly engaged community. The documentation is better, and the maintainers are responsive.
Verdict: LangChain for community size, LlamaIndex for documentation quality.
Pros and Cons
LangChain: Strengths and Weaknesses
Strengths: - Massive ecosystem of integrations - Flexible for non-RAG applications (agents, chains, tools) - Huge community and tutorial base - Excellent for complex, multi-step workflows
Weaknesses: - Steep learning curve - API churn between versions - Overkill for simple RAG - Documentation is sprawling and sometimes outdated
LlamaIndex: Strengths and Weaknesses
Strengths: - Focused on data indexing and retrieval - Clean, intuitive API - Excellent documentation - Built-in hybrid search and re-ranking - Better RAGAS integration
Weaknesses: - Less flexible for non-RAG applications - Smaller community - Fewer integrations for external tools
Summary Table
| Aspect | LangChain | LlamaIndex |
|---|---|---|
| Learning curve | Steep | Gentle |
| Flexibility | High | Moderate |
| Data loaders | 100+ | Good coverage |
| Retrieval | Good | Excellent |
| Evaluation | Commercial tool | RAGAS native |
| Community | Large | Growing |
| Best for | Complex LLM apps | Data-centric RAG |
Building a Simple RAG Pipeline: Step-by-Step with Both Frameworks
Let's build the same pipeline in both frameworks. We'll use a local LLM (Llama 2 via Ollama), sentence-transformers for embeddings, and Chroma as the vector database. The goal is to load a PDF, chunk it, embed it, store it, retrieve relevant chunks, and generate an answer.
Setting Up the Environment
# Install dependencies
pip install langchain llama-index chromadb sentence-transformers pypdf
# Install Ollama and pull Llama 2
curl -fsSL https://ollama.ai/install | sh
ollama pull llama2
Step 1: Load Documents
LangChain:
from langchain_community.document_loaders import PyPDFLoader
loader = PyPDFLoader("manual.pdf")
documents = loader.load()
LlamaIndex:
from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader(input_files=["manual.pdf"]).load_data()
Step 2: Chunk the Documents
LangChain:
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)
LlamaIndex:
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(chunk_size=500, chunk_overlap=50)
nodes = splitter.get_nodes_from_documents(documents)
Step 3: Generate Embeddings
LangChain:
from langchain_community.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
LlamaIndex:
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
Settings.embed_model = HuggingFaceEmbedding(model_name="all-MiniLM-L6-v2")
Step 4: Store in a Vector Database
LangChain:
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(chunks, embeddings)
LlamaIndex:
from llama_index.core import VectorStoreIndex
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
chroma_client = chromadb.Client()
chroma_collection = chroma_client.get_or_create_collection("manual")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_documents(nodes, vector_store=vector_store)
Step 5: Retrieve Relevant Chunks
LangChain:
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
results = retriever.invoke("How do I reset the device?")
LlamaIndex:
retriever = index.as_retriever(similarity_top_k=3)
results = retriever.retrieve("How do I reset the device?")
Step 6: Generate an Answer
LangChain:
from langchain_community.chat_models import ChatOllama
from langchain.chains import RetrievalQA
llm = ChatOllama(model="llama2")
qa_chain = RetrievalQA.from_chain_type(llm, retriever=retriever)
answer = qa_chain.invoke("How do I reset the device?")
LlamaIndex:
from llama_index.llms.ollama import Ollama
Settings.llm = Ollama(model="llama2")
query_engine = index.as_query_engine()
response = query_engine.query("How do I reset the device?")
Code Complexity Comparison
Both pipelines are roughly 15–20 lines of code. LlamaIndex's approach is more declarative—you set global Settings and the framework handles the rest. LangChain requires more explicit wiring but gives you finer control over each step.
Key Takeaway: For a simple pipeline, LlamaIndex gets you to a working system faster. LangChain's verbosity pays off when you need custom behavior.
Performance and Quality Considerations
The framework you choose matters less than the choices you make within it. Here's what actually affects RAG quality:
Chunking Impact
Chunk size is the single most important lever. Small chunks (200–300 tokens) retrieve more precisely but may lack context. Large chunks (800–1000 tokens) provide context but dilute relevance. Start with 500 tokens and 10% overlap, then tune based on your evaluation results.
Embedding Model Selection
all-MiniLM-L6-v2 is a good default but not the best. BGE models (like BAAI/bge-large-en-v1.5) and E5 models typically produce better embeddings at the cost of speed and memory. Test a few models on your specific document type.
Hybrid Search vs. Vector-Only
Vector search handles semantic similarity but can miss exact keywords. BM25 handles exact matches but misses synonyms. Hybrid search combines both. A benchmark study by Karpukhin et al. found that hybrid retrieval improved accuracy by up to 20% over dense-only methods. Use hybrid search when your documents contain domain-specific terminology.
Re-Ranking
Retrieval returns the top-k chunks by similarity, but "similar" doesn't always mean "relevant." Re-ranking with a CrossEncoder model like cross-encoder/ms-marco-MiniLM-L-6-v2 improves answer quality by reordering chunks based on deeper semantic analysis.
Evaluation Metrics
RAGAS provides four key metrics:
- Faithfulness: Is the answer grounded in the retrieved context?
- Answer relevance: Does the answer address the question?
- Context precision: Are the retrieved chunks relevant to the question?
- Context recall: Did the retrieval miss relevant information?
Run these metrics on a test set of 20–50 questions before deploying.
Real-World Use Cases and Examples
The same pipeline pattern applies across domains:
Customer Support Chatbot. Load product manuals and FAQs. Users ask questions in natural language; the bot retrieves relevant sections and answers with citations. This is the most common RAG use case because documentation is readily available and users ask the same questions repeatedly.
Research Assistant. Load a corpus of academic papers. Query with a research question; the system retrieves relevant papers and summarizes findings. The 2020 RAG paper showed this works well on open-domain QA benchmarks like Natural Questions and TriviaQA.
Legal Document Review. Load contracts and legal documents. Ask for clauses related to indemnification, termination, or liability. Metadata filtering (by date, party, or contract type) becomes essential here.
Personal Knowledge Base. Load your notes, journal entries, and saved articles. Ask questions about what you've written. This works surprisingly well with local models, keeping your data private.
News Summarization. Load recent articles on a topic. Ask for a summary that covers the key points across multiple sources. Retrieval ensures the summary reflects current events, not stale training data.
Common Pitfalls and Misconceptions
Poor Chunking. The most common failure mode. Chunks that split mid-sentence or split across semantic boundaries produce garbage retrieval. Use a sentence-aware splitter and test different chunk sizes.
Ignoring Metadata Filtering. If your documents have dates, authors, or categories, use metadata filters to narrow retrieval before similarity search. This dramatically improves precision.
Over-Reliance on Vector Search. Vector search fails on exact-match queries like product codes or legal citations. Add BM25 or hybrid search to cover this gap.
RAG Does Not Eliminate Hallucinations. RAG grounds answers in retrieved context, but the LLM can still generate incorrect claims if the context is incomplete or if the prompt doesn't enforce strict grounding. Always instruct the model to say "I don't know" when the answer isn't in the context.
Mistaking RAG for Fine-Tuning. RAG changes what the model reads at inference time. Fine-tuning changes the model's weights. They solve different problems. RAG is for knowledge access; fine-tuning is for style, format, and domain-specific reasoning. You can combine them, but they're not interchangeable.
Verdict: Which Framework Should You Choose?
When to Choose LangChain
Choose LangChain if you're building a general LLM application, not just RAG. If you need agents, multi-step chains, tool use, or integrations with external systems, LangChain's ecosystem is unmatched. It's also the better choice if you want to move beyond RAG later. The learning curve is real, but the payoff is flexibility.
When to Choose LlamaIndex
Choose LlamaIndex if RAG is your primary use case. The API is cleaner, the documentation is better, and retrieval features like hybrid search and re-ranking are first-class citizens. If you're building a document Q&A system, a knowledge base, or a research assistant, LlamaIndex gets you there faster with less code.
Final Recommendation
For most RAG projects, start with LlamaIndex. It's easier to learn, produces working systems faster, and has better built-in retrieval features. If you hit its limits—which happens when your application grows beyond data retrieval—you can always migrate to LangChain later.
Key Takeaway: LangChain is a general-purpose Swiss Army knife for LLM apps. LlamaIndex is a specialized tool for data retrieval. Use the right tool for the job.
Conclusion
RAG is the most practical way to make LLMs useful for your specific data, and the open-source ecosystem has matured to the point where building a pipeline takes minutes, not weeks. LangChain and LlamaIndex both deliver working systems, but they serve different needs.
LlamaIndex is the better starting point for RAG-focused projects. Its clean API, excellent documentation, and built-in retrieval features let you focus on your data rather than framework plumbing. LangChain is the better choice when your application needs to grow beyond retrieval into agents, tools, and complex workflows.
The pipeline itself—load, chunk, embed, store, retrieve, generate—is the same regardless of framework. Master the concepts, and the framework becomes a detail.
Further Resources:
- LangChain Documentation
- LlamaIndex Documentation
- ChromaDB Documentation
- RAGAS
- Lewis et al. (2020), "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"
- Gao et al. (2023), "Retrieval-Augmented Generation for Large Language Models: A Survey"
FAQ
What are the essential open-source tools for building a RAG pipeline?
You need an embedding model (sentence-transformers), a vector database (Chroma or FAISS), an LLM (Llama 2, Mistral, or Zephyr via Ollama), and a framework to tie them together (LangChain or LlamaIndex). All are free and run locally.
How do I choose a chunk size for my documents?
Start with 500 tokens and 10% overlap. If retrieval results are too noisy, reduce to 300. If they lack context, increase to 800. Evaluate with a small test set to find the sweet spot for your documents.
Can I run a RAG pipeline entirely on a local machine?
Yes. Use Ollama for the LLM, sentence-transformers for embeddings, and Chroma for storage. A 16GB RAM laptop can handle this comfortably. A GPU helps with larger models but isn't required.
What is the difference between RAG and fine-tuning?
RAG retrieves relevant information at query time and grounds the answer in that context. Fine-tuning modifies the model's weights to improve performance on specific tasks or domains. RAG changes what the model reads; fine-tuning changes how the model thinks.
How do I evaluate my RAG pipeline?
Use RAGAS metrics: faithfulness (is the answer grounded in context?), answer relevance (does it answer the question?), and context precision/recall (did retrieval find the right chunks?). Run these on a test set of 20–50 questions.
What are common pitfalls in building RAG pipelines?
Poor chunking, ignoring metadata filtering, relying only on vector search, and expecting RAG to eliminate hallucinations completely. Also, confusing RAG with fine-tuning.
Do I need a GPU to run a RAG pipeline?
No. Small embedding models and 7B-parameter LLMs run on CPU, just slower. For production, a GPU helps, but for learning and prototyping, CPU is fine.
Can I use RAG with non-English documents?
Yes. Use a multilingual embedding model like paraphrase-multilingual-MiniLM-L12-v2 and a multilingual LLM like Llama 3 or Mistral. The pipeline architecture is language-agnostic.
Ready to build your own RAG pipeline? Start with our step-by-step guide and choose the framework that fits your needs. Subscribe to our newsletter for more AI tutorials and updates.