PDF to Markdown for LangChain and LlamaIndex: Cleaner RAG Ingestion

LangChain and LlamaIndex are the two most popular frameworks for building retrieval-augmented generation (RAG) systems. Both let you load documents, split them into chunks, embed those chunks, and retrieve the relevant ones at query time. And both ship with PDF loaders that make it tempting to point a directory of PDFs at your pipeline and call it a day.

The problem is that the default PDF loaders extract plain text. They flatten your documents into an undifferentiated stream of characters, losing the headings, lists, and tables that give a document its structure. When you then chunk that text, you split on arbitrary character counts instead of natural boundaries, and your retrieval quality suffers. Headings end up glued to unrelated paragraphs. Tables become word soup. A single logical section gets sliced across three chunks.

Converting your PDFs to Markdown first fixes this at the source. Markdown preserves the document hierarchy in a format your splitters understand, so your chunks line up with real sections and your retriever surfaces coherent, self-contained passages. This article walks through the ingestion pipeline for both frameworks, with working code for each loader and the tradeoffs you should know before you commit to an approach.

Why the Default PDF Loaders Hurt Retrieval

To see the problem clearly, it helps to understand what a RAG pipeline actually does with your document. The steps are always the same:

1. Load: Read the source file and turn it into text.
2. Split: Break the text into chunks small enough to embed and retrieve.
3. Embed: Convert each chunk into a vector.
4. Store: Save the vectors in a vector database.
5. Retrieve: At query time, find the chunks whose vectors are closest to the query.

Retrieval quality is decided almost entirely in steps 1 and 2. If the text you loaded has lost its structure, no amount of clever embedding will put it back. A common default in both frameworks is a recursive character splitter that tries to break on paragraph breaks, then sentences, then words. That works reasonably well on clean prose. It falls apart on PDF text extraction, where line breaks are physical (they mark the edge of the page, not the end of a thought) and headings are indistinguishable from body text.

Markdown gives the splitter something to hold onto. A heading is a line that starts with #. A list is a run of lines that start with -. A table is a block of pipe-delimited rows. Both frameworks ship splitters that are aware of these markers, so your chunks respect the document outline instead of cutting across it. If you want the full argument with benchmarks, we covered it in detail in our guide to building better RAG pipelines with clean Markdown.

The Ingestion Pipeline, Step by Step

The recommended flow is the same regardless of framework. Convert first, then load the Markdown:

1. Convert each PDF to Markdown using a converter that preserves headings, lists, and tables.
2. Load the Markdown files with a Markdown-aware loader.
3. Split on structure using a header-aware or Markdown-aware splitter.
4. Attach metadata so you can filter and cite retrieved chunks.
5. Embed and store as usual.

The one new step is the conversion. You can run a local library, or you can call a hosted converter and skip the dependency and OCR headaches. Either way, the output you want is a .md file per document that a human could read and understand. If the Markdown looks right to you, it will chunk right for your retriever.

LangChain: Loading Markdown Instead of Raw PDF

The instinct in LangChain is to reach for PyPDFLoader. It works, but it returns one text-only document per page with no structural markup. Here is what most people start with:

# The tempting default: plain text, no structure
from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("report.pdf")
docs = loader.load()  # one Document per page, plain text
print(docs[0].page_content[:200])
# "Q3 Financial Report Revenue grew 14% ... "  headings are gone

Instead, convert the PDF to Markdown first, then load it with UnstructuredMarkdownLoader and split with the header-aware splitter. This keeps section boundaries intact and lets you promote each heading into chunk metadata:

# Better: load Markdown and split on headers
from langchain_community.document_loaders import UnstructuredMarkdownLoader
from langchain_text_splitters import (
    MarkdownHeaderTextSplitter,
    RecursiveCharacterTextSplitter,
)

# report.md was produced by your PDF-to-Markdown step
loader = UnstructuredMarkdownLoader("report.md")
docs = loader.load()
markdown_text = docs[0].page_content

# First pass: split on the document's own headings
headers_to_split_on = [
    ("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),
]
header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False,
)
sections = header_splitter.split_text(markdown_text)

# Second pass: keep long sections within the embedding window
char_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
)
chunks = char_splitter.split_documents(sections)

print(len(chunks), "chunks")
print(chunks[0].metadata)  # {'h1': 'Q3 Financial Report', 'h2': 'Revenue'}

Notice what the metadata gives you. Every chunk now carries the heading trail it came from, so at retrieval time you can tell the model not just what the passage says but where it sits in the document. You can also filter on those fields, for example restricting a query to chunks under a specific section.

From here the rest of the pipeline is standard LangChain. Embed the chunks and push them into your vector store:

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=OpenAIEmbeddings(),
    persist_directory="./chroma_db",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

If you are batch-converting a folder of PDFs, run the conversion step over the whole directory first, write out .md files, then point DirectoryLoader at the folder with a *.md glob. Keeping conversion and loading as separate stages makes the pipeline easier to debug: you can open the intermediate Markdown and see exactly what your retriever will index.

LlamaIndex: MarkdownNodeParser for Structure-Aware Nodes

LlamaIndex calls chunks nodes, and it has a parser built specifically for Markdown. The MarkdownNodeParser splits a document along its headings and records the header path on each node, which is exactly the structure you want to preserve from a PDF. As with LangChain, the trick is to feed it Markdown rather than raw PDF text.

from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser import MarkdownNodeParser

# Read the Markdown your conversion step produced
with open("report.md", "r", encoding="utf-8") as f:
    markdown_text = f.read()

document = Document(text=markdown_text)

parser = MarkdownNodeParser()
nodes = parser.get_nodes_from_documents([document])

for node in nodes[:3]:
    print(node.metadata)   # includes the header path
    print(node.text[:80])

Each node keeps the heading hierarchy in its metadata, so when a node is retrieved the model sees the section it belongs to. Build an index from the nodes and query it as usual:

index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine(similarity_top_k=4)

response = query_engine.query("How much did revenue grow in Q3?")
print(response)

If you prefer to load a whole directory, SimpleDirectoryReader reads .md files out of the box. Point it at the folder of converted Markdown, then run the same parser over the documents it returns:

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader(
    input_dir="./markdown_docs",
    required_exts=[".md"],
).load_data()

nodes = MarkdownNodeParser().get_nodes_from_documents(documents)
index = VectorStoreIndex(nodes)

The takeaway is the same in both frameworks: the parser is only as good as the text you give it. A Markdown-aware parser fed plain PDF text has no headings to split on and quietly falls back to naive chunking. Give it real Markdown and it does its job.

Handling Tables, the Hardest Part

Tables are where the difference between plain text and Markdown is most dramatic. When a PDF table is extracted as plain text, the columns collapse. A row that read Region | Revenue | Growth becomes Region Revenue Growth, and the numbers on the next line lose their association with the headers. Once that happens, the model has no way to reconstruct which number belongs to which column.

A Markdown table keeps the relationships explicit with pipes and a header separator row. When that survives into your chunks, the model can read the table as a table. Keep each table in a single chunk where possible: a table split across two chunks is nearly as useless as no table at all. If your tables are large, consider raising the chunk size for table-heavy documents, or converting each table into a short natural-language summary that lives alongside the raw Markdown. We go deeper on this in our guide to PDF to text versus PDF to Markdown for AI.

Metadata and Citations

RAG systems live or die on their ability to cite sources. Users trust an answer more when it points to the exact section it came from, and you need that trail for debugging when a retrieval goes wrong. Because Markdown-aware splitters attach the heading path to each chunk, you get citation-ready metadata for free.

Enrich it further at ingestion time. A useful metadata payload for each chunk includes:

• Source file: the original PDF name, so you can trace a chunk back to its document.
• Heading path: the h1/h2/h3 trail from the Markdown splitter.
• Page or section number: if your converter preserves it, useful for pointing users to the right place.
• Document type or category: a tag you set per folder, for filtering at query time.

Both frameworks let you merge a metadata dictionary onto every chunk or node during ingestion, so add these fields once and every downstream query benefits.

Local Library or Hosted Converter?

You have two ways to produce the Markdown. Run a conversion library inside your own pipeline, or call a hosted converter over an API. Each has clear tradeoffs.

If your documents are sensitive and must never leave your infrastructure, a local library is the right call, and you can pair it with the Python approaches in our guide to converting PDF to Markdown in Python. If you would rather not maintain OCR and layout logic, a hosted converter like PDFtoMD gives you clean Markdown from a single call. PDFtoMD has a free tier of 3 conversions per month with no credit card, so you can test the output quality against your own documents before wiring it into a pipeline. See the full range of workflows on our use cases page.

A Complete Minimal Pipeline

Putting it together, here is the shape of a complete LangChain ingestion pipeline that starts from a folder of Markdown files (already converted from PDF) and ends with a working retriever:

from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# 1. Load all converted Markdown files
loader = DirectoryLoader(
    "./markdown_docs",
    glob="*.md",
    loader_cls=TextLoader,
)
raw_docs = loader.load()

# 2. Split on headings, then cap chunk length
header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
    strip_headers=False,
)
char_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)

chunks = []
for doc in raw_docs:
    sections = header_splitter.split_text(doc.page_content)
    for section in sections:
        section.metadata["source"] = doc.metadata.get("source", "")
    chunks.extend(char_splitter.split_documents(sections))

# 3. Embed and store
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=OpenAIEmbeddings(),
    persist_directory="./chroma_db",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

Swap the embedding model or vector store for whatever you already use. The two things that matter are that you loaded Markdown, not raw PDF text, and that you split on headings before you split on length.

Frequently Asked Questions

Do I really need Markdown, or can I just use the built-in PDF loader?

You can use the built-in loader, and for short, simple, prose-only PDFs it may be fine. The difference shows up on structured documents: reports with headings, contracts with numbered clauses, and anything with tables. On those, plain text extraction loses the structure that makes retrieval accurate. Markdown preserves it, so your chunks match real sections and your retriever returns coherent passages.

Which chunk size should I use?

Start by splitting on headings first, then cap the length of any oversized section. A chunk size of 800 to 1200 characters with 100 to 200 characters of overlap is a reasonable default for most embedding models. The key insight is that heading-aware splitting does most of the work: length capping is only a safety net for unusually long sections.

Does this work with scanned PDFs?

Only if your conversion step can read them. A scanned PDF is an image, so plain text extraction returns nothing. You need OCR before you get any Markdown at all. A hosted converter typically handles OCR for you, while a local pipeline means wiring up an OCR engine yourself. Either way, the RAG steps after conversion are identical.

Can I mix LangChain and LlamaIndex?

Yes. Both accept Markdown as input, so the conversion step is shared. Many teams convert PDFs to Markdown once, store the .md files, and then feed the same files into whichever framework a given project uses. Decoupling conversion from the framework is one of the practical benefits of standardizing on Markdown as your intermediate format.

Where should the conversion step live in production?

Treat it as its own stage that runs when a document is ingested, before anything touches your vector store. Convert on upload, write the Markdown to storage, and let your indexing job read from there. This keeps conversion failures out of your query path and lets you re-index without re-converting.

The Takeaway

LangChain and LlamaIndex both give you the machinery for great retrieval, but that machinery can only work with the structure you feed it. Loading raw PDF text throws away the headings, lists, and tables that your splitters need, and no downstream tuning recovers them. Converting to Markdown first is the single highest-leverage change you can make to a document-heavy RAG pipeline.

Convert once, load the Markdown with a structure-aware loader, split on headings, and attach metadata for citations. Do that and your chunks will line up with real sections, your tables will survive, and your retriever will return answers you can trust. The frameworks are ready. Give them clean Markdown and let them do the rest.