Motion graphic

Project: Generative AI Applications with RAG and LangChain

9 hours · 3 modules · LangChain + watsonx + Chroma + Gradio

The capstone. You build a QA bot that answers questions from a PDF you upload. Everything in courses 7–12 converges here. This is also the portfolio piece — it's the one thing from this certificate you can show someone in 30 seconds.

---

Module 1 — Document Loader Using LangChain

Loaders

Every loader returns a list of Document objects with .page_content (text) and .metadata (source, page number, whatever else).

from langchain_community.document_loaders import (
    PyPDFLoader, TextLoader, CSVLoader, WebBaseLoader,
    UnstructuredMarkdownLoader, JSONLoader, DirectoryLoader, WikipediaLoader,
)

docs = PyPDFLoader("policy.pdf").load()          # one Document per PAGE
docs = TextLoader("notes.txt").load()            # one Document
docs = WebBaseLoader("https://example.com").load()
docs = DirectoryLoader('./docs', glob="**/*.pdf", loader_cls=PyPDFLoader).load()

docs[0].page_content[:200]
docs[0].metadata          # {'source': 'policy.pdf', 'page': 0}

Metadata is not decoration. It's how you cite sources, filter by date or department, and debug retrieval. Preserve it through the whole pipeline.

PDF caveats worth knowing before you lose an afternoon: scanned PDFs have no text layer and need OCR (UnstructuredPDFLoader with OCR, or Tesseract); multi-column layouts often extract in the wrong reading order; tables usually come out mangled. If retrieval is bad, print the extracted text first — the problem is frequently at the loader, not the retriever.

Why you can't just put the whole document in the prompt

The second lab makes this concrete by having you try it. It fails for four reasons:

1. Context window — a 200-page PDF exceeds even large windows
2. Cost — you pay per token, on every single query
3. Latency — long prompts are slow
4. "Lost in the middle" — models attend well to the beginning and end of long contexts and

measurably worse to the middle. Burying the answer at 60% depth degrades accuracy even when it technically fits.

That last point is the real argument for retrieval, and it's the one people miss. Retrieval isn't only a workaround for context limits — it improves accuracy even when everything would fit.

Text splitting

from langchain_text_splitters import (
    RecursiveCharacterTextSplitter, CharacterTextSplitter,
    MarkdownHeaderTextSplitter, TokenTextSplitter,
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],     # tried in order
    length_function=len,
)
chunks = splitter.split_documents(docs)

RecursiveCharacterTextSplitter is the default you should reach for. It tries to split on paragraph breaks first, then line breaks, then sentences, then words — only falling back to mid-word if it must. Structure is preserved wherever possible.

TokenTextSplitter splits by tokens instead of characters, which matters when you're budgeting context precisely.

Choosing chunk_size:

chunk_overlap ≈ 10–20% of chunk_size. Overlap exists so a sentence spanning a boundary appears whole in at least one chunk.

Verify your splits. Print the first few chunks and check they're coherent — this five-second check catches most bad RAG before you build anything on top of it:

print(len(chunks))
print(chunks[0].page_content)
print(chunks[0].metadata)

---

Module 2 — RAG Using LangChain

Embeddings

from langchain_ibm import WatsonxEmbeddings
from ibm_watsonx_ai.foundation_models.utils.enums import EmbeddingTypes

embeddings = WatsonxEmbeddings(
    model_id=EmbeddingTypes.IBM_SLATE_30M_ENG.value,
    url="https://us-south.ml.cloud.ibm.com",
    project_id="skills-network",
)
vec = embeddings.embed_query("What is the retention policy?")
len(vec)          # e.g. 384

Alternatives you'll see: HuggingFaceEmbeddings (local, free, all-MiniLM-L6-v2 is the standard small model), OpenAIEmbeddings, sentence-transformers directly.

Choosing an embedding model, in order of what actually matters:

1. Dimension — 384 (fast, small) to 1536+ (more expressive, more storage)
2. Max input length — chunks longer than this get truncated silently
3. Domain — general vs code vs multilingual vs legal
4. Symmetric vs asymmetric — some models are trained for short-query→long-document retrieval

specifically; those perform better for RAG

Check the max sequence length against your chunk_size. A 512-token embedding model with 2000-character chunks is discarding the tail of every chunk and you will never see an error.

Vector store

from langchain_chroma import Chroma

vectordb = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",       # omit for in-memory
    collection_name="policy_docs",
)

results = vectordb.similarity_search("retention policy", k=4)
results = vectordb.similarity_search_with_score("retention policy", k=4)   # with distances

Inspect the scores while developing. If your top result has a poor score, retrieval is the problem, not generation — and you've just saved yourself from debugging the wrong half of the system.

Retrievers

# 1. plain similarity
retriever = vectordb.as_retriever(search_kwargs={"k": 4})

# 2. MMR — relevance balanced against diversity
retriever = vectordb.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 4, "fetch_k": 20, "lambda_mult": 0.5})

# 3. similarity score threshold — return nothing rather than something bad
retriever = vectordb.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.5, "k": 4})

# 4. multi-query — LLM generates query variants, unions the results
from langchain.retrievers.multi_query import MultiQueryRetriever
retriever = MultiQueryRetriever.from_llm(retriever=vectordb.as_retriever(), llm=llm)

# 5. parent-document — search small chunks, return their larger parents
from langchain.retrievers import ParentDocumentRetriever

# 6. self-query — LLM extracts metadata filters from natural language
from langchain.retrievers.self_query.base import SelfQueryRetriever

The two that earn their complexity:

• MMR when your corpus has redundancy — otherwise the top 4 are four copies of one passage.

lambda_mult: 1.0 = pure relevance, 0.0 = pure diversity.

• Parent-document because it resolves the chunk-size dilemma directly: embed small chunks for

precise matching, then hand the model the surrounding larger chunk for context. Best of both.

Score threshold is underrated: it lets the system return nothing, which is what should happen when the answer isn't in the corpus. Most RAG systems have no way to say "not found" and that's where hallucination gets in.

Reading: InstructLab fine-tuning vs RAG

Same conclusion as Course 10 and 12, stated once more because it's the certificate's thesis:

Production systems use both. Say that in an interview and you're answering correctly.

---

Module 3 — Create a QA Bot to Read Your Document

Gradio

Python UI in a few lines, with a public share link — which is exactly what you want for a portfolio piece.

import gradio as gr

def answer(file, question):
    ...
    return response

demo = gr.Interface(
    fn=answer,
    inputs=[gr.File(label="Upload a PDF", file_types=['.pdf'], type="filepath"),
            gr.Textbox(label="Question", lines=2, placeholder="Ask about the document…")],
    outputs=gr.Textbox(label="Answer", lines=10),
    title="Document QA Bot",
    description="Upload a PDF and ask questions about it.",
)
demo.launch(server_name="0.0.0.0", server_port=7860)

Notes that save time:

• type="filepath" gives you a path string — what the PDF loader wants. The default gives a file

object and the loader will reject it.

• In the Skills Network lab environment, server_name="0.0.0.0" is required or the proxy can't

reach it.

• gr.Blocks() instead of gr.Interface when you need custom layout.
• share=True gives a temporary public URL — handy for showing someone, expires in 72 hours.

The complete application

This is the final deliverable, end to end:

import gradio as gr
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain.chains import RetrievalQA
from langchain_ibm import WatsonxLLM, WatsonxEmbeddings

def build_qa_chain(pdf_path):
    docs   = PyPDFLoader(pdf_path).load()
    chunks = RecursiveCharacterTextSplitter(
        chunk_size=1000, chunk_overlap=200).split_documents(docs)

    vectordb  = Chroma.from_documents(chunks, embeddings)
    retriever = vectordb.as_retriever(search_kwargs={"k": 4})

    return RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True,
    )

def answer(file, question):
    if file is None:
        return "Upload a PDF first."
    qa     = build_qa_chain(file)
    result = qa.invoke({"query": question})
    sources = {d.metadata.get('page') for d in result['source_documents']}
    return f"{result['result']}\n\nSources — pages: {sorted(sources)}"

gr.Interface(
    fn=answer,
    inputs=[gr.File(label="PDF", file_types=['.pdf'], type="filepath"),
            gr.Textbox(label="Question")],
    outputs=gr.Textbox(label="Answer"),
    title="QA Bot",
).launch(server_name="0.0.0.0", server_port=7860)

chain_type options — asked on the quiz:

Use stuff unless k is large.

return_source_documents=True is what lets you cite. Do it — grounded answers with page numbers is the difference between a demo and something credible.

Custom prompt — the anti-hallucination piece

Don't leave the default prompt in place. This is the single highest-value change:

from langchain_core.prompts import PromptTemplate

template = """Use only the context below to answer the question.
If the answer is not in the context, say "I couldn't find that in this document."
Do not use outside knowledge. Quote the relevant passage where possible.

Context:
{context}

Question: {question}

Answer:"""

qa = RetrievalQA.from_chain_type(
    llm=llm, retriever=retriever,
    chain_type_kwargs={"prompt": PromptTemplate.from_template(template)},
    return_source_documents=True,
)

Three instructions doing the work: only the context, permission to say "not found", and quote the passage. Each independently reduces hallucination.

Final project — the graded deliverables

Six screenshots/tasks, which map directly to the labs. Take each screenshot as you complete the lab; going back to reproduce them is a waste of an evening.

1. PDF loader — code + output showing the document loaded
2. Code splitter — chunks produced, with size and overlap visible
3. Embedding — an embedding generated, showing its dimension
4. Vector database — documents stored and a similarity search returning results
5. Retriever — a retriever returning relevant segments for a query
6. QA bot — the Gradio interface, with the PDF uploaded and a real answer displayed

Screenshot requirements are strict: the code and its output must both be visible, and the question and answer must both be legible in the QA bot shot. Peer graders mark against exactly this list.

Common failures

Making it a portfolio piece

Worth an extra hour beyond the rubric, because this is the artefact you'll actually show people:

• Show sources with page numbers on every answer
• Handle the empty state and errors gracefully
• Add a conversation history (ConversationalRetrievalChain for follow-up questions)
• Point it at a document from your domain — broadcast delivery specs, a codec manual, a

post-production style guide. A QA bot over material you know cold demonstrates far more than one over a generic sample PDF, and you can immediately tell when it's wrong.

• Write a short README: the problem, the architecture, the chunking choices and why

---

Course 13 in one page

• Loaders return Document(page_content, metadata) — keep the metadata, it's your citations
• Whole document in the prompt fails on window, cost, latency and "lost in the middle"
• RecursiveCharacterTextSplitter, 1000/200, is the sensible default — and inspect your chunks
• Embedding model max input length must exceed chunk size, or the tail is silently discarded
• Chroma for local; similarity_search_with_score to debug retrieval before blaming the LLM
• MMR for redundant corpora; parent-document to get precise matching and wide context; score

threshold so the system can return nothing

• chain_type: stuff by default; map_reduce for many chunks; refine for quality; map_rerank

for single-chunk answers

• return_source_documents=True and a custom prompt that forbids outside knowledge and permits

"I couldn't find that"

• Gradio: type="filepath", server_name="0.0.0.0"
• Screenshot each of the six deliverables as you go