Motion graphic

Fundamentals of AI Agents Using RAG and LangChain

9 hours · 3 modules · LangChain + Hugging Face + IBM watsonx

Where the certificate becomes application engineering. Fine-tuning changes the model; RAG and agents change the system around the model.

---

Module 1 — Fundamentals of AI Agents

In-context learning

The model learns from examples placed in the prompt, with no weight updates at all. An emergent property of scale — small models can't do it, large ones can.

• Zero-shot — instruction only.

Classify the sentiment: "The film dragged."

• One-shot — one worked example first.
• Few-shot — several examples establishing the pattern and the output format.

Review: "Brilliant pacing."  Sentiment: positive
Review: "I walked out."      Sentiment: negative
Review: "The film dragged."  Sentiment:

Few-shot's main value is often format control — showing the model the exact output shape you want is more reliable than describing it.

Limits: it consumes context window, it doesn't persist between calls, and beyond a handful of examples returns diminish. When you need consistent behaviour on thousands of calls, fine-tune instead — that's cheaper per call and more reliable.

Prompt engineering

Core components of a good prompt:

1. Role / persona — "You are a broadcast post-production engineer."
2. Task — a specific, unambiguous instruction
3. Context — background, constraints, retrieved documents
4. Examples — few-shot demonstrations
5. Output format — explicit; give a schema or a template
6. Tone / constraints — length, audience, what to avoid

Patterns the course names:

Why CoT works: the model has a fixed amount of computation per token. Forcing it to write intermediate steps gives it more tokens — therefore more computation — before it must commit to an answer. It isn't "thinking harder"; it's being given room to work.

Practical rules:

• Be specific. "Summarise in 3 bullets, max 15 words each" beats "summarise briefly".
• Positive instructions beat negative ones — say what to do, not what to avoid.
• Delimit sections clearly (###, XML tags, triple backticks).
• Put the instruction before long context for long documents.
• Explicitly permit "I don't know" — it measurably cuts hallucination.
• Iterate. Prompts are code; test them on a fixed set of cases.

Decoding parameters

You'll set these on every LLM call for the rest of the certificate:

Rule of thumb: temperature ~0 for extraction, classification and factual QA; 0.7–1.0 for creative work. Tune temperature or top_p, not both.

AI agents

Definition: a system where an LLM decides what to do, not merely what to say — it can choose actions, call tools, observe results, and iterate toward a goal.

The loop:

Goal → Think (plan) → Act (call a tool) → Observe (read the result) → repeat → Answer

Components:

1. LLM — the reasoning engine
2. Tools — search, calculator, code execution, database queries, APIs
3. Memory — conversation history, and longer-term storage
4. Planning — decomposing a goal into steps
5. Executor — the loop that runs it

ReAct, the classic agent pattern, made concrete:

Thought: I need current data for this.
Action: search("Cloudflare Pages pricing 2026")
Observation: <results>
Thought: I now have what I need.
Final Answer: ...

Chain vs agent — the distinction that gets quizzed:

• Chain — a fixed, predetermined sequence of steps. Predictable, cheap, debuggable.
• Agent — the LLM chooses the steps at runtime. Flexible, and unpredictable, expensive, and

capable of looping forever.

Use a chain when you know the steps. Reach for an agent only when you genuinely don't.

---

Module 2 — RAG Framework

The problem RAG solves

LLM knowledge is frozen at training time, has no access to your private data, hallucinates confidently, and can't cite sources.

RAG (Retrieval-Augmented Generation): retrieve relevant documents at query time and put them in the prompt. The model reads rather than recalls.

Query → embed → search vector DB → top-k chunks → prompt(query + chunks) → LLM → grounded answer

Why it's the default answer to hallucination: you've converted a memory problem into a reading-comprehension problem. Reading comprehension is what these models are best at.

RAG vs fine-tuning — the decision table

The single most examinable table in the second half of the certificate:

They compose. Fine-tune for domain vocabulary and output format; use RAG for the facts. That's what a production system does.

The two phases

Indexing (offline, once per document):

1. Load documents
2. Split into chunks
3. Embed each chunk into a vector
4. Store vectors + text + metadata in a vector database

Retrieval + generation (online, per query):

1. Embed the query with the same embedding model
2. Similarity-search the store for the top-k chunks
3. Build a prompt from query + chunks
4. Generate
5. Optionally return sources

Embeddings and vector search

Text → dense vector. Semantically similar text → nearby vectors. Compared with cosine similarity.

This is why RAG beats keyword search: a query about "revenue decline" retrieves a passage about "falling sales" even with no shared words.

Non-negotiable rule: query and documents must be embedded by the same model. Different models produce incompatible vector spaces, and the failure is silent — you get results, they're just meaningless.

Vector databases: Chroma (local, simple — what the labs use), FAISS (Meta's library, fast, in-process), Pinecone / Weaviate / Qdrant / Milvus (managed or production-grade), pgvector (Postgres extension). They use approximate nearest-neighbour indexes (HNSW, IVF) because exact search over millions of vectors is too slow — you trade a small amount of recall for enormous speed.

Chunking

The parameter that most determines RAG quality, and the one most people neglect.

• Too large → the retrieved chunk contains the answer plus a lot of noise; you burn context

and dilute attention

• Too small → context is severed; the chunk mentions "he" with no antecedent

Typical: 500–1000 characters with 10–20% overlap. Overlap prevents a sentence that straddles a boundary from being lost to both chunks.

Strategies (Course 13 covers the code):

• Fixed-size — simple, ignores structure
• Recursive character — splits on paragraph → sentence → word, preserving structure where it

can. The sensible default.

• Document-structure-aware — markdown headers, HTML tags, code functions
• Semantic — split where embedding similarity drops between sentences

Retrieval quality

• top-k — usually 3–5. More context is not automatically better; irrelevant chunks measurably

degrade answers.

• Hybrid search — combine dense (semantic) with sparse BM25 keyword search. Catches exact

identifiers, product codes and names, which pure embeddings often miss.

• Re-ranking — retrieve 20 cheaply, then re-score with a cross-encoder and keep the best 5.

A cross-encoder reads query and document together, so it's much more accurate but too slow to run over the whole corpus.

• Query rewriting — have the LLM expand or clarify the query before retrieving.
• MMR (Maximal Marginal Relevance) — trade relevance against diversity so you don't retrieve

five near-identical chunks.

RAG failure modes

Worth knowing because debugging RAG is the actual job:

1. Retrieval miss — the answer exists but wasn't retrieved → chunking, embedding model, k,

hybrid search

1. Retrieved but ignored — it's in the context and the model didn't use it → prompt

instruction, chunk ordering ("lost in the middle" — models attend better to the start and end of long contexts)

1. Contradictory chunks — two sources disagree → metadata, recency filters, tell the model

how to resolve

1. Generation ignores grounding — model answers from parametric memory → explicit "answer

only from the context provided; if it isn't there, say so"

1. No answer exists — the model should decline, and usually won't unless told it may

---

Module 3 — Prompt Engineering and LangChain

What LangChain is

A framework for composing LLM applications: a common interface across model providers, plus building blocks for prompts, chains, retrieval, memory and agents.

The pitch: swap OpenAI for watsonx for a local model by changing one line.

The honest caveat, worth holding: LangChain adds abstraction layers that can obscure what's actually being sent to the model. For simple applications, direct API calls are often clearer. Learn it because it's on every job description and because the certificate uses it — but know when it's overhead.

The components

Chat models and LLMs

from langchain_ibm import WatsonxLLM
llm = WatsonxLLM(model_id='meta-llama/llama-3-70b-instruct',
                 url=..., project_id=..., params={'temperature': 0.5, 'max_new_tokens': 500})

Prompt templates

from langchain_core.prompts import PromptTemplate, ChatPromptTemplate

prompt = PromptTemplate(input_variables=['topic', 'audience'],
                        template="Explain {topic} to {audience} in three sentences.")

chat_prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise technical writer."),
    ("human",  "{question}"),
])

Output parsers — turn text into structured data

from langchain_core.output_parsers import StrOutputParser, JsonOutputParser

LCEL (LangChain Expression Language) — the pipe syntax that composes them:

chain = prompt | llm | StrOutputParser()
chain.invoke({'topic': 'colour grading', 'audience': 'a producer'})

| passes each component's output to the next. Chains built this way support .invoke(), .batch(), .stream() and their async equivalents for free.

Memory — LLM APIs are stateless; memory is what makes a conversation.

from langchain.memory import ConversationBufferMemory, ConversationSummaryMemory

• Buffer — keeps everything. Simple; eventually overflows the context window.
• Window — keeps the last k turns.
• Summary — LLM-summarises older turns to stay compact.
• Summary-buffer — recent turns verbatim, older ones summarised. The usual production choice.

Tools and agents

from langchain.agents import AgentExecutor, create_react_agent
from langchain_core.tools import tool

@tool
def get_frame_rate(codec: str) -> str:
    """Return the standard frame rate for a given broadcast codec."""
    ...

agent = create_react_agent(llm, tools=[get_frame_rate], prompt=react_prompt)
executor = AgentExecutor(agent=agent, tools=[get_frame_rate],
                         verbose=True, max_iterations=5,
                         handle_parsing_errors=True)
executor.invoke({'input': 'What frame rate should I use for XDCAM HD422?'})

The docstring is not documentation — it is the tool's specification to the model. The LLM reads the name, the docstring and the type hints to decide when and how to call it. A vague docstring means a tool that never gets used, or gets used wrongly. This is the highest-leverage detail in agent building.

max_iterations is a required safety valve — agents loop.

Document loaders and retrievers — Course 13's subject:

from langchain_community.document_loaders import PyPDFLoader, WebBaseLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma

A complete RAG chain

The pattern the final project builds on:

from langchain_core.runnables import RunnablePassthrough

retriever = vectorstore.as_retriever(search_kwargs={'k': 4})

template = """Answer the question using only the context below.
If the context does not contain the answer, say you don't know.

Context:
{context}

Question: {question}

Answer:"""
prompt = ChatPromptTemplate.from_template(template)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
rag_chain.invoke("What is the retention policy for raw footage?")

RunnablePassthrough() forwards the original input unchanged while the retriever branch runs in parallel — that's how one input feeds two slots in the prompt.

---

Course 12 in one page

• In-context learning: zero/one/few-shot, no weight updates; great for format control
• CoT works by buying the model more computation before it commits
• Temperature ~0 for factual, 0.7+ for creative; tune temperature or top_p
• Agent = LLM chooses actions; chain = you choose them. Prefer chains; cap agent iterations
• RAG supplies knowledge, fine-tuning supplies behaviour — and they compose
• Index: load → split → embed → store. Query: embed → retrieve top-k → prompt → generate
• Same embedding model for queries and documents — mismatches fail silently
• Chunk 500–1000 chars, 10–20% overlap; chunking quality dominates RAG quality
• top-k 3–5; hybrid search for exact terms; re-rank with a cross-encoder for accuracy
• Debug RAG in order: did it retrieve? did the model use it? did the prompt permit "I don't know"?
• LangChain: prompt | llm | parser; memory types; @tool docstrings ARE the tool spec