Two Layers That Stop the Bot from Making Things Up

The portfolio bot once scored 0.601 on What university did he attend? None of the seven facts mention a school, but the question barely cleared the 0.6 threshold — so generation ran anyway. The grounded prompt still answered I don't know based on the provided context instead of inventing a university. That near-miss is why a cutoff alone is not enough.

This is part 5 of Learning AI Engineering in Public.

One-sentence answer: RAG retrieves the best documents first, then asks the LLM to answer only from that context — or refuses before generation when nothing is close enough.

The pipeline

Part 4 stopped at ranked documents. This post adds generation. The code lives in rag.ts. Same seven facts. Same embeddings. New pieces: a threshold bouncer, a grounded prompt, citations, and a model fallback list.

Analogies again:

• Retrieval still picks the 3 most relevant library books.
• The 0.6 threshold is the bouncer at the door. If the best book is not relevant enough, we never ask the LLM.
• The grounded prompt says: answer only from those books. If the answer is not on the page, say you do not know.

Shared pieces: documents, embed, retrieve

const documents = [
  "Yeremia builds PrimaKu's web platform — clinician-facing UI and integrations on a pediatric health product whose consumer app has 1M+ downloads.",
  "On Mazecare, Yeremia shipped FHIR-based clinic flows and AI-assisted clinical notes.",
  "At KiriminAja, Yeremia built high-volume logistics UI using Web Workers and virtualization.",
  "Yeremia's main frontend stack is Next.js, React 19, TypeScript, and Tailwind CSS.",
  "Yeremia deploys with Docker, GitHub Actions CI/CD, and Vercel.",
  "Yeremia is building a construction project platform in Indonesia with his brother.",
  "Yeremia is currently learning RAG, embeddings, and AI agents.",
];
 
const documentVectors = await embed(documents, "RETRIEVAL_DOCUMENT");
 
async function retrieve(query: string) {
  const [queryVector] = await embed([query], "RETRIEVAL_QUERY");
  return documents
    .map((text, index) => ({
      id: index + 1,
      text,
      score: cosineSimilarity(queryVector, documentVectors[index]),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, TOP_K);
}

What this block does:

• Documents and embed / cosine similarity match part 4 (RETRIEVAL_DOCUMENT vs RETRIEVAL_QUERY).
• retrieve returns { id, text, score }. The id is index + 1 so citations can say [1], [2], [7].
• TOP_K is 3. Only three chunks become context.

Constants for the bouncer and the chat call

const EMBEDDING_MODEL = "gemini-embedding-001";
const CHAT_MODEL = "gemini-3.8-flash";
const SIMILARITY_THRESHOLD = 0.6;
const TOP_K = 3;

What this block does:

• SIMILARITY_THRESHOLD is 0.6 — the bouncer's cutoff from the part 4 lesson.
• TOP_K caps context size.
• CHAT_MODEL names the preferred chat model. In practice the fallback list below is what kept the demo running.

When backoff is not enough: model fallback

I still had withBackoff from part 2 — exponential backoff for 503 and 429. That was not enough when my preferred model kept returning retryable errors. Waiting longer on the same model did not help.

I listed available models with ai.models.list() and added generateWithFallback(), which tries CHAT_MODELS in order and moves to the next model on 503 or 429.

const CHAT_MODELS = [
  "gemini-3.8-flash",
  "gemini-3.5-flash",
  "gemini-3.1-flash-lite",
];
 
async function generateWithFallback(prompt: string) {
  let lastError: unknown;
  for (const model of CHAT_MODELS) {
    try {
      const stream = await ai.models.generateContentStream({
        model,
        contents: prompt,
        config: { temperature: 0.2 },
      });
      console.log(`(using ${model})`);
      return stream;
    } catch (error: any) {
      const isRetryable = error?.status === 503 || error?.status === 429;
      if (!isRetryable) throw error;
      console.log(`(${model} busy, trying next model...)`);
      lastError = error;
    }
  }
  throw lastError;
}

What this block does:

• Tries each model in order with generateContentStream and temperature 0.2.
• On 503 or 429, logs that the model is busy and tries the next one.
• On any other error, throws immediately — same idea as part 2: do not retry a bad request or a bad key.
• If every model fails with a retryable error, it throws the last error.

Real run: my preferred model stayed busy, so fallback picked gemini-3.5-flash. The same three Google flash IDs power auto-fallback on /lab.

answer(): threshold, grounded prompt, stream, sources

async function answer(question: string) {
  console.log(`\nQuestion: ${question}`);
  const results = await retrieve(question);
 
  if (results[0].score < SIMILARITY_THRESHOLD) {
    console.log(
      "Answer: I don't have that in my portfolio yet. Feel free to contact me directly!",
    );
    return;
  }
 
  const context = results
    .map((result) => `[${result.id}] ${result.text}`)
    .join("\n");
 
  const prompt = `You are "Ask Yeremia", an assistant on Yeremia's portfolio website.
Answer the question using ONLY the context below.
If the context does not contain the answer, say you don't know. Never make things up.
Cite sources using their numbers, like [1] or [2].
 
Context:
${context}
 
Question: ${question}`;
 
  const stream = await generateWithFallback(prompt);
 
  process.stdout.write("Answer: ");
  for await (const chunk of stream) {
    process.stdout.write(chunk.text ?? "");
  }
  console.log(
    `\nSources: ${results.map((result) => `[${result.id}] ${result.score.toFixed(3)}`).join(", ")}`,
  );
}

What this block does, in order:

1. retrieve(question) gets the top-3 { id, text, score } rows.
2. If the best score is below 0.6, print a refusal and return. The LLM is never called. That is cheaper and safer: no tokens spent, no chance to invent a recipe.
3. Otherwise build context as numbered lines like [1] text. Those numbers are the citation handles.
4. Build a grounded prompt: answer using ONLY the context; if it is not there, say you do not know; never make things up; cite like [1].
5. Stream the answer through generateWithFallback.
6. Print the sources with their similarity scores so a reader can check the evidence.

The refusal message's tone can be made friendlier later. The important part is refusing before generation.

Real outputs

Healthcare

Question: "Has he worked in healthcare?"

The answer was Yes — it cited PrimaKu [1] and Mazecare [2]. KiriminAja was still in the retrieved top-k (same pattern as part 4), but the LLM ignored it for the written answer. Retrieval can surface a weak neighbor; the grounded prompt is what keeps that neighbor out of the sentence.

Learning

Question: "Is he learning anything new?"

The answer pointed at document [7]: currently learning RAG, embeddings, and AI agents.

Rendang

Question: "What is a good rendang recipe?"

Refused before the LLM was called. The top similarity score stayed under 0.6, so the bouncer stopped the request. No generation, no invented recipe.

University

Same question as the hook: top score 0.601, no university in the facts, honest I don't know from the grounded prompt.

Two layers, one rule

Stack defenses, then measure the cutoff on your own Q&A pairs — the number is not portable.

• Retrieval + top-k: embed with RETRIEVAL_DOCUMENT / RETRIEVAL_QUERY, keep the three closest chunks (healthcare cites PrimaKu and Mazecare; rendang never gets that far).
• Threshold before generation: if the best score is below 0.6, refuse and skip the LLM — no tokens, no invented recipe.
• Grounded prompt + citations: when a question barely passes (university at 0.601), only use context and numbered sources still block a made-up school; print scores so claims are checkable.
• Temperature + streaming: answer at 0.2 through generateWithFallback after withBackoff on a single model is not enough.
• Eval per dataset: this demo used top-3 and 0.6; live /lab runs the same pattern on a bigger corpus at top-5 and 0.45 — exactly why you need evals before you trust one threshold everywhere.

That is the end of this series: What an LLM Actually Is → My First LLM API Call → Embeddings and Cosine Similarity → Semantic Search → this pipeline. Try the questions yourself on /lab.

What I would improve next: a small eval set for the similarity cutoff and a friendlier refusal line — not a promise of another post, just the next lever I would pull.