LLM Interview Questions (2026): 15 Debugging Scenarios With Answers

Key Takeaways

• Some applied LLM interviews start from a production symptom ("answers are cut off", "the bill doubled") and ask you to find the cause, rather than asking for a definition. This guide is built around that format.
• This guide gives 15 scenario and debugging questions, each tagged junior, mid or senior, with what the interviewer is testing, an answer outline and one follow-up.
• The questions cover long context, retrieval misses, stale indexes, truncation, logprobs, citations, speculative decoding, flaky extraction, chunk boundaries, JSON failures, judge drift, cost spikes, silent model changes, prompt caching and long chats.
• Every vendor or research claim links to the official documentation or the paper. Where an answer is engineering judgment, we say so.
• None of the 15 is a main question in our paid interview kit. If you want broader questions too, the free sample PDF has 12 of the kit's 120, with full answers.

LLM interview questions for application engineers often go beyond "explain temperature" to "here is what users are seeing, what do you check first?" The interviewer wants to watch you narrow down a failure in a system made of a prompt, a retriever, a model call and some parsing code. Below are 15 of those questions with the answer shape that tends to work: name the likely causes, say how you would tell them apart, then say what you would change.

Each question has a level: junior, mid or senior, the depth where a good answer starts. The levels are our editorial judgment, not data from any employer. If you are preparing for the broader loop (rounds, company types, coding), start with our AI engineer interview questions guide and come back here for the debugging round.

LLM Interview Questions on Context and Retrieval

1. The answer ignores a fact we know is in the middle of a long prompt. Why, and what do you do? (mid)

What it tests: whether you know that a long context window is not the same as uniform attention to everything in it.

Answer outline:

• Name the known effect. The "Lost in the Middle" study found that performance "is often highest when relevant information occurs at the beginning or end of the input context" and "significantly degrades when models must access relevant information in the middle of long contexts" (Liu et al., 2023). That was reported for the models it tested, so measure your own.
• Reproduce it: move the fact to the start or end and see if the answer changes. That separates position from a retrieval or wording problem.
• Fix it with less context, not more: retrieve fewer, better passages, put the most relevant ones at the edges, and restate the question after long documents.
• Note that single-fact tests flatter long context. Google's Gemini docs say needle tests "consider the most basic setup, where you have a single needle", and that performance "can vary to a wide degree" when the model must find several items (Gemini long context).

Follow-up: How would you build an eval that catches this before users do? (Vary the position of a known fact across a set of prompts and track accuracy by position.)

2. A valid question returns top-k results that are all irrelevant. How do you detect that, and what should the system say? (mid)

What it tests: whether you treat "nothing relevant found" as a state the product handles, instead of letting the model answer from weak context.

Answer outline (engineering practice, no vendor rule applies):

• A retriever always returns k results, so "top-k came back" says nothing about relevance. You need a signal: a similarity threshold tuned on labeled queries, a reranker score, or a cheap model check of "does this passage answer the question?"
• Below the threshold, branch: ask a clarifying question, say the answer is not in the knowledge base, or route to a human. Do not pass weak passages to the model with "answer using the context", because that invites a confident wrong answer.
• Log these queries. A cluster of valid questions with no good match is a content gap, and it is the cheapest roadmap input you have.

Follow-up: How do you pick the threshold? (Label a few hundred real queries as answerable or not, then choose the cutoff from the precision you need on the "refuse" side.)

3. The policy changed last week, but the bot still quotes the old one. (mid)

What it tests: whether you can trace an answer back through the index to the source data.

Answer outline:

• Find the chunk the bot cited, check its source and timestamp. Three usual causes: the new document was never ingested, the old one was never removed, or both are indexed and the old one ranks higher.
• Indexes do not update themselves. Amazon Bedrock's docs, for example, say that "each time you add, modify, or remove files from your data source, you must sync the data source so that it is re-indexed", and that changed documents are re-parsed, re-chunked, re-embedded and re-indexed (Bedrock knowledge base sync).
• Expect lag even after a sync: the same page warns it "could take a few minutes for the vector embeddings of the newly synced data to reflect in your knowledge base" for some vector stores.
• Fix the process: trigger ingestion on document change, delete superseded versions, store an effective date in chunk metadata, and filter or boost on it.

Follow-up: How would you monitor freshness? (Track the age of the newest indexed version per source document against the source system, and alert on drift.)

Generation and Outputs

4. Answers are cut off mid-sentence, but only in production. (junior)

What it tests: whether you read the stop reason before guessing.

Answer outline:

• Log the stop field on every response. In OpenAI's Chat Completions, finish_reason is "length if the maximum number of tokens specified in the request was reached" and "stop if the model hit a natural stop point or a provided stop sequence" (OpenAI chat object). Anthropic's stop_reason is max_tokens when "the response reached your max_tokens limit" (Anthropic stop reasons).
• "Only in production" points to a difference in inputs or config: longer real inputs, a lower token limit in the prod config, or a stop sequence that appears in real text but not in test data.
• Reasoning models add a twist: the output cap includes reasoning tokens, and OpenAI notes (for the Responses API) the limit can be reached "before any visible output tokens are produced" (OpenAI reasoning guide).
• Fix: raise the limit or continue the response, and treat a truncated tool call as a failure. Anthropic's docs say a response with an incomplete tool use block needs a retry "with a higher max_tokens value".

Follow-up: What would you alert on? (The share of responses ending on the length limit, per endpoint.)

5. Product wants to flag low-confidence answers using logprobs. What do you tell them? (senior)

What it tests: whether you know what token probabilities measure and what they do not.

Answer outline:

• Define it: "a logprob is log(p), where p = probability of a token occurring at a specific position" (OpenAI logprobs cookbook).
• It works best where the answer is a short label: a classification or yes/no, where one token carries the decision. For long free text, the per-token numbers are hard to turn into one score.
• Confidence is not correctness. The same cookbook says confidence "measures certainty in the classification, not whether the classification is correct", and advises validating thresholds "with task-specific evals".
• Propose a plan: pick a threshold on labeled data, measure how often flagged answers are actually wrong, and only then ship the flag.

Follow-up: What if the provider or model you use does not return logprobs? (Use a separate verifier step, or agreement across several samples, and evaluate it the same way.)

6. The model cites a source that does not contain the claim. (senior)

What it tests: whether you can tell "the citation points somewhere real" from "the citation supports the sentence".

Answer outline:

• Check how citations are produced. If the model writes them as free text, they can be invented. Anthropic's citations feature returns structured pointers, and its docs say they "are guaranteed to contain valid pointers to the provided documents" (Anthropic citations).
• A valid pointer still does not prove support. Your application has to check: does the cited text actually entail the claim? Run an entailment check (a small model or a judge prompt) on each claim and its cited span.
• Look at retrieval too: if the right passage was never retrieved, the model cites the nearest thing it has.
• Product choice: drop unsupported sentences, mark them, or refuse when too few claims are supported.

Follow-up: How do you measure citation quality over time? (Sample answers weekly, label each claim as supported or not, and track the unsupported rate.)

7. Why does speculative decoding cut latency without changing the output distribution? (junior)

What it tests: whether you understand the idea well enough to explain it simply.

Answer outline:

• A small, fast draft model proposes several tokens. The large model checks all of them in one forward pass, which costs about the same as generating one token, and keeps the accepted prefix.
• The acceptance rule is what keeps the output distribution the same. Chen et al. describe "a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics" (Chen et al., 2023).
• Gains depend on how often the draft is right. The papers report 2-3x on T5-XXL with "identical outputs" (Leviathan et al., 2022) and 2-2.5x on Chinchilla 70B. Those are their test setups, not a general figure.

Follow-up: When would it help least? (When the draft model often disagrees with the large one, for example on unusual domains, so most proposals are rejected.)

Pipelines in Production

8. The same extraction prompt returns different values run to run. (junior)

What it tests: whether you treat model output as non-deterministic and design around it.

Answer outline:

• First, find out whether the input is really the same: templating, retrieved context or document order often differ between runs. Diff the exact prompts.
• Then reduce variation: lower the sampling temperature, give a strict output schema, and add worked examples of tricky fields.
• If the field is genuinely ambiguous in the source, more prompting will not fix it. Make the schema allow "unknown" and send those cases for review.
• Do not promise identical outputs to downstream systems unless you have measured it on your model and settings. Validate every output instead.

Follow-up: How would you quantify flakiness? (Run each test document N times and report the share of fields that change.)

9. Retrieval finds the right document, but the fact is split across two chunks. (mid)

What it tests: whether you can trace one miss to chunk boundaries.

Answer outline:

• Inspect the chunks for the failing query. If the fact ("the limit is" ... "50 per day") straddles a boundary, neither chunk matches well alone.
• Fixes, cheapest first: add overlap between chunks, split on structure (headings, list items, table rows) instead of a fixed size, or retrieve the neighbors of a hit chunk.
• Re-run your retrieval eval after the change. Fixes like overlap trade one failure for another (more near-duplicate results).

Follow-up: How do you find other queries with the same problem? (Label answer spans in your eval set and count the cases where the span crosses a chunk boundary.)

10. About 2% of structured responses fail to parse in production. (mid)

What it tests: triage: splitting one failure rate into its causes before fixing any of them.

Answer outline:

• Sort the failures into buckets: truncated JSON (check the stop reason, as in question 4), valid JSON with the wrong shape, extra prose around the JSON, or a refusal instead of an answer.
• Truncation means a bigger limit or a smaller output. Wrong shape means the provider's schema-constrained output mode where it exists, or validation with one repair retry. Refusals need their own handling path, not a parser fix.
• Count the buckets before and after each fix, so you know which change moved the 2%.

Follow-up: Is one retry on parse failure safe? (Yes, if the call is idempotent and you cap retries. Log retried requests so the retry rate itself is visible.)

11. LLM-judge scores keep rising, but user thumbs-down is flat. (senior)

What it tests: whether you notice when an eval stops measuring what users care about.

Answer outline (engineering judgment):

• Suspect the eval first. The eval set may no longer look like real traffic, or the prompts may have been tuned to the judge (for example, longer answers the judge likes).
• Check agreement: have people label a fresh sample of real conversations and compare with the judge. Low agreement means the judge needs a new rubric or examples.
• Read the thumbs-down conversations and group them by cause. Add those cases to the eval set so the next score reflects them.

Follow-up: How often do you refresh the eval set? (On a schedule, and whenever a new product surface or user group launches.)

12. The bill doubled this week, and traffic did not. (mid)

What it tests: whether you can find a cost regression from per-request data.

Answer outline:

• Break cost into requests × tokens per request × price, by endpoint and model. One of those moved.
• The usual suspects: a retry loop, an agent that calls tools more times per task, a prompt or retrieved context that grew, chat history that is no longer trimmed, or traffic routed to a more expensive model.
• If you use prompt caching, check that cache reads did not fall (question 14).
• Fix the cause, then add budgets: alerts on tokens per request and requests per user, and a cap on agent steps.

Follow-up: What would you log so this takes minutes next time? (Per request: model, input and output tokens, cached tokens, retries, and the feature that made the call.)

13. Behavior changed overnight, and we deployed nothing. (senior)

What it tests: whether you know what can change outside your codebase.

Answer outline:

• List what is not in your deploy: the model behind the ID you call, retrieved data, upstream tools and APIs, and the traffic itself.
• Check which model ID you call. With Anthropic, for models before the 4.6 generation "the alias is a convenience pointer that resolves to the dated ID", while "every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on" (Anthropic models overview). Other providers have their own alias rules, so check yours.
• Compare yesterday's and today's logged prompts and responses for the same inputs. Run your eval set now and compare with the last stored run.
• Prevention: pin model versions in production, run the eval set on a schedule, and log the model version returned with each response.

Follow-up: How do you upgrade a pinned model safely? (Run the eval set on the new version, then shadow or canary real traffic before switching.)

14. The prompt cache hit rate fell to near zero after a deploy. (mid)

What it tests: whether you know that caching works on an exact prefix.

Answer outline:

• Cache hits need an identical prefix. Anthropic's docs say "cache hits require 100 percent identical prompt segments", and the prefix is built in the order tools, system, then messages, with "changes at each level" invalidating "that level and all subsequent levels" (Anthropic prompt caching).
• Diff the prompt before and after the deploy. Common causes: a timestamp, request ID or user name moved into the system prompt, tool definitions reordered, or a new tool added.
• Confirm with the usage fields: cache_read_input_tokens and cache_creation_input_tokens show reads and writes per request.
• Also check traffic timing: Anthropic's default cache lifetime is 5 minutes, so a quieter endpoint can miss even with an unchanged prefix.

Follow-up: How do you keep this from regressing? (Keep stable content first and per-request content last, and alert on the cache read share.)

15. Long chat sessions get worse after about 20 turns. (mid)

What it tests: conversation-history management.

Answer outline:

• Check what you send each turn. Full history grows cost and latency every turn, and important early instructions end up in the middle of a long context, which is where question 1's effect applies.
• Check for truncation that drops the start of the conversation, including the system prompt or the user's original goal.
• Fixes: keep the system prompt and a running summary of decisions at the top, keep only the last few turns verbatim, and retrieve older turns when they are relevant.

Follow-up: How do you test a summary strategy? (Replay long real sessions with and without it and compare answer quality on the late turns.)

More LLM Interview Questions to Practice

Debugging rounds sit next to concept and design questions. For those, see our AI engineer interview questions guide (13 worked answers by level and company type). The AI Engineer Interview Kit has 120 more questions with full answers and follow-ups across 12 chapters, including RAG, evals, inference and agents. The free sample PDF has 12 of them.

How We Wrote This

We wrote these 15 questions for this page. None is a main question in the paid kit, and we checked that the question and answer text does not repeat the kit. Levels are our editorial judgment. Vendor and research claims link to pages we read on November 1, 2026. Provider documentation changes without dates, so check limits and field names on the provider page before an interview. Research results are reported for the models and setups each paper tested.

FAQ

What LLM interview questions do application engineers get?

Often a mix: concept questions on tokens, sampling and embeddings, plus some scenarios that start from a production symptom such as truncated answers, stale citations or a cost spike and ask you to find the cause.

How should I answer an LLM debugging question in an interview?

Name the likely causes, say which log or experiment tells them apart (stop reasons, retrieved chunks, token counts, cache usage fields), then say what you would change and how you would confirm the fix with an eval or a metric.

Do LLM interviews ask about research papers?

Sometimes. Lost in the Middle and speculative decoding are ideas worth knowing because they explain behavior engineers see in production: position effects in long prompts, and latency gains that preserve the target model's output distribution.

Where can I get more LLM interview questions with answers?

The free AI Engineer Interview Kit sample has 12 questions with full answers and follow-ups. The full kit has 120 across 12 chapters.