Integrating a large language model into a production system is a fundamentally different engineering discipline from calling a traditional API. The response is probabilistic, the latency is high, the cost is per-token, and failure modes are qualitative rather than binary. Teams that treat LLM integration like REST API integration ship systems that work in demos and fail in production.
This guide covers what production-grade LLM integration actually requires.
Choosing the Right Model
Model selection is the first significant decision, and it is not simply a question of "which model is most capable." The relevant dimensions are:
Capability vs cost trade-off. The frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) deliver the best reasoning quality but cost 5–20x more per token than smaller models (GPT-4o-mini, Claude Haiku, Gemini Flash). For most production use cases, the right architecture uses the cheapest model that meets the quality bar for each specific task — not the most powerful model for everything.
Context window requirements. If your use case involves processing long documents, multi-turn conversations with extensive history, or complex multi-step reasoning, context window size matters. Models like Claude 3.5 (200K tokens) and Gemini 1.5 Pro (1M tokens) handle long-context tasks that GPT-4o (128K) cannot.
Function calling / tool use reliability. Not all models are equally reliable at structured tool use. Test function calling accuracy against your specific tools with your specific prompts before committing to a model for an agentic use case.
Data residency and compliance. If you are processing personally identifiable information, check whether the model provider's data processing terms meet your compliance requirements. Some enterprise use cases require models deployed in specific geographic regions or on isolated infrastructure.
The Prompt Engineering Layer
Prompts are code. They should be version-controlled, tested, and deployed with the same rigour as application code. Several practices separate production prompts from quick experiments:
System prompt architecture. A well-structured system prompt establishes the model's identity, capabilities, constraints, and output format requirements. Break it into clearly labelled sections rather than one continuous paragraph. Models respond better to structured instructions.
Output format contracts. Instruct the model to respond in a specific JSON schema and use structured outputs (supported by OpenAI, Anthropic, and Google) to enforce the schema at the API level. Never parse free-text LLM output with regex in production — it will break on edge cases.
const response = await openai.chat.completions.create({
model: "gpt-4o",
response_format: { type: "json_object" },
messages: [
{ role: "system", content: systemPrompt },
{ role: "user", content: userInput }
]
});Few-shot examples in the prompt. For complex extraction or classification tasks, include 2–5 examples of correct input/output pairs in the system prompt. This consistently improves accuracy more than lengthening the instruction.
Chain-of-thought for reasoning tasks. For tasks requiring multi-step reasoning, instruct the model to think step by step. The extended reasoning traces in Claude 3.5 Sonnet and o1/o3 models make this even more effective for complex problems.
Building Agent Architectures
Single-turn LLM calls handle simple tasks. Agents — systems where the model can use tools, receive results, and continue reasoning — handle complex multi-step workflows.
The ReAct Pattern
The ReAct (Reason + Act) pattern is the most reliable agent architecture for production use:
- Model receives a task and a set of available tools
- Model reasons about what action to take
- Model calls a tool
- Tool result is returned to the model
- Model reasons again — continues until the task is complete or a stop condition is reached
Implementation with the Anthropic SDK:
const tools = [
{
name: "search_database",
description: "Search the customer database for records matching a query",
input_schema: {
type: "object",
properties: {
query: { type: "string" },
limit: { type: "number" }
},
required: ["query"]
}
}
];
async function runAgent(task: string) {
const messages = [{ role: "user", content: task }];
while (true) {
const response = await anthropic.messages.create({
model: "claude-3-5-sonnet-20241022",
max_tokens: 4096,
tools,
messages
});
if (response.stop_reason === "end_turn") break;
if (response.stop_reason === "tool_use") {
const toolUse = response.content.find(c => c.type === "tool_use");
const result = await executeToolCall(toolUse);
messages.push({ role: "assistant", content: response.content });
messages.push({ role: "user", content: [{ type: "tool_result", tool_use_id: toolUse.id, content: result }] });
}
}
}Multi-Agent Orchestration
For complex workflows requiring specialised expertise, a single general-purpose agent is less effective than a team of specialised agents coordinated by an orchestrator.
A typical pattern:
- Orchestrator agent: Receives the high-level task, decomposes it, and delegates to specialists
- Specialist agents: Each has a specific set of tools and a system prompt optimised for a specific domain (data extraction, validation, writing, calculation)
- Critic/validator agent: Reviews the output of specialist agents before results are returned
This architecture allows parallelisation of independent subtasks and produces more reliable results on complex multi-step workflows.
Cost Management
LLM API costs are a function of input tokens + output tokens × price per token. At scale, unmanaged costs become a significant operational burden.
Caching. If the same or similar prompt is sent multiple times, caching the response avoids redundant API calls. Anthropic offers prompt caching (cache the system prompt across calls). For application-level caching, semantic similarity search against a vector store of previous calls can retrieve cached responses for semantically equivalent inputs.
Context management. The most common cost blowout in conversational systems is unbounded context growth. As conversations grow, pass only a summarised version of the history (generated by a cheap model) rather than the full transcript.
Model routing. Route tasks to the cheapest model that can handle them. A simple classification task should use Haiku or 4o-mini. A complex reasoning task uses Sonnet or GPT-4o. Only use frontier reasoning models (o1, Claude 3.5 Sonnet extended thinking) for tasks that genuinely require it.
Token budget guards. Set `max_tokens` appropriately for the expected output. An extraction task that should return a 200-token JSON object should not have a 4096 token budget — a runaway response is a cost anomaly and usually indicates a prompt error.
Observability
Without observability, debugging production LLM failures is guesswork. You need structured logs that capture:
- The exact prompt sent (system + user messages)
- The model and parameters used
- Input token count, output token count, and cost
- Latency (time to first token and total response time)
- The raw response and parsed output
- Whether the output passed validation
- The trace ID linking the LLM call to the user request
Tools like LangSmith, Langfuse, and Helicone provide LLM-specific observability dashboards out of the box. At minimum, emit structured JSON logs with these fields to whatever observability stack you already operate.
Error Handling and Fallbacks
LLM calls fail. Rate limits, server errors, and timeouts are all common. Production systems need:
Exponential backoff with jitter for rate limit errors (HTTP 429)
Response validation — if the model returns invalid JSON or fails a schema check, retry with an explicit correction prompt rather than propagating the error
Model fallback — if the primary model is unavailable, fall back to a secondary model. Maintain a provider abstraction layer that makes this transparent to the calling code
Graceful degradation — define what the system does when LLM calls are unavailable. For some use cases, a rule-based fallback is acceptable. For others, the feature should be transparently unavailable rather than silently incorrect.
Building for the Long Term
LLM integration is not a fire-and-forget operation. Models are updated, deprecated, and repriced. Evaluation pipelines — automated test suites that run your prompts against a golden dataset and measure accuracy — are the only way to detect regressions when models change.
Invest in evaluation infrastructure from the start. It is far cheaper to detect a quality regression before deployment than after.