LLM applications that survive month three
Integrating a model takes an afternoon. Making the output reliable, the cost predictable and the latency acceptable at real volume is what separates a product from an experiment.
Key takeaways
- Integrating a language model takes an afternoon; what decides whether the product survives is structured output, cost control and latency at real volume.
- Task-based routing — a cheap model classifies and summarizes, a capable one writes the final answer — is the single biggest lever on cost without touching perceived quality.
- Choosing between prompting, RAG and fine-tuning moves the budget by an order of magnitude: RAG when the model does not know the information, fine-tuning only for tone or domain vocabulary.
- Quarl works with OpenAI, Anthropic, Google, Azure OpenAI and Amazon Bedrock behind an abstraction layer, so switching providers is configuration rather than a quarter of work.
Operating track record
Three years operating the system in production — not one delivery and an exit.
The most frequent failure in RAG architectures, solved with structured metadata at ingestion.
The proposal arrives within 48 hours with fixed scope, price and date. If scope changes, it is quoted separately and approved first.
The hard part is not calling the API
Anyone can wire a model to a form in an afternoon. What decides whether the product survives is everything else: that the output always has the shape your system expects, that cost per request does not explode as usage grows, that latency stays tolerable, that you can switch providers without rewriting the application, and that you can measure whether a change improved or degraded the answers.
None of that shows up in the demo. All of it shows up once there are real users.
The pieces that make the difference
- Guaranteed structured output. The model returns JSON that validates against a schema, with retries and repair when it does not comply. Your system never receives something it cannot process.
- Tool use and function calling. The model queries your APIs and databases instead of answering from memory, with limits and validation on every call.
- Task-based routing. A cheap model classifies, summarizes and filters; a capable one generates the final answer. It is the single biggest lever on cost without touching perceived quality.
- Prompt caching and context trimming. Reduces cost and latency in applications with long, repeated instructions.
- Streaming. The answer starts appearing immediately. It does not reduce real latency but it completely changes the user’s perception.
- Provider abstraction. Moving from OpenAI to Claude or to a self-hosted model should not cost a quarter of engineering time.
- Prompt injection defenses. Instruction hierarchy, strict separation between system and user content, and input and output filtering.
Prompting, RAG or fine-tuning
The most frequent question, and the one that wastes the most money when answered badly.
| If your problem is… | What applies |
|---|---|
| The model does not know your information | RAG |
| The model does not answer in the right format | Structured output |
| The tone or style is not yours | Prompting first, fine-tuning later |
| A very specific domain with its own vocabulary | Fine-tuning |
| Cost is unsustainable | Routing and caching |
| Nobody knows whether it works | Evaluation layer |
Ranges and timelines
| Scope | What it includes | |
|---|---|---|
| Integration into your product | One LLM-powered feature inside an existing application, with evaluation and cost control. | 3–5 weeks |
| LLM-native product | Full application: backend, orchestration, interface, evaluation and observability. | 6–12 weeks |
| Cost and latency optimization | Routing, caching, context trimming and before-and-after measurement. | 2–4 weeks |
| Fine-tuning | Dataset preparation, training, evaluation against the baseline. | 3–6 weeks |
Providers and tooling
Related services
RAG systems
Chunking, reranking and hybrid search, evaluated with recall@k and NDCG.
View serviceAI agents
LangGraph orchestration, durable state and human approval on steps with consequences.
View serviceLangChain and LangGraph
Implementation and observability instrumented with LangSmith.
View serviceAI consulting
Architecture, evaluation criteria and cost per query before writing code.
View serviceFrequently asked questions
Which model should we use?
It depends on the task, and it is almost always several within the same system. For classifying, extracting and summarizing, a cheap model is enough and costs a fraction. For complex reasoning or the final user-facing answer, a capable one. We work with OpenAI, Anthropic, Google, Azure OpenAI and Amazon Bedrock, and the choice is justified with numbers in the proposal, not by fashion.
How do you control cost?
Four levers: task-based routing, context trimming so you are not sending everything just in case, prompt caching for long repeated instructions, and hard limits per request and per user. In systems we have operated this is the difference between predictable cost and cost that makes scaling impossible. We also instrument spend so you can see it without asking.
Can we switch providers later?
Yes, and we design for it from the start. The application talks to an abstraction layer rather than directly to one provider’s API, so switching models is configuration plus an evaluation run to confirm quality holds. Without that layer, migrating costs a quarter.
What is structured output and why does it matter?
It is forcing the model to return data in an exact shape — JSON that satisfies a schema — instead of free text. It matters because your system has to process the response: if it occasionally returns a field under a different name or a number as a string, the integration breaks in production in ways that are hard to reproduce. We validate against a schema and retry with repair when it does not comply.
What is prompt injection and how do you handle it?
It is when someone embeds instructions inside content the model processes — a document, an email, a message — to make it ignore its rules. Mitigated with instruction hierarchy in the system prompt, strict separation between system and user content, input and output filtering, and limiting which tools the model can invoke. In agent systems this stops being optional.
When does fine-tuning make sense?
When you need a tone, format or vocabulary that prompting cannot achieve consistently, and you have at least a few hundred high-quality examples. It is not for injecting updatable knowledge — that is what RAG is for. In practice, nine out of ten cases that arrive asking for fine-tuning are better solved with RAG and better prompting, and we say so.
How do you know an improvement worked?
With a set of real cases and their expected outcomes, run before and after each change. We measure answer quality, cost per request and latency. Without that, "we improved the prompt" is an opinion — and that is exactly how systems degrade without anyone noticing.
Is this useful for processing documents at volume?
Yes, and it is one of the clearest-return use cases: extracting data from invoices, contracts or forms in variable formats where rigid templates fail. It combines with schema validation and human review on low-confidence cases. We also cover this from the automation side.
Book 15 minutes
Tell us what you are building, or what stopped working. You leave the call with a concrete answer: it can be fixed, it can be built, or it isn't worth it.