LLM applications that survive real users

It depends on the task, and it is almost always several within the same system. For classifying, extracting and summarizing, a cheap model is enough and costs a fraction. For complex reasoning or the final user-facing answer, a capable one. We work with OpenAI, Anthropic, Google, Azure OpenAI and Amazon Bedrock, and the choice is justified with numbers in the proposal, not by fashion.

Four levers: task-based routing, context trimming so you are not sending everything just in case, prompt caching for long repeated instructions, and hard limits per request and per user. In the systems we have operated, those four levers are what keeps the bill predictable as usage multiplies. We also instrument spend so you can see it without asking.

Yes, and we design for it from the start. The application talks to an abstraction layer rather than directly to one provider’s API, so switching models is configuration plus an evaluation run to confirm quality holds. Without that layer, migrating costs a quarter.

It is forcing the model to return data in an exact shape — JSON that satisfies a schema — instead of free text. It matters because your system has to process the response: if it occasionally returns a field under a different name or a number as a string, the integration breaks in production in ways that are hard to reproduce. We validate against a schema and retry with repair when it does not comply.

It is when someone embeds instructions inside content the model processes — a document, an email, a message — to make it ignore its rules. Mitigated with instruction hierarchy in the system prompt, strict separation between system and user content, input and output filtering, and limiting which tools the model can invoke. In agent systems this stops being optional.

When you need a tone, format or vocabulary that prompting cannot achieve consistently, and you have at least a few hundred high-quality examples. It is not for injecting updatable knowledge — that is what RAG is for. In practice, nine out of ten cases that arrive asking for fine-tuning are better solved with RAG and better prompting.

With a set of real cases and their expected outcomes, run before and after each change. We measure answer quality, cost per request and latency. Without that, "we improved the prompt" is an opinion, and that is exactly how systems degrade without anyone noticing.

Yes, and it is one of the clearest-return use cases: extracting data from invoices, contracts or forms in variable formats where rigid templates fail. It combines with schema validation and human review on low-confidence cases. We also cover this from the automation side.