LLM inference too slow. Users churning.Optimised to <1s this session.

PM assigned in 10 minutes. AI infrastructure engineer starts diagnosis within the hour. An 8-second latency is significantly above the 2-3 second threshold where users begin abandoning. The engineer diagnoses the bottleneck layer: if you are using the OpenAI or Anthropic API, the bottleneck is almost always the prompt length (a 4,000-token prompt takes significantly longer to process than a 1,000-token prompt fix: prompt compression and context pruning) or the output length (requesting more tokens than necessary fix: add a max_tokens constraint). If you are self-hosting, the bottleneck is inference configuration batch size too small, quantisation not enabled, KV cache configuration suboptimal, or GPU underprovisioned for the model size. The engineer implements the appropriate optimisation and measures P50 and P99 latency before and after.

Yes. Significant latency reductions (50-80%) are achievable without changing your LLM model through infrastructure and prompt optimisation alone. The techniques, in order of impact for API-based deployments: implementing streaming responses (users see the first token in under 1 second rather than waiting for the complete response dramatically improves perceived responsiveness even if total generation time is unchanged), prefix caching (for prompts with a shared system prompt or few-shot examples, the API caches the prefix KV state and only charges/processes the new tokens reduces latency by 40-60% for cached prefixes), prompt compression removing redundant instructions and context, and output length constraints preventing the model from generating more tokens than necessary for your use case.

Both. For OpenAI API optimisation: the engineer implements streaming responses (reducing perceived latency dramatically), evaluates which model tier is appropriate (GPT-4o-mini runs 3-5x faster than GPT-4o for tasks where the quality difference is acceptable), implements prompt caching for your system prompt and few-shot examples (30-60% latency reduction for cached prefixes), compresses your prompt to the minimum tokens needed for accuracy, adds a max_tokens constraint to prevent over-generation, and evaluates whether fine-tuning on a smaller faster model (gpt-4o-mini fine-tune) can match your quality requirements at lower latency. For self-hosted: deploys vLLM with continuous batching, configures appropriate GPU instance types, enables quantisation (FP8 or INT4), and tunes KV cache sizing.

Latency targets for customer-facing AI: perceived responsiveness (time to first token when the user sees the response start appearing) should be under 1 second for a good experience and under 2 seconds to avoid frustration. Total response time (time to complete generation) matters less when streaming is implemented because users read as the response generates rather than waiting for completion. For non-streaming responses (a single API call returning a complete answer), the target is under 3 seconds for simple queries and under 5 seconds for complex multi-step queries before users start showing dissatisfaction signals. Document processing or complex reasoning tasks can tolerate up to 10-15 seconds if the UI shows a progress indicator. The engineer designs for the streaming path first since it is architecturally harder to retrofit than to build in from the start.

Techniques ranked by typical latency reduction: first, streaming responses does not reduce actual generation time but reduces perceived latency from 8 seconds to under 1 second (time to first token); second, prompt prefix caching reduces time to first token by 40-60% for prompts with shared system prompt or few-shot examples by caching the KV state of the repeated prefix; third, speculative decoding uses a small draft model to predict tokens and the large model to verify, reducing forward passes by 3-4x for typical output lengths; fourth, KV cache quantisation storing the attention cache in INT8 or FP8 instead of FP16 doubles effective context capacity and reduces memory bandwidth bottlenecks; fifth, model quantisation AWQ or GPTQ 4-bit reduces model weights by 4x, enabling larger batch sizes on the same GPU and increasing throughput.

QuickHire Instant is built for immediate engineering execution - book a vetted developer and dedicated PM in under 10 minutes, pay per session with no contracts. QuickHire Enterprise is for long-term team partnerships: dedicated developers, managed engineering pods, staff augmentation at scale, and enterprise-grade contracts (MSA, NDA, SLA) with a dedicated account manager. Both models use the same vetted talent network. Visit enterprise.quickhire.services to explore Enterprise engagements.