Definition: Inference is the stage where a trained model is used to produce outputs, such as a prediction, classification or generated text, from new inputs, as opposed to training, where the model learns its weights.
An application sends input to a model served through a provider API, a managed cloud service such as AWS Bedrock or Azure OpenAI, or a self-hosted server such as vLLM. For language models, inference has two phases: processing the prompt, then generating output tokens one at a time.
Training is often a one-off cost, but inference runs every time a user or process calls the model, so it usually dominates ongoing AI spend. Key metrics are latency, time to first token, throughput and cost per request.
Teams reduce cost with caching, batching, model routing, quantization and distillation, and set a latency budget per feature so the user experience stays predictable.
Ryz AWS Bedrock engineers and MLOps engineers run inference in your cloud accounts, working on your team.
Thanks — your message has been sent. We’ll get back to you soon.
Something went wrong while sending your message. Please try again or email info@ryzlabs.com.