What is model serving?

Definition: Model serving is the infrastructure and process of hosting a trained machine learning model behind an API so applications can send it inputs and receive predictions reliably, quickly and at scale.

What a serving layer handles

Loading model weights onto CPUs or GPUs, exposing a REST or gRPC endpoint, batching requests for throughput, autoscaling with traffic, versioning, health checks and logging inputs and outputs.

Serving patterns

Real-time online serving answers each request in milliseconds to seconds. Batch serving scores large datasets on a schedule. Streaming serving scores events as they arrive. For LLMs, specialized servers optimize memory and token throughput, and quantization can cut hardware needs.

Model serving vs inference

Inference is the act of running a model on an input. Model serving is the production system that makes inference available as a dependable service, with the monitoring, rollout and rollback practices of MLOps.

Self-hosted or managed

Managed endpoints from cloud providers reduce operations work. Self-hosting on Kubernetes gives more control over cost, latency and data location, and is common for open-weight models.

How Ryz helps

Ryz MLOps engineers build serving stacks in your cloud, working on your team.

Related

• MLOps services
• Hire MLOps engineers
• AI inference
• MLOps