Introduction

Model deployment is the process of taking a trained model and making it available for actual use — serving predictions to real users, applications, or systems — moving it from a training environment (like a notebook or training script) into a production setting where it reliably handles real-world requests. Building on the previous topic's coverage of saving models, deployment addresses what comes next: how a saved model actually gets put to work.

Deployment introduces an entirely different set of concerns than training — reliability, latency, scalability, and monitoring — that don't come up during model development, making it a distinct discipline in its own right, often overlapping with software engineering and infrastructure work as much as machine learning itself.

Why Does Model Deployment Matter?

Model deployment helps to:

• Turn a trained model into something that actually delivers value to real users
• Serve predictions reliably and efficiently at whatever scale is needed
• Bridge the gap between machine learning experimentation and real production software
• Ensure a model continues performing well after it's released into the real world
• Support different deployment targets — servers, mobile devices, browsers — depending on the use case
• Complete the full lifecycle from training through saving to actual real-world use

The Deployment Landscape

Whiteboard

Common Deployment Targets

1. Server-Side Deployment (TensorFlow Serving)

Hosting a model on a server, typically accessed via an API, to serve predictions to applications or other services in real time.

docker run -p 8501:8501 \
  --mount type=bind,source=/path/to/saved_model,target=/models/my_model \
  -e MODEL_NAME=my_model tensorflow/serving

This runs a production-grade serving container that exposes the model through a REST/gRPC API, handling incoming prediction requests.

2. Mobile and Edge Deployment (TensorFlow Lite)

Converting a model into a lightweight format optimized for running directly on mobile phones, embedded devices, or other resource-constrained hardware.

3. Browser Deployment (TensorFlow.js)

Converting a model to run directly within a web browser using JavaScript, enabling client-side inference without a server round-trip.

tensorflowjs_converter --input_format=tf_saved_model \
  saved_model_directory web_model_directory

Building a Simple Prediction API

This simple example demonstrates the core deployment pattern: load a saved model once at startup, then use it repeatedly to serve predictions for incoming requests — directly reflecting the training-vs-inference distinction covered in an earlier topic, where inference happens continuously using a fixed, already-trained model.

Key Deployment Considerations

Model Optimization for Deployment

Techniques commonly used to make models more efficient for
deployment, especially on resource-constrained devices:

- Quantization: reducing numerical precision (e.g., float32 to
  int8) to shrink model size and speed up inference
- Pruning: removing less-important weights or connections
- Model Distillation: training a smaller "student" model to
  mimic a larger "teacher" model's behavior

These techniques trade some accuracy for significant gains in
speed, memory usage, and deployment feasibility, particularly
important for mobile and edge deployment via TensorFlow Lite.

Deployment Targets Compared

Real-Time Inference vs Batch Inference

Model Monitoring After Deployment

Deployment isn't a one-time event — models need ongoing
monitoring to catch issues like:

- Data Drift: real-world input data gradually differing from
  the data the model was originally trained on
- Performance Degradation: prediction quality declining over time
- System Issues: latency spikes, errors, or downtime

This connects directly back to the Model Evaluation Basics
topic — the same metrics used during development remain
relevant for tracking a model's ongoing health in production.

Key Properties of Model Deployment

• Deployment moves a trained model from a development environment into real-world, production use.
• Different deployment targets (server, mobile, browser) require different formats and optimization approaches.
• TensorFlow Serving, TensorFlow Lite, and TensorFlow.js each address a distinct deployment scenario.
• Techniques like quantization and pruning help optimize models for resource-constrained deployment targets.
• Ongoing monitoring after deployment is essential, since model performance can degrade over time due to data drift.

Where Does Model Deployment Matter Most?

Advantages

• Turns machine learning work into something that delivers actual, real-world value
• Multiple deployment options accommodate very different use cases and constraints
• Optimization techniques make even large models feasible on constrained hardware
• Established tools (TF Serving, TFLite, TF.js) handle much of the deployment complexity
• Enables the full lifecycle from training through to real, ongoing usage

Limitations

• Introduces new engineering challenges (scalability, latency, monitoring) beyond core ML work
• Model optimization techniques can trade away some accuracy for efficiency
• Requires ongoing maintenance and monitoring, not just a one-time deployment effort
• Different deployment targets require learning different tools and formats
• Production issues (data drift, performance degradation) require dedicated processes to catch and address

Real-World Examples

Best Practices

• Choose a deployment target and format based on your specific latency, hardware, and scale requirements.
• Apply optimization techniques like quantization when deploying to mobile or edge devices.
• Build in monitoring from the start to catch data drift or performance degradation early.
• Plan for model versioning so updates can be rolled out without disrupting live traffic.
• Test deployment thoroughly under realistic load conditions before relying on it for production traffic.

Interview Tip

A common interview question is:

"What factors would influence your choice between deploying a model with TensorFlow Serving versus TensorFlow Lite?"

A strong answer is:

The choice comes down primarily to where the model needs to run and what constraints apply. TensorFlow Serving is designed for server or cloud-based deployment, well-suited for high-throughput prediction APIs where the model runs on infrastructure you control, with resources available to run a full-sized model efficiently. TensorFlow Lite, by contrast, is built for mobile and edge devices with limited compute, memory, and power — it typically requires the model to be converted and optimized, often using techniques like quantization, to run efficiently on that constrained hardware. If low-latency, on-device inference without server round-trips matters most, TensorFlow Lite is the better fit; for centralized, high-throughput serving, TensorFlow Serving is the standard choice.

Framing the decision around infrastructure control and hardware constraints makes your answer stronger.

Conclusion

Model deployment completes the full journey from training and saving a model to actually putting it to real-world use, with different tools — TensorFlow Serving, TensorFlow Lite, and TensorFlow.js — suited to server, mobile, and browser deployment respectively, alongside ongoing monitoring to maintain performance over time. With Model Deployment Basics now covered, the entire TensorFlow & Keras section is complete, spanning from initial installation and core tensor concepts through custom components, training, and finally real-world deployment.