Introduction
A tensor is PyTorch's fundamental data structure — a multi-dimensional array, conceptually similar to a NumPy array, that represents all data flowing through a PyTorch program, from raw input data to model weights to final predictions. While the concept mirrors what was covered in the earlier TensorFlow Tensors topic, PyTorch tensors have their own distinct API, conventions, and — most notably — built-in support for automatic differentiation via the requires_grad flag, which connects directly to Autograd, the next topic in this section.
Understanding PyTorch's tensor API specifically — its creation syntax, operations, and how it differs from both NumPy arrays and TensorFlow's tensors — is the essential first technical building block before working with any real PyTorch model code.
Why Do Tensors Matter in PyTorch?
Tensors help to:
- Provide a consistent data structure for representing all data in PyTorch
- Support automatic differentiation directly through the
requires_gradattribute - Enable efficient computation on both CPUs and GPUs
- Represent data of any dimensionality — scalars, vectors, matrices, and beyond
- Integrate closely with NumPy, while adding deep learning-specific capabilities
- Form the foundation that every PyTorch operation, layer, and model ultimately builds upon
Creating Tensors in PyTorch
Key Tensor Attributes
| Attribute | Description | Example |
|---|---|---|
| shape (or .size()) | The size of the tensor along each dimension | torch.Size([2, 3]) |
| dtype | The data type of the tensor's elements | torch.float32, torch.int64 |
| device | Which hardware (CPU/GPU) the tensor resides on | cpu, cuda:0 |
| requires_grad | Whether PyTorch should track operations on this tensor for gradient computation | True or False |
The requires_grad Flag: PyTorch's Key Distinction
Setting requires_grad=True tells PyTorch to track every
operation performed on that tensor, building a computational
graph behind the scenes specifically so gradients can later be
computed automatically — this is the direct entry point into
PyTorch's Autograd system, covered in full depth in the next topic.
This is a meaningful design difference from TensorFlow, where
gradient tracking is instead controlled by wrapping code in a
tf.GradientTape context, rather than being a per-tensor attribute.Moving Tensors Between CPU and GPU
Explicitly moving tensors to a device using .to(device) is a distinctly PyTorch pattern — both a model and its input data must reside on the same device (CPU or GPU) for operations between them to work, a requirement that comes up constantly in real PyTorch code.
Tensor Operations
In-Place Operations: A PyTorch-Specific Convention
PyTorch has a widely-used convention where methods ending in
an underscore (like add_(), mul_(), or relu_()) perform their
operation IN PLACE, directly modifying the original tensor
rather than returning a new one — this naming convention is
specific to PyTorch and doesn't have a direct TensorFlow
equivalent, since TensorFlow tensors work differently around
mutability.Converting Between Tensors and NumPy Arrays
Note: tensors created with torch.from_numpy() SHARE memory
with the original NumPy array — modifying one will modify the
other. This is different from simply copying data, and is worth
being aware of to avoid unexpected side effects.PyTorch Tensors vs TensorFlow Tensors
| Aspect | PyTorch Tensors | TensorFlow Tensors |
|---|---|---|
| Gradient Tracking | Per-tensor flag: requires_grad=True | Context manager: tf.GradientTape() |
| Mutability | Tensors are mutable by default; in-place ops use a trailing underscore | Distinguishes tf.constant (immutable) from tf.Variable (mutable) |
| Device Placement | Explicit .to(device) calls | Often handled more automatically, though explicit placement is also possible |
| Creation | torch.tensor(), torch.zeros(), etc. | tf.constant(), tf.zeros(), etc. |
| Default Execution Mode | Eager by default (dynamic, immediate execution) | Eager by default since TF 2.x, with optional graph compilation via @tf.function |
PyTorch Tensors vs NumPy Arrays
| Aspect | PyTorch Tensors | NumPy Arrays |
|---|---|---|
| GPU Acceleration | Native support via .to(device) | Not natively supported |
| Automatic Differentiation | Built-in via requires_grad and Autograd | Not supported |
| Memory Sharing | torch.from_numpy() shares memory with the source array | N/A — standalone |
| Interoperability | Converts easily to/from NumPy via .numpy() / torch.from_numpy() | Converts easily to/from PyTorch tensors |
Key Properties of PyTorch Tensors
- A PyTorch tensor is a multi-dimensional array supporting operations similar to NumPy, plus GPU acceleration and autograd.
- The
requires_gradattribute controls whether operations on a tensor are tracked for automatic gradient computation. - Tensors must be explicitly moved between CPU and GPU using
.to(device), and operands must share the same device. - Methods ending in an underscore (e.g.,
add_()) perform their operation in place, a PyTorch-specific naming convention. torch.from_numpy()creates a tensor that shares memory with its source NumPy array, unlike a full copy.
Where Do Tensors Matter Most in PyTorch?
| Context | Why Tensors Matter |
|---|---|
| Model Input/Output | All data fed into or produced by a model is represented as tensors |
| Model Weights | Trainable parameters are tensors with requires_grad=True |
| GPU-Accelerated Training | Explicit device placement determines where computation actually runs |
| Custom Training Loops | Direct tensor manipulation is central to writing training logic (covered later in this section) |
| Data Preprocessing | Converting raw data into tensors is typically the first pipeline step |
Advantages
- Provides a consistent, NumPy-like data structure with added GPU and autograd support
requires_gradoffers an intuitive, per-tensor way to control gradient tracking- In-place operations (with the underscore convention) can help manage memory efficiently when needed
- Eager execution by default makes tensor behavior immediately visible and easy to debug
- Seamless, well-documented interoperability with NumPy
Limitations
- Explicit device management (
.to(device)) adds a manual step compared to some automatic handling elsewhere - Mismatched devices between tensors (e.g., one on CPU, one on GPU) cause runtime errors that require careful handling
- In-place operations, while efficient, can sometimes interfere with autograd's gradient computation if used carelessly
- The underscore naming convention for in-place operations must be learned, since it's not immediately obvious
- Shared memory behavior with
torch.from_numpy()can cause subtle, unexpected bugs if not understood
Real-World Examples
| Application | Tensor Use |
|---|---|
| Image Classification | Input images represented as 4D tensors (batch, channels, height, width) |
| Natural Language Processing | Token sequences represented as 2D tensors (batch, sequence length) |
| Model Weights | Neural network parameters stored as tensors with requires_grad=True |
| GPU Training | Tensors explicitly moved to a CUDA device for accelerated computation |
| Custom Loss Calculations | Tensor operations directly computing prediction error |
Best Practices
- Set
requires_grad=Truespecifically on tensors that need gradient tracking, typically model parameters. - Always ensure a model and its input tensors are on the same device before running operations between them.
- Use in-place operations (trailing underscore) deliberately and carefully, being aware of their interaction with autograd.
- Remember that
torch.from_numpy()shares memory with its source array — copy explicitly if that's not desired. - Check tensor
.shapeand.dtypewhen debugging unexpected errors, just as with any tensor-based framework.
Interview Tip
A common interview question is:
"How does gradient tracking with PyTorch tensors differ from TensorFlow's approach?"
A strong answer is:
In PyTorch, gradient tracking is controlled per-tensor via the
requires_gradattribute — setting it toTruetells PyTorch to record every operation performed on that tensor into a computational graph, which Autograd later uses to compute gradients. TensorFlow, by contrast, uses a context manager approach withtf.GradientTape(), where any operations performed inside thatwithblock are recorded for differentiation, regardless of which specific tensors are involved. This reflects a broader design difference: PyTorch's gradient tracking is a property of the tensor itself, while TensorFlow's is tied to the surrounding code context in which operations are executed.
Explaining the underlying design philosophy difference, not just the syntax, makes your answer stronger.
Conclusion
PyTorch tensors provide the same fundamental multi-dimensional array capability as TensorFlow tensors and NumPy arrays, but with PyTorch's own distinct conventions — particularly the per-tensor requires_grad flag, explicit .to(device) placement, and the underscore convention for in-place operations. With tensors now covered specifically within PyTorch's API, the next topic dives into Autograd, PyTorch's automatic differentiation engine that requires_grad directly plugs into.