Autograd System¶

MiniTorchBR implements reverse-mode automatic differentiation (backpropagation). This page explains the design so you can understand what happens under the hood.

Core Concepts¶

Variable¶

Variable is the central object — it wraps a NumPy array and optionally participates in a computation graph.

from MiniTorch.core.variable import Variable
import numpy as np

x = Variable(np.array([1.0, 2.0, 3.0]), name="x")

Key attributes:

Attribute	Description
`data`	The underlying NumPy array
`grad`	Accumulated gradient as another `Variable`
`creator`	The `Function` that produced this variable
`generation`	Topological depth used while constructing the graph

Function¶

Every differentiable operation is a subclass of Function with two methods:

forward(*inputs) — computes the output value
backward(grad_output) — returns gradients w.r.t. each input

from MiniTorch.core.function import Function

class MySquare(Function):
    def forward(self, x):
        return x ** 2

    def backward(self, grad_output):
        return 2 * self.inputs[0] * grad_output

    def backward_array(self, grad_output):
        return 2 * self.input_data(0) * grad_output

backward preserves a differentiable graph for higher-order derivatives. backward_array is the fast NumPy path used by ordinary first-order training.

Computation Graph¶

Each call to a Function builds a node in the graph:

x ──┐
    ├─► Mul ──► z ──► Sum ──► loss
y ──┘

When you call loss.backward(), MiniTorchBR:

Starts with gradient 1.0 at the loss node
Walks the graph in reverse topological order
Calls each Function.backward() to propagate gradients

Example: Manual Inspection¶

import numpy as np
from MiniTorch.core.variable import Variable

x = Variable(np.array([2.0, 3.0]))
y = Variable(np.array([4.0, 5.0]))

z = x * y        # element-wise multiply
loss = z.sum()   # scalar

loss.backward()

print(x.grad.data)    # [4. 5.]  (dL/dx = y)
print(y.grad.data)    # [2. 3.]  (dL/dy = x)

Gradient Accumulation¶

Gradients accumulate across multiple .backward() calls (like PyTorch). Zero them before each training step:

optimizer.zero_grad()   # or manually: param.grad = None
loss.backward()
optimizer.step()

Numerical Gradient Check¶

Use the built-in checker to verify custom ops:

from MiniTorch import numerical_diff

approximate = numerical_diff(my_function, x)

Compare this finite-difference result with x.grad.data when implementing a new operation.

no_grad Context¶

from MiniTorch.core.config import no_grad

with no_grad():
    prediction = model(x_test)  # no graph allocated