自动求导机制

了解这些并不是绝对必要的，但我们建议您熟悉它，因为它将帮助您编写更高效，更简洁的程序，并可帮助您进行调试。

从后向中排除子图

每个变量都有两个标志：requires_grad和volatile。它们都允许从梯度计算中精细地排除子图，并可以提高效率。

>>> x = Variable(torch.randn(5, 5))
>>> y = Variable(torch.randn(5, 5))
>>> z = Variable(torch.randn(5, 5), requires_grad=True)
>>> a = x + y
>>> a.requires_grad
False
>>> b = a + z
>>> b.requires_grad
True

model = torchvision.models.resnet18(pretrained=True)
for param in model.parameters():param.requires_grad = False
# Replace the last fully-connected layer
# Parameters of newly constructed modules have requires_grad=True by default
model.fc = nn.Linear(512, 100)# Optimize only the classifier
optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)

>>> regular_input = Variable(torch.randn(5, 5))
>>> volatile_input = Variable(torch.randn(5, 5), volatile=True)
>>> model = torchvision.models.resnet18(pretrained=True)
>>> model(regular_input).requires_grad
True
>>> model(volatile_input).requires_grad
False
>>> model(volatile_input).volatile
True
>>> model(volatile_input).creator is None
True

自动求导如何编码历史信息

Variable上的In-place操作

In-place正确性检查

CUDA语义

torch.cuda会记录当前选择的GPU，并且分配的所有CUDA张量将在上面创建。可以使用torch.cuda.device上下文管理器更改所选设备。

x = torch.cuda.FloatTensor(1)
# x.get_device() == 0
y = torch.FloatTensor(1).cuda()
# y.get_device() == 0with torch.cuda.device(1):# allocates a tensor on GPU 1a = torch.cuda.FloatTensor(1)# transfers a tensor from CPU to GPU 1b = torch.FloatTensor(1).cuda()# a.get_device() == b.get_device() == 1c = a + b# c.get_device() == 1z = x + y# z.get_device() == 0# even within a context, you can give a GPU id to the .cuda calld = torch.randn(2).cuda(2)# d.get_device() == 2

最佳实践

使用固定的内存缓冲区

当副本来自固定（页锁）内存时，主机到GPU的复制速度要快很多。CPU张量和存储开放了一个pin_memory() 方法，它返回该对象的副本，而它的数据放在固定区域中。

另外，一旦固定了张量或存储，就可以使用异步的GPU副本。只需传递一个额外的async=True参数到 cuda() 的调用。这可以用于将数据传输与计算重叠。

通过将pin_memory=True 传递给其构造函数，可以使DataLoader将batch返回到固定内存中。

使用 nn.DataParallel 替代 multiprocessing

大多数涉及批量输入和多个GPU的情况应默认使用DataParallel来使用多个GPU。尽管有GIL的存在，单个python进程也可能使多个GPU饱和。

从0.1.9版本开始，大量的GPU(8+)可能未被充分利用。然而，这是一个已知的问题，也正在积极开发。和往常一样，测试你的用例吧。

调用multiprocessing来利用CUDA模型存在重要的注意事项；使用具有多处理功能的CUDA模型有重要的注意事项; 除非就是需要谨慎地满足数据处理需求，否则您的程序很可能会出现错误或未定义的行为。

扩展PyTorch

本篇文章中包含如何扩展 torch.nn, torch.autograd和使用我们的 C 库 编写自定义的C 扩展。

扩展 torch.autograd

# Inherit from Function
class Linear(Function):# bias is an optional argumentdef forward(self, input, weight, bias=None):self.save_for_backward(input, weight, bias)output = input.mm(weight.t())if bias is not None:output += bias.unsqueeze(0).expand_as(output)return output# This function has only a single output, so it gets only one gradientdef backward(self, grad_output):# This is a pattern that is very convenient - at the top of backward# unpack saved_tensors and initialize all gradients w.r.t. inputs to# None. Thanks to the fact that additional trailing Nones are# ignored, the return statement is simple even when the function has# optional inputs.input, weight, bias = self.saved_tensorsgrad_input = grad_weight = grad_bias = None# These needs_input_grad checks are optional and there only to# improve efficiency. If you want to make your code simpler, you can# skip them. Returning gradients for inputs that don't require it is# not an error.if self.needs_input_grad[0]:grad_input = grad_output.mm(weight)if self.needs_input_grad[1]:grad_weight = grad_output.t().mm(input)if bias is not None and self.needs_input_grad[2]:grad_bias = grad_output.sum(0).squeeze(0)return grad_input, grad_weight, grad_bias

现在，为了可以更简单的使用自定义的operation，我们建议将其用一个简单的 helper function 包装起来。 functions:

def linear(input, weight, bias=None):# First braces create a Function object. Any arguments given here# will be passed to __init__. Second braces will invoke the __call__# operator, that will then use forward() to compute the result and# return it.return Linear()(input, weight, bias)

你可能想知道你刚刚实现的 backward方法是否正确的计算了梯度。你可以使用 小的有限的差分进行数值估计。

from torch.autograd import gradcheck# gradchek takes a tuple of tensor as input, check if your gradient
# evaluated with these tensors are close enough to numerical
# approximations and returns True if they all verify this condition.
input = (Variable(torch.randn(20,20).double(), requires_grad=True),)
test = gradcheck.gradcheck(Linear(), input, eps=1e-6, atol=1e-4)
print(test)

扩展 torch.nn

class Linear(nn.Module):def __init__(self, input_features, output_features, bias=True):self.input_features = input_featuresself.output_features = output_features# nn.Parameter is a special kind of Variable, that will get# automatically registered as Module's parameter once it's assigned# as an attribute. Parameters and buffers need to be registered, or# they won't appear in .parameters() (doesn't apply to buffers), and# won't be converted when e.g. .cuda() is called. You can use# .register_buffer() to register buffers.# nn.Parameters can never be volatile and, different than Variables,# they require gradients by default.self.weight = nn.Parameter(torch.Tensor(input_features, output_features))if bias:self.bias = nn.Parameter(torch.Tensor(output_features))else:# You should always register all possible parameters, but the# optional ones can be None if you want.self.register_parameter('bias', None)# Not a very smart way to initialize weightsself.weight.data.uniform_(-0.1, 0.1)if bias is not None:self.bias.data.uniform_(-0.1, 0.1)def forward(self, input):# See the autograd section for explanation of what happens here.return Linear()(input, self.weight, self.bias)#注意这个Linear是之前实现过的Linear

多进程最佳实践

共享CUDA张量

最佳实践和提示

避免和抵制死锁

重用经过队列的缓冲区

异步多进程训练（例如Hogwild）

import torch.multiprocessing as mp
from model import MyModeldef train(model):# Construct data_loader, optimizer, etc.for data, labels in data_loader:optimizer.zero_grad()loss_fn(model(data), labels).backward()optimizer.step()  # This will update the shared parametersif __name__ == '__main__':num_processes = 4model = MyModel()# NOTE: this is required for the ``fork`` method to workmodel.share_memory()processes = []for rank in range(num_processes):p = mp.Process(target=train, args=(model,))p.start()processes.append(p)for p in processes:p.join()

序列化语义

PACKAGE参考

参考文献

参考链接：https://pytorch-cn.readthedocs.io/zh/latest/