PyTorch Developer Podcast

Latest Episodes:

Expect tests Jun 18, 2021

Show notes

What's an expect test? Why should you use them? Why is inline expect test better than out of line? How to write a good expect test?

Further reading. expecttest source implementation https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/expecttest.py (only 311 lines!)

XLA Jun 17, 2021

Show notes

What's PyTorch XLA? Why should you care? How is it implemented? How does PyTorch XLA trade off functionality versus ease of performance debugging? What are some new developments in this space?

Further reading.

XLA's repo has lots of really good docs. Check out https://github.com/pytorch/xla/blob/master/OP_LOWERING_GUIDE.md and also the main https://github.com/pytorch/xla/blob/master/README.md
Alex Suhan's RFC about lazy core https://github.com/pytorch/rfcs/pull/18

TH Jun 16, 2021

Show notes

What is TH? Why might you care? What is so horrible about it? What the heck is the generic/ folder? Why are we porting everything to C++? What are some downsides of having ported all our TH code to C++?

Further reading.

The TH to ATen porting guide has lots of explanations of old school TH idioms https://github.com/pytorch/pytorch/wiki/TH-to-ATen-porting-guide
Old notes about refcounting in TH https://github.com/pytorch/pytorch/blob/master/aten/src/README.md

TorchScript Jun 15, 2021

Show notes

There is a really good TorchScript overview at https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/OVERVIEW.md and in this 20min podcast, I want to give you some of the highlights from this document.

CMake Jun 14, 2021

Show notes

Why is PyTorch's build so g-dang complicated. How to avoid having to deal with cmake at all? And if you have to deal with cmake, what are the most important things to know? And if you were going to improve our cmake, how would you go about doing it...

Further reading.

The official CMake documentation is a great help and well worth reading https://cmake.org/documentation
If you work in torch/csrc chances are you'll need to edit this file https://github.com/pytorch/pytorch/blob/master/tools/build_variables.bzl

Liner notes.

multiple build systems: cmake, buck, xplat buck, ovrsource buck, bazel
- tools/build_variables.bzl is read from cmake! append_filelist
  - but not used uniformly for all components! (ouch!)
mashed together ATen and Caffe2 build systems (e.g., main library libtorch_cpu is defined in caffe2/CMakeLists.txt)
cmake: not very much syntax, "everything is a function". This means you can look up constructs relatively easily; e.g., even if() is a command
the general cmake model: "set a bunch of variables, run a bunch of commands". cmake is VERY GREPPABLE
- but not everything is in CMakeLists.txt; check *.cmake too
- the directory structure makes no sense, you really need to grep.
  (doing a lot of set PARENT_SCOPE to propagate stuff)
- renaming a file? grep for it
- primary hazard of refactoring: need to make sure all the variables
  are setup at the new location
many directories are not recursive glob, beware of adding new directories
old school cmake: literally everything is stuffed in variables (CMAKE_CXX_FLAGS). new school cmake: attach things to targets, things propagate when you depend on targets (public/private dependencies)
add_library: the most important thing
don't randomly change things and pray: have hypotheses and test them

torchdeploy Jun 11, 2021

Show notes

torchdeploy is a way of running multiple Python interpreters inside the same process. It can be used to deploy Python PyTorch programs in situations where the GIL is a problem, not the CPython interpreter. How does it work, and what kind of challenges does it pose for people who want to write code that calls from C++ to Python?

Further reading.

How the torchdeploy build system works https://dev-discuss.pytorch.org/t/torch-deploy-the-build/238
Description of the single interpreter per Tensor invariant https://github.com/pytorch/pytorch/issues/57756
Recent work on making it possible to load C extensions into torchdeploy https://dev-discuss.pytorch.org/t/running-multiple-python-interpreters-via-custom-dynamic-loading/241

C++ frontend Jun 10, 2021

Show notes

What's the C++ frontend? Why is avoiding templates so important? Why is Tensor a reference type? How do we simulate keyword arguments in C++? Where did the nn Module support in the C++ API come from? Why did we reimplement all modules in C++? How are modules implemented in C++? What are some performance challenges of writing Python in C++, and how are we working around them?

Further reading.

C++ frontend tutorial https://pytorch.org/tutorials/advanced/cpp_frontend.html
Writing Python in C++ (a manifesto) https://github.com/pytorch/pytorch/wiki/Writing-Python-in-cpp-(a-manifesto)
MaybeOwned PR https://github.com/pytorch/pytorch/pull/53317

PyObject preservation Jun 09, 2021

Show notes

Given two separately refcounted objects, how can you arrange for each of them to stay live so long as the other is live? Why doesn't just having a strong-strong or strong-weak reference between the two objects work? What is object resurrection in CPython? What's a finalizer and why does it make things more complicated? How does Python GC work?

Further reading.

PyObject preservation PR https://github.com/pytorch/pytorch/pull/56017
Sam Gross's original PoC, which works fine if the two objects in question are both PyObjects https://github.com/colesbury/refcount/
PEP 442 Safe object finalization https://www.python.org/dev/peps/pep-0442/
Essential reading about Python GC https://devguide.python.org/garbage_collector/

Mobile selective build Jun 08, 2021

Show notes

What is mobile selective build? Why are we so obsessed with reducing binary size? How does selective build work? Why doesn't static linking just work? Why can't you just read out the ops used in a TorchScript model to determine what operators you actually need? What are the tradeoffs of statically determining the operator dependency graph versus tracing? What's up with the SELECTIVE_NAME macro? How the heck does selective build work at all when you have multiple mobile apps in a single Buck build system? What takeaways should I have as a regular PyTorch developer?

Further reading:

Official open source mobile documentation on custom selective builds https://pytorch.org/mobile/android/#custom-build
How to rebuild the op dependency yaml https://github.com/pytorch/pytorch/blob/master/tools/code_analyzer/build.sh

Liner notes:

binary size is premium; ship only what you actually need
big idea:
- get the ops your model needs -> apply this to build of pytorch
get the ops your model needs
- TorchScript ~> read it out directly from the model itself
- but what if ops use other ops?
  - need a dependency graph. done with static analysis llvm (jiakai) ~> with a (possibly inaccurate) yaml checked in for easy kickstart if you don't want to run the pass (updated by bot, not operational since Feb, recommend rebuilding from scratch if you run into trouble)
- other possibility: dynamic tracing
  - pro: no need for dependency graph, just look at what was called; works for dtypes
  - con: need representative inputs, if control flow might not cover everything
apply this to build of pytorch
- ordinarily: static linking ensures stuff that isn't used gets pruned
  - but this doesn't work with distributed operator registration based on static initializers
- how?
  - codegen - just don't generate it
  - no codegen - SELECTIVE_NAME - C++ doesn't support string in macro
- build system integration
  - buck constraint: only one library
    - therefore: generate multiple copies of glue library
  - alt: atomize library into each operator. caffe2 used to do this; each library takes a long time to build (1m) and crashes xcode because there's too many
common hiccups
- modify implementation details, some op is/isn't called anymore ~> error! usually just means some yaml needs regenerating. PyTorch Edge developers are very friendly and can help

torch.nn Jun 07, 2021

Show notes

What goes into the implementation of torch.nn? Why do NN modules exist in the first place? What's the function of Parameter? How do modules actually track all the parameters in question? What is all of the goop in the top level NN module class? What are some new developments in torch.nn modules? What are some open problems with our modules?

Further reading:

Implementation of nn.Module https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py
nn.Module is complicated and that means its sometimes a bit slow. Some analysis at https://dev-discuss.pytorch.org/t/overhead-in-nn-module-causing-massive-slowdowns-compared-to-raw-cublas-or-torchscript/110
Lazy modules PR https://github.com/pytorch/pytorch/pull/44538 and factory kwargs https://github.com/pytorch/pytorch/pull/54508

Liner notes:

python for hackability (c++ is reimplemented)
parameters
- parameter collection (for optimization)
- buffers: not considered optimizable
modules
- functorial operation (_apply)
- jit script: staged computation (init is not scripted)
- __call__ to forward (extra instrumentation)
- serialization / state_dict
new stuff: device kwarg (joel schlosser)
new stuff: lazy modules (emcastillo)
open problems: parameter initialization

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected