Five Tensor Operations
May 30, 2020
Last week I enrolled in FreeCodeCamp’s ZeroToGANs deep learning with PyTorch course. The motivation behind doing so was mainly because of quarantine and wanting to learn something new while being on lockdown.
As a software engineer mainly focused on app development, who by the way, barely completed college level algebra and trig, I can say in total confidence that I have no clue what the hell I am getting myself into or even if I have the capacity to complete it.
But, the cool thing about this course is that the instructors take a
code-first rather than a
math-first approach in teaching the lessons and I think this works well for me because one thing
I can learn to userstand is code.
Now whether the
code-first approach is the right way to get introduced into the space of machine
learning is something I won’t get into and of course there are concepts that I would have to
learn in order to contribute anywhere effectively but the way I see it is:
I have to start somewhere right?
So with all that said, this is by no means a hardcore introduction to data science. I won’t be going into things like gradient descent, linear regression, neural networks etc. because there are plenty of authority figures who are better equipped to do so. Instead, what you’ll find below are 5 PyTorch methods I found interesting that I think are a good way to get introduced to the space starting with code.
A little intro to PyTorch and torch.tensor
PyTorch is a python library built and maintained by Facebook, full of programs to help facilitate building of deep learning projects.
PyTorch at its core processes multi-dimensional matrix called a tensor. A tensor can contain integers, floats and *booleans.
To create a tensor in PyTorch, use the
torch.tensor API likeso:
import torch torch.tensor([1, 2, 3, 4]) # Output tensor([1., 2., 3., 4.])
In the example above I created a one dimensional tensor with integers valued 1 - 4.
torch.tensor, the five methods I’ll go over are:
These are just methods I found that might be a good intro into PyTorch API but of course, if you want to learn more check out the documentation.
Function #1 - torch.arange
Returns a one dimensional tensor with values from start to end incremented by steps.
# Example 1: 1-D tensor with size of 10 with steps of 1 torch.arange(10) # Output tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Notice that the first value in our tensor starts with 0. If no start parameter is provided, it defaults to 0.
# Example 2 - More parameters torch.arange(1., 2, .1, requires_grad = True) # Output tensor([1.0000, 1.1000, 1.2000, 1.3000, 1.4000, 1.5000, 1.6000, 1.7000, 1.8000, 1.9000], requires_grad=True)
Notice here the start and steps parameters are float points. We can also provide a range of other parameters such as requires_grad, out, dtype, layout and device.
# Example 3 - breaking (to illustrate when it breaks) length = 10 width = 5 torch.arange(length * width, step = 1, dtype = torch.get_default_dtype()) # Output TypeError: arange() received an invalid combination of arguments
Be mindeful of how you organize your parameters. Correct way to write this would be like so:
torch.arange(start = 0, end = length * width, step = 1, dtype = torch.get_default_dtype())
torch.arange can be useful to create a one dimensional tensor of indices.
Function #2 - torch.split
Splits the tensor into chunks. Each chunk is a view of the original tensor.
# Example 1 - Basic split input_tensor = torch.tensor([50, 80, 10, 2]) torch.split(input_tensor, split_size_or_sections = 1) # Output (tensor(), tensor(), tensor(), tensor())
Above we created a input tensor and used the split method to break our tensor data into equal chunks of size 1.
# Example 2 - Split a tensor of ones, size 10, into chunks of at most size 2 x = torch.ones(10) torch.split(x, 2) # Output (tensor([1., 1.]), tensor([1., 1.]), tensor([1., 1.]), tensor([1., 1.]), tensor([1., 1.]))
In the example above, first we created a tensor of 1’s size 10. After which we use the split method to split the tensor into chucks of seperate tensors with a size of atmost 2.
# Example 3 - breaking (to illustrate when it breaks) y = torch.zeros([3, 3]) torch.split(y, 1, 3) # Output IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 3)
We get this error IndexError: Dimension out of range when we attempt increase the dim size. This causes an issue with the shape. Keep in mind the dimensions along which you want to split the tensor.
torch.split is perfect when you need to split a Tensor into equal-size chunks!
Function #3 - torch.stack
Concatenates sequence of tensors along a new dimension.
# Example 1 - Basic example a = torch.ones([3, 3]) b = torch.zeros([3, 3]) torch.stack([a, b]) # Output tensor([[[1., 1., 1.], [1., 1., 1.], [1., 1., 1.]], [[0., 0., 0.], [0., 0., 0.], [0., 0., 0.]]])
In the above example we create two seperated tensors using ones and zeros. After which was concatenated into one tensor using the stack method.
# Example 2 - Stack two tensor on one dimension c = torch.ones(5) d = torch.zeros(5) torch.stack([c, d], 1) # Output tensor([[1., 0.], [1., 0.], [1., 0.], [1., 0.], [1., 0.]])
In the above example we created another two tensors with the ones and zeros method. This time when we concatenated them using the stack method we added an extra parameter for our dimension we want to stack along.
# Example 3 - breaking (to illustrate when it breaks) e = torch.ones(3) f = torch.zeros(4) torch.stack([e, f]) # Output RuntimeError: stack expects each tensor to be equal size, but got  at entry 0 and  at entry 1
We get the error above stack expects each tensor to be equal size because the shapes of tensor e and f are not the same.
torch.stack function is perfect when you need to concatenate two tensors of equal size.
Function #4 - torch.abs
Computes the element-wise absolute value of the given input tensor.
# Example 1 - Basic example ex_tensor = torch.tensor([-1, -2, -3]) torch.abs(ex_tensor) # Output tensor([1, 2, 3])
In the above example, first we created a tensor with all negative integers and passed that tensor through torches *abs which turned every element to a positive integer.
# Example 2 m = torch.tensor() n = torch.tensor() print('negative:', m - n) torch.abs(m - n) # Output negative: tensor([-1]) tensor()
Similar to the first example, here we create two tensors using tensor method, after which we run some arithmitic on both tensors which returns a negative integer but as you can see when we pass that same expression to abs method it returns as positive.
# Example 3 - breaking (to illustrate when it breaks) o = numpy.array([1, [-2]]) torch.abs(o) # Output TypeError: abs(): argument 'input' (position 1) must be Tensor, not numpy.ndarray
Make sure you pass a tensor as a argument to torch.abs unless it will throw the above error: abs(): argument ‘input’ (position 1) must be Tensor, not numpy.ndarray.
torch.abs is great when you want to work with all positive element in your tensor!
Function #5 - torch.add
Adds the scalar other to each element of the input and returns a new resulting tensor.
# Example 1 - Basic example p = torch.ones(10) torch.add(p, 9) # Output tensor([10., 10., 10., 10., 10., 10., 10., 10., 10., 10.])
So in the above example we create a tensor of ones with size of 10 using ones method. Next we pass our tensor of ones to the add method with an accumulator of 9 which takes each element of our tensor and add 9 to it.
# Example 2 q = torch.randn(4) r = torch.randn(1) torch.add(q, r, alpha = 10) # Output tensor([17.4783, 17.0044, 17.4107, 19.7463])
In this example we create a tensor of random floats with size of 4. Next we create a tensor of a random integer with size of one. Finally we add all the elements of q with r with a scalar multiplier of 10.
# Example 3 - breaking (to illustrate when it breaks) s = complex(1, 1) t = torch.zeros(1) torch.add(t, s) # Output RuntimeError: Complex dtype not supported.
In the above example we create a complex number using the complex method. Then we create a tensor of 0’s with size of one and then attempt to add the element of our tensor to our complex number. We get the error: Complex dtype not supported. because only integers and floats are supported in the add method.
This function is great when you need to accumulate each element in your tensor by the same value.
And that’s about it
If you enjoyed this little intro I recommend checking out the rest of the documentation and also the course. I’ll link them below.
If you want to play around with the code, I’ve created a Jupyter notebook. There you can run the examples I’ve written above. You can also fork it and play around with the API yourself if you so please.
A blog by Ruben Profit