Okay, let's start with TensorFlow which will be one of our of our main tools in this course on specializations, alongside some other libraries such as. What is TensorFlow? TensorFlow is a high quality production ready and the resource friendly library for machine learning. After it was Open Source by Google in November 2015, it quickly became one of the most popular frameworks, both in the community of machine learning researchers and among practitioners. As an example, I can only mention that most of relevant tasks that Google uses in its business are nowadays powered by TensorFlow. And soon, this might give you an idea of what power you can get once you master this great tool. As you will see, in a bit later, TensorFlow is not only a great tool for machine learning, but also is very useful to the applied math in general due to its flexibility, and computation, and analytical power. Let's start with how TensorFlow is organized. The main idea implemented this time in TensorFlow is the idea of a data flow computational model. This idea, as simple as it may sound turns out critical for prioritization, scalability and portability of practical, machine learning algorithms. Let's see how it works. A data flow graph is a graph with nodes that represent some units of computation, and edges that represent data consumed or produced by any given node. In other words, edges represent the flow of data on the graph. And that's why it's called a data flow graph. Let's see how we can express a simple function of two variables, x and y, shown here x squared times y + y + 2. To this end, we first introduce a few sorts of nodes. First, we will have blue squares. Like these ones that represent inputs to the function. They can be variables, like these ones, x and y, or they can be constants like the node that for a fixed number, 2 in this function. Other nodes like these ovals represent operations, or ops for short. That are performed on these inputs and on outputs of other ops. Now let's make one more step with this dataflow graph and see what happens if our inputs to the graph, not the regular numbers, but other tensors. But what are tensors? If you are familiar with multidimensional arrays in Numpy, they are exactly these tensors. In other words tensors are multidimensional mathematical objects that can be stored as multidimensional arrays in Python or other languages. If you've never worked with multidimensional arrays in Python, but are only familiar with matrices or two dimensional arrays, you can think of a list of such matrices. This would be tensor of rank-3. Why a rank-3, what is a rank of a tensor? The rank of a tensor is simply the number of dimensions of the tensor. And this number would be 3 for our example because we will have three coordinates to store our list of matrices. The number of the matrix, the row and the column. One very natural example of using rank-3 tensors in finance would be stock data. In such tensor one dimension would stand for the date, another one for the stock ticker, and the third one for all features of this topic that you will want to include in your analysis. For example, if you fetch stock data for a set of stocks, and a set of dates from who using it will return an object called a data panel which implements a rank feed tensor of exactly this type. Please note that, no matter how many fishers you want to include in this analysis, your data will still be a rank-3 tensor. It's just the number of elements along the third dimension that will increase. So once we let tensors live on the dataflow graph, we get TensorFlow. It's name is an exact description of what it does. It computes a flow of data called its Tensor on a graph. More precisely, does it in two steps. First it allows you to construct a dataflow graph for your problem using a high level Python API. Then it lets you execute this graph on a single device such as a CPU or a GPU, or on a network of such devices. As I mentioned earlier, TensorFlow was open sourced by Google in 2015. And since then became one of the most popular tools for machine learning. We will see why this is so by the end of this week, and even more so going forward with this visualization. Now, let's come back to the main architectural ideas of TensorFlow. As we have just said, the first one is the idea of a data flow graph. The other one is the idea of a lazy evaluation. What is lazy evaluation? Lazy evaluation is exactly this separation of your calculation into two steps that I have just mentioned. The first step you just construct a graph. You can think of it as a detailed project specification that you can give to your assistant that explains in details what and in what sequence should be done in a project. The actual computation happens on the second step when you pass your graph to a device such as CPU or a GPU. This device will translate your graph into a highly optimized C++ code and will do the calculation. You can also make some nodes on your graph instead of the whole graph lazy. And wait for some other nodes to be executed before kicking in. While lazy evaluation is very important for an efficient working of TensorFlow, it's even more important for you, as a user, to understand another key feature of TensorFlow, namely automating differentiation. More specifically, TensorFlow implements a particular algorithm for automatic differentiation of any function with many inputs with respect to these inputs. This algorithm is called reverse-mode autodiff, and here is how it works. It first perform a forward pass from inputs to outputs to evaluate the function given its inputs. The it performs a backward pass on the graph, from outputs to inputs. That allows it to compute all derivatives sequentially. The technique that is used in such sequential calculation is just the chain rule for a derivative of a composite function. To recall the chain rule, it says the following. If an argument of function f is given by another function n of an independent argument x, then the derivative for function f with respect to x is given by a product of the derivative of the function respect to its argument n, times the derivative of function n with respect to x. Let's now see how it works using our previous example. Let's first assign numbers to our nodes on the graph. So we'll have node 1, 2, 3, etc. Let's start with the last node n7. Because the output of this node is the value of our function itself we'll write the derivative of the function with respect to the node value in this case will be 1. Now, let's move to node 6. We use the chain rule to compute the derivative of the output with respect to this node. We have a product of two derivatives here, but the first derivative is already known and it's equal 1. So the derivative of the output with respect to node 6 will be given by this derivative, which is again 1, because node n7 implements an addition iteration. Next we compute the derivatives with respect to nodes n5 and n4. Please note that in the second expression, we end up with the derivative of node n5 with respect to node n4. Because the node n5 multiplies the outputs of nodes n4 and n3, the derivative is equal to the value of node n3. Now we calculate the derivative with respect to the node n3, which carries the input y. In this case we have to remember that the node n3 is connected to two nodes, n5 and n6, on the graph. Therefore, we will have two terms in the general, that we use to compute the derivative. The final answer for this derivative will be x squared + 1, which is certainly a correct answer, as we can check it by differentiation of our function f. And finally we compute the derivative with respect to x. Again, the number 2xy here is exactly what we get by differentiation of function f with respect to x. So this is it for our first theoretical introduction to TensorFlow. To recap, TensorFlow offers great flexibility and clean design construction in order to build machine learning models. It has a very powerful functionality to automatically compute derivatives of arbitrary functions. It also has many other powerful features such as visualization tools that you will see when we start working hands-on with TensorFlow. And that is exactly what we are going to do next. In the next video we will run our first standard flow code to see how the graphs works in very simple arithmetic examples.
