Here are some property of autoencoder. First of all, it's unsupervised process, so you don't need explicit labels to train this neural network. This process of compression decompression is data specific. They're trying to learn the patterns from the original training data. When you use the autoencoder, similar data, they will extract similar patterns. It's actually designed to compress similar data to the training data. Finally, this compression decompression process is lossy, meaning that autoencoder would not exactly reconstruct original input. It will be close, but with degraded representation. In fact, we want this lossy compression because it can capture the essential patterns in the data while ignoring some of the artifacts or noise in the original data. With this intuition in mind. Here's a quiz question about autoencoder to help us understand intuition behind this process. Keeping in mind autoencoder is trying to learn some patterns from the original data, then using the pattern to reconstruct the original data. Here, if you look at the original input vector, looks pretty random, 17, 35, 10, 21, 5 and 11. Then, give you some hints. This output reconstruction vector r is the following; 17, 34, 10, 20, 5, 10. Do you see some patterns in this output vector r? Use that patterns, can you figure out what the latent pattern h, this three-dimensional vector should be? Here's the answer. H, in this case, capture the essential input. You notice that the input, every other elements is like, you can double that, then repeat, for example, 17 times 2 equal to 34, and the second element is 35 is close to that. 10 times 2 equal to 20, and the element follow 10 is actually 21. So it's close. Then five follows, it should be 10. But input is actually 11, so they're still close. We can consider this latent pattern should be just three values, 17, 10 and 5. We can reconstruct this six element by just double this and copy those value again. That's intuition. Now, can you specify the corresponding W and W prime matrix? In this case, we don't need a bias vector, so the B and B prime equals 0, just for simplicity. Can you figure out the corresponding matrix W and W prime and such that you can carry out this compression decompression process. For W, it turns out what we want to do is, we just want to keep true value, every other values. In that case, the input is six-dimensional vector, and output is three-dimensional vector. This W is of size three by six. Then we want to keep the first value, the third value, and the fifth value. So the first row corresponding to the output of the first value, so we copy the first element out of the six, then the second row corresponding the second output value, and we are going to take the third element. The third row corresponding to this third value. In this case, we copy the fifth element. If you apply this, W over multiply with input x, you will get 17, 10 and 5, by copying the first, third and fifth element. Now, can we reconstruct that back to something similar to the original input? It turns out this W prime matrix is very similar. In order to generate the six-dimensional output from a three-dimensional input, we need this matrix to be six by three. In this particular case, what we want to do is, we want to first copy the original input, 17, 10 and 5, then we also want to double those original input and copy those values again. In this case, the first input element will be output twice, where one as itself and also as doubled. Then the second element, which is 10, also we'll do the same. I'll put 10 itself at the third element, but also double that as the forth element. Then the third element, we do the same. We first copy the original value as the fifth element, then double that as the sixth element. That's how we get this output, 17, 34, 10, 20, 5 and 10. In this case, it' s just a simple illustration of the intuition behind autoencoder, and the sigma and sigma prime activation function is just identity.
