Hi. When it comes to testing, the goal is to find defects. This can mean anything and anywhere from requirements all the way through deployed code. Test selection is the process by which we select the test cases that we want to run against our software to attempt to discover those defects. Like so many other elements of software engineering, the more we can take humans, ourselves out of the process of running tests, out of that process, the better it tends to go. Before we can truly step back and see what happens, we need to provide some parameters for what can actually be tested. What will be tested? These parameters can make or literally break the testing process. You'd be surprised how well random testing does. It isn't easy to come up with strategies that beat it, whether that's based on the complexity of the work or bias and how we select the test, that's the mark to beat. Everything else we do is an attempt to do better, better than random and better than the last best technique. Before I move on, I want to make sure that there are some definitions played out. Test cases and test data. Tests are not just the input, that's test data, but a test is the input, the test data, and the expected output. It's not enough to say that the test is going to be, I'm testing square root, so I'm going to give it 2, I'm going to give it 4, I'm going to give it 0, I'm going to give it -1. Those work, those are good test data, but they're test data that will hopefully cause some issues if the code goes wrong. A test case says, "If I provide 4 and it's square rooted, the answer will be 2." That's a test case, and we need to more formally define test cases and test data, especially those expected outputs as we move into more automated processes. Test suites are just sets of test cases. They can be large or small and software under test is SUT. That's the unit of the system which we're trying to execute to perform the tests. The unit could be a single class or it could be the entire system. Expected versus actual output, comes back to what the test cases really are. Expected is how we believe the system should behave, the expected output given those inputs, the test data. Actual output is what we actually observe after running the tests on the software under test. The oracle, then is what determines whether or not the test actually passes, whether or not the actual output matches the expected output. For example, you might try something like square root of 4, that's 2. Who decides when you run it, whether that matches? For the most part, it's us, it's humans. Humans are the oracle. We pick some test data, we enter it in, we hit "Enter", we see how the program responds. We look at the output and say, "Yes. That's what I expected." That's a passed test. But we become even more reliant on automated oracles, oracles that are more sophisticated than that. If you've ever worked in a *Unit, so JUnit, PyUnit, PHPUnit, et cetera, you've known about test cases and are providing them. That's automated oracle. Given an input and a set of expected output, it'll run the code, check the comparison between expected and actual output and tell you this test passed, this test failed, or this many tests failed. Don't forget though, that we also want to look at negative testing. When I give square root the letter A, what's it going to do? If your expected output is that it's going to error or throw an exception because it can't deal with letters, and it actually returns the letter B, that's bad. It'll return the number 8 because it's converted to ASCII for example. That's something you want to know. So, negative testing that doesn't fail when I thought it would, is just as important as positive testing. Code coverage is a measure of how well the tests work, by measuring the extent to which they cover the code structure. For example, statement coverage means that after running your test suite, all of your tests, how many lines of code actually executed? We usually talk about this in terms of percentages. It's usually difficult to provide, say, 100% statement coverage because there will be some statements in your code that are likely error handling that are difficult to force to happen in a testing situation. So usually talking about meeting some benchmark of code coverage. So, you need at least 80% statement coverage. But you will find in some regulated industries like the United States FAA, the FDA, they actually require 100 percent statement coverage or even more, 100% of a higher-level code coverage, which we'll get into in a little while. This means that if you can't actually run a test for that line, you need to explain why that isn't run ... why the code can't work with a test. Manual versus automated testing is effectively this modernization idea. It's whether or not a developer ran the test themselves or did a tool run the test and report the results. Test selection is the basis of the rest of this lecture, and it's how we select inputs to run on the SUT, the software under test. That's versus test adequacy, which is measuring how well our tests have done in finding the bugs. Code coverage is one example of this, and we'll take that as a proxy, the amount of code coverage as a test adequacy measure. Now when it comes to test, there's a bit of a distinction here. We really want developers not to test their own code. This is different than when you're building. When you're building, you run your own little test just to make sure things are going well until you think it's ready. But real tests, the benchmark of whether or not you are really done, whether it really works, shouldn't only be in the hands of the developer, you need a tester. Developers for many different reasons, whether it's ego, we want our things to be good, if my thing doesn't work, if there's a bug, I'm a poor developer, which isn't true, those things make a developer, as you can see, baby the code a little bit versus testers. It's literally their job to try and break the code. In fact, having your best developer be the teams tester, is often the best case. Now hear me out. Rather than writing all the code and no one else understanding what it does, the best developer can focus on best practice, and is more likely at finding your poor code in unique test cases, which will find more defects. They likely take quality very seriously and is in the best position to help others through feedback and guidance. So, having that best developer, the person who has the best understanding of the development practice, having them sit above and more of a technical team lead, providing input and code review across the entire development system, can not only increase the quality of your code but also increase the capabilities of your developers as they're getting constant feedback from their mentor, from that high-level developer. Now this is not at all a comprehensive list, but most testing approaches seem to stop at the manual unit testing stage. Having developers write tests for their own small pieces of code while they're writing. Those tests go on to serve as the integration test suite, regression testing, system testing, everything. But manually written tests, written by developers is not a good way to approach testing. We need some other metrics for how we're going to select the tests, what tests we run, especially when it comes to things like quality attributes, like security and performance. Randomized testing is still the standard to be. The problem with most testing selection techniques that are used without a formal process behind them, is that we tend to select values to try to cause problems. We talked about this with square root. You test 4, 16 values you know the answer to, then you test negative one. You know what that's going to do. Randomized testing does the same thing but rather organize all the outputs into equivalence classes, so that, for example, from 1 to 1,000 maybe we know the values. We shouldn't only test 16, 64 values we know. We should be randomizing the actual value we use to try and find out maybe this one value really causes a problem. Having parameterized tests where we can throw in a randomized value can really be beneficial. So now, let's go back to code coverage. Code coverage is trying to make sure that while you're looking at the code, your tests run a significant amount of the complexity of that code. Statement coverage is pretty simple, did you run every line of code? It sounds like a pretty good idea. Should every line of code run if you test the whole thing? Sure, but that's actually not enough. There are more sophisticated and higher levels of code coverage that go beyond just the If statement. Within an If, you might have something like "A && B && C". How do you make that true or false? If you have everything be true, then it's going to be true, and then everything false, it's going to fail. The question instead though is, what if A was false and the other two were true? That's not tested necessarily in statement coverage. That could cause a big problem. So, higher levels of testing code coverage are things like condition coverage, that every condition within the boolean result, has taken on at least true and false once. So A and B and C all take true and false once. Compare that to decision coverage. The entire decision has to be true or false once. Again, as we saw with the "and" example, A and B and C, either true, true, true or false, false, false. There are easy ways to get the decision to be true and false, but it doesn't really do a good job of code coverage. So, the even higher ones are things like MC/DC, modified condition/decision coverage. This is the level of testing that we require regulated software. We won't get into MC/DC in this lecture. We may not get into it in the specialization, but there are other lectures and other specializations that cover this code coverage in much more detail. The important thing to know is that you have to look into this as a metric for how good is our code, how good are our tests? Requirements testing is similarly difficult. When you do proper requirements testing, you create the test when you don't have code. Requirements tests should be entirely based on the user's goal. It's what the user tries to do and what should happen when they try and do that. They come straight out of use cases or user stories if you're in Agile. We want to be able to test the high-level behavior and in some cases, we know what the inputs will be, but we're not focusing on the code itself. This is black-box testing. We don't see inside the code, when we know where this input comes in. Here's the output that should come out, the behavior of the system, that's it. We know what goes in, we know what should come out that behavior. We don't know how it happens. This is just a brief introduction to these kinds of issues with test selection. There are plenty of other more in-depth investigations of the variety of ways that we could select tests. We'll get into a tiny bit more in the next lecture with test adequacy. But for the most part, if you're looking for more information, I would strongly recommend looking up the other Coursera courses, or other resources to find more information about how these testing and test practices, best practices are actually put into the field. Test selection is incredibly important. How we select our tests can define whether or not we find the bug, find the defect, or if we don't. Manual testing by developers tends to be biased. It tends to orient towards certain things, especially if you do it after you're done. If you test after you're done completing the code, you tend to test how you built it. You test based on what you know it should do, rather than what it should do. Knowing what it does and knowing what it should do are two different things, which is why requirements testing and testing up front is so important.
