In previous videos, we discussed that evaluating the performance of a model based on accuracy and specificity or receiver operating characteristic is not sufficient, when to examining how well the model generalizes in an external validation cohort and also whether the model is well-calibrated. In this video, we are going to discuss how to incorporate clinical consequences by reporting the net benefit of the model and comparing it with relation to other strategies. For example, treating all patients versus treating none of the patients versus treating patients according to the model. A prediction model provides the risk of an event based on a set of factors and variables. The risk in typical studies is expressed as a probability of the event occurrence. We have discussed about discrimination accuracy, as well as internal, and external validation. However, is this do not incorporate clinical usefulness. Consider a scenario where a false negative result is much more harmful than a false-positive result. For example, in cancer detection, missing a positive biopsy sample of breast cancer would delay the treatment of a patient with critical consequences to their health. Remember that specificity shows how many negative predictions are really negative, whereas sensitivity shows how many positive predictions are really positive. Therefore, a model that has a much greater specificity, but slightly lower sensitivity than another could have a higher area under the curve, but would be a poor choice for clinical use in the scenario of cancer detection. In a typical decision analysis, firstly, we identify possible consequences of a clinical decision. Subsequently, the expected outcomes of alternative clinical management strategies are simulated using estimates of the probability and sequence of events in a hypothetical cohort of patients. Decision analysis requires explicit valuation of health outcomes. Such as a number of complications prevented, the life-years saved, or the quality-adjusted of the life years that have been saved. Therefore, in a decision analysis of a prognostic model, the optimal model is the one that maximizes the outcome of all or some of these factors. There are two general problems associated with applying traditional decision analytic methods to prediction models. Firstly, they require data, such as cost or quality-adjusted life years, which are not found in the validation dataset. Remember, that the validation dataset will only include the results of the model and the true disease state or outcome. This implies that the prediction model cannot be evaluated in a decision analysis without further information being obtained. Furthermore, as we've seen, decision-analytic methods often require explicit valuation of health, state, or risk-benefit ratios for a range of outcomes. The second problem is that decision analysis typically requires that the test or prediction model being evaluated keeps a binary result so that the rate of true and false positive and negative results can be estimated. On the other hand, prediction models often provide a resulting continuous form, such as probability of an event from 0 - 1. In order to evaluate such a model using decision-analytic methods, the analyst must dichotomize the continuous result at a given threshold, and use this to evaluate a wide range of such thresholds. Clinical usefulness can be considered in terms of net benefit. Net benefit is calculated across a range of threshold probabilities. It can be seen as the treatment threshold weighted sum of through my nose, false-positive classifications for each strategy. Net benefit, typhus from other performance metrics, such as discrimination and calibration, because it incorporates the consequences of the decisions made based on a model. Net benefit is plotted against an entire range of treatment thresholds. The build also incorporates information with relation to the scenarios where all patients are diagnosed as positive or all patients are diagnosed as negative. This helps to weigh the benefits of an intervention version, the risk. It is a subjective term defined by the experience of experts as well as the patient, particular circumstances. In decision curve analysis, we include results for intervention for all and intervention for non-patients. Let's discuss some examples of why this is important. One reasonable strategy in the prostate biopsy study would be to biopsy all patients with an elevated prostate-specific antigen test irrespectively of the results of the diagnostic test. Indeed, this is generally what happens in current clinical practice. This is because the risk of cancer outweighs any possible benefit of avoiding the biopsy. The type of intervention depends on the clinical decision. For example, in patients with an infection, the intervention would mean antibiotics. In preventing heart disease, intervention might be providing specific medication. Here I show you an example of a decision curve analysis with relation to the model we have discussed earlier that predicts outcome in first-episode psychosis. In order to evaluate this model with relation to its clinical usefulness, we plot the net benefit with relation to the threshold probability. The black line represents a strategy where we don't treat any patient. Whereas the gray line represents a strategy where we treat all patients. The red line represents the strategy which we treat patients based on the duration of untreated psychosis. This is a very common approach because it is a consistent predictor for outcome in first-episode psychosis. Finally, with the dotted black line is the proposed model that have been developed and validated in a recent publication. In order to understand the clinical usefulness of the prediction model, we need to specify a probability threshold above which we would consider the treatment. This is subjective and it depends on expert knowledge. For this reason, the study have consulted specialist, psychiatrist to confirm the probability threshold at which they would consider treatment. The range of threshold that they reported varied from 40 percent to 60 percent. In other words, they would adopt an assertive monitoring and intervention approach when an individual's probability of emission was above 40 to 60 percent to balance the likelihood of benefits versus the harms. Based on the decision curve analysis reported here, it is reasonable then to assume that in this thresh hold, the net benefit of the prediction model outperforms all other approaches. Taking everything together, the predictive model have been evaluated with internal validation for its performance. But we have also discussed about external validation with relation to discrimination and calibration. Here, we used external validation cohort to examine its clinical utility. Let's look into some theoretical examples of decision curve analysis. We built here, again, the net benefit with relation to the threshold probability, and in this case, the disease incidence is 20 percent. The horizontal line shows the net benefit if no patient is treated. The dotted thin line shows the net benefit if all patients are treated, assuming that all patients have the disease. The gray line shows the net benefit of a near perfect binary predictor. Like with 99 percent sensitivity and 99 percent specificity. Whereas the thick black line shows a perfect prediction model. The solid line shows a predictor with very good sensitivity, 99 percent, and 50 percent specificity. Whereas the dashed line shows a binary predictor with 50 percent sensitivity and 99 percent specificity. This is a good example to show the predictors with high sensitivity are better to a predictor with higher specificity when the threshold probability is low. In this case, the harm of a false negative is greater than the harm of a false positive. However, the situation is reversed when the threshold probability is high. We might be able to understand better the relationship between preference and threshold probability. If we look into the odds, the risk, for example, of 10 percent is a nodes of one to nine. In using a threshold probability of 10 percent, the doctor is telling us that missing a high-grade cancer, for example, is nine times worse than doing an unnecessary biopsy. Let's see here, another theoretically example. Now, we would like to evaluate a predictor which is a normally distributed laboratory marker. The thin line assumes that no patients have the disease, like we saw earlier. Whereas the dotted line assumes that we treat all patients, assuming they have the disease. The solid lines from left to right represent a mean shift from 0.33 to 0.5 to 1 and to 2 standard deviation in patients with disease. Through what you see here, a predictor that is two standard deviation higher in patients who have the event, is better across nearly all the range of threshold probabilities. Summarizing, in this video, we discuss how to use decision curve analysis to incorporate clinical consequences in clinical decision support systems. Decision curve analysis provides us with an intuitive way to compare prediction models. However, we should be careful that this is not a method that replaces measures of accuracy, such as sensitivity and specificity. These measures are important in the early stages of developing diagnostic and prognostic strategies. We should also highlight the decision curve analysis can be used for prediction models that gives a probability of an event but also it can be used for standard diagnostic tests that produce a simple binary result.
