Separation Management
| Full Text: |
 Pdf
(0.55 MB) |
|
Document Number: |
DOT/FAA/TC-TN-12/65 |
|
Publication Date: |
11-2012 |
| Authors: |
Anthony J. Masalonis James Messina Jonathan Rein, PhD. Ben Willems |
Rein, J. R., Masalonis, A. J., Messina, J., & Willems, B. (2012). Separation Management: Automation reliability meta-analysis and conflict probe reliability analysis (DOT/FAA/TC-TN-12/65). Atlantic City International Airport, NJ: Federal Aviation Administration William J. Hughes Technical Center.
Abstract
Objective: The purpose of this study is to establish whether a valid performance criterion exists to determine the acceptability of the En Route Automation Modernization (ERAM) Conflict Probe’s conflict-detection accuracy and to evaluate observed accuracy against this criterion.
Background: The Conflict Probe can exhibit very high or very low accuracy depending on the analysis technique. It is necessary to establish both an empirically backed criterion for accuracy and the appropriate accuracy analyses and metrics.
Method: A meta-analysis was conducted on Human Factors automation reliability literature; an additional analysis was done on the results of probe reliability studies by the Federal Aviation Administration (FAA) Concept Analysis branch to derive various accuracy metrics. The results were compared to determine the acceptability of the conflict probe accuracy.
Results: The meta-analysis produced an estimated criterion of 65% correct responses for automation to improve performance, but this estimate is subject to a broad confidence interval due to variability in the source data from the literature. The probe performance exceeded the 65% value when giving credit for all correct rejections, but it fell short when not giving credit for correct rejections. Another metric, Positive Predictive Value (PPV, the percent of alerts that are valid), is operationally meaningful and its values demonstrated large accuracy improvements over baseline with the FAA Concept Analysis’ parametric adjustments, but a PPV cutoff criterion could not be established from the meta-analysis.
Conclusion: The present results provide insight on several fronts, but operational input is essential to determine (1) a more justifiable air traffic control-specific accuracy criterion and (2) which aircraft encounters are appropriate to include in a test set for accuracy assessment. Automation responses should only increase the value of an accuracy metric to the extent that the responses add informational value for the controller. Applications: The present results will be used in the design of an evaluation to derive a set of operationally meaningful aircraft encounters. Different measures of accuracy, such as PPV, also merit further exploration.
