Testing 101: Beyond Sensitivity and Specificity

Test accuracy (sensitivity and specificity, collectively) is clearly the first and most important variable to consider when assessing the quality of a test. We need to be able to trust what the test is telling us, otherwise what’s the point? More precisely, given that no test will give a perfect result every time, we need to know how much to trust the result, and this is what sensitivity and specificity tell us. If a test is 99% accurate, it will be right 99% of the time.

But test accuracy is by no means the sole criterion by which all tests should be compared, nor should a nation rely exclusively on the one test with the highest accuracy. There are many factors that determine the real-world usefulness of a medical test, and in this piece we discuss a few that are particularly relevant to Ireland.

At the core of the discussion, is the complexity of the test process.

Test Complexity

While a test for COVID-19 is conceptually quite simple – the virus is present or it is not – the test process and the time and resources required to get a result, vary greatly.

The RT-PCR testing process – from specimen to result – is long, complicated, and resource-intensive for all involved. The individual must get themselves to and from the test centre, then self-isolate until the result has been returned – a period which could last several days. The specimens must be collected by a professional, transported to the lab, and stored in appropriate conditions until selected for testing. The lab has its own supply chain, requiring staff, chemicals, and equipment to operate quickly and accurately, and a lack of any one of these inputs can create bottlenecks and backlogs. What use is perfect accuracy if you don’t have the necessary inputs, or there are insufficient staff available to run the tests?

The RT-PCR testing process has many points of failure and is susceptible to exogenous shocks. Thankfully, less complex and less resource-intensive tests also exist.

Rapid lateral flow antigen tests use the same technology as a pregnancy test and are simple enough to be used by untrained individuals. The taker performs a nasal swab (not the deeper naso-pharyngeal swab), creates a solution with the sample, drops it on to the paper strip and waits up to 30 minutes for a result. The inputs are basic, the tests can be mass produced, and once the user has the test kits, they can use them as needed as there are no external dependencies.

Greater simplicity does come at the cost of lower accuracy, but if the rapid tests are used in the appropriate circumstances (high volume, repeated, screening not diagnostics) a lower sensitivity relative to PCR is a tractable problem.

The RT-LAMP COVID-19 testing process uses similar technology to the PCR test and it too needs health care professionals to operate. However, the LAMP equipment is portable and the test can completed at the point of care, so there is no need for transportation to a central laboratory, less opportunity for bottlenecks, and no delay in communicating the result.

Test Turnaround Time

For a non-communicable disease that develops over many years, the ‘turnaround time’ on the test would not be important. Whether it takes a couple of days or a couple of weeks to produce a result, it is unlikely to have a meaningful impact on the path of the disease or the course of treatment. In which case, a slower but more accurate test would usually be preferred, as the cost of waiting for greater accuracy is minimal.

COVID-19 is different: it is a communicable disease and it has a short life-cycle. For both reasons, the test turnaround time is an important variable. We want to find infected individuals before they suffer serious damage to their health, and we want to minimise the likelihood that they will pass the infection on. Given that this virus exhibits pre-symptomatic and asymptomatic transmission, the speed and responsiveness of our testing regimen is even more important again.

A national testing program that relies solely on PCR testing cannot achieve these goals. Even in the best of circumstances it would not be feasible to book a PCR test, get tested, and get the result back within 24 hours. A 4-day turnaround time is more realistic for Ireland, but we should also note the downside risks. We need testing most when cases are high and the virus is spreading exponentially. Yet that is exactly the time when the PCR process tends to perform the worst, becoming jammed up and creating backlogs such that by the time people get their results back, the disease has come and gone (or they have spent an unnecessary week in quarantine).

COVID-19 spreads quickly and out of sight. A comprehensive risk management response would need a testing program that can react in real time and find hidden transmission before the virus has the opportunity to spread into the community. Both rapid antigen tests and LAMP can produce accurate results within an hour, and that is the kind of speed we need to get ahead of the virus.

Test Capacity

If the maximum number of tests that can be performed in a given period is finite – whether through a lack of the raw materials needed to produce the tests, or a lack of funds to purchase them – then there is also the possibility that the limit will be reached, and our test capacity will need to be rationed.

Rationing medical resources is rarely a sign of good things to come. Whether it is PPE, tests, ICU beds, or vaccines, a shortage of any one of these key resources will lead to a deterioration in health outcomes, including hospitalisations and death. These situations usually point to a serious lack of preparation and foresight on the part of policymakers.

Global PCR testing capacity was already stretched when global cases started to rise faster in October 2020. The global supply of PCR tests was effectively reached its limit and policymakers were forced to search for alternatives (this is one of the reasons why you are hearing so much more about rapid tests these days). As vaccination becomes more widespread, we should expect the demand for PCR testing to fall and the market should balance again. Until then, we need to source extra capacity.

The rapid antigen tests described above require only basic inputs and do not appear to have a meaningful cap on supply. They can be produced in large quantities and at short notice, which means their supply should be more reliable too. The technology underlying LAMP testing for COVID-19 is relatively new and manufacturers are still building capacity, however a supply already exists and we can expect this to expand over time.

Ease of Distribution

The easier it is for people to get a test, the more likely they are to get themselves tested. If policymakers can find ways to lower the barriers to testing and to remove the day-to-day frictions in the process, greater numbers will come forward for testing – and more regularly too. One of the ways policymakers can support that outcome is by moving the testing facilities closer to the people.

The complexity and inflexibility of the PCR process is a major impediment to its swift and widespread distribution across the country. A trained health care worker is required to perform the test, so if there is a shortage of healthcare workers in a region, there will be a shortage of test capacity too. Even when there is a surplus of healthcare workers and swabs, the samples will still have to be sent back to a laboratory with the fixed equipment and machinery needed to complete the analysis, which slows the whole testing process.

The PCR is a centralised testing process – there is one key processing centre through which all tests must pass – whereas rapid testing is fundamentally decentralised. The entire life cycle of a rapid test can be completed in one place – any place – in less than an hour. Rapid antigen tests can be distributed directly into homes, schools, and workplaces. LAMP tests can be performed in airports and other transport hubs, at entrances to nursing homes or conference centres, or in stand-alone drive-through test centres.

The simplicity and flexibility of rapid testing technology allows for greater geographic decentralisation, making it easier for people all over the country to get tested and making our testing regimen more powerful.

Ireland Needs Rapid Tests

Test accuracy is just one factor to consider when assessing the real-world effectiveness of a medical test. In this piece we discussed test complexity, and showed how it affects test turnaround time, ease of use and distribution, and the constraints and bottlenecks that can emerge when the entire world is trying to implement the same complex test process at the same time. Policymakers must take account of all of these factors when deciding how to allocate resources to a national testing program, as a weakness in any one area can impede the whole testing regimen, or even bring it to a complete halt.

The multivariate nature of testing naturally complicates the analysis, but we can simplify the decision by asking ourselves, what the purpose of testing is and what we are trying to achieve with our testing program. When we have a clear understanding of why we are testing, decisions around how to test will be simplified.

The PCR is a very accurate diagnostic test, and I’m sure there are many situations where that level of accuracy is desirable. A contagious outbreak is not one of them. Contagions need speedy, decisive responses from policymakers, and a test that is as complex and resource-intensive as the PCR is a drag on both data-collection and decision-making. Test accuracy can be sacrificed in a contagion if it means we get much timelier data on which to base our decisions. The need for a fast turnaround time dictates that we should have a significant amount of rapid testing capacity in our national arsenal with rapid antigen tests and the LAMP test process forming the backbone of our testing regimen.