Is it time to plan to retire the A/B testing?
Copyright© Schmied Enterprises LLC, 2021
The pandemic showed that there is a lot that we can do to enhance science and apply it into practice.
The traditional way of doing science – the A/B testing – faced a strong challenge. The world needed lifesaving Covid-19 vaccines fast. The traditional approach to test the real vaccine and a placebo on a random set of people has proven to have issues.
Science has a new ally, and that is data. Data is available and experiencing exponential growth. We collect more as devices become more affordable. Historic information is kept longer. This will have a tremendous impact on research. Access can be purchased cheaply. Raw information can be shared, making collection and storage affordable, even in the long run. Historic results can even be compared to point out the dynamics.
Clean comparative analysis can also be done quickly. Your data is there; you can correlate it with other research. There is no interference between studies. You can slice, dice, and clean the data set so that you can verify your results.
Statistical regression, correlation analysis, and machine learning can be applied. All these provide strong evidence only when used with computational tools. As they become cheaper, research becomes more affordable as well, and it can scale up to find granular results. It will allow us to research rare illnesses. It will allow us to find unexpected interactions.
Data is very easy to reuse. There are only a few limitations. One limitation is privacy, but there are numerous methods to limit the harm that data can do with careful filtering. A lot can be done in this space. Most importantly, as data becomes abundant, the focus will shift to protecting individuals by limiting the actions that can be done using it.
One important fact is that data comes from the real world rather than from an experimental scenario. While laboratory results are important as final proof of concept, rich information can be collected and verified to test a hypothesis and the impact on real people and society.
Money. First, it takes time and money to carry out tests. You need to collect participants, do the legal work and obtain consent, and collect the results. All this costs money. It is also hard to find volunteers and it is even more difficult to pro-rate the results to the general population.
Freedom. Second, A/B testing was invented when society was less democratic. The general population has more access to information in the data age. Hierarchies that were built on the superiority of one's knowledge have become less important. Hieararchical ethical practices were all based on information that was not available to the public. A test that is based on withholding truth will slowly lose its credibility.
Information. A/B testing also withdraws a part of the freedom of an individual. This may not have an impact in the case of a single test, but if science uses more A/B testing, eventually people will anticipate being lied to all the time. Do I have the freedom to know what I eat? Am I being tested? Do I get real stuff or just a placebo?
Psychology. A lot of psychological research can be done on this topic. Does A/B testing have a long-term psychological impact on an individual who is never tested? Does it change behavior or personality traits? Does it affect future results? I believe A/B testing has a long-term psychological cost embedded in it.
Interference. A/B testing usually relies on a statistically significant data set. This has a limitation as it does not scale well. Eventually, it requires a general population, limiting the usability of it on a town, for example. Moreover, multiple tests on the subset of the same population can have negative effects on each other. Let us take the example of the Covid vaccine. Is the accidental death of six vaccinated people because of the vaccine alone, or is it due to the interaction with the A/B test of another experimental medicine or a psychological test carried out through one's browser? What is the loss of statistical evidence in this case? Not many serious A/B tests can be done on the same population at the same time without anticipating these effects. Repeating helps, but it incurs costs.
Learning. The EU has recently pushed into AI territory. There are proposals to limit the learning on the real general population, for example, to prevent accidents. Even more important is the impact on human learning. Will humans and AI learn and give different reactions when realizing they may be under testing? Humans may learn the effects of medicine and the immune system may simulate responses. How much do tests and artificial learning affect individual freedom and happiness?
Temporary. Also, society changes. There may be issues that affect the current results. Only repeated expensive tests can prove that these results were temporary. All this takes time and money.
Ethics. Conscience also matters. Using data does not require changing the real world. It still provides the evidence necessary. A/B testing needs intrusion and intervention. Will you exclude expectant mothers? Will you exclude people with mental illness to prevent distress or suicide? What is the impact of these on the usefulness of the results?
Psychosomatic. Statistical medicine also has issues in that it embraces the general population. Side effects happen and they usually affect only a subset of the population. Is this because of a domino effect of experiences in the past? We all have life experiences with learned responses and emotional attachment. Is someone's cultural background or race important to be excluded or included? Should extensive quantitative analysis be done before accepting participation?
Theatre. There was a case of an alleged terrorist who pressed a button, which the FBI created in Portland, Oregon. The case was very interesting. Somebody commits a criminal act but in a simulated environment. No harm could be done; the environment is fake. Still, the individual would carry out the act otherwise, not knowing about the situation. Is this crime, or should they be treated for delusions and mental illness? A lot of research will be done in the future regarding such issues, I think.
Equality. A/B testing divides people, even if it is random and not arbitrary. Still, if the impact is strong enough, this carries a sad message. Those people will do similar acts in their private and professional lives. Is dividing people ethical? Is telling fake news to test the reactions worth a positive relationship?
Visibility. Also, participating in impactful research may improve visibility, and the life experience makes people stand out. The resulting personality changes may affect them in the long term. A/B participants may favor more visibility, or they may be open to more observation. Some, on the other hand, may become withdrawn or hostile.
Betting. A/B testing has the economic structure of betting. This may collide with the ethics of people who avoid agent-principal relationships. A bet is made on the health or happiness of others, and they face many consequences as stated above.
The author has two important suggestions regarding A/B testing. If existing data can be used, it is better to avoid A/B testing to prevent the side effects above. If A/B testing must be carried out anyway due to scientific and regulatory reasons, consider that full information is provided and consent is requested. It is also important to accept that A/B testing can have long-term health and psychological impacts, so that participants need to be checked periodically for the acceptance of the full result of the research.