OpinionDid you know code can kill?
Opinion
Did you know code can kill?
The conversation on system bugs can start with user experience and data but can very quickly reach human life, writes Eran Agmon, Chief Product Officer at DreaMed
We have all become accustomed to the fast pace of technology. Every developer and product manager has a duty to push forward at all times, to release the latest innovation everyone will talk about even though it may not be ready. The mentality is to bring the idea to market first and improve it later, bringing developers to the forefront. For years, the common practice in tech has found development and product teams working and building products together. The development has now become a process of trial and error, or A/B testing, as it is commonly known in professional jargon. In A/B testing, you try several options and see what actually works in the field, the level of user understanding, and general reactions to the product.
But what happens when an unfinished product can kill?
Until recently, innovation in the medical world focused primarily on medical devices for diagnostic purposes, in addition to new drug treatments. But the real change is now coming from the world of software and artificial intelligence. Technological initiatives now enable physicians to treat patients better, faster, and more accurately – including early error detection and information collection. Beyond improving physician efficiency and convenience, new capabilities make previously inaccessible treatments a reality. With the help of smart systems, general practitioners can provide a solution to problems previously required by specialists, alleviating high costs and lengthy waitlists. Patients can receive many services remotely, unlocking high-level medical care for populations far from medical centers. Medical technological advancement also aids doctors to catch human errors and mistakes in treatment and diagnostics, supporting decision-making that can save lives.
These new capabilities come with new challenges. After all, every program has bugs. While people are accustomed to living with them, it is almost impossible to develop a complex system without them. User interfaces should always be clear, but everyone has a story where they face an unfamiliar or buggy system. And of course, every algorithm has results that make everyone raise an eyebrow.
Algorithm development touches on the question of how many mistakes a program can make while staying relevant. If a customer accesses a clothing store's website, the algorithm will likely offer them products that interest them alongside products that don’t. As long as the customer finds something interesting to buy, they probably will not think too much about the “wrong” options. If Waze offers a route that turns out to be slightly longer than another every once in a while, it may cause frustration but not much else.
When software is responsible for human life, whether it chooses proper treatments, detects cancerous tumors, or assists other medical problems, these mistakes are not acceptable. In this case, the responsibility rests with people who in other situations are not accustomed to carrying that weight. Developers, testers, UX designers, and product managers involved in creating these important systems face a special challenge that echoes the first law of robotics: build a system that will never harm human life.
Developers and algorithm designers must create systems with a high level of scrutiny. This includes built-in safety mechanisms that even in unexpected situations will remain effective and undangerous. UX designers and product managers are entrusted with building clear systems for users navigating these systems to catch faults in human error and prevent harmful decisions from being implemented.
The conversation on system bugs can start with user experience and data but can very quickly reach human life. This work prolongs and increases the cost of medical software development. The challenges teams face change constantly and are altered by market demands and available technology. But when you look at the results of the medical software, the viewpoint shouldn’t be sales, but the potential to save lives. This could be early detection to successfully treat a disease, adjusting treatment options to suit complex needs, or helping patients deal with situations where they lack understanding and feel helpless. These are results that are not typical in other categories of technological and software development. However, the payoff is immeasurably greater than one that is strictly commercial or superficial. The medical world's development carries a great moral responsibility for professionalism and caution to avoid costly mistakes and save lives.
Eran Agmon is the Chief Product Officer at DreaMed
