IBM Watson for Oncology unsafe treatment recommendations
Internal IBM documents reported that the Watson for Oncology AI system had, in multiple instances, recommended unsafe or incorrect cancer treatments to doctors using the tool.
The damage was public. The root cause was preventable.
Internal IBM documents reported that the Watson for Oncology AI system had, in multiple instances, recommended unsafe or incorrect cancer treatments to doctors using the tool.
What the software actually got wrong
The system was trained largely on a small number of hypothetical 'synthetic' patient cases developed with a single US hospital rather than on large volumes of real, diverse patient data, so its recommendations often did not generalize well to real-world cases and local treatment guidelines.
The system was trained largely on a small number...
The system was trained largely on a small number of hypothetical 'synthetic' patient cases developed with a single US hospital rather than on large volumes of real, diverse patient data, so its recommendations often did not generalize well to real-world cases and local treatment guidelines.
Requs AI Edge Case flags this exact pattern at the requirements and architecture stage — before a single line of code implementing it exists — so the assumption behind it gets challenged while it is still cheap to fix.
Beyond code coverage and "shall" testing
Root causes like this one rarely show up in code coverage or requirements-compliance testing, because nobody wrote a requirement anticipating the specific edge case that broke. Requs AI Edge Case and Requs AI Software FMEA are built to surface exactly this class of overlooked failure mode — before the software is written.
Surfaces this before code exists
Identifies edge cases like this one at requirements and architecture time, using the Common Defect Enumeration to catalog failure patterns seen across hundreds of real-world software failures — including this one.
Connects the failure mode to the hazard
Traces this class of root cause directly to the system-level hazard it can produce, so a defect pattern like this one gets flagged during design review instead of after it ships.
Find the overlooked root causes before they ship.
Schedule a demonstration, or explore how Requs AI Edge Case and Software FMEA use the Common Defect Enumeration.
See real failures this would have caught
Failures like this one can be found before release. See how Requs AI finds them early:
- Software FMEA tool — find failure modes and root causes before code ships
- Software reliability prediction — predict defects before code is written (IEEE 1633)
- Software risk assessment checklist — spot the risks behind failures like this