NYC 'MyCity' government chatbot giving illegal business advice
New York City's official AI chatbot for small business owners was found to routinely give incorrect and sometimes illegal advice, including wrongly saying landlords could refuse tenants on rental assistance and that businesses could refuse cash payments; the roughly half-million-dollar tool was eventually ordered shut down by the incoming mayor as 'functionally unusable.'
The damage was public. The root cause was preventable.
New York City's official AI chatbot for small business owners was found to routinely give incorrect and sometimes illegal advice, including wrongly saying landlords could refuse tenants on rental assistance and that businesses could refuse cash payments; the roughly half-million-dollar tool was eventually ordered shut down by the incoming mayor as 'functionally unusable.'
What the software actually got wrong
The Microsoft Azure-powered chatbot generated its answers using a general-purpose language model without sufficiently reliable grounding in New York City's actual laws and regulations, so it would confidently state plausible-sounding but incorrect legal guidance, and the city had not built adequate safeguards to catch or correct these errors before the tool was made available to the public.
The Microsoft Azure-powered chatbot generated its answers using a...
The Microsoft Azure-powered chatbot generated its answers using a general-purpose language model without sufficiently reliable grounding in New York City's actual laws and regulations, so it would confidently state plausible-sounding but incorrect legal guidance
Requs AI Edge Case flags this exact pattern at the requirements and architecture stage — before a single line of code implementing it exists — so the assumption behind it gets challenged while it is still cheap to fix.
The city had not built adequate safeguards to catch...
the city had not built adequate safeguards to catch or correct these errors before the tool was made available to the public
Requs AI Software FMEA traces this failure mode back to the system-level hazard it feeds, tagging it against the Common Defect Enumeration so it surfaces in review instead of in the field.
Beyond code coverage and "shall" testing
Root causes like this one rarely show up in code coverage or requirements-compliance testing, because nobody wrote a requirement anticipating the specific edge case that broke. Requs AI Edge Case and Requs AI Software FMEA are built to surface exactly this class of overlooked failure mode — before the software is written.
Surfaces this before code exists
Identifies edge cases like this one at requirements and architecture time, using the Common Defect Enumeration to catalog failure patterns seen across hundreds of real-world software failures — including this one.
Connects the failure mode to the hazard
Traces this class of root cause directly to the system-level hazard it can produce, so a defect pattern like this one gets flagged during design review instead of after it ships.
Find the overlooked root causes before they ship.
Schedule a demonstration, or explore how Requs AI Edge Case and Software FMEA use the Common Defect Enumeration.