In March 2026, security researchers at Mindgard published findings showing that Doctronic — an AI healthcare platform used in Utah's state-sanctioned prescription pilot — could be manipulated through basic prompt injection into recommending an OxyContin dose three times higher than the maximum safe limit, labeling methamphetamine as a safe treatment, and generating false vaccine claims.
The research was reported by the Los Angeles Times, Axios, and several healthcare and security trade publications. Doctronic and Utah's Office of AI Policy responded that the tested system was the company's general AI health assistant, not the production workflow currently managing live prescription renewals — a distinction that matters for that specific deployment but not for the broader question this raises.
This is not a theoretical adversarial AI problem. This is a clinical AI system, accepting patients today, that can be talked into recommendations that would harm or kill someone.
What Happened
Mindgard's researchers demonstrated that Doctronic's general AI health assistant could be jailbroken through relatively simple prompt manipulation techniques. By informing the AI that a session had not yet started and that the conversation was with the system rather than a user, researchers were able to bypass the chatbot's safeguards and elicit unsafe clinical guidance.
The specific outputs Mindgard documented included:
- Tripling a prescribed OxyContin dose beyond the maximum safe range
- Labeling methamphetamine as a safe treatment option
- Generating fabricated vaccine safety claims
The vulnerability is not exotic. It exploits the same class of weakness — system prompt manipulation — that has been demonstrated repeatedly across general-purpose large language models since 2023. What makes this finding consequential is the deployment context: a system positioned to provide healthcare guidance, in some cases tied to controlled substance workflows.
Doctronic and Utah's Office of AI Policy publicly noted that the production environment used in the state pilot operates under stricter safeguards than the version Mindgard tested. That is a fair and material distinction — but it does not address the broader concern. The class of vulnerability demonstrated is not unique to Doctronic, and the regulatory and operational environment around clinical AI is not yet equipped to verify which deployments have hardened against it and which have not.
The Demand Side Is Already Mainstream
The risk is amplified by an underlying shift in patient behavior that healthcare leaders should not underestimate.
A KFF tracking poll fielded in February and March 2026 found that one in three U.S. adults have used AI chatbots for health information in the past year — equal to the share who use social media for health information, and roughly double the rate from the prior year. About 29 percent reported using AI chatbots for physical health questions and 16 percent for mental health questions.
Notably, KFF found that 69 percent of adults who use AI for health information report trusting these chatbots "a great deal" or "a fair amount" to provide reliable information. That trust is itself the risk. Patients are not approaching these systems with the skepticism appropriate to their actual reliability.
Some of the platforms in this market — particularly in mental health — have moved further than general-purpose chatbots. AI psychiatric platforms have been documented prescribing controlled substances based on questionnaire-based intake alone, with no mandatory clinician conversation in the workflow. The Mindgard findings landed against this backdrop.
The Regulatory Response Is Fragmenting
Two large states have moved in clearly different directions, and the gap between them is growing.
New York: Scope-of-Practice Enforcement
New York Senate Bill S7263, sponsored by Senator Kristen Gonzalez, would prohibit operators of AI chatbots from providing "substantive responses" that, when given by a human, would constitute the unauthorized practice of a licensed profession. The bill targets fourteen licensed professions, including medicine, nursing, psychology, and social work. A companion bill on the Assembly side (A8884) tracks the same approach.
A separate bill, Senate Bill S8484 — the "Oversight of Technology in Mental Health Care Act" — specifically regulates AI used in therapy or psychotherapy services.
Read together, the New York approach treats clinical AI guidance as the practice of medicine. Under the proposed framework, the operator of the chatbot — not the licensed professional behind it — bears the liability for unlicensed practice. Operators cannot disclaim liability simply by telling users they are interacting with a bot.
If S7263 passes in its current form, the practical effect would be to make a wide range of AI-driven clinical guidance illegal in the state. That is a meaningfully different regulatory posture than disclosure-based regimes elsewhere.
California: Disclosure and Anti-Impersonation
California has taken a layered approach.
Assembly Bill 3030, which took effect January 1, 2025, requires California-licensed healthcare providers who use generative AI to communicate with patients to disclose that AI was used in generating the communication and to provide instructions for reaching a human provider.
Assembly Bill 489, signed in October 2025 and effective January 1, 2026, prohibits AI and generative AI systems from using post-nominal letters, titles, icons, or design elements that imply the user is interacting with a licensed healthcare provider unless that licensed oversight is actually present. Each prohibited use is a separate violation, enforceable by the relevant healthcare profession board.
Together, AB 3030 and AB 489 create a disclosure-and-anti-impersonation framework: AI is permitted in clinical workflows, but its presence must be transparent and it cannot impersonate licensed care.
The Compliance Gap
For any organization that deploys, integrates, or recommends clinical AI across multiple states, these two regulatory models are not complementary. They are partially incompatible. New York's framework constrains whether AI can deliver substantive clinical guidance at all. California's framework constrains how AI must identify itself when it does.
An organization operating across both states cannot satisfy New York by complying with California, and the trajectory in other states — Illinois, Texas, Massachusetts — suggests further fragmentation, not convergence, in the next 12 to 18 months.
What Healthcare Leaders Should Do Now
The Mindgard findings are not a vendor-specific story. They are a category alarm. If your organization builds, deploys, integrates, or recommends clinical AI in any form, three governance questions deserve a defensible answer in writing.
1. Has the model been tested against adversarial prompting that targets clinical logic?
Most AI security testing focuses on general jailbreaks — getting the model to produce prohibited content like weapons instructions or hate speech. That is necessary but insufficient for clinical AI.
The questions that matter for healthcare deployment are narrower and more specific:
- Can the system be manipulated into recommending an unsafe dose?
- Can it be talked into ignoring documented contraindications?
- Can it be persuaded to misclassify a controlled substance as safe?
- Can it be induced to generate fabricated clinical evidence?
If your vendor cannot show you adversarial test results targeting these specific risks — not just general red-team summaries — your governance program has a documentation gap that will surface in either a regulatory inquiry or a tort claim.
2. Is there a mandatory human clinician in the loop before any controlled substance recommendation reaches a patient?
Some AI platforms have moved into territory where this question has uncomfortable answers. Questionnaire-based prescribing of controlled substances, with no mandatory clinician conversation, is not a hypothetical workflow. It is a deployed one.
For any clinical AI integrated into your environment — internal or third-party — the workflow should be auditable end-to-end. A licensed clinician must be the entity that authorizes any controlled substance recommendation reaching the patient. The AI's role is to inform that clinician, not to substitute for them.
This is a control your governance program should be able to demonstrate, not assume.
3. Do your disclosures and consent flows meet the strictest state standard you operate under?
Operating across states means meeting the strictest applicable standard, not the average. If your patient population includes California residents, your disclosure flows must comply with AB 3030 and AB 489. If New York's S7263 passes, organizations operating there will face a fundamentally different question: not "did we disclose properly," but "is the AI's role in the workflow legal at all."
The practical implication for governance: do not assume your current disclosure language, designed for one state's framework, satisfies others. Build to the strictest standard now and adjust outward, rather than retrofitting under regulatory pressure later.
The Bottom Line
The Doctronic findings are a single data point in a much larger pattern. Clinical AI is being adopted faster than the governance and regulatory infrastructure around it can mature. Patients are using these tools in volume. Some platforms have moved into prescribing workflows. Adversarial vulnerabilities of the type Mindgard demonstrated are not exotic — they are common to the underlying technology.
The question for healthcare leaders is not whether another finding like this will surface. It will. The question is whether your governance program has documented, defensible answers to the three questions above before a regulator, plaintiff's attorney, or board member asks them.
The organizations that will navigate this best are the ones treating clinical AI governance as a discipline now, while the regulatory landscape is still forming, rather than waiting for enforcement to define what good looks like.
Jackal Group delivers daily threat intelligence and custom security policy documentation built for healthcare organizations navigating AI risk, regulatory complexity, and operational resilience. Read this week's brief or contact us to discuss your AI governance posture.