Patient Advocacy

Artificial Intelligence in Pennsylvania Hospitals

Artificial intelligence is here. Very rapidly, it’s become embedded in nearly every aspect of our daily lives – including at the bedside, where AI’s potential for improved outcomes is undeniably real. But its potential for misuse and the subversion of in-the-moment clinical judgement is also very real.

The fact is, AI in many of its applications in healthcare is wrong approximately 20% of the time.  And it has a feature that causes it to “hallucinate” or make up data that doesn’t exist when that data is missing.  This is why it is PASNAP’s strong stance that AI must be regulated and challenged at the bedside on an ongoing basis – to make sure that it is learning and helping doctors and nurses and other caregivers, not hurting us and our patients.

This page explains what’s happening, why it matters, and how it affects the care we provide.  

AI Can Help, But Caregivers Have to Lead

On December 15, 2025, PASNAP President Maureen May testified before the House Communications and Technology Committee in Harrisburg in support of House Bill 1925 (see below for more details), which seeks to regulate AI in health care in Pennsylvania.  House Bill 1925, titled Regulation of the Use of Artificial Intelligence in Healthcare, is a new legislative effort aimed at putting guardrails around how AI is used in healthcare across the Commonwealth.

“We know from experience,” Maureen said in her remarks, “that we cannot rely on our employers to ensure that innovation serves caregivers and patients before it serves the bottom line. We need to make sure that clinical personnel remain in charge of making clinical decisions, even if AI is in the mix.” 

Watch Our Video

How AI Is Showing Up in PA Hospitals

In healthcare, AI and its algorithms are often produced based on patients' Electronic Health Records (EHRs) and other sources of data collected from patients, their environment, and from those who care for them.

  1. Staffing & Acuity Algorithms

    Hospitals are increasingly relying on software to determine patient acuity levels, staffing levels and patient assignments — even when the underlying data is incomplete or unrealistic. When the system is wrong, staffing is wrong.

  2. “Predictive” Clinical Alerts

    Some systems claim they can identify deterioration earlier than staff can. Instead, workers are often pulled away from care to chase false alarms, while subtle changes only a trained professional would notice go unrecognized.

  3. Remote & Virtual Monitoring

    Several PA facilities are experimenting with remote observation models that reduce the number of skilled workers physically present with patients. This fragments care and shifts responsibility onto off-site staff or lower-skilled roles.

  4.  Auto-Generated Notes & Care Plans

    Documentation systems can now produce templated notes and care plans automatically — sometimes missing the behavioral, emotional, or clinical details that matter most. These tools are often introduced without explanation, without training, and without any accountability when they fail. This introduces a liability for practitioners who use those templates.


PASNAP's Position: Additional Safeguards Needed for AI in Pennsylvania

In an internal PASNAP survey regarding AI in healthcare, conducted in December 2025, respondents reported being almost entirely shut out of decisions about AI at the bedside, distrustful of management’s intentions, and deeply concerned about where liability for AI decisions lies.  

If AI is going to be in healthcare — and it already is — it is PASNAP’s conviction that frontline caregivers must be at the table and the state must back us up in protecting the privacy and care of all Pennsylvanians. Healthcare workers in PA hospitals need:


Implementation and Ongoing Assessment: 

We need to institute AI implementation teams at every healthcare institution to assess, on an ongoing basis, where and for whom AI should be deployed as well as its successes and failures. These teams need to assess, prior to implementation of AI: 

  • Is AI appropriate in this setting and for these patients?
  • How is the system being piloted?
  • Is the staff training the system, is the system training the staff -- or is it both?
  • What is the system's success rate? And who is caught up in the failure group? Are there systemic ways that the particular AI tool is negatively impacting a certain group?

Training

Caregivers who are being asked to use AI need to be trained on how the AI works in order to be able to assess how to incorporate it into their clinical thinking. Caregivers need to know up front:

  • How the system was trained. What population was it trained on?
  • Is it a fixed model or can it learn?
  • What are the decision steps the AI performs? Or is it just a "black box"? Black bost systems should be avoided when it comes to diagnoses or assessments; it's critical that caregivers understand the basis upon which judgements are being made.

Protection of Clinical Judgment and Clinical Staff

Caregivers need guarantees that AI will support, not replace, clinical judgment – including assurance that they will not be held liable for AI mistakes. Multiple studies have shown that AI is wrong approximately 20% of the time. 

  • Clinical staff should not be subject to discipline or liability for not following a suggestion from AI, even if that suggestion turns out to be right and clinical staff wrong.
  • Clinical staff should also not be subject to discipline or liability for following a suggestion from AI that turns out not to be right.
  • AI should not be mandated -- it should be a tool, and there should be a choice to use it or not as the basis of any particular clinical decision.


Challenging and Honing the Tool

We need to encourage frontline caregiving staff at every institution in which AI is deployed to challenge the tool, to override its decisions when they are inappropriate or not in the best interests of the patient, and to teach the tool to be better. Caregivers with years of clinical experience and intuition are either adding to the data sets that AI uses,  or the tool – which is wrong approximately 20% of the time – is teaching us.


Patient Privacy and Disclosure

We need protections for patient privacy. At one of our hospitals, AI is used to write case notes by recording entire patient conversations — without patients being fully informed.


Tracking and Surveillance

We need institutions to be transparent about tracking employee behavior with AI and to what end they are doing so. Any time staff are being recorded or tracked in any way, that needs to be disclosed in advance. 


Protections for Newer Staff

Newer staff are more likely than experienced staff to follow AI suggestions uncritically, leading to worse outcomes. Those worse outcomes then become ingrained into the system as the system continues to get trained.

  • Doctors, nurses, and other clinicians in their first two years of practice should not be using AI for assessment or diagnostic purposes. Newer staff need time to develop the skill set to make clinical judgments on their own, without AI. If they don't, they may fail to develop thse skills or lose the skills they have.

What HB 1925 Does

HB 1925, titled Regulation of the Use of Artificial Intelligence in Healthcare, creates a statewide framework to ensure that AI is used with guardrails, including:

  1. Transparency

    Hospitals, insurers, and clinicians must disclose when and where they are using AI — including in clinical decision-making, patient communications, and insurance determinations. Patients and the public have a right to know.

  2. Human Clinical Judgment Comes First

    Any use of AI must still result in an actual, individualized assessment by a qualified human decision-maker. AI cannot replace the clinical judgment or professional responsibilities of healthcare workers.


  3. Bias & Discrimination Safeguards

    Hospitals and insurers must attest — with evidence — that their A.I. tools minimize bias and discrimination already prohibited by law. Regulators must be able to evaluate whether that is true in practice.


  4. Accountability & Oversight

    HB 1925 requires facilities and insurers to submit compliance information to the Department of Health or Insurance, creating public oversight where none currently exists. This sets a baseline for transparency and gives the Commonwealth tools to address misuse or harmful practices.

What This Bill Means for PASNAP Members

HB 1925 represents an important shift: If the bill passes, hospitals and insurers for the first time will not be able to quietly roll out AI systems without explaining what they do, how they work, or whether they are safe. For PASNAP members, this means:

  • You cannot be replaced by an algorithm.

  • Your clinical assessment cannot be overridden by software.

  • Facilities must disclose when AI is being used.

  • Insurers must reveal when AI drives a denial or prior authorization decision.

  • The Commonwealth gains oversight powers it has never had before.


This legislation sets a foundation PASNAP can build on as AI becomes more common across the healthcare industry.


footer

We use our collective strength to advocate for things like safe staffing, universal access to healthcare, and prevention of harassment and violence against healthcare workers. Our advocacy was instrumental in passing Act 102, Pennsylvania's ban on mandatory overtime for healthcare workers.

Group 3
Flexible Box
Flexible content lorem ipsum dolor sit amet orci eiusmod imperdiet quam curabitur vel lobortis ornare.
Watch Our Video
Relevant Contracts Your Rights Opt In for Email and Text Alerts from Your Local
Name
By providing your cell phone number you consent to receive recurring updates from PASNAP by automated text message. Msg & Data rates may apply. Text HELP for help, STOP to end. Privacy Policy

3031 Walton Road, Suite C-104 Plymouth Meeting, PA 19462-2326

[email protected]

Copyright © 2022-2025 PASNAP. All Rights Reserved.