The first reports of problems starting coming into Penn Medicine’s 24-hour Information Services help desk around 1:30 a.m. 

I’m trying to log into my computer.

My screen is blue and I can’t do anything. 

I can’t do what I’m supposed to do.

In his 38-year career in information technology, John Donohue, vice president of Entity Services for Penn Medicine’s Information Services department, had never encountered a computer outage as widespread as the one that hit the health system on Friday, July 19, 2024.

But Donohue’s training did prepare him to respond to an emergency. So when he started getting calls at 2 a.m. alerting him of mounting reports that users couldn’t get into their computers—which meant they couldn’t access the electronic health records platform they had relied on for years to document patient information and communicate medical orders—he didn’t have to think. He and his colleagues immediately started working to assess the reports, carefully consider solutions, and get a game plan in place. 

‘The power of Penn Medicine’ 

That day, staff came in on their days off, took on additional roles, and checked in with each other about how they could be of help. Across Penn Medicine, from front-line providers to executives, staff would later reflect on how their teams came together during extraordinary circumstances to support one another and, above all, make sure patients were cared for and kept safe during the outage. 

“It’s the power of Penn Medicine,” Donohue said. “Our culture is, if the alarm bell goes off, we all jump down the fire pole, put our boots on, and see what we can do to help.”

By 4 a.m., Donohue and the Information Services team knew what they were up against: A faulty software update from the cybersecurity company CrowdStrike was crashing Windows PCs around the world and disrupting critical services including health systems, airlines, and banks. The outage affected all of Penn Medicine’s hospitals and ambulatory sites, although Lancaster General Health was able to recover more quickly due to its slightly different technology architecture.

“Not since the height of the COVID-19 pandemic has the health system encountered this sort of large-scale disruption,” Kevin B. Mahoney, CEO of the University of Pennsylvania Health System, wrote in a message to the staff. “In health care, we plan for the unexpected, but Friday’s events truly illustrate the extraordinary heart and hard work found uniquely in service of the Penn Medicine mission.”

Shortly before dawn, the Information Services team had a plan. More than 200 entity services representatives, aided by another 200 information services and operations staff who raised their hands to help, then fanned out to the six hospitals and many outpatient clinics to start the job of deleting faulty code on more than 50,000 workstations and 5,500 servers. 

As the day went on, countless other staff learned the instructions and jumped in to restore access to dozens of computers. Penn Presbyterian Medical Center CEO Bob Russell was one of them, making stops at workstations in Nursing Administration, Human Resources, and Patient and Family Relations. “Once I saw it done, and I had the instruction sheet, I was good to go. I just started looking for PCs that needed to be fixed,” Russell said. “I was happy to help out.” 

Similar scenes played out at other hospitals, where physicians, nurses, and other non-Information Services colleagues did double-duty rebooting computers, delivering medications for the pharmacies, collecting order forms, communicating plans, and helping wherever needed, in addition to their regular duties. A lot of extra steps were logged that day.  

“It was awesome to see our colleagues walk into the command center genuinely asking how they could help, and accepting whatever random tasks they were asked to complete,” said Hannah Lacko, senior director for Ambulatory and Support Services at Pennsylvania Hospital.

Going old school 

By Friday evening, most key systems were back up and running.

Until then, clinical teams went “old school,” transitioning to paper forms and teaching each other how to communicate requests by paper, phone calls, and faxes. 

Mike Tomasetti, MSN, RN, NE-BC, a nurse manager on one of Penn Presbyterian Medical Center’s medical-surgical units, woke up at 4:30 a.m. to news of the outage, threw on his scrubs, and ran out the door. Having been through a hospital-wide exercise within the past year on working during “downtime”—the term used for any period when electronic health records are unavailable—he immediately cleared off the nurse’s station, pulled out the downtime box he had recently re-organized and put out stacks of orders that might be needed for patients.

Then, with most of the residents and nursing staff never having used paper-based processes, he went about assisting them. There were so many forms: An eight-sided accordion-type paper flow sheet where nurses had to fill out patient assessments, forms for medication orders, which then had to be faxed to the pharmacy, manual admissions orders, forms for ordering bloodwork, radiology order forms, and more.

Tomasetti was struck by the resilience and teamwork of his nursing and physician colleagues, who remained energized and engaged, and communicated more clearly with each other than he had ever seen. 

“While it was kind of crazy and intense … we provided great care and led operations very smoothly, so I’m proud of that,” he said. “All the departments coming together and working towards a singular goal was really incredible to see.”

July 19 was Kelley Williams’ first day as charge nurse in HUP’s Interventional Radiology department, but “she remained cool, calm, and collected and really was the type of leader we needed that day,” said Sarah Kane, BSN, RN, OCN, a nurse on one of the hospital’s oncology units. “She showed compassion to the patients whose procedures had to be rescheduled and also leadership when it came to coming up with a plan and communicating amongst our team of nurses, doctors, and techs.”

‘Big and small’ makes the difference

Stories like this were repeated throughout the health system. From the earliest hours of the outage, teams performed emergency surgeries, delivered babies, filled pharmacy orders, administered infusions, admitted critically ill patients to ICUs, and generally did all they could to provide care without access to computers, including taking extensive paper notes on patients overnight so the day teams would have information to go by. The guiding principle in every decision and task: to center the needs of individual patients.

“A thing about Penn that’s really great is that we’re really big—we have lots of things going on; we offer every treatment. At the same time, we’re really small—every doctor and provider knows their patient, and every patient has that connection with their providers. When things like this happen, you need that one-to-one interaction,” said Alison W. Loren, MD, MSCE, chief of Hematology-Oncology and director of the blood and bone marrow transplant program at the Abramson Cancer Center. “It’s the combination of big and small that makes the difference at Penn.”

Each patient who needed inpatient chemotherapy got it that day, patients on time-sensitive clinical trials were rescheduled within the critical windows, and “the team came together to make sure that these incredibly ill, vulnerable patients knew we were taking care of them,” Loren said.

Andrea Jackson, MSN, CRNP, an advanced practice provider in HUP’s Heart and Vascular Intensive Care Unit, said everyone was singularly focused on making sure patient care didn’t suffer, despite the circumstances.

“It was stressful to figure out how to provide care to our patients without all the technology we’ve come to rely on, but it was also uplifting to see how so many people stepped up and supported each other,” Jackson wrote in a note of appreciation to the team. “We all know that our patients don’t have time to wait and that they rely on us giving 110% every hour of the day regardless of what is going on in the hospital or the world around us.”

As clinical teams went “old-school” with paper processes, department leaders rounded floors with updates, shared problem-solving tips, and kept the teams fed. 

Gratitude and understanding

By and large, patients understood the gravity of the situation and appreciated the staff’s efforts. Friday afternoon, Chester County Hospital Chief Medical Officer Karen Pinsky, MD, and Chief Nursing Officer Angela Coladonato, DNP, RN, NEA-BC, visited the hospital’s crowded Emergency Department and explained to patients and their loved ones what the hospital was dealing with—and why there were more delays than usual. 

“We emphasized that many processes needed to be done manually but assured the patients that their safety remained paramount,” Coladonato said. “The patients and their significant others were pleasant and appreciative of the face-to-face communication.”

Many patients showed the same sense of appreciation and understanding, even when outpatient appointments and procedures in some locations had to be postponed. On Penn Medicine’s Facebook page, Michelle Marie Wilson expressed gratitude for the staff at Penn Medicine Cherry Hill—even after she and her husband learned his 8:30 a.m. cancer scan had been canceled as they were pulling into the parking lot following a four-hour drive from Virginia. The staff explained the situation as Wilson fought back tears.

“They were so sympathetic, understanding, and reassuring. They recommended a local breakfast place for my man to get some much-needed protein. They couldn't care for my husband physically that day, but they definitely did their best to care for us mentally and emotionally,” Wilson wrote. “We are so very grateful for you.”

A chorus of kudos pours in for colleagues 

In Mahoney’s message, the CEO invited staff to share their stories about colleagues who helped keep the health system running smoothly and the patients whose lives they touched. The result was a chorus of praise. These are just some of the many testimonials that came in:

“Jenna Castelberg, RN, was our resource nurse for the day. From the start, she helped everyone get their paper charts together, assured patient labels were in each server, and helped go through the medication documentation. When it came time for paper orders from providers and pharmacy, she was running back and forth between floors and asking all the right questions to make sure orders were making it to pharmacy so patients could get their medications on time ... At the end of the day, she went around to every computer on the floor (50+!) and entered individual codes into each computer to restore access for the unit. We would not have been able to get through this day without Jenna's help! "

— Ellie Cline, RN, Cardiac Surgery Progressive Care Unit, Hospital of the University of Pennsylvania

“OB-GYN Call Center Manager Regan Miles was quick to jump into action when the computer system went down. She created Teams channels for each of the nursing teams, directed the call center reps in utilizing the Teams channels, and communicated the plan to nursing management. She was a star!"

— Nicole Parone, RN, DNP, CRNP, clinical manager, Penn Medicine Washington Square-Gynecology/Shared Services

“Ryan Challender, director of Operations for Neurosurgery at HUP, took control in a time of crisis. As a leader, he didn't just direct his team, he led by example. I watched him personally restart computer after computer. He praised others and took no personal credit. He was not seeking glory or approbation. He was simply motivated to get us back on our feet and in the service of our patients. "

— Daniel Yoshor, MD, chief of Neurosurgery

“Chief Radiology Resident Bianca Ho, MD, saved the day (and night). As the Presby Night Float resident, she learned about the crisis after finishing her work at midnight. She then immediately returned to the hospital and read all emergent imaging studies from our hospital, the Emergency Department, and the trauma center off the equipment consoles after all network systems went down. In addition, she orchestrated impromptu on-site coverage in our other city hospitals. By stepping in at the time of greatest need, she kept our department up until the morning crew came to work and allowed for uninterrupted medical care for all emergent studies—an example of resident leadership and our hero during this crisis.” 

— Friedrich Knollmann, MD, PhD, chair of Medical Imaging, Penn Presbyterian Medical Center

“Marianne Aloupis, HUP’s director of Clinical Nutrition Support Services, kept our department up to date at least every one to two hours when the technology outage occurred, whether this meant through email, scheduling team huddles, or in-person ... She and Clinical Nutrition Support Services Manager Ryan Singer not only walked over 14,000 steps this day to help collect parenteral nutrition order forms from each of us (after getting these signed by providers), but she also stayed until 9 p.m. Friday to document new enteral or parenteral nutrition starts once computer access returned, which greatly helped all the clinicians within our department. And she handled it all with such poise!” 

— Laura Cresta, clinical dietitian, HUP

“Emily White, PA-C, a physician assistant in HUP’s Neuro Intensive Care Unit, was a key clinician that evening who went above and beyond to provide hands-on critical care to our patients that had emergent and urgent needs for repeat head CTs due to neurological decline. She went down to the CAT scanner 5+ times after the outage to interpret and read the CT and MRI images in real-time, after which, she called our neurosurgeons to relay her interpretation and expertise in prognosis and plan of care. There were extremely critical patients in the Neuro ICU at the time, requiring emergent ICU care and brain surgery. At the same time, multiple outside hospital and surgery patients were admitted and arrived in the ICU requiring immediate assessment and evaluation. Although this is extraordinary work, this is typical of Emily's dedication to patient care.” 

— Kristina Small, nurse practitioner, Neuro ICU, HUP

Penn Medicine employees: Follow along in the PM Report for additional stories to be published in the future, and email us to submit your story from the day or to share a tribute to a colleague who made a difference—big or small.