Inpatient video has massive potential as a new form of patient monitoring and as a new kind of medical data type. One obstacle to overcome to realize this potential is the fact that video generates massive amounts of data, way more than any human can interpret (much less clinicians who are pressed for time taking care of sick patients). AI-powered computer vision has the potential to process vast quantities of data in real-time and deliver actionable insights to doctors, nurses, and other care providers. One approach to AI-powered computer vision is intelligent video summary: condense many hours of video into minute-long (if not shorter) video snippets for interpretation by clinicians.
In this blog post, I review LookDeep Health’s work with Duke Medical School on intelligent video summary. The work discussed was presented at the 2022 Machine Learning for Healthcare (MLHC) conference in an abstract entitled “Patient and Room Activity Video Summary (PRAVS): Rapidly Interpretable, Machine Learning-Generated Clinical Video Summaries of the Overnight Period.” As suggested by the title, I’ll talk about the significance of patient and bedside activity, the significance of the overnight period, and how we bring it together into a video summary prototype intended for use during morning rounds and similar clinical activities.
Video monitoring of patient and bedside activity
In prior work with Duke, LookDeep Health demonstrated the ability of AI-powered computer vision to quantify patient activity as well as bedside activity. This demonstration was in an IRB-approved study in the medical intensive care unit, and generated over 2000 hours of patient video data.
Patient activity is an emerging readout of patient health, including physical recovery/functional status, mental status (e.g. agitation, hypo/hyper-activity), and rest/sleep. Wearables have potential for quantifying patient activity; however, they are intrusive, only measure one extremity, and often aren’t tuned to patient activity in the hospital. Bedside activity, that is, activity attributable to bedside visitors (e.g. staff) as well as the patient, is a measure of clinical attention and a potential metric for patient acuity. Video is a natural method for quantifying bedside activity and highlights the limitations of wearables (one device per person) to track all motion that is relevant to patient care.
The significance of the overnight period
High acuity patients require frequent (if not continuous) monitoring. At the same time, it is critical to minimize overnight interruptions to promote patient rest and sleep. Computer vision technologies such as video summary can potentially reduce the number of requires overnight staff interruptions. Likewise, there is an ever-increasing recognition in the medical and popular press that patients need to get more sleep in the hospital. Video summary and associated data streams provide a means of assessing the impact and efficacy of care redesign efforts to promote patient rest and sleep.
Patient and room activity video summary
The core idea of patient and room activity video summary (PRAVS) is that minute-by-minute computer vision data streams can highlight busy periods in the overnight period. These busy periods can be related to patient activity or bedside activity. Once identified, the busiest period from overnight hour can be shown in a simple user interface for clinical review each morning during rounds.
The figure shows this concept in action. For each hour in the overnight period (9p-6a), the busiest 5-minute period is looped at 30x speed. The 3 busiest hours are flagged with a red border; the least busy, a green border; the middle busy, a yellow border. Finally, the underlying bedside activity time-series data are plotted above each video summary.
Clinical impact
I am very excited by the potential for overnight video summary in patient care in a variety of use-cases, especially rest/sleep monitoring and hypo/hyper-activity in delirium. By extending these video summaries to the daytime period, additional impact may be realized in tracking physical recovery for discharge planning and in assessing patient motion to prevent hospital-acquired pressure injury.
Duke authors
Samantha Parker, MD
Daniel Gilstrap, MD
Armando Bedoya, MD, MMCi
Patty J. Lee, MD
Kaivalya Deshpande, MD
LookDeep Health authors
Michael A. Choma, MD, PhD
Narinder Singh, MBA, MTM
Paolo G. Gabriel, PhD
Laura Urbisci, PhD
Dewei Hu
Tyler Troy, PhD
Tom Hata, PhD