Publications

The OpenOximetry Dataset stores clinical and lab pulse oximetry data. It supports measurements of arterial oxygen saturation (SaO2) by arterial blood gas co-oximetry and pulse oximetry (SpO2), alongside processed and unprocessed photoplethysmography (PPG) data and other metadata. This includes skin color measurements, finger diameter, vital signs (e.g., arterial blood pressure, end-tidal carbon dioxide), and arterial blood gas parameters (e.g., acid-base balance, hemoglobin concentration). All data, from desaturation studies to clinical trials, are collected prospectively to ensure accuracy. A common data model and standardized protocols for consistent archival and interpretation ensure consistent data archival and interpretation. The dataset aims to facilitate research on pulse oximeter performance across diverse human characteristics, addressing performance issues and promoting accurate pulse oximeters. The initial release includes controlled lab desaturation studies (CLDS), with ongoing updates planned as further data from clinical trials and CLDS become available.

Background: Some pulse oximeters perform worse in people with darker skin, and this may be due to inadequate diversity of skin pigment in device development study cohorts. Guidance is needed to accurately and equitably characterize skin pigment to ensure diversity in research cohorts. We tested multiple methods for characterizing skin pigment to assess comparability and impact on cohort diversity.
Objectives:

• Assess reliability and comparability of common skin pigment measurement methods
• Compare findings from different anatomical sites
• Demonstrate that pigment cannot be assumed from US National Institutes for Health (NIH) race categories

Methods: We used three subjective methods (perceived Fitzpatrick pFP, Monk Skin Tone MST and Von Luschan VL) and two objective methods (Konica Minolta CM-700d spectrophotometer and Delfin Skin Color Catch DSCC colorimeter) for individual typology angle (ITA), across multiple measurement sites in adults. We calculated ΔE to estimate operator perceptibility thresholds for subjective methods and to determine reproducibility for objective methods. We used each method to categorize participants as ‘light, medium, or dark’ and compared the impact of method selection on cohort diversity.

Results: We studied 789 participants, with 33,856 assessments. The MST had the widest luminosity range, and VL had the least discernible adjacent categories. With ‘dark’ defined as ITA <-30°, 14% of participants were categorized ‘dark’ as compared to 26% by pFP or 16% by MST. Approximately half of the ‘dark’ cohort had an ITA <-50°. With an ITA threshold <-50°, only 7% of the cohort was categorized as ‘dark.’ When ‘Black or African American’ self-identification was used to define ‘dark’, 23% of the cohort was categorized as such. Each self-assigned NIH race category included a wide range of ITA and subjective scale categories. Both ITA and L* from the KM-700d and DSCC demonstrated strong correlation (⍴ > 0.7).

Conclusion:  Common methods for skin pigment characterization, especially the use of race or subjective scales, have significant limitations. When applied to the same cohort, different methods yield significantly different results, and some may overestimate diversity. Previously published ITA thresholds for defining ‘dark’ skin are too light and lead to underrepresentation of people with darker skin.

The OpenOximetry Repository is a structured database storing clinical and lab pulse oximetry data, serving as a centralized repository and data model for pulse oximetry initiatives. It supports measurements of arterial oxygen saturation (SaO2) by arterial blood gas co-oximetry and pulse oximetry (SpO2), alongside processed and unprocessed photoplethysmography (PPG) data and other metadata. This includes skin color measurements, finger diameter, vital signs (e.g., arterial blood pressure, end-tidal carbon dioxide), and arterial blood gas parameters (e.g., acid-base balance, hemoglobin concentration). Data contributions are encouraged. All data, from desaturation studies to clinical trials, are collected prospectively to ensure accuracy. A common data model and standardized protocols for consistent archival and interpretation ensure consistent data archival and interpretation. The dataset aims to facilitate research on pulse oximeter performance across diverse human characteristics, addressing performance issues and promoting accurate pulse oximeters. The initial release includes controlled lab desaturation studies (CLDS), with ongoing updates planned as further data from clinical trials and CLDS become available.

Background

Fingertip pulse oximeters are widely available, inexpensive, and commonly used to make clinical decisions in many settings. Device performance is largely unregulated and poorly characterised, especially in people with dark skin pigmentation.

Methods

Eleven popular fingertip pulse oximeters were evaluated using the US Food and Drug Administration (FDA) Guidance (2013) and International Organization for Standardization Standards (ISO, 2017) in 34 healthy humans with diverse skin pigmentation utilising a controlled desaturation study with arterial oxygen saturation (SaO 2) plateaus between 70% and 100%. Skin pigmentation was assessed subjectively using a perceived Fitzpatrick Scale (pFP) and objectively using the individual typology angle (ITA) via spectrophotometry at nine anatomical sites.

Findings

Five of 11 devices had a root mean square error (ARMS) > 3%, falling outside the acceptable FDA performance range. Nine devices demonstrated worse performance in participants in the darkest skin pigmentation category compared with those in the lightest category. A commonly used subjective skin colour scale frequently miscategorised participants as being darkly pigmented when compared to objective quantification of skin pigment by ITA.

Interpretation

Fingertip pulse oximeters have variable performance, frequently not meeting regulatory requirements for clinical use, and occasionally contradicting claims made by manufacturers. Most devices showed a trend toward worse performance in participants with darker skin pigment. Regulatory standards do not adequately account for the impact of skin pigmentation on device performance. We recommend that the pFP and other non-standardised subjective skin colour scales should no longer be used for defining diversity of skin pigmentation. Reliable methods for characterising skin pigmentation to improve diversity and equitable performance of pulse oximeters are needed.

Funding

This study was conducted as part of the Open Oximetry Project funded by the Gordon and Betty Moore Foundation, Patrick J McGovern Foundation, and Robert Wood Johnson Foundation. The UCSF Hypoxia Research Laboratory receives funding from multiple industry sponsors to test the sponsors’ devices for the purposes of product development and regulatory performance testing. Data in this paper do not include sponsor’s study devices. All data were collected from devices procured by the Hypoxia Research Laboratory for the purposes of independent research. No company provided any direct funding for this study, participated in study design or analysis, or was involved in analysing data or writing the manuscript. None of the authors own stock or equity interests in any pulse oximeter companies. Dr Ellis Monk’s time utilised for data analysis, reviewing and editing was funded by grant number: DP2MH132941.

Background

Retrospective clinical trials of pulse oximeter accuracy report more frequent missed diagnoses of hypoxemia in hospitalized Black patients than White patients, differences that may contribute to racial disparities in health and health care. Retrospective studies have limitations including mistiming of blood samples and oximeter readings, inconsistent use of functional versus fractional saturation, and self-reported race used as a surrogate for skin color. Our objective was to prospectively measure the contributions of skin pigmentation, perfusion index (PI), sex, and age on pulse oximeter errors in a laboratory setting.

Methods

We enrolled 146 healthy subjects, including 25 with light skin (Fitzpatrick class I and II), 78 with medium (class III and IV), and 43 with dark (class V and VI) skin. We studied 2 pulse oximeters (Nellcor N-595 and Masimo Radical 7) in prevalent clinical use. We analyzed 9763 matched pulse oximeter readings (pulse oximeter measured functional saturation [Spo2]) and arterial oxygen saturation (hemoximetry arterial functional oxygen saturation [Sao2]) during stable hypoxemia (Sao2 68%-100%). PI was measured as percent infrared light modulation by the pulse detected by the pulse oximeter probe, with low perfusion categorized as PI < 1%. The primary analysis was to assess the relationship between pulse oximeter bias (difference between Sao2 and Spo2) by skin pigment category in a multivariable mixed-effects model incorporating repeated-measures and different levels of Sao2 and perfusion.

Results

Skin pigment, PI, and degree of hypoxemia significantly contributed to errors (bias) in both pulse oximeters. For PI values of 1.0% to 1.5%, 0.5% to 1.0%, and <0.5%, the P value of the relationship to mean bias or median absolute bias was <.00001. In lightly pigmented subjects, only PI was associated with positive bias, whereas in medium and dark subjects bias increased with both low perfusion and degree of hypoxemia. Sex and age was not related to pulse oximeter bias. The combined frequency of missed diagnosis of hypoxemia (pulse oximeter readings 92%-96% when arterial oxygen saturation was <88%) in low perfusion conditions was 1.1% for light, 8.2% for medium, and 21.1% for dark skin.

Conclusions

Low peripheral perfusion combined with darker skin pigmentation leads to clinically significant high-reading pulse oximeter errors and missed diagnoses of hypoxemia. Darkly pigmented skin and low perfusion states are likely the cause of racial differences in pulse oximeter performance in retrospective studies.

Anemia and hypoxemia are common clinical conditions that are difficult to study and may impact pulse oximeter performance. Utilizing an in vitro circulation system, we studied performance of three pulse oximeters during hypoxemia and severe anemia. Three oximeters including one benchtop, one handheld, and one fingertip device were selected to reflect a range of cost and device types. Human blood was diluted to generate four hematocrit levels (40%, 30%, 20%, and 10%). Oxygen and nitrogen were bubbled through the blood to generate a range of oxygen saturations (OHb) and the blood was cycled through the in vitro circulation system. Pulse oximeter saturations (SpO) were paired with simultaneously-measured OHb readings from a reference CO-oximeter. Data for each hematocrit level and each device were least-squares fit to a 2nd-order equation with quality of each curve fit evaluated using standard error of the estimate. Bias and average root mean square error were calculated after correcting for the calibration difference between human and in vitro circulation system calibration. The benchtop oximeter maintained good accuracy at all but the most extreme level of anemia. The handheld device was not as accurate as the benchtop, and inaccuracies increased at lower hematocrit levels. The fingertip device was the least accurate of the three oximeters. Pulse oximeter performance is impacted by severe anemia in vitro. The use of in vitro calibration systems may play an important role in augmenting in vivo performance studies evaluating pulse oximeter performance in challenging conditions.

It has long been known that many pulse oximeters function less accurately in patients with darker skin. Reasons for this observation are incompletely characterized and potentially enabled by limitations in existing regulatory oversight. Based on decades of experience and unpublished data, we believe it is feasible to fully characterize, in the public domain, the factors that contribute to missing clinically important hypoxemia in patients with darkly pigmented skin. Here we propose 5 priority areas of inquiry for the research community and actionable changes to current regulations that will help improve oximeter accuracy. We propose that leading regulatory agencies should immediately modify standards for measuring accuracy and precision of oximeter performance, analyzing and reporting performance outliers, diversifying study subject pools, thoughtfully defining skin pigmentation, reporting data transparently, and accounting for performance during low-perfusion states. These changes will help reduce bias in pulse oximeter performance and improve access to safe oximeters.

Background

Universal access to pulse oximetry worldwide is often limited by cost and has substantial public health consequences. Low-cost pulse oximeters have become increasingly available with limited regulatory agency oversight. The accuracy of these devices often has not been validated, raising questions about performance.

Methods

The accuracy of 6 low-cost finger pulse oximeters during stable arterial oxygen saturations (SaO2) between 70% and 100% was evaluated in 22 healthy subjects. Oximeters tested were the Contec CMS50DL, Beijing Choice C20, Beijing Choice MD300C23, Starhealth SH-A3, Jumper FPD-500A, and Atlantean SB100 II. Inspired oxygen, nitrogen, and carbon dioxide partial pressures were monitored and adjusted via a partial rebreathing circuit to achieve 10 to 12 stable target SaO2 plateaus between 70% and 100% and PaCO2 values of 35 to 45 mm Hg. Comparisons of pulse oximeter readings (SpO2) with arterial SaO2 (by Radiometer ABL90 and OSM3) were used to calculate bias (SpO2 – SaO2) mean, precision (SD of the bias), and root mean square error (ARMS).

Results

Pulse oximeter readings corresponding to 536 blood samples were analyzed. Four of the 6 oximeters tested showed large errors (up to -6.30% mean bias, precision 4.30%, 7.53 ARMS) in estimating saturation when SaO2 was reduced <80%, and half of the oximeters demonstrated large errors when estimating saturations between 80% and 90%. Two of the pulse oximeters tested (Contec CMS50DL and Beijing Choice C20) demonstrated ARMS of <3% at SaO2 between 70% and 100%, thereby meeting International Organization for Standardization (ISO) criteria for accuracy.

Conclusions

Many low-cost pulse oximeters sold to consumers demonstrate highly inaccurate readings. Unexpectedly, the accuracy of some low-cost pulse oximeters tested here performed similarly to more expensive, ISO-cleared units when measuring hypoxia in healthy subjects. None of those tested here met World Federation of Societies of Anaesthesiologists standards, and the ideal testing conditions do not necessarily translate these findings to the clinical setting. Nonetheless, further development of accurate, low-cost oximeters for use in clinical practice is feasible and, if pursued, could improve access to safe care, especially in low-income countries.

Introduction

Pulse oximetry may overestimate arterial oxyhemoglobin saturation (Sao2) at low Sao2 levels in individuals with darkly pigmented skin, but other factors, such as gender and oximeter probe type, remain less studied.

Methods

We studied the relationship between skin pigment and oximeter accuracy in 36 subjects (19 males, 17 females) of a range of skin tones. Clip-on type sensors and adhesive/disposable finger probes for the Masimo Radical, Nellcor N-595, and Nonin 9700 were studied. Semisupine subjects breathed air-nitrogen-CO2 mixtures via a mouthpiece to rapidly achieve 2- to 3-min stable plateaus of Sao2. Comparisons of Sao2 measured by pulse oximetry (Spo2) with Sao2 (by Radiometer OSM-3) were used in a multivariate model to assess the source of errors.

Results

The mean bias (Spo2 – Sao2) for the 70%-80% saturation range was 2.61% for the Masimo Radical with clip-on sensor, -1.58% for the Radical with disposable sensor, 2.59% for the Nellcor clip, 3.6% for the Nellcor disposable, -0.60% for the Nonin clip, and 2.43% for the Nonin disposable. Dark skin increased bias at low Sao2; greater bias was seen with adhesive/disposable sensors than with the clip-on types. Up to 10% differences in saturation estimates were found among different instruments in dark-skinned subjects at low Sao2.

Conclusions

Multivariate analysis indicated that Sao2 level, sensor type, skin color, and gender were predictive of errors in Spo2 estimates at low Sao2 levels. The data suggest that clinically important bias should be considered when monitoring patients with saturations below 80%, especially those with darkly pigmented skin; but further study is needed to confirm these observations in the relevant populations.

Background

It is uncertain whether skin pigmentation affects pulse oximeter accuracy at low HbO2 saturation.

Methods

The accuracy of finger pulse oximeters during stable, plateau levels of arterial oxygen saturation (Sao2) between 60 and 100% were evaluated in 11 subjects with darkly pigmented skin and in 10 with light skin pigmentation. Oximeters tested were the Nellcor N-595 with the OxiMax-A probe (Nellcor Inc., Pleasanton, CA), the Novametrix 513 (Novametrix Inc., Wallingford, CT), and the Nonin Onyx (Nonin Inc., Plymouth, MN). Semisupine subjects breathed air-nitrogen-carbon dioxide mixtures through a mouthpiece. A computer used end-tidal oxygen and carbon dioxide concentrations determined by mass spectrometry to estimate breath-by-breath Sao2, from which an operator adjusted inspired gas to rapidly achieve 2- to 3-min stable plateaus of desaturation. Comparisons of oxygen saturation measured by pulse oximetry (Spo2) with Sao2 (by Radiometer OSM3) were used in a multivariate model to determine the interrelation between saturation, skin pigmentation, and oximeter bias (Spo2 – Sao2).

Results

At 60-70% Sao2, Spo2 (mean of three oximeters) overestimated Sao2 (bias +/- SD) by 3.56 +/- 2.45% (n = 29) in darkly pigmented subjects, compared with 0.37 +/- 3.20% (n = 58) in lightly pigmented subjects (P < 0.0001). The SD of bias was not greater with dark than light skin. The dark-light skin differences at 60-70% Sao2 were 2.35% (Nonin), 3.38% (Novametrix), and 4.30% (Nellcor). Skin pigment-related differences were significant with Nonin below 70% Sao2, with Novametrix below 90%, and with Nellcor at all ranges. Pigment-related bias increased approximately in proportion to desaturation.

Conclusions

The three tested pulse oximeters overestimated arterial oxygen saturation during hypoxia in dark-skinned individuals.