Proceedings of the ASME 2015 International Design Engineering Technical Conferences &

Computers and Information in Engineering Conference

IDETC/CIE 2015

August 2-5, 2015, Boston, Massachusetts, USA

DETC2015-46309

NON-CONTACT MONITORING OF PRETERM INFANTS USING RGB-D CAMERA

Annalisa Cenci Department of Information Engineering Universita Politecnica delle Marche`Via Brecce Bianche, 12 Ancona 60131 Email: a.cenci@pm.univpm.it

Daniele Liciotti, Emanuele Frontoni,Adriano Mancini, and Primo ZingarettiDepartment of Information Engineering Universita Politecnica delle Marche`Via Brecce Bianche, 12 Ancona 60131

ABSTRACT

In this study, we present a medical imaging method to estimate the preterm infant respiratory rate with a non invasive embedded RGB-D sensor. The respiratory rate is derived by measuring morphological chest wall movements with a depth sensor. The performances of our method are evaluated by comparing the values of respiratory rate measurements obtained using our method with those resulting from a standard reference device, used as a benchmark. Experimental results showed that the proposed method can correctly measure the respiratory rate in preterm infants and activates an alarm signal when respiratory rate values go out of the physiological range or when the infant remains stationary for a long time. As future development, the project can be used as multi-purpose non invasive test on movements and their symmetry or on the infants’ ability to correctly inflate.

1. INTRODUCTION

The respiratory rate is defined as the number of breaths taken by an individual over a minute. Every single respiratory cycle is composed of an inspiratory phase, during which the chest wall expands itself, and an expiratory phase, during which the chest wall goes back to the previous state. The two most widely used methods for evaluating the respiratory activity are spirometer and pneumotacography. Other invasive devices are impedance pneumography [1], thoracic impedance [2], respiratory inductance pletysmography [3], photopletysmograEmail: {d.liciotti, e.frontoni, a.mancini, p.zingaretti}@univpm.it

phy [4], strain gauges [5], thermistor [6], magnetometers [7] and acoustic monitoring [8]. However the measurement of the respiratory rate in all these techniques needs the direct contact with the monitored people and it may cause interference in their normal activity breath. For this reason, in the last years some non-contact respiratory measurement methods were investigated, such as Doppler radar [9], microwave sensors [10, 11], infrared temperature sensors [12], thermal imaging [13], structured light pletysmography [14] and optoelectronic pletysmography [15,16]. Most recently, the RGB-D camera is used to detect the respiratory rate [17–20] but in literature, there is a little number of works describing this technique, which, however, is used only for adults. The respiratory rate of an adult at rest is 12−20 breaths per minute [21]. In the newborn and throughout the first year of age, the respiratory rate is about 40 breaths per minute. In preterm infants it is higher, about 40 − 50 per minute, and irregular. Preterm infants are babies born between the 22^nd and 36^th week of gestation [22] and the respiratory system is a critical aspect for these babies. In fact, preterm birth affects the anatomical and functional development of all organs, inversely with gestational age and the degree of immaturity is primarily in the respiratory tract. The rhythm of the breath is usually irregular for the immaturity of the respiratory centers. Periodic breathing (Cheyne-Stokes rhythm) is frequently encountered, characterized by recurrent periods of apnea (about 5-10 s) within periods with close breaths. A respiratory rate greater than 60 breaths per minute creates alert. Even if the baby has an apnea, i.e., breathing stops for more than 10 seconds, it means that there is a respiratory problem. For all these reasons it is necessary to continuously monitor the respiratory rate in preterm infants. Generally, the respiratory activity of hospitalized infants is monitored using the thoracic impedance technique. In literature, we find that some of the methods previously cited are adapted and tested also on newborns and preterm infants, such as respiratory inductance pletysmography [23], impedance plethysmography [24], structured light plethysmography [25] and optoelectronic plethysmography [16]. In the neonatal intensive care unit (NICU) of the Pediatric Hospital G.Salesi in Ancona, where these babies are taken into care, the respiratory activity of preterm infants is continuously monitored using a cardiomonitor, used as our benchmark, obtaining the respiratory rate through the ECG electrodes by impedance pneumography.

In this work, we describe our contribution for monitoring the respiratory rate of preterm infants, developing a low cost embedded video system that measures preterm infant respiratory rate without any physical contact. We derive the respiratory rate by calculating morphological chest wall motions using a depth camera positioned above the infant lying on the infant warmer. The advantage of the proposed method is that it is contactless, noninvasive and suitable to be used in an indoor environment with poor lighting, as might be rooms in the neonatal intensive care unit.

We use the depth camera also to evaluate the movements of the monitored preterm infants. In fact, in the last years it was found that the spontaneous motility of preterm infants during their first weeks of life has an important clinical significance. The study of the early motor repertoire in preterm infants is an important predictor of nervous system dysfunctions such as cerebral palsy and minor neurological dysfunction [26]. In literature, there are several studies that prove the importance of qualitative observation of spontaneous movements. In these studies, the authors compare the results obtained from the video recording of infant movements and the consequent assessment of these movements, done by several qualified observers, with the results of magnetic resonance imaging and neurobehavioral assessment [27]. Recently, also the predictive values of quantitative aspects of the motor repertoire have been investigated [28]. In this context, we develop a system that automatically measures the percentage of time that the infant is in motion discriminating also which part of the body he/she is moving.

The paper is mainly focused on the basic hardware and software architecture of the system reporting preliminary tests in the experimental phase. Experimental results show that the proposed method can correctly measure the respiratory rate in preterm infants and it activates an alarm signal when respiratory rate values go out of the physiological range or when the infant remains stationary for a long time. The project opens also to future evolutions in the direction of multi-purpose non invasive tests on movements and their symmetry or on the infant ability to correctly inflate.

FIGURE 1: REPRESENTATION OF THE CONFIGURATION SCHEME OF THE SYSTEM.

The paper is organized as follow: next sections describe, respectively, the system architecture and the proposed method for data processing ; section 4 presents results, followed by discussions and conclusions in section 5.

2. THE SYSTEM ARCHITECTURE

Figure 1 shows the configuration scheme of the system. It consists of two main components:

1.    Single Board Computer: is a complete computer built on a single circuit board, with microprocessor(s), memory, input/output (I/O) and other features required for a functional computer. Single-board computers were made as demonstration or development systems, for the use as embedded computer controllers [29].

2.    RGB-D sensor: is composed by an infrared sensor, an RGB sensor and some microphones. It is able to provide in output a RGB representation of the scene and it also allows to reconstruct a depth map of the same. In the depth map the value of each pixel codifies the distance of each element in a 3D scene.

The architecture design is based on the choice of hardware and software components. In order to satisfy both functional and non-functional requirements, an embedded board, such as a CubieBoard2, has been used, since it is sufficiently small and suited to manage all algorithms. Asus Xtion Pro live has been chosen as depth sensor because it has a low cost, it has smaller dimensions than Microsoft Kinect, and the power supply is provided only by the USB port: it does not need an external power. Moreover, it is easy to install over the crib. The Cubieboard2 is a SoC based on ARM Cortex-A7 Dual-Core processor running on 1 GHz, and on ARM Mali 400MP2GPU, containing 1 GB of DDR3 memory and 4 GB of internal NAND flash. On that, a version of Debian OS (CubianX) is installed because compatible with the hardware and easy to setup the drivers of depth sensor (OpenNI2), the vision libraries (OpenCV), and the Qt Framework (Qt5).

3. THE PROPOSED METHOD

In this part, we describe the main steps of the algorithm that analyzes the respiratory rate and the infant movements.

The subjects under examination are 3 preterm infants (males) recovered in the NICU. Their gestational age ranged from 32 and 34 weeks, with a mean age of 33⁺³ weeks of age and the weight is between 1400g and 1800g, as shown in table 1. In order to evaluate the performances of the proposed algorithm, five tests, lasting 30 seconds each one, for each subject were performed.

TABLE 1: PATIENT CHARACTERISTICS (WEIGHT AND GESTATIONAL AGE)

Patient Identificator	Weight(g)	Gestational Age(weeks+days)
1	1420	32+3
2	1618	33+3
3	1762	34+2
Mean	1600	33+3

The depth sensor has been placed perpendicularly above the infant lying on the crib at a distance of 70cm, normally directed to the subject skin. Other tests have been performed putting it in a slightly inclined position, however this different position does not affect the measurement. For the analysis of the respiratory rate, we have chosen the measurement points in the thoracoabdominal area, considering where the possibility to detect the movements due to inspiration and expiration acts is greater and at the same time where it is possible to work on an available part of the skin surface of the infant, that is wearing a diaper and has sensors attached to the body. The mechanical movements of this area due to inspiration end expiration acts cause the variation of the depth signal acquired.

3.1         Respiratory Rate Analysis

Our method for the analysis of respiratory rate is based on the depth sensor. The idea is to analyze the time trend of different points in the thoraco-abdominal surface area of the subject monitored, present in the depth frame. The particular configuration of the system facilitates the monitoring of the respiratory rate. We created a graphical interface in C++ that allows viewing video streams (RGB and depth) from the sensor.

The algorithm includes a first step where the user can choose to analyze different strategic points. These, along with the depth values and the timestamps, are exported to a .csv file. Then, thanks to the aid of GNU Octave, for each frame, we calculated the average value of depth of the chosen points. After this first analysis, we applied a filter (fourth-order Butterworth low-pass filter) to the raw data so collected to remove the unwanted noise components due to the cardiac activity and to the motion artifacts. Then we used a find-peak function to detect the respiratory signal peaks and we calculated the time elapsed between two successive peaks, that is the instantaneous respiration period (RR). Finally for each test performed, we compared the period RR obtained from the RGB-D sensor and derived from the cardiomonitor, used as reference.

3.2        Movements Analysis

We propose an algorithm that can detect the baby’s movements in real time and it is based on vision techniques. Initially, the user selects a region of interest in the GUI where it is possible to visualize the infant. This procedure is used to filter the noise produced by the edges of the crib, creating zero pixels in the depth image. After this stage, the process of acquiring depth stream begins. The frames are stored in a vector so it is possible a subtraction operation between the first and the last frame. The size of the vector has been experimentally defined. Subsequently, an image filtering is performed: all points that, at the time of the acquisition have zero value are filtered. After filtering, the image is segmented using a threshold value, so as to further discriminate positive signals that indicate moving of limbs, by false positives due to background noise. At this point, the segmented image contains some blobs that represent the variations of the depth value due to the movements of the baby. These regions are identified and filtered depending on the area. The centers of mass are extracted from the blobs and are tracked for the duration of the movement. The detail of the algorithm is shown in Algorithm 1.

4. RESULTS

We realized an application with a user-friendly interface that doctors and nurses of the hospital ward where preterm infants are taken into care, can easily use. From the interface, they can choose the measurement points (figure 5) and then in real input : Depth image d(x,y) input : Number of frame delay N_fdelayoutput: Vector of points pointswhile sensor.isOpen() do

ddif f → dlist.last()−dlist.first(); d_list.pop back(); d_fil→ filtering(d_dif f ); d_seg→ segmentation(d_fil); contours → find contours(d_seg);

points → moments(contours);

tracking(points);

else skip; end

list.pushfront(d(x,y));

end

Algorithm 1: Movements Analysis algorithm.

FIGURE 2: EXAMPLE OF RESPIRATION SIGNAL MEASURED FROM RGB-D CAMERA (BLU LINE) AND OF THE FILTERED SIGNAL (RED LINE) ON A TIME WINDOW OF

10s.

time, our system displays the path of temporal depth measured by RGB-D camera from which the respiratory signal and the respiratory rate are derived.

In figure 2, we report an example of the respiration signal acquired with the RGB-D camera (blu line) with the filtered signal superimposed (red line) on a time window of 10s. We used the filtered signal to detect better the peaks of the respiratory signal and to calculate the mean value of the RR period compared with that obtained from the cardiomonitor.

The scatter plot and the regression line, reported in figure 3, show a good correlation between the measured data with respect

FIGURE 3: SCATTER PLOT AND REGRESSION LINE OF RR MEAN VALUES.

TABLE 2: TABLE MEAN RR VALUES MEASURED BY RGB-D SENSOR AND CARDIOMONITOR.

PatientIdentificator	Mean RR [s]	Test 1	Test2	Test3	Test4	Test5
1	Cardiomonitor	0.98	0.91	0.92	0.88	0.96
RGB-D Sensor	0.96	0.94	0.91	0.88	0.95
2	Cardiomonitor	1.17	1.33	1.15	1.03	0.99
RGB-D Sensor	1.21	1.18	1.18	1.02	0.96
3	Cardiomonitor	0.93	0.94	0.79	0.74	0.88
RGB-D Sensor	0.95	0.95	0.79	0.82	0.91

to the data derived from the reference instrument. The agreement between the two methods is also confirmed by the Pearson’s correlation coefficient that is equal to 0.95. In table 2 we report the mean values of the RR period measured both from the RGB-D sensor and the cardiomonitor, for each test and for each patient.

Our system also displays in real time the heat map of the movements of the infant, as it is possible to see in figure 4. The heat map is a graphical representation of data where the values contained in the depth matrix are represented as colors. The proposed system is a valid instrument for the real time visual monitoring and for the registration of the infant movements. Moreover, applying our system, it is possible to establish, with high confidence, which part of the body the infant is moving and for how long.

5. CONCLUSION

In this study we presented a medical imaging method to estimate preterm infant respiratory rate with a non invasive embedded RGB-D sensor. The paper is mainly focused on the basic hardware and software architecture of the proposed system and on preliminary tests. We evaluated our method by compar-

FIGURE 4: HEAT MAP OF THE MOVEMENTS MADE WITH THE SOFTWARE.

FIGURE 5: SCREENSHOT OF INTERFACE DEVELOPED WITH THE QT FRAMEWORK WHILE IT IS RUNNING ON CUBIEBOARD2.

ing the respiratory rate measurements estimated by our method with those deriving from a standard reference device, used as a benchmark. Experimental results showed that the method can correctly measure the respiratory rate in preterm infants, activating an alarm signal when respiratory rate values go out of the physiological range or when the infant remains stationary for a long time. The project opens also to future evolutions including the exploitation of multi-domain information gathered from audio and video frameworks [30] and in the direction of multipurpose non invasive tests on movements and their symmetry or on the infant ability to correctly inflate. Future works will involve clinical benchmarking, the development of multi point measurements, the beginning of the design of specific tests on infants movements taking into account symmetries, amount and speed.

ACKNOWLEDGMENT

The authors thank prof. Virgilio P. Carnielli (Neonatal Intensive Care Unit, Universita Politecnica delle Marche, Ancona,` ITALY) for his precious collaboration and the continuous support to the work.

REFERENCES

[1]       Seppa, V.-P., Viik, J., and Hyttinen, J., 2010. “Assessment of pulmonary flow using impedance pneumography”. Biomedical Engineering, IEEE Transactions on, 57(9), pp. 2277–2285.

[2]       Houtveen, J. H., Groot, P. F., and de Geus, E. J., 2006. “Validation of the thoracic impedance derived respiratory signal using multilevel analysis”. International Journal of Psychophysiology, 59(2), pp. 97–106.

[3]       Cantineau, J. P., Escourrou, P., Sartene, R., Gaultier, C., and Goldman, M., 1992. “Accuracy of respiratory inductive plethysmography during wakefulness and sleep in patients with obstructive sleep apnea.”. CHEST Journal, 102(4), pp. 1145–1151.

[4]       Allen, J., 2007. “Photoplethysmography and its application in clinical physiological measurement”. Physiological measurement, 28(3), p. R1.

[5]       Carskadon, M. A., Harvey, K., Dement, W. C., Guilleminault, C., Simmons, F. B., and Anders, T. F., 1978. “Respiration during sleep in children”. Western Journal of Medicine, 128(6), p. 477.

[6]       Farre, R., Montserrat, J., Rotger, M., Ballester, E., and Navajas, D., 1998. “Accuracy of thermistors and thermocouples as flow-measuring devices for detecting hypopnoeas”. European Respiratory Journal, 11(1), pp. 179– 182.

[7]       Sharp, J., Druz, W., Foster, J., Wicks, M., and Chokroverty, S., 1980. “Use of the respiratory magnetometer in diagnosis and classification of sleep apnea.”. CHEST Journal, 77(3), pp. 350–353.

[8]       Harper, V. P., Pasterkamp, H., Kiyokawa, H., and Wodicka, G. R., 2003. “Modeling and measurement of flow effects on tracheal sounds”. Biomedical Engineering, IEEE Transactions on, 50(1), pp. 1–10.

[9]       Droitcour, A. D., Seto, T. B., Park, B.-K., Yamada, S., Vergara, A., El Hourani, C., Shing, T., Yuen, A., Lubecke, V. M., and Boric-Lubecke, O., 2009. “Non-contact respiratory rate measurement validation for hospitalized patients”. In Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE, IEEE, pp. 4812–4815.

[10]    Uenoyama, M., Matsui, T., Yamada, K., Suzuki, S., Takase, B., Suzuki, S., Ishihara, M., and Kawakami, M., 2006. “Non-contact respiratory monitoring system using a ceiling-attached microwave antenna”. Medical and Biological Engineering and Computing, 44(9), pp. 835–840.

[11]    Dei, D., Grazzini, G., Luzi, G., Pieraccini, M., Atzeni, C., Boncinelli, S., Camiciottoli, G., Castellani, W., Marsili, M., and Lo Dico, J., 2009. “Non-contact detection of breathing using a microwave sensor”. Sensors, 9(4), pp. 2574–2585.

[12]    Boccanfuso, L., and O’Kane, J. M., 2012. “Remote measurement of breathing rate in real time using a high precision, single-point infrared temperature sensor”. In Biomedical Robotics and Biomechatronics (BioRob), 2012 4th IEEE RAS & EMBS International Conference on, IEEE, pp. 1704–1709.

[13]    Chekmenev, S. Y., Rara, H., and Farag, A. A., 2005. “Noncontact, wavelet-based measurement of vital signs using thermal imaging”. In The first international conference on graphics, vision, and image processing (GVIP), Cairo, Egypt, pp. 107–112.

[14]    Levai, I., Sidoroff, V., and Iles, R., 2012. “An introduction to the non-invasive non-contact assessment of respiratory function”. Respiratory Therapy, 7(5), p. 43.

[15]    Aliverti, A., Dellaca, R., Pelosi, P., Chiumello, D., Pedotti, A., and Gattinoni, L., 2000. “Optoelectronic plethysmography in intensive care patients”. American journal of respiratory and critical care medicine, 161(5), pp. 1546–1552.

[16]    Dellaca, R. L., Ventura, M. L., Zannin, E., Natile, M., Pedotti, A., and Tagliabue, P., 2010. “Measurement of total and compartmental lung volume changes in newborns by optoelectronic plethysmography”. Pediatric research, 67(1), pp. 11–16.

[17]    Yu, M.-C., Liou, J.-L., Kuo, S.-W., Lee, M.-S., and Hung, Y.-P., 2012. “Noncontact respiratory measurement of volume change using depth camera”. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, IEEE, pp. 2371–2374.

[18]    Benetazzo, F., Freddi, A., Monteriu, A., and Longhi, S.,` 2014. “Respiratory rate detection algorithm based on rgb-d camera: theoretical background and experimental results”. Healthcare Technology Letters, 1(3), pp. 81–86.

[19]    Bernacchia, N., Scalise, L., Casacanditella, L., Ercoli, I., Marchionni, P., and Tomasini, E. P., 2014. “Non contact measurement of heart and respiration rates based on kinect”. In Medical Measurements and Applications (MeMeA), 2014 IEEE International Symposium on, IEEE, pp. 1–5.

[20]    Aoki, H., Miyazaki, M., Nakamura, H., Furukawa, R., Sagawa, R., and Kawasaki, H., 2012. “Non-contact respiration measurement using structured light 3-d sensor”.

In SICE Annual Conference (SICE), 2012 Proceedings of, IEEE, pp. 614–618.

[21]    Tortora, G. J., and Derrickson, B. H., 2008. Principles of anatomy and physiology. John Wiley & Sons.

[22]    Organization, W. H., et al., 2012. “Born too soon: the global action report on preterm birth”.

[23]    Stick, S., Ellis, E., LeSouef, P., and Sly, P., 1992.¨ “Validation of respiratory inductance plethysmography (respitrace R ) for the measurement of tidal breathing parameters in newborns”. Pediatric pulmonology, 14(3), pp. 187–191.

[24]    Daily, W. J., Klaus, M., Belton, H., and Meyer, P., 1969. “Apnea in premature infants: monitoring, incidence, heart rate changes, and an effect of environmental temperature”. Pediatrics, 43(4), pp. 510–518.

[25]    Usher Smith, J., Wareham, R., Lasenby, J., Cameron, J., Bridge, P., and Iles, R., 2009. “Structured light plethysmography in infants and children-a pilot study”.

[26]    Einspieler, C., and Prechtl, H. F., 2005. “Prechtl’s assessment of general movements: a diagnostic tool for the functional assessment of the young nervous system”. Mental retardation and developmental disabilities research reviews, 11(1), pp. 61–67.

[27]    Constantinou, J. C., Adamson-Macedo, E. N., Mirmiran, M., and Fleisher, B. E., 2007. “Movement, imaging and neurobehavioral assessment as predictors of cerebral palsy in preterm infants”. Journal of perinatology, 27(4), pp. 225–229.

[28]    Bruggink, J. L., Einspieler, C., Butcher, P. R., Stremmelaar, E. F., Prechtl, H. F., and Bos, A. F., 2009. “Quantitative aspects of the early motor repertoire in preterm infants: do they predict minor neurological dysfunction at school age?”. Early human development, 85(1), pp. 25–36.

[29]    Liciotti, D., Contigiani, M., Frontoni, E., Mancini, A., Zingaretti, P., and Placidi, V., 2014. “Shopper analytics: A customer activity recognition system using a distributed rgb-d camera network”. In Video Analytics for Audience Measurement. Springer International Publishing, pp. 146–157.

[30]    Liciotti, D., Ferroni, G., Frontoni, E., Squartini, S., Principi, E., Bonfigli, R., Zingaretti, P., and Piazza, F., 2014. “Advanced integration of multimedia assistive technologies: A prospective outlook”. In Mechatronic and Embedded Systems and Applications (MESA), 2014 IEEE/ASME 10th International Conference on, IEEE, pp. 1–6.