Remote assessment of pulse wave parameters from face video recordings for the computer vision dataset creation

Introduction. Telemedicine is becoming a powerful tool for increasing the availability and timely provision of medical care. However, many monitoring methods become unavailable due to the loss of direct contact between the doctor and the patient. All this requires new methodological approaches to the organization of diagnostics and treatment at the junction of information and communication technologies and medical sciences.

Aim: To develop a method for extracting the pulse wave from a human face video and to form a dataset using a telemedicine complex, traditional and remote photoplethysmography for artificial intelligence systems.

Material and Methods. The study included 450 practically healthy individuals who were examined using a telemedicine system, traditional and videophotoplethysmography at rest and after physical exertion. For each subject, three-minute video recordings were made from three webcams and smartphone video cameras with different resolutions, compared with a classic photoplethysmogram with a frequency of 100 Hz and vital health parameters recorded using a domestic telemedicine system.

Results. The resulting dataset contains 2,700 video files that can be used for training and testing artificial neural networks for remote photoplethysmography. Also, when collecting data, we studied the important scientific problem of synchronizing video files and photoplethysmograms for their correct comparison. To solve this problem, a new measurement coordination method based on comparing time series of measurement moments has been proposed. The results obtained were used to compare and analyze several existing pulse wave extraction algorithms using artificial neural networks in comparison with data obtained from a photoplethysmograph and a telemedicine system.

Conclusion. Based on the use of a telemedicine complex, traditional and remote photoplethysmography, a dataset has been assembled that can be used to extract physiological indicators of human health from facial video from user devices. The use of various well-known neural network computer vision algorithms has demonstrated the possibility of implementing remote photoplethysmography in medical diagnostics and health monitoring.

1. Casado C.Á., Miguel B.L. Face2PPG: an unsupervised pipeline for blood volume pulse extraction from faces. IEEE Journal of Biomedical and Health Informatics. 2023;27(11):5530–5541. https://doi.org/10.1109/JBHI.2023.3307942

2. Perpetuini D., Di Credico A., Filippini C. et al. Is it possible to estimate the average heart rate from facial thermal imaging? Eng. Proc. 2021;8(1):10. https://doi.org/10.3390/engproc2021008010

3. Dae-Yeol Kim D.-Y., Goh E., Lee K.K. et al. Remote bio-sensing: open source benchmark framework for fair evaluation of rPPG. arXiv:2307.12644. https://doi.org/10.48550/arXiv.2307.12644

4. Haleem A., Javaid M., Singh R., Suman R. Telemedicine for healthcare: capabilities, features, barriers, and applications. Sensors International. 2021;2:100117. https://doi.org/10.1016/j.sintl.2021.100117

5. George A.S., George A.S.H. Telemedicine: A New Way to Provide Healthcare. Partners Universal International Innovation Journal (PUIIJ). 2023;01(03):98–129. https://doi.org/10.5281/zenodo.8075850

6. Ionov M.V., Konradi A.O. Telehealth in hypertensive patients: to whom, how and for how long? Arterial’naya Gipertenziya (Arterial Hypertension). 2022;28(2):108–125. (In Russ.) https://doi.org/10.18705/1607-419X-2022-28-2-108-125

7. Hegazy A., Abdelhai R., Aboushady A., Elessawy R., Seddik S. Telemedicine during the COVID-19 pandemic: Egyptian healthcare professionals’ views. Oxford Open Digital Health. 2023;1. https://doi.org/10.1093/oodh/oqad009

8. Ikumapayi O., Kayode J., Afolalu A. et al. Telehealth and telemedicine -an overview. Proceedings of the 4th African International Conference on Industrial Engineering and Operations Management Nsukka. 2022, Nigeria, April 5-7. https://doi.org/10.46254/AF03.20220258

9. Muller A., Haneke H., Kirchberger V. et al. Integration of mobile sensors in a telemedicine hospital system: remote-monitoring in COVID-19 patients. Z. Gesundh. Wiss. 2022;30(1):93–97. https://doi.org/10.1007/s10389-021-01655-2

10. Olmedo-Aguirre J., Reyes-Campos J., Alor-Hernández G. et al. Remote healthcare for elderly people using wearables: a review. Biosensors (Basel). 2022;12(2):73. https://doi.org/10.3390/bios12020073

11. Prieto-Avalos G., Cruz-Ramos N., Alor-Hernández G. et al. Wearable devices for physical monitoring of heart: a review. Biosensors (Basel). 2022;12(5):292. https://doi.org/10.3390/bios12050292

12. Kalasin S., Surareungchai W. Challenges of emerging wearable sensors for remote monitoring toward telemedicine healthcare. Analytical chemistry. 2023; Jan. 11. https://doi.org/10.1021/acs.analchem.2c02642

13. Vaghasiya J.V., Mayorga-Martinez C.C., Pumera M. Wearable sensors for telehealth based on emerging materials and nanoarchitectonics. Npj Flex Electron. 2023;7(1):26. https://doi.org/10.1038/s41528-023-00261-4

14. Donati M., Celli A. Ruiu A. Saponara S. Fanucci L. A telemedicine service platform exploiting BT/BLE wearable sensors for remote monitoring of chronic patients. 2018 7th International Conference on Modern Circuits and Systems Technologies (MOCAST), Thessaloniki, Greece. 2018:1–4. https://doi.org/10.1109/MOCAST.2018.8376643

15. Gusev A.V., Vladzymyrskyy A.V., Sharova D.E. et al. Evolution of research and development in the field of artificial intelligence technologies for healthcare in the Russian Federation: results of 2021. Digital Diagnostics. 2022;3(3):178–194. (In Russ). https://doi.org/10.17816/DD107367

16. Mouloudj K., Oanh L., Bouarar A. et al. Adopting Artificial Intelligence in healthcare: a narrative review. In book: The Use of Artificial Intelligence in Digital Marketing: Competitive Strategies and Tactics. 2024:1–20. https://doi.org/10.4018/978-1-6684-9324-3.ch001

17. Choudhury A., Sarma K. Intelligent wearable healthcare monitoring framework: trends in sensor-deep learning approaches. In book: Investigations in Pattern Recognition and Computer Vision for Industry 4.0. 2023:127–179. https://doi.org/10.4018/978-1-6684-8602-3.ch008

18. Palavicini G. Intelligent health: progress and benefit of artificial intelligence in sensing-based monitoring and disease diagnosis. Sensors (Basel). 2023;23(22):9053. https://doi.org/10.3390/s23229053

19. Huang X., Ju Z., Zhang R. Real-time heart rate detection method based on 77 GHz FMCW Radar. Micromachines (Basel). 2022;13(11):1960. https://doi.org/10.3390/mi13111960

20. Schif D., Forst H., Schwarz U.T. Methods for sweat detection in cars based on thermal images of the face. IEEE Sensors Journal. 2021;21(13):15342–15348. https://doi.org/10.1109/JSEN.2021.3071800