Skip to main content

Towards Using Biometric Transcripts for Analyzing Multimodal Sensory Information Processing Data

  • Conference paper
  • First Online:
Augmented Cognition (HCII 2025)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15778))

Included in the following conference series:

  • 260 Accesses

Abstract

In immersive environments like virtual reality (VR), task performance can be affected by different visual and auditory stimuli. Analyzing multiple aspects of user experience, such as cognitive load and cognitive alertness, in these scenarios can yield multimodal datasets that may make it difficult to detect patterns or relationships between different data modalities. Moreover, these large data sets would cause the researcher to split their attention. This study proposes a synchronized method for analyzing temporal multimodal data more efficiently, called Biometric Transcript for Information Processing (BTIP). In this paper, we demonstrate the proposed approach and the usefulness of viewing temporal multimodal data concurrently. We used the following data modalities collected during an interactive VR task in the presence of visual and auditory distractors: electroencephalogram (EEG) (alpha and theta waves), pupil dilation, fixation location, and gaze duration for each fixation location for the first engagement of the first instance with each distractor. We assessed sensory overloading and changes to attention through the fluctuations of alpha and theta wave activity and explored the connection to pupil dilation by analyzing the patterns found in the biometric transcript. Our data indicated a significantly positive correlation between alpha and theta waves. Peaks in alpha activity correspond to enhanced focus, whereas peaks in theta wave activity correspond to overloading. We further discuss the implications of using this method to research nuances in sensory processing research, specifically with biometric feedback. These insights suggest that this method is promising to be used effectively during the analysis process of temporal multimodal data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
€34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 69.54
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 87.73
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

') var buybox = document.querySelector("[data-id=id_"+ timestamp +"]").parentNode var buyingOptions = buybox.querySelectorAll(".buying-option") ;[].slice.call(buyingOptions).forEach(initCollapsibles) var buyboxMaxSingleColumnWidth = 480 function initCollapsibles(subscription, index) { var toggle = subscription.querySelector(".buying-option-price") subscription.classList.remove("expanded") var form = subscription.querySelector(".buying-option-form") var priceInfo = subscription.querySelector(".price-info") var buyingOption = toggle.parentElement if (toggle && form && priceInfo) { toggle.setAttribute("role", "button") toggle.setAttribute("tabindex", "0") toggle.addEventListener("click", function (event) { var expandedBuyingOptions = buybox.querySelectorAll(".buying-option.expanded") var buyboxWidth = buybox.offsetWidth ;[].slice.call(expandedBuyingOptions).forEach(function(option) { if (buyboxWidth buyboxMaxSingleColumnWidth) { toggle.click() } else { if (index === 0) { toggle.click() } else { toggle.setAttribute("aria-expanded", "false") form.hidden = "hidden" priceInfo.hidden = "hidden" } } }) } initialStateOpen() if (window.buyboxInitialised) return window.buyboxInitialised = true initKeyControls() })()

Institutional subscriptions

Similar content being viewed by others

References

  • Albus, P., Vogt, A., Seufert, T.: Signaling in virtual reality influences learning outcome and cognitive load. Comput. Educ. 166, 104154 (2021)

    Article  Google Scholar 

  • Antonenko, P., Paas, F., Grabner, R., Van Gog, T.: Using electroencephalography to measure cognitive load. Educ. Psychol. Rev. 22(4), 425–438 (2010)

    Article  Google Scholar 

  • Baceviciute, S., Mottelson, A., Terkildsen, T., Makransky, G.: Investigating representation of text and audio in educational VR using learning outcomes and EEG. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020)

    Google Scholar 

  • Başar, E., Başar-Eroğlu, C., Karakaş, S., Schürmann, M.: Are cognitive processes manifested in event-related gamma, alpha, theta and delta oscillations in the EEG? Neurosci. Lett. 259(3), 165–168 (1999)

    Article  Google Scholar 

  • Baudisch, P., Good, N., Bellotti, V., Schraedley, P.: Keeping things in context: a comparative evaluation of focus plus context screens, overviews, and zooming. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 259–266 (2002)

    Google Scholar 

  • Belchior, P.P., et al.: Video game training to improve selective visual attention in older adults. Comput. Hum. Behav. 29(4), 1318–1324 (2013)

    Article  Google Scholar 

  • Bezemer, J., Jewitt, C.: Multimodal analysis: key issues. Res. Methods Linguist. 180 (2010)

    Google Scholar 

  • Callaghan, T.C.: Dimensional interaction of hue and brightness in preattentive field segre-gation. Percept. Psychophys. 36(1), 25–34 (1984)

    Article  Google Scholar 

  • Chun, M.M.: Contextual cueing of visual attention. Trends Cogn. Sci. 4(5), 170–178 (2000)

    Article  Google Scholar 

  • Chiossi, F., Arias, J.J., Borghese, N.A.: EEG-based adaptive virtual reality: modulating visual complexity to enhance user experience and cognitive performance. arXiv preprint arXiv:2311.10447. https://arxiv.org/abs/2311.10447 (2023)

  • Chun, M.M., Jiang, Y.: Contextual cueing: implicit learning and memory of visual context guides spatial attention. Cogn. Psychol. 36(1), 28–71 (1998)

    Article  MathSciNet  Google Scholar 

  • Clark, J.: Visual Information Processing in Virtual Reality: Merging Theory and Practice. In International Conference on Information, pp. 461–468. Springer, Cham (2023)

    Google Scholar 

  • Clark, J., Boz, L.: Exploring the effects of multi-sensory extraneous load on attention and task performance in VR. In: Proceedings of the 27th International Conference, HCI International 2025 (2025a)

    Google Scholar 

  • Clark, J., Boz, L.: Exploring the effects of visual and auditory distractors in virtual reality on perceived cognitive load and cognitive alertness. In: WSEAS Transactions on Information Science and Applications (2025b)

    Google Scholar 

  • Debue, N., Van De Leemput, C.: What does germane load mean? An empirical contribution to the cognitive load theory. Front. Psychol. 5, 1099 (2014)

    Article  Google Scholar 

  • Gevins, A., Smith, M.E.: Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci. 4(1–2), 113–131 (2003)

    Article  Google Scholar 

  • Gevins, A., Smith, M.E., McEvoy, L., Yu, D.: High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice. Cerebral cortex (New York, NY: 1991), 7(4), 374–385 (1997)

    Google Scholar 

  • Gonçalves, G., Monteiro, P., Melo, M., Vasconcelos-Raposo, J., Bessa, M.: A comparative study between wired and wireless virtual reality setups. IEEE Access 8, 29249–29258 (2020)

    Article  Google Scholar 

  • Hebb, D.O., Donderi, D.C.: Textbook of Psychology (Psychology Revivals). Psychology Press (2013)

    Book  Google Scholar 

  • Hoffman, D.M., Girshick, A.R., Akeley, K., Banks, M.S.: Vergence–accommodation conflicts hinder visual performance and cause visual fatigue. J. Vis. 8(3), 33 (2008)

    Article  Google Scholar 

  • Hsia, H.J.: The information processing capacity of modality and channel performance. ECTJ. 19, 51 (1971). https://doi.org/10.1007/BF02768431

    Article  Google Scholar 

  • Jerald, Jason. The VR book: Human-centered design for virtual reality. Morgan & Claypool, 2015

    Google Scholar 

  • Klepsch, M., Schmitz, F., Seufert, T.: Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Front. Psychol. 8, 1997 (2017)

    Article  Google Scholar 

  • Klimesch, W.: EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Rev. 29(2–3), 169–195 (1999)

    Article  Google Scholar 

  • Ma, C., Huang, J.B., Yang, X., Yang, M.H.: Hierarchical convolutional features for visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3074–3082 (2015)

    Google Scholar 

  • Mack, A.: Inattentional blindness: looking without seeing. Curr. Dir. Psychol. Sci. 12(5), 180–184 (2003)

    Article  Google Scholar 

  • Makransky, G., Terkildsen, T.S., Mayer, R.E.: Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn. Instr. 60, 225–236 (2019)

    Article  Google Scholar 

  • Mayer, R.E.: Cognitive theory of multimedia learning. Cambridge Handb. Multimedia Learn. 41, 31–48 (2005)

    Article  Google Scholar 

  • Mayer, R.E.: Research-based principles for designing multimedia instruction. In: Benassi, V.A., Overson, C.E., Hakala, C.M. (eds.), Applying science of learning in education: Infusing Psychol. Sci. Curriculum, 59–70 (2014)

    Google Scholar 

  • Mayer, R.E., Fiorella, L.: 12 principles for reducing extraneous processing in multimedia learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity principles. In: The Cambridge Handbook of Multimedia Learning, vol. 279. Cambridge University Press (2014)

    Google Scholar 

  • Mind Monitor. https://mind-monitor.com/. Accessed 13 Feb 2025

  • Moreno, R., Mayer, R.E.: Visual presentations in multimedia learning: Conditions that overload visual working memory. In: International Conference on Advances in Visual Information Systems, pp. 798–805. Springer, Heidelberg (1999)

    Google Scholar 

  • Neurofeedback EEG device - how it works. Muse. https://choosemuse.com/pages/science. Retrieved 13 Feb 2025

  • Petersen, A., Petersen, A.H., Bundesen, C., Vangkilde, S., Habekost, T.: The effect of phasic auditory alerting on visual perception. Cognition 165, 73–81 (2017)

    Article  Google Scholar 

  • Sterman, M.B., Mann, C.A., Kaiser, D.A., Suyenobu, B.Y.: Multiband topographic EEG analysis of a simulated visuomotor aviation task. Int. J. Psychophysiol. 16(1), 49–56 (1994)

    Article  Google Scholar 

  • Unity Real-Time Development Platform. https://unity.com/. Accessed 13 Feb 2025

  • VIVE Pro Eye Overview. https://www.vive.com/sea/product/vive-pro-eye/overview/. Accessed 13 Feb 2025

  • Vorländer, M. Shinn-Cunningham, B.G.: Virtual Auditory Displays. Handbook of Virtual Environments, 2nd ed (2014)

    Google Scholar 

  • Zhang, L.: Towards using eye-tracking and consumer-grade electroencephalogram devices to detect usability issues in mobile applications. Doctoral dissertation. School of Information, University of Arizona, USA (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jack Clark .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Clark, J., Boz, L. (2025). Towards Using Biometric Transcripts for Analyzing Multimodal Sensory Information Processing Data. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Augmented Cognition. HCII 2025. Lecture Notes in Computer Science(), vol 15778. Springer, Cham. https://doi.org/10.1007/978-3-031-93724-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-93724-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-93723-1

  • Online ISBN: 978-3-031-93724-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics