Skip to main content
Springer Nature Link
Log in

The Ontology of Mixed Reality Agents Memorializing the Dead and Dying

  • Conference paper
  • First Online:
Culture and Computing. Design Thinking and Cultural Computing (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12795))

Included in the following conference series:

  • 1662 Accesses

  • 2 Citations

Abstract

Beginning with the 2009 publication of Michael Massimi’s Dying, Death, and Mortality: Towards Thanatosensitivity in HCI, researchers have been exploring how to design UX and UI for end-of-life systems and memorials. This work has resulted in the design of interactive posthumous personhoods memorializing those who have lived by utilizing the data the deceased produced while they were alive. HCI designers and researchers have implemented Natural Language Processing (NLP) to create interactive posthumous personhoods in the form of chatbots. Over the last five years, Mixed Reality (MR) has been integrated by practitioners to create posthumous personhoods that they claim offer a greater fidelity with the departed when they were alive than previous technologies. How MR intersects with those previous technologies, cultural, personal heritage, and memorialization to achieve this greater fidelity requires further exploration if the media is to be used to elevate the evidentiary status of these posthumous personhoods. Insights from HCI end-of-life literature, death and dying studies, and interactive non-fiction are used to explore the ontology of these interactive posthumous personhoods to discover what MR is doing. This paper elucidates connected ethical, ontological, and thanatological issues that the HCI field needs to address as this alternative practice of mourning and memorialization becomes a mainstream practice. The paper argues that MR increases the social and spatial presence of an MR posthumous personhood’s identity and subjectivity with greater fidelity than previous technologies, and that this supports an audience’s active creation of belief in the agent as being evidentiary of a posthumous personhood.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+
from €39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 67.40
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 85.59
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

') var buybox = document.querySelector("[data-id=id_"+ timestamp +"]").parentNode var buyingOptions = buybox.querySelectorAll(".buying-option") ;[].slice.call(buyingOptions).forEach(initCollapsibles) var buyboxMaxSingleColumnWidth = 480 function initCollapsibles(subscription, index) { var toggle = subscription.querySelector(".buying-option-price") subscription.classList.remove("expanded") var form = subscription.querySelector(".buying-option-form") var priceInfo = subscription.querySelector(".price-info") var buyingOption = toggle.parentElement if (toggle && form && priceInfo) { toggle.setAttribute("role", "button") toggle.setAttribute("tabindex", "0") toggle.addEventListener("click", function (event) { var expandedBuyingOptions = buybox.querySelectorAll(".buying-option.expanded") var buyboxWidth = buybox.offsetWidth ;[].slice.call(expandedBuyingOptions).forEach(function(option) { if (buyboxWidth buyboxMaxSingleColumnWidth) { toggle.click() } else { if (index === 0) { toggle.click() } else { toggle.setAttribute("aria-expanded", "false") form.hidden = "hidden" priceInfo.hidden = "hidden" } } }) } initialStateOpen() if (window.buyboxInitialised) return window.buyboxInitialised = true initKeyControls() })()

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Wendy Moncur, a professor of Digital Living at the University of Dundee has broken down these processes into one, the conduct of the funeral; two, the burial disposition (how the physical body is handled); three, the identification of the deceased [55, 56]. Technology and emerging media touch these processes in different ways.

  2. 2.

    I adopt Milgram’s original definition of Mixed Reality as, “one in which real world and virtual world objects are presented together within a single display, that is, anywhere between the extrema of the [reality-virtuality] continuum.” [57] In 2019, a longitudinal study of papers on Mixed Reality was unable to meaningfully move beyond this general definition [58].

  3. 3.

    I use the term “identify” in the manner of Kenneth Burke, in that a group of individuals are persuaded by the fidelity of a representation because it aligns with their expectations of what that representation should be and how it acts [59, 60].

  4. 4.

    As a person of Jewish heritage, I am cognizant of the problematic nature of critiquing representations of the Holocaust, whether they are interactive or not. This issue has been discussed at length by survivors and scholars and is outside the scope of this paper. USC’s Shoah Foundation is at the forefront of this work. However, I would like to underscore that I am not critiquing the history shared by these MR IAMs but exploring the use of MR in the design and development of an interactive posthumous personhood to share its history after these survivors have passed on.

  5. 5.

    The creation of this data on social networks such as Facebook and Twitter prompted them to develop rules governing the stewardship of data upon a user’s death [2]. In 2014, Facebook enabled users to choose a “Legacy Contact” who could manage their user data [61]. Facebook does not allow a user to act through a deceased person’s profile. Twitter does allow this.

  6. 6.

    EoL literature in HCI has been grappling with posthumous personhood by looking primarily at social networks. There are numerous examples on Twitter, Facebook, and others that have kept accounts active and online even after people have passed. This can look like a user’s scheduled posts occurring regularly after they pass, birthday messages, and other forms of automated or generated content. This ongoing interaction with those that have died is an extension of existing human behaviors of mourning from a variety of different cultures [19]. In this contemporary moment, instead of prayer or totems, “digital codes and computational texts are key sites for posthumous personhood.” [15] After famous movie critic Roger Ebert passed, his wife Chaz Ebert maintained his twitter account—only occasionally inserting her own voice by signing a Tweet “–Chaz” [62]. Others rely on a data-centric approach. Deadsoci.al will send out pre-recorded messages for users after they pass [63]. Others, such as LivesOn [64], analyze a user’s Twitter Feed to discover a user’s syntax as well as their likes and dislikes. The service will then algorithmically continue tweeting for a user after they have died.

  7. 7.

    The distinction between person and personhood is etymological. The conception of a “persona” comes from Latin and was a concept for a, “fundamental entity”. The concept of a person came from this foundation. The original idea of persona never referred to a physical, human confined to a biological shell. In Rome, personhood could be lost if one became a slave to the enemy but regained once emancipated and returned home. A persona is then the subjective recognition of a role played by an individual whereas personhood denotes the subject as a person.

  8. 8.

    In parallel, the identities and personas seen on social media platforms are the, “performative construction of individual and collective entities” [20, 65].

  9. 9.

    EoL HCI scholars have compared these kinds of aesthetics to different forms of physical memorialization from grave stones, mausoleums, to 1920s death photography and totems [3]. We might extend this understanding to different forms of interactive death aesthetics. These aesthetics include pedagogical ones, such as the MR IAMs at Holocaust museums, to a performance death aesthetic, such as the MR IAM of Tupac [66].

  10. 10.

    Without exception, those represented at the Holocaust museums have their other personas explored through a film that plays before the interactive experience. However, Minhua Ma, who worked on the MR IAMs at the National Holocaust Museum in the UK, noted that the presence of the survivor within the MR IAM is directly related to the fullness of their testimony and the openness of the subject domain [28]. Ma recognized that the person presented by an MR IAM is directly related to the quantity, diversity, and quality of the recorded media.

  11. 11.

    Survivors who were unable to make it to California are able to be filmed in their home with a mobile-rig that contains four to ten cameras [27].

References

  1. Graham, C., Arnold, M., Kohn, T., Gibbs, M.R.: Gravesites and websites: a comparison of memorialisation. Vis. Stud. 30, 37–53 (2015). https://doi.org/10.1080/1472586X.2015.996395

    Article  Google Scholar 

  2. Van Der, N.E., et al.: Death and the Internet (2017)

    Google Scholar 

  3. Massimi, M., Charise, A.: Dying, death, and mortality: Towards thanatosensitivity In: Proceedings of the HCI Conference on Human Factors in Computing Systems, pp. 2459–2468 (2009). https://doi.org/10.1145/1520340.1520.349

  4. Odom, W., Uriu, D., Kirk, D., Banks, R., Wakkary, R.: Experiences in designing technologies for honoring deceased loved ones. Des. Issues 34, 54–66 (2018). https://doi.org/10.1162/DESI

    Article  Google Scholar 

  5. Handayani, B., Tinggi, S., Bali, P.: Virtual Dark Tourism : The Role of Sound Branding and Augmented Reality for Death Sites. vol. 2, p 1–17 (2018). https://doi.org/10.4018/IJCMHS.2018070101

  6. Dow, S., Lee, J., Oezbek, C., MacIntyre, B., Bolter, J.D., Gandy, M.: Exploring spatial narratives and mixed reality experiences in Oakland Cemetery. In: ACE’05 Proceedings of the 2005 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, pp. 51–60 (2005). https://doi.org/10.1145/1178477.1178484

  7. Fisher, J.A., Schoemann, S.: Toward an ethics of interactive storytelling at dark tourism sites in virtual reality. In: International Conference on Interactive Digital Storytelling, pp. 577–590 (2018)

    Google Scholar 

  8. Fisher, J.A., David Bolter, J.: Ethical considerations for AR experiences at dark tourism sites. Adjun. In: Proceedings of the 2018 International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018, pp. 365–369 (2018). https://doi.org/10.1109/ISMAR-Adjunct.2018.00106

  9. Getty, E., Cobb, J., Gabeler, M., Nelson, C., Weng, E., Hancock, J.T.: I said your name in an empty room: Grieving and continuing bonds on Facebook. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 997–1000 (2011). https://doi.org/10.1145/1978942.1979091

  10. Brubaker, J.R., Hayes, G.R., Dourish, P.: Beyond the grave: Facebook as a site for the expansion of death and mourning. Inf. Soc. 29, 152–163 (2013). https://doi.org/10.1080/01972243.2013.777300

    Article  Google Scholar 

  11. Gould, H., Kohn, T., Gibbs, M.: Uploading the ancestors: experiments with digital Buddhist altars in contemporary Japan. Death Stud. 43, 456–465 (2019). https://doi.org/10.1080/07481187.2018.1544948

    Article  Google Scholar 

  12. Walter, T., Hourizi, R., Moncur, W., Pitsillides, S.: Does the internet change how we die and mourn? Overview and analysis. Omega J. Death Dying 64, 275–302 (2011). https://doi.org/10.2190/OM.64.4.a

    Article  Google Scholar 

  13. Massimi, M., Odom, W., Banks, R., Kirk, D.: Matters of life and death: locating the end of life in lifespan-oriented HCI research. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 987–996 (2011). https://doi.org/10.1145/1978942.1979090

  14. Engberg, M.: Augmented And Mixed Reality Design For Contested And Challenging Histories

    Google Scholar 

  15. Meese, J., Nansen, B., Kohn, T., Arnold, M., Gibbs, M.: Posthumous personhood and the affordances of digital media. Mortality 20, 408–420 (2015). https://doi.org/10.1080/13576275.2015.1083724

    Article  Google Scholar 

  16. Bassett, D.: Death and Anti-Death, vol. 16: 200 Years After Frankenstein, p. 16 (2018)

    Google Scholar 

  17. Brunt, S.D.: Performing beyond the grave: the posthumous duet. In: Strong, C., Lebrun, B. (eds.) Death and the Rock Star, pp. 165–173. Ashgate Publishing Ltd, Burlington (2015)

    Google Scholar 

  18. Bode, L.: No longer themselves framing digitally enabled posthumous “Performance”. Soc. Cine. Media Stud. 49, 46–70 (2010)

    Google Scholar 

  19. Gibbs, M., Mori, J., Arnold, M., Kohn, T.: Tombstones, uncanny monuments and epic quests: memorials in world of warcraft. Game Stud. 12, 2 (2012)

    Google Scholar 

  20. Lehner, N.: The work of the digital undead: digital capitalism and the suspension of communicative death. Continuum (N. Y). 33, 475–488 (2019). https://doi.org/10.1080/10304312.2019.1627289

  21. Gibson, M., Watkins, R.: The futures of grief. TEXT (2018). https://doi.org/10.1057/palgrave.jors.2601268

  22. Fisher, J.A.: Interview Transcripts at the Illinois Holocaust Museum (2020)

    Google Scholar 

  23. Traum, D.: New dimensions in testimony: digitally preserving a holocaust survivor’s interactive storytelling. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 9445, 269–281 (2015). https://doi.org/10.1007/978-3-319-27036-4_26

  24. Ballis, A., Barricelli, M., Gloe, M.: Interactive digital 3-D certificates and Holocaust Education - Development, Presentation and research, pp. 1–26 (2020)

    Google Scholar 

  25. Dratel, S.H.L.: “An Immersive Journey:” Analyzing the Use of Survivor Testimony in Holocaust Museums (2018). https://doi.org/10.1017/CBO9781107415324.004

  26. Monteiro, J., Morais, C., Carvalhais, M.: Interactive Storytelling for the Maintenance of Cultural Identity: The Potential of Affinity Spaces for the Exchange and Continuity of Intergenerational Cultural Knowledge. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). vol. 10690 LNCS, pp. 299–302 (2017). https://doi.org/10.1007/978-3-319-71027-3_30

  27. Ballis, A., Barricelli, M., Gloe, M.: Interaktive digitale 3-D-Zeugnisse und Holocaust Education – Entwicklung. Präsentation und Erforschung. Springer Fachmedien Wiesbaden (2019). https://doi.org/10.1007/978-3-658-24205-3_22

    Article  Google Scholar 

  28. Ma, M., Coward, S., Walker, C.: Question-answering virtual humans based on pre-recorded testimonies for holocaust education. Serious Games Edutainment Appl. II, 391–409 (2017). https://doi.org/10.1007/978-3-319-51645-5_18

    Article  Google Scholar 

  29. Artstein, R., Gainer, A., Georgila, K., Leuski, A., Shapiro, A., Traum, D.: New Dimensions in Testimony Demonstration. vol. 2016, pp. 32–36 (2016). https://doi.org/10.18653/v1/n16-3007

  30. Lycan, B., Artstein, R.: Direct and mediated interaction with a holocaust survivor. Lect. Notes Electr. Eng. 510, 161–167 (2019). https://doi.org/10.1007/978-3-319-92108-2_17

    Article  Google Scholar 

  31. Artstein, R., Silver, K.: Ethics for a combined human-machine dialogue agent. AAAI Spring Symposium Series - Technical Report SS-16-01-, pp. 184–189 (2016)

    Google Scholar 

  32. Holocaust Encyclopedia: Anne Frank: Diary|The Holocaust Encyclopedia. https://encyclopedia.ushmm.org/content/en/article/anne-frank-diary. Accessed 12 Jan 2021

  33. Davidson, S.: A hatchet job and the Holocaust. Institute of Public Affair, vol. 60 (2008)

    Google Scholar 

  34. Ong, W.J.: Orality and Literacy, the Technologizing of the Word. Routledge, London (1982)

    Book  Google Scholar 

  35. Huberman, J.M.: The digital double burial: resurrecting Hertz in the digital age. Mortality 23, 334–349 (2018). https://doi.org/10.1080/13576275.2017.1351937

    Article  Google Scholar 

  36. Bassett, D.J.: Ctrl + Alt + Delete: the changing landscape of the uncanny valley and the fear of second loss. Curr. Psychol. (2018). https://doi.org/10.1007/s12144-018-0006-5

    Article  Google Scholar 

  37. Engberg, M., Bolter, J.D.: The aesthetics of reality media. J. Vis. Cult. 19, 81–95 (2020). https://doi.org/10.1177/1470412920906264

    Article  Google Scholar 

  38. Murray, J.H.: Virtual/reality: how to tell the difference. J. Vis. Cult. 19, 11–27 (2020). https://doi.org/10.1177/1470412920906253

    Article  Google Scholar 

  39. Murray, J.: Hamlet on the Holodeck: The Future of Narrative in Cyberspace. The MIT Press (1998)

    Google Scholar 

  40. Dennett, D.: Conditions of Personhood

    Google Scholar 

  41. Savin-Baden, M., Burden, D., Taylor, H.: The ethics and impact of digital immortality. Knowl. Cult. 5, 178–196 (2017). https://doi.org/10.22381/KC52201711

  42. Fisher, J.A.: Empathic Actualities: Toward a Taxonomy of Empathy in Virtual Reality (2017). https://doi.org/10.1007/978-3-319-71027-3_19

  43. Graham, C., Constable, M., Fernando, J.: Guest editors’ introduction: the aesthetic of death and after-death in an Internet age. Vis. Stud. 30, 32–36 (2015). https://doi.org/10.1080/1472586X.2015.996393

    Article  Google Scholar 

  44. Films, B.: 116 Cameras. Op-Docs, USA (2017)

    Google Scholar 

  45. Foundation, S.: Introductory Film for Aaron Elster. Shoah Foundation, USA (2017)

    Google Scholar 

  46. Huberman, J.: Dearly departed: communicating with the dead in the digital age. Soc. Anal. 61, 91–107 (2017). https://doi.org/10.3167/sa.2017.610306

    Article  Google Scholar 

  47. Gaudenzi, S.: The Living Documentary: from representing reality to co-creating reality in digital interactive documentary, pp. 1–309 (2013)

    Google Scholar 

  48. Aston, J., Gaudenzi, S., Aston, J.: Studies in Documentary Film Interactive documentary : setting the field. vol. 3280, pp. 37–41 (2014). https://doi.org/10.1386/sdf.6.2.125

  49. Holz, T., Campbell, A.G., Ohare, G.M.P., Stafford, J.W., Martin, A., Dragone, M.: MiRA-mixed reality agents. Int. J. Hum Comput Stud. 69, 251–268 (2011). https://doi.org/10.1016/j.ijhcs.2010.10.001

    Article  Google Scholar 

  50. Bolter, J.D., Grusin, R.: Introduction: The Double Logic of Remidiation. Remediat. Underst. New Media. vol. 2 (1999)

    Google Scholar 

  51. Hill, R.W.: Ethics of immersive technologies. In: Abbas, A.E.E. (ed.) Next-Generation Ethics: Engineering a Better Society. pp. 39–53. Cambridge University Press (2019). https://doi.org/10.1017/9781108616188.004

  52. Sallnäs, E.-L.: Effects of communication mode on social presence, virtual presence, and performance in collaborative virtual environments. Presence Teleoper. Virtual Environ. 14, 434–449 (2005). https://doi.org/10.1162/105474605774785253

    Article  Google Scholar 

  53. Abramson, D.I., Johnson, J.: Creating a Conversational Chat Bot of a Specific Person, (2020)

    Google Scholar 

  54. Hogan, B.: The presentation of self in the age of social media: distinguishing performances and exhibitions online. Bull. Sci. Technol. Soc. 30, 377–386 (2010). https://doi.org/10.1177/0270467610385893

    Article  Google Scholar 

  55. Moncur, W., Kirk, D.: An emergent framework for digital memorials. In: Proceedings of the Conference on Designing Interactive Systems: Processes, Practices Methods, Techniques, DIS, pp. 965–974 (2014). https://doi.org/10.1145/2598510.2598516

  56. Moncur, W., Bikker, J., Kasket, E., Troyer, J.: From death to final disposition: roles of technology in the post-mortem interval. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 531–540 (2012). https://doi.org/10.1145/2207676.2207750

  57. Milgram, P., Kishino, F.: Taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D, 1321–1329 (1994). https://doi.org/10.1.1.102.4646

    Google Scholar 

  58. Speicher, M., Hall, B.D., Nebeling, M.: What is mixed reality? In: Proceedings of the Conference on Human Factors in Computing Systems (2019). https://doi.org/10.1145/3290605.3300767

  59. Schoen, S.W.: The rhetoric of evidence in recent documentary film and video. ProQuest Diss. Theses. 195 (2012)

    Google Scholar 

  60. Burke, K.: A Rhetoric of Motives. University of California Press, Berkeley (1969)

    Google Scholar 

  61. Facebook Finally Figured Out What to Do With Dead People. https://www.newsweek.com/facebook-legacy-death-306344. Accessed 12 Jan 2021

  62. Brubaker, J.R., Callison-Burch, V.: Legacy contact: Designing and implementing post-mortem stewardship at Facebook. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 2908–2919 (2016). https://doi.org/10.1145/2858036.2858254

  63. DeadSocial - Prepare for Death Digitally & Build Your Digital Legacy – About. https://deadsocial.org/about. Accessed 12 Jan 2021

  64. LivesOn LLC.: LivesOn

    Google Scholar 

  65. Cheney-Lippold, J.: New algorithmic identity, soft biopolitics and the modulation of control. Theory Cult. Soc. 28, 164–181 (2011)

    Google Scholar 

  66. Dre, Snoop D., Philip A., Dylan B., Digital Domain: Digital Domain Virtual Tupac, https://www.digitaldomain.com/work/virtual-tupac/. Accessed 15 Jan 2021

Download references

Acknowledgements

I am deeply indebted to my friend and colleague Dr. Anna Weisling for her attentive notes on this paper. I would also like to acknowledge the kindness and hospitality shown to me by the docents and staff at the Illinois Holocaust Museum.

Author information

Authors and Affiliations

  1. Department of Interactive Arts and Media, Columbia College Chicago, 600 S Michigan Ave, Chicago, IL, 60605, USA

    Joshua A. Fisher

Authors
  1. Joshua A. Fisher
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Joshua A. Fisher .

Editor information

Editors and Affiliations

  1. Eindhoven University of Technology, Eindhoven, The Netherlands

    Matthias Rauterberg

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fisher, J.A. (2021). The Ontology of Mixed Reality Agents Memorializing the Dead and Dying. In: Rauterberg, M. (eds) Culture and Computing. Design Thinking and Cultural Computing. HCII 2021. Lecture Notes in Computer Science(), vol 12795. Springer, Cham. https://doi.org/10.1007/978-3-030-77431-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77431-8_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77430-1

  • Online ISBN: 978-3-030-77431-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics