Skip to main content
Springer Nature Link
Log in

Adaptive Planning: Comparing Human and AI Responses in Premortem Planning

  • Conference paper
  • First Online:
HCI International 2024 – Late Breaking Papers (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15382))

Included in the following conference series:

  • 536 Accesses

Abstract

Premortems are a structured analytic technique designed to evaluate a plan before it is implemented. They have been used to improve estimates and generate alternative predicted outcomes. One psychological mechanism that underlies the Premortem is prospective hindsight with people generating reasons for a plan’s failure. The reasons reduce people’s confidence and overconfidence in their plans relative to other plan critiquing methods. In this case study, we qualitatively compare student-generated responses for evaluating a plan and a large language model-generated (e.g., Chat GPT-4) evaluation of the same plan. To the extent that the reasons for failure generated by the large language model are similar or different to the human participants is the first step in a conceptual evaluation. The results have implications for a variety of structured analytic techniques to improve human-AI teaming in decision making. Furthermore, they provide initial suggestions for where adaptive systems could support collaborative intelligence.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
€34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 53.49
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 70.61
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

') var buybox = document.querySelector("[data-id=id_"+ timestamp +"]").parentNode var buyingOptions = buybox.querySelectorAll(".buying-option") ;[].slice.call(buyingOptions).forEach(initCollapsibles) var buyboxMaxSingleColumnWidth = 480 function initCollapsibles(subscription, index) { var toggle = subscription.querySelector(".buying-option-price") subscription.classList.remove("expanded") var form = subscription.querySelector(".buying-option-form") var priceInfo = subscription.querySelector(".price-info") var buyingOption = toggle.parentElement if (toggle && form && priceInfo) { toggle.setAttribute("role", "button") toggle.setAttribute("tabindex", "0") toggle.addEventListener("click", function (event) { var expandedBuyingOptions = buybox.querySelectorAll(".buying-option.expanded") var buyboxWidth = buybox.offsetWidth ;[].slice.call(expandedBuyingOptions).forEach(function(option) { if (buyboxWidth buyboxMaxSingleColumnWidth) { toggle.click() } else { if (index === 0) { toggle.click() } else { toggle.setAttribute("aria-expanded", "false") form.hidden = "hidden" priceInfo.hidden = "hidden" } } }) } initialStateOpen() if (window.buyboxInitialised) return window.buyboxInitialised = true initKeyControls() })()

Institutional subscriptions

Similar content being viewed by others

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

References

  1. Bettin, B., Steelman, K.S., Wallace, C., Pontious, D., Veinott, E.S.: Identifying and addressing risks in the early design of a sociotechnical system through premortem. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 66, No. 1, pp. 1514–1518. SAGE Publications, Sage, Los Angeles, CA (2022)

    Google Scholar 

  2. Buehler, R., Griffin, D., Ross, M.: Exploring the “Planning fallacy”: why people underestimate their task completion times. J. Pers. Soc. Psychol. 67(3), 366–381 (1994)

    Article  Google Scholar 

  3. Chen, X.A., et al.: Next steps for human-centered generative AI: a technical perspective. arXiv preprint arXiv:2306.15774 (2023)

  4. DeRosa, D. M., Smith, C. L., & Hantula, D. A. (2007). The medium matters: Mining the long-promised merit of group interaction in creative idea generation tasks in a meta-analysis of the electronic group brainstorming literature. Computers in Human Behavior, 23(3), 1549–1581–239

    Google Scholar 

  5. Dell'Acqua, F., et al.: Navigating the jagged technological frontier: field experimental evidence of the effects of AI on knowledge worker productivity and quality (September15, 2023). Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 24-013 (2023). https://ssrn.com/abstract=4573321 or https://doi.org/10.2139/ssrn.4573321

  6. Eapen, T.T., Finkenstadt, D.J., Folk, J., Venkataswamy, L.: How generative AI can augment human creativity. Harv. Bus. Rev. 101(4), 56–64 (2023)

    Google Scholar 

  7. Gallop, D., Willy, C., Bischoff, J.: How to catch a black swan: measuring the benefits of the premortem technique for risk identification. J. Enterprise Transform. 6(2), 87–106 (2016)

    Article  MATH  Google Scholar 

  8. Gigone, D., Hastie, R.: The impact of information on small group choice. J. Pers. Soc. Psychol. 72(1), 132–140 (1997). https://doi.org/10.1037/0022-3514.72.1.132

    Article  MATH  Google Scholar 

  9. Gupta, P., Ding, B., Guan, C., Ding, D.: Generative AI: aA systematic review using topic modelling techniques. Data Inform. Manage. (2024)

    Google Scholar 

  10. Hackman, J.R.: Collaborative intelligence: using teams to solve hard problems. Berrett-Koehler Publishers (2011)

    Google Scholar 

  11. Hagendorff, T., Fabi, S., Kosinski, M.: Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nat. Comput. Sci. 3(10), 833–838 (2023)

    Article  Google Scholar 

  12. Hoffman, B.G.: Red teaming: how your business can conquer the competition by challenging everything. Crown business (2017)

    Google Scholar 

  13. Klein, G.: Performing a project premortem. Harvard Bus. Rev. 85(9), 18–19 (2007)

    Google Scholar 

  14. Klein, G., Pliske, R., Crandall, B., Woods, D.D.: Problem detection. Cogn. Technol. Work 7, 14–28 (2005)

    Article  Google Scholar 

  15. Kohn, N.W., Paulus, P.B., Choi, Y.: Building on the ideas of others: an examination of the idea combination process. J. Exp. Soc. Psychol. 47, 554–561 (2011)

    Article  MATH  Google Scholar 

  16. Lehman, B., Veinott, E.S.: Changing perspectives: examining factors related to counterfactual thinking in ambiguous social judgments. In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 44, no. 44 (2022)

    Google Scholar 

  17. McDermott, A.F., et al.: Developing an adaptive framework to support intelligence analysis. In: Sottilare, R., Schwarz, J. (eds.) 3rd International Conference for Adaptive Instructional Systems as part of HCI International (HCII) (2021)

    Google Scholar 

  18. Mosier, K.L., Fischer, U.M.: Judgment and decision making by individuals and teams: issues, models, and applications. In: Decision Making in Aviation, pp. 139–198. Routledge (2017)

    Google Scholar 

  19. Mitchell, D.J., Russo, J.E., Pennington, N.: Back to the future: Temporal perspective in the explanation of events. J. Behav. Decis. Mak. 2, 25–38 (1989)

    Article  MATH  Google Scholar 

  20. Mueller, S.T., et al.: Principles of explanation in human-AI systems. arXiv preprint arXiv:2102.04972 (2021)

  21. Mueller, S.T., Hoffman, R.R., Clancey, W., Emrey, A., Klein, G.: Explanation in human-AI systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. arXiv preprint arXiv:1902.01876 (2019)

  22. Orasanu, J., Fischer, U.: Finding decisions in natural environments: the view from the cockpit. In: Naturalistic Decision Making, pp. 343–357. Psychology Press (2014)

    Google Scholar 

  23. Parnes, S.J., Meadow, A.: Effects of ‘brainstorming’ instructions on creative pro lem solving by trained and untrained subjects. J. Educ. Psychol. 50(4), 171–176 (1959)

    Article  Google Scholar 

  24. Pearson, C.M., Misra, S.K., Clair, J.A., Mitroff, I.I.:. Managing the unthinkable. Organiz. Dyn. 26(2), 51–64 (1997)

    Google Scholar 

  25. Peabody, M., Veinott, E.: Focus shift: differences in reasons generated using Premortem and Worst-Case Scenario plan evaluation methods. Naturalistic Dec. Mak. 259–261 (2017)

    Google Scholar 

  26. Peabody, M.: Improving planning: quantitative evaluation of the premortem technique. Unpublished Masters of Science Thesis. Michigan Technological University (2017)

    Google Scholar 

  27. Putman, V.L., Paulus, P.B.: Brainstorming, brainstorming rules and decision making. J. Creat. Behav. 43, 23–39 (2009)

    Article  MATH  Google Scholar 

  28. Sunstein, C.R.: Worst-case scenarios. Harvard Press (2009)

    Google Scholar 

  29. Roose, K.M., Lehman, B.R., Veinott, E.S.: Premortems in game development teams: impact and potential. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 67, no. 1, pp. 1856–1861. SAGE Publications, Sage, Los Angeles, CA (2023). https://doi.org/10.1177/216950672311936

  30. Shaer, O., Cooper, A., Mokryn, O., Kun, A.L., Ben Shoshan, H.:. AI-augmented brainwriting: investigating the use of LLMs in group ideation. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–17 (2024)

    Google Scholar 

  31. Shin, J.G., Koch, J., Lucero, A., Dalsgaard, P., Mackay, W.E.: Integrating AI in human-human collaborative ideation. In: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–5 (2023)

    Google Scholar 

  32. Valacich, J.S., Dennis, A.R., Connolly, T.: Idea generation in computer-based groups: a new ending to an old story. Organ. Behav. Hum. Dec. Proces. 57, 448–467 (1994)

    Article  MATH  Google Scholar 

  33. Veinott, B., Klein, G., Wiggins, S.: Evaluating the effectiveness of the premortem technique on plan confidence. In: 7th International ISCRAM Conference. Seattle (2010)

    Google Scholar 

  34. Veinott, E.S.: Adaptive collaborative intelligence: key strategies for sensemaking in the wild. In: Stephanidis, C., et al. (eds.) HCI International 2021 - Late Breaking Papers: Cognition, Inclusion, Learning, and Culture. HCII 2021. LNCS, vol. 13096. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-90328-2_8

  35. Wadinambiarachchi, S., Kelly, R.M., Pareek, S., Zhou, Q., Velloso, E.: The effects of generative AI on design fixation and divergent thinking. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–18 (2024)

    Google Scholar 

  36. Whitaker, E., Trewhitt, E., Veinott, E.S.: Intelligent tutoring design alternatives in a serious game. In: Sottilare, R., Schwarz, J. (eds.) First International Conference for Adaptive Instructional Systems as part of HCII 2019, pp. 151–165. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_13. ISBN 978-3-030–22341-0

  37. Whitaker, E., Trewhitt, E., Veinott, E.S.: Heuristica II: updating a 2011 game-based training architecture using generative AI tools. In: Sottilare, R.A., Schwarz, J. (eds.) Adaptive Instructional Systems. HCII 2024. LNCS, vol. 14727. Springer, Cham (2024). https://doi.org/10.1007/978-3-031-60609-0_23 (2024)

  38. Woolley, A.W., Aggarwal, I., Malone, T.: Collective intelligence and group performance. Curr. Dir. Psychol. Sci. 24, 420–424 (2015)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Michigan Technological University, Houghton, MI, 49931, USA

    Elizabeth S. Veinott & Betsy R. Lehman

Authors
  1. Elizabeth S. Veinott
    View author publications

    Search author on:PubMed Google Scholar

  2. Betsy R. Lehman
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Elizabeth S. Veinott .

Editor information

Editors and Affiliations

  1. Siemens Corporation, Princeton, NJ, USA

    Helmut Degen

  2. and Technology - Hellas, Foundation for Research, Heraklion, Crete, Greece

    Stavroula Ntoa

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Veinott, E.S., Lehman, B.R. (2024). Adaptive Planning: Comparing Human and AI Responses in Premortem Planning. In: Degen, H., Ntoa, S. (eds) HCI International 2024 – Late Breaking Papers. HCII 2024. Lecture Notes in Computer Science, vol 15382. Springer, Cham. https://doi.org/10.1007/978-3-031-76827-9_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-76827-9_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-76826-2

  • Online ISBN: 978-3-031-76827-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics