Keywords

1 Introduction

27% of older adults are completely inactive [1]. If elderly citizens are motivated to do more cognitive and physical exercises, it may have the potential to slow down mental and physical decline [2, 3]. Even 75 min a week of light physical activity could reduce the risk to develop a cardiovascular disease by as much as 14% [4]. Furthermore, the risk of developing dementia can also be reduced by playing challenging games, interacting with other people and doing physical exercises [5, 6]. Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR) technologies offer novel opportunities for an interactive learning style and new ways of interaction. There are some existing solutions for the elderly regarding physical or cognitive training using VR or AR but not MR like in TACTILE. Regarding VR, various solutions and projects can be found like the project Rehabilitation Gaming System (RGM) that reports a system for the rehabilitation of elderly people at home after suffering a brain lesion due to stroke [7]. Quintero et al. [8] survey the use of AR for education including minorities and students with impairments, noting that only 2% of the reviewed studies aim elderly people to improve their physical and mental health. Further research regarding VR and AR applications for the elderly target group can be found in [9,10,11,12,13,14]. However, existing solutions mainly aim to entertain, some have a focus on rehabilitation, but there is no communication or possibility to play together with a real person especially combined with a training aspect.

The goal of the EU project TACTILE is to foster an active lifestyle and well-being of older adults, through a solution that enables training both, the cognitive and physical state, and maintaining social contacts by connecting seniors with their family and friends. TACTILE provides an easy to use MR software for elderly people to play board games and do physical training exercises on MR glasses (Fig. 1).

Fig. 1.
figure 1

© AIT/www.einstellungssache.at

The idea of the TACTILE system. Left: User plays Ludo in Mixed Reality. Right: User is doing physical exercises together with a virtual avatar.

Full size image

2 The TACTILE System

The TACTILE system consists of the MR app that provides board games and physical trainings; and the portal that holds user accounts that can be managed by the secondary end-users like caregivers. The app runs on the MR glasses “Magic Leap One” (ML1)Footnote 1, which was chosen because of user requirements and market availability at project start. As development environment we are using the game engine Unity3DFootnote 2. The final system will offer two board games – Ludo and Halma – as well as eight training exercises. At the current stage of the project, we implemented the first version of the board game – Ludo - along with four exercises. This setup is currently tested at the end-user partners GedächtnistrainingsAkademie (GTA) in Austria and TanteLouise (TAL) in the Netherlands.

2.1 Board Games

The MR app provides familiar board games that are played in a MR setting with a remote partner, which can be a family member, a friend or another secondary end-user like a caregiver. The user plays with physical game pieces on a real game board that he is looking at. The game board and pieces are detected and tracked via computer vision (CV) algorithms. To detect the game board, AprilTag [15] visual markers are used, which are mounted on the game board, as depicted in Fig. 2. Based on the detected pose of the markers, the location and rotation of the game board in real-world coordinates can be determined. Game pieces are detected using the geometry information of the detected game board and the derived well-known locations of the game fields, as well as CV algorithms based on image segmentation. The detected game pieces are displayed in the right position on the opponent’s game board as virtual game figures. To avoid problems regarding occlusions and different light conditions, repainted pink Checkers figures are used for the first prototype to improve their detection. Figure 2 shows the board game in the MR glasses from the user’s perspective.

Fig. 2.
figure 2

Left: Virtual game figures are projected on the real game board. The red figures are belonging to the real Checker figures of the local player; the green ones are the projected figures of the remote player. Right: Training menu is overlaid on the avatar.

Full size image

2.2 Physical Exercises

The TACTILE system provides physical trainings that are done together with a virtual avatar. The avatar explains the exercises and guides the user through the training. Since the virtual avatar is standing next to the user in the real environment in MR, the user is expected to be able to follow the instructions and demonstrations more easily. To adapt the physical trainings to the individual user needs, the trainings can be adjusted via a user portal in the browser. The secondary end-user can choose different exercises and combine them to a training. Difficulty levels can be defined and adapted according to the individual user needs. This makes sure that users with physical limitations are not excluded by the developed system.

3 Design and Development Methodology

TACTILE follows a user-centered design process. In a first step, end-user requirements were collected both in Austria and in the Netherlands by the end-user partners GedächtnistrainingsAkademie (GTA) and TanteLouise (TAL). In Austria, a requirements analysis with 11 workshops took place. 75 primary users (elderlies at an age >65 years) and 35 secondary users (relatives and caregivers) were asked by two psychologists about basic requirements. The requirements workshop for the primary users included questions about favorite and well-known board games, physical and visual disorders and conditions at home like the amount of space they have for the physical training. The secondary users were asked about requirements regarding the online portal (e.g. statistics, options to adjust the settings for the primary users). The exercises for the physical training were selected by literature research and expert interviews with occupational therapists and physiotherapists.

During development, end-users are constantly providing feedback via end-user workshops that are taking place at the end-user organizations in Austria and the Netherlands. The first user workshop took place in October 2019, where focus was on the user interface and interaction design as well as on the avatar design. Further, the first two training exercises were evaluated. The results of the first end-user workshop are described in Sect. 4. In the second workshop, which takes place in March 2020, four new exercises as well as the first board game – Ludo are tested by the end-users. In the final workshop, which will take place in August 2020, the end-users will evaluate the 3rd prototype, which will include all planned exercises as well as both board games. The feedback will support the further improvement of the prototype for the field trials that will start in February 2021 in Austria and the Netherlands.

3.1 UI Design and Interaction

Since older adults often have troubles handling technical devices, it is from uttermost importance to pay special attention to the user interaction during the development process of the MR solution. MR offers new opportunities to interact in a natural way that is familiar to this generation. In TACTILE, the MR application is designed in a way that only a minimal number of UI elements - such as buttons – are implemented, because it is important to keep the natural interaction that is familiar to the target group. Since the UI exists in the same 3D space as the other virtual elements, it is challenging to position it as to not disturb the user interaction when playing board games and doing physical exercises (see Fig. 2). For the first user workshop, we developed a UI prototype, where the user had to choose buttons of different size (see Fig. 3) with three different modes of interaction. One interaction mode is controller-based, the others are based on hand movement. The controller-based approach used the controller as laser pointer, selecting buttons with the laser beam and pressing a controller button. For both hand-based modes, a button is highlighted by holding the open hand over it. In the first mode, selection is achieved by keeping the hand over it for a few seconds and in the other one by closing the hand. Apart from the interaction modes, we investigated how the UI design must look in MR to easily select UI elements. Therefore, the interaction modes were tried out on UI screens with different number of buttons. Further, different font sizes and the color scheme were reviewed. The results are described in Sect. 4.

Fig. 3.
figure 3

Different button sizes with different interaction methods were tested in the first user workshop to find out how buttons have to be designed for the elderlies.

Full size image

3.2 Avatar Design & Training Exercises

According to the collected end-user requirements, the first training exercises were implemented. Two avatars – one male and one female – were designed with respect to the older target group. In the first end-user workshop the avatar design as well as the first exercises were evaluated (see Fig. 4). Further, the sequence of the exercises and overall understanding of the avatar’s explanation were assessed. The results are described in Sect. 4.

Fig. 4.
figure 4

Left: A virtual avatar (male or female) is guiding the user through the training exercises. Middle: User is following the movements of the avatar. Right: User is choosing a button

Full size image

3.3 Board Games

Based on the outcome of the collected user requirements and the technical feasibility, the board games Ludo and Halma are chosen. Since elderlies often have problems with their fine motor skills, game pieces and dice larger than found in common board games are used. This also comes handy for the developed algorithms to detect the game board and the game pieces. As described in Sect. 2.1, we decided to choose game figures of the game “Checkers”. In the second end-user workshop we want to find out if these figures are accepted by the elderlies or if they prefer the original game figures. Further, the end-users test the first board game - Ludo. The end-user feedback will define our future developments towards the third prototype.

4 Results

In October 2019 the first end-user workshop took place in Austria and the Netherlands. In the following the results of the first workshop in Austria are described. N = 20 primary users (PUs) (13 female, 7 male) and N = 19 secondary users (SUs) (13 female, 6 male) participated. The workshop lasted for one and a half hours and included interviews, several exercises wearing the ML1, and questions about usability and accessibility. The sociodemographic data shows that the PUs in Austria were on average 76.7 years old (±5.57 standard deviation N = 20) and the SUs were 42.2 years old (±14.2 standard deviation N = 19). Three of the PUs had no laptop, PC or mobile phone. 75% of PUs and 90% of SUs agreed to the sentence “technologies enrich my daily life” and 80% of the PUs and 95% of the SUs agreed to the sentence “I am critical about new technology”. None of the PUs had any previous experience with AR or VR systems. 25% of the SUs had tried VR glasses for gaming before.

4.1 Test Sequence During the Workshop

The first user workshop was split in four different tasks. In the first task the participants had to put on the MR glasses, adjust them, turn them on and start the application. The exercise has been completed by all participants and only 15% of the PUs needed help by one of the workshop leaders. In the second task the participants had to select virtual buttons in the user interaction prototype as described in Sect. 3.1 the different number and sizes of the buttons made no difference for the participants. Great differences appeared when the participants rated the different interaction methods for choosing the buttons as shown in Table 1. The PUs and SUs rated the three possibilities from 1 “very good” to 5 “very bad”. In the third task two players tried to connect to each other and choose a board game. All participants described the procedure as easy to understand and straightforward.

Table 1. Percentage of participants that rated the interaction methods as “very good” or “good”.
Full size table

The last task was about the physical training. The participants had to choose between the male and female avatar, place it into the room, start the physical training and provide feedback about the training exercises. None of the SUs and 5 PUs needed help to complete this task. The main problems were the positioning of the avatar and the differentiation between the exercise explanation and the avatar’s instruction to actually start the training exercise.

4.2 Usability/Accessibility

After completing the tasks, participants were asked to answer questions regarding the usability and accessibility. A 5-point Likert scale ranging from “I absolutely agree” to “I absolutely disagree” was used. 18 PUs and 18 SUs completed the questionnaire. 50% of the PUs and 88.89% of the SUs claimed that they want to use the system more often. Another 25% of the PUs answered neutral and claimed that they have to see more of the system in the next workshop to evaluate it properly. 83.3% of SUs and 63.3% of PUs rated the system as not unnecessarily complex. 100% of SUs and 83.3% of PUs claimed that the system was easy to use. 27.76% of PUs and 5.56% of the SUs claimed that they would need help of a technically more experienced person, but 94.45% of SUs and 66.67% of PUs think that they would easily learn to use the system. Only one SU and 3 PUs said that the system was strange to use. 94.44% of the SUs and 77.78% of the PUs agreed to the sentence “I felt very comfortable while using the system”. In terms of accessibility 100% of SUs and 83.3% of the PUs agreed that the meaning of the buttons was clear, but only 50% in each group agreed that the information in the system was highlighted properly. 2 PUs and 1 SU rated the contrast between foreground and background in the user interface as not clear enough. None of the asked participants disagreed to the sentence “the navigation to go ahead in the system was clear” and 100% of the asked participants agreed that the feedback of the system was sufficient to know if the steps were done right.

5 Summary and Outlook

This work highlights the aim, the technical prototype, and the first user involvement results of the EU co-funded project TACTILE. The paper underlines that MR-based solutions have various supporting potentials but are not widely used, especially not in the target group of older adults. As outlined in Sect. 4, the TACTILE prototype highly motivates seniors to use, learn from, and enjoy the solution even if it is still in an early development stage. The feedback of the first user workshop was incorporated into the second prototype that is currently tested in Austria and the Netherlands. In our future activities we will evaluate and incorporate the feedback of the current end-user workshops and implement the second board game “Halma” and the remaining training exercises that will be tested in the 3rd end-user workshop. These results will further support the final development phase before the final TACTILE prototype will be tested and evaluated by the primary- and secondary target group in a long-lasting field trial phase in Austria and the Netherlands.