WP4 Impact Evaluation

Spotlight on potential users

The task of Laurea University of Applied Sciences was to assess the impacts of the new service (to be launched on the market within the next three years) on the social connectedness of older adults during the user-centric service development process. At the first glance the task seemed rather impossible to execute. A totally new approach was needed.

Following the user-centric approach adopted in the Express to Connect project we decided to rely on the opinions of the potential users, the test persons themselves, and to ask them to anticipate the impacts of the service concepts and prototypes. During the last phase of the project when the Storyville games had been finalized we discussed the anticipated impacts of Storyville gaming with the municipal and other experts in social and health care, and with the representatives of other projects focusing in developing preventive activities for the elderly.

The evaluation of the anticipated impacts by the users

We started our impact evaluation planning process by studying the well-known questionnaires on wellbeing, for example: WHOQOL, 15D, PGC Morale Scale and LEIPAD (De Leo, Diekstra etc. 1998) and scales on social connectedness (e.g. Lee & Robbins 1995).

We compared the items in the scales to the desired outcomes identified in the Express to Connect ethnographic study. We came up to the conclusion: the generic scales are not sensitive enough to be able to identify the changes in the everyday life of a senior the new service is designed to achieve. Therefore it was necessary to design a new questionnaire based on the understanding of the social connectedness among older adults adopted in the Express to Connect project.

The questionnaire focusing on the anticipated impacts was used in all the tests starting from the experience prototype test in November – December 2010. In the demo test of the digital game (April– May 2010) the participants of the test anticipated for example the following impacts:

Demo tests (April– May 2010): Anticipated impacts, % of YES-answers, N=16.

 

Percentage of “Yes”- answers in 2011 (N=16)

I am able to better outline the life I have lived so far by reminiscence.

81 %

I perceive the joy of learning and the feeling of being capable more often than before.

 

75 %

The feeling of being related to other people has become stronger.

69 %

I’m able to tell and share the memories of my life, especially of those persons who have been important to me.

 

69 %

 

As part of the final field test a more detailed study on the interaction and game experience during the gaming sessions was conducted.  We maintain that it is necessary to keep on playing the games and keep on discussing on the stories heard during the gaming sessions for a longer period of time to strengthen the social connectedness of the players, and thus prevent loneliness. It is the accumulated positive game experiences, and the accumulation of empowering interaction between the players that will in the long run lead to social connectedness, thus avoiding the risks of loneliness to come true.

The interaction during the game sessions

The interaction during the gaming sessions (Final field trial, Oct.– Nov. 2012, N=107) was perceived very positive by the older test persons (50 years or older). Almost all of them (94 %) said that the atmosphere amongst the players was relaxed. Nine out of ten players reported that the players showed willingness to help each other (90 %) and approval towards each other (86 %). Three players out of four felt close to other players (77 %) – thus it can be concluded that the gaming sessions offer opportunities for strengthening social connectedness among the players.

The gaming experience

Playing the Storyville games was perceived as fun (83 %) and the gaming experience was perceived as a positive surprise (69 %) by the older test persons. Two thirds of them maintained that it was interesting to get to know the games (67 %).  Using the iPad was easy for two thirds of the older test persons (67 %), and the same amount of them (63 %) said that the game flow was easy to follow. The majority of the older test players (65 %) had experienced the joy of success during the game play. Six out of ten older test players thought that playing was interesting throughout the whole gaming session, and the playing was suitably challenging during all the gaming sessions.

The anticipated impacts

In the final field trial we followed the same procedure with impact evaluation as in the previous tests. The participants were asked to imagine that they had played the games for months, even for a year. At least six out of ten older test players agreed with the following items:

When the gaming sessions create opportunities for high quality encounters between family members, next of kin, friends, acquaintances, neighbours, and peers at the meetings of senior associations the social connectedness of the players will be strengthened and thus the risk of falling into loneliness will be diminished significantly.  When interpreting the answers given to the questions concerning the anticipated impacts we have to keep in mind the shortness of the testing period.

Lessons learnt

In the Express to Connect project we have learnt that assessing the anticipated impacts of an innovation during its development process is a mission possible.The solution lies in the user-centric approach. The evaluation of the anticipated impacts of an innovation has to be based on the desired outcomes the users themselves have defined. In the case of Express to Connect project a new questionnaire was designed to be used in the evaluation.

Cost-efficiency analyses based on the Portraits

The Personas created and described by Copenhagen Living Lab in the context of the first work package focusing on gathering insights on the focal phenomenon, social connectedness, were the starting point for our work on developing a new approach for estimating the cost-efficiency of a new service to be launched on the market in the near future.We chose three Personas(later called Portraits) out of six, and started writing scenarios for the persons described in the Portrait.

Two kinds of scenarios for a period of one year were written. The first ones describe the lives of the Portraits when the risks of falling into loneliness come true.The second scenarios describe the lives of the Portraits when the Storyville gaming sessions work as a trigger for strengthening connectedness.

We presented the E2C Impact Model, the descriptions of the three Portraits, and the scenarios to ten Finnish and four Danish experts in elderly care and wellbeing of older adults. In the meetings with the experts we gathered feedback on the E2C Impact Model, and identified several potential chains of effects in the lives of the Portraits.

One of our tasks included in the evaluation of the impacts of the Storyville gaming was to analyse the cost-efficiency of the new service, that is the games to be played on an iPad together with family members, next of kin, and friends. The task was extremely challenging as the games had not been launched on the market. We decided to work with the Portraits, and the chains of effects identified in the discussions with the experts.

For each Portrait we chose one chain of negative effects, and one chain of positive effects. These two chains were kind of mini scenarios, or opposite sides of one coin.  We started putting the price tags on the chain of negative effects. The following table present the costs of negative effects for one of the Portraits and for the public sector.

The costs of the chain of negative effects for Kirsten, The Professional Human Being, and for the public sector.

 

Costs for the individual and her family

 

Costs for the public sector

Visiting doctor in local health center twice a year, including EKG, labs.

13,80 € x 2

        ->27,60 €

242 € x 2

      ->484,00 €

Visiting emergency unit at night time because of the flu (instead of day time visit to health center)

 

27,50 €

 

295,00 €

Her daughter’s family has to hire a babysitter 15 times 4 hours/ year (Kirsten used to babysit)

8,20 € x 60 h           

-> 492,00 €

Tax reduction 45%

 ->271,00 €

 

221,00 €

Safety services

361,00 €

0,00 €

TOTAL

687,10 €

1.000,00 €

 

When we manage to avoid the risks of falling into loneliness, the costs presented in the above table, will not come true. When assessing the cost-efficiency of the new service from the individual’s point of view, the costs avoided can be regarded as “savings”.

To be able to calculate the net effect of the new service we have to assess the costs of the usage. In this case Kirsten has to buy an iPad if she does not already own one, and she has to download the games from the iTunes Store.  In addition to buying the game board and the games, some money has to spent on organising the gaming sessions. 

 

The net effect for Kirsten

“Savings” (see table above)

+687,10 €

Three games (iTunes Store)

– 5,00 €

Refreshments during the visits of neighbouring ladies (gaming sessions)

 

– 150,00 €

New table cloth, new coffee cups, and other tableware

 

– 175,00 €

The net effect

+357,10 €

 

Lessons learnt

The idea of utilising Portraits as the basis for the assessment of the anticipated impacts in monetary terms is a new one. It is promising but more work has to be done to develop the approach further. Especially it will be very interesting to try to find out ways and procedures how to scale up the net effects from the level of one Portrait to the level of a larger population.