Evaluation: Questionaire on second release

In this post we explain how we collected and processed the feedback on the second release (version 0.2). We analyze the results from the questionnaires and from the timings.

Goal

First of all we wanted to check wether our refinement of the invite partners screen was successful (remember that a lot of users didn’t know what the “Done” button did, so we now show a warning message when the button is pressed while there are still pending invites). Version 0.2 had another addition: fridge management. We want to know whether this works fast enough, because it is not the “fun” part of our app. We also want to check our SUS score to track the overall progress, which was rather low in our previous evaluation.

Method

As in the first iteration, we asked users to fill in a questionnaire while or after they used our application. Apart from the SUS questions, we asked whether the user used the app for real, and how often they would do so if it was complete. We also asked for the user’s favorite dishes and additional remarks.

Additionally we timed the execution of a simple fridge management scenario by some users. We also asked them if they thought it was fast enough to really use.

Rationale

We asked the questions that came up during development (e.g. which dishes to add to the app) to get answers to those questions. We use SUS to track our overall progress. We timed fridge management to know whether this is fast enough.

Who, what, where

The user counter of Google Analytics still points to 21 users. This time we convinced only 8 people to fill in the questionnaire. We tested the fridge management with 5 users. All users are fellow students or siblings of ours (probably, the Google Docs questionnaire is anonymous).

Results

Alas, nobody used the application to organize a real party. 3 out of 8 said they would do so weekly, the rest answered monthly. Nobody said they would never use our app! We won’t elaborate on the favorite dishes of our users, apart from saying 5 out of 8 mentioned spaghetti 🙂

The results of the SUS questions are given in the following boxplots:

boxplot_sus1-3

boxplot_sus_4-6

boxplot_sus_7-9

boxplot_sus_10

The total SUS score is 79.4%, an improvement of  7.5 points. The timings for the different actions in fridge management are given in the next boxplots:

boxplot1 boxplot2

Conclusions

  • First of all, we conclude that while nobody has used the app for real, there is hope for the future. A problem is that we are entering the exam-period now, and our testing public might not want to use the app for real. We will have to find a way to motivate them to do so.
  • Secondly we decide from the SUS score that our refinement of the invite partners screen is really an improvement. We think this might have something to do with the prettier layout that makes our app seem more attractive.
  • Considering fridge management, we conclude that while nobody found they lost too much time, they did use more of it then we expected. We measure an average of 15.5s to add one ingredient, 24.8s to change its quantity and 3.4s to delete it. We do note that the users never used the functionality before, and that the clicking away of some errors and loading time is included in these timings.

 

Advertisements