The Users for the first session were 5 males and 1 female, with age ranging from 27 to 49. For the second, there were 4 males with age ranging from 26 to 52, with significant Brainstorming sessions experience, using Mind-mapping software, whiteboard computer based systems. The room setup consisted from a large space around Surface, with chairs surrounding it. All users were in the Contributor role and they had the opportunity to switch to Session Master any time. The application was started, with no Brainstorming Ideas on the tabletop.
The Evaluation session started with the Prototype presentation. This was targeted only on the User Interfaces, explaining how and which actions can be performed with each area of interest (UserControls and Main SurfaceWindow from Implementation Chapter). The users were encouraged to experience the interface and ask questions during the presentation, in order to understand better the interaction with the prototype.
Following the presentation phase was designed as a game, in order to exercise the prototype’s interface and evaluate the system and its features. The game was built around a imaginary, newly created, travel agency. The participants should create the company’s portfolio in a Brainstorming session, composed from the four phases (scenarios) presented in Typical use cases. They were disclosed one by one, in order to not affect the brainstorming process.
[nivo source=”current-post” ]
In the first stage was brainstorming was Brainstorming items creation with the theme Holiday Destinations. The users were encouraged to use all the means to add an item: by using the software keyboard, speaking to the microphone and drawing. The amount of ideas was not limited. They were also remembered that there are two additional brainstorming helpers: Twitter search for specific messages and Flickr search for images.
In the next stage, the users were supposed to group them into three categories by using the Prototype’s feature, structuring the imaginary company’s portfolio in three categories. They were encouraged to discuss about their decisions with the other users from the Brainstorming Session.
The final step was to relate them in logical structures by using the Prototype’s built-in Relating feature. The users were asked to create the structure of offered holidays.
The author of this paper was also the organizer of the User Evaluation session, observing the group dynamics, helping and reminding users the steps of specific interactions. The time of the session was limited to 45 minutes, from which the biggest share of time was allocated to the Brainstorming Item generation. The session concluded with the filling of the Questionnaire.
Hypothesis and methodology
The target of the first test was towards answering the following questions, which are the initial study goals, mentioned in Introduction:
- Q1. How was the User Experience using brainstorming applications running on multiuser multitouch tabletops?
- Q2. Is it useful to have post-brainstorming ideas organization?
- Q3. Is the brainstorming process successfully supported by mashing with other systems (e.g. Twitter and Flickr)?
- Q4. Is integration with small mobile devices a good solution to address the physical limitation of the tabletop?
- Q5. How important is to be able to persist the result of the brainstorming session?
By focusing on these ideas, the users will provide insight in the proposed process and also prove if the presented Breiny prototype is usable in real case scenarios. The hypotheses that were established before the process begun are presented below and state the author’s expectations from the prototype, expectations which can be proved or disproved by the Evaluation Session users:
- H1. Using tabletop improves the collaboration and communication among users
- H2. Multiple edit modes can improve creation of ideas
- H3. Grouping and Relating are important to structure the Brainstorming items
- H4. Fresh ideas from external sources improve the Brainstorming process
- H5. Interfacing with additional mobile devices improves the idea generation process
- H6. Sharing the outcome of the session is a useful feature
The questionnaire mentioned above was given to the Brainstorming Session participants and consisted from 7 pages for the first one and 8 pages for the second one. It started with creating a baseline of the user’s profile in brainstorming experience, tools used and familiarity with the tabletop used, Microsoft Surface. The next sections focused on the session itself, the social experience it brings by sharing a device, the input, grouping and relating possibilities offered by the Breiny prototype; following was the evaluation of external sources usability. For the second questionnaire, the next section was evaluating the impact of using additional devices in collaboration with the tabletop. The questionnaire ended with the evaluation of importance of sharing and saving features of the prototype and the final overview from the usability, utility and overall experience of the Breiny prototype.
Between the two User Evaluation sessions, several modifications were made, based on the user’s feedback. The most noticeable was the temporary deactivation of the Microphone Edit mode as the users felt that the session’s discussions were impeding using this device and several modifications to the mobile clients to accommodate new features offered by the prototype.
Conclusions
The following discussion is based on the results presented in Appendix A: User Evaluation Questionnaire, where the Brainstorming Session users’ feedback is presented. The questionnaire is focused on getting answers to the research questions and the hypothesis formulated in the previous section.
As presented in the beginning of the User Evaluation with Users in this section, the group can be considered balanced. All the questions presented to them are closed questions:
- grading difficulty types, in which the users must give a grade on a scale from 1 (very easy) to 5(very difficult);
- yes-no questions, to verify subjective opinions;
- free text areas, to allow the users to express their opinions or suggestions directly;
The first set of questions were targeted to the social, communication and collaboration side of the process, 90% of the users declaring that their movements were not impeded, the personal space was respected and they were confortable during the Brainstorming Session. The visibility was only 60%, the users mentioning that they focused on their idea generation and editing them, hence focusing on the tabletop and not on their colleagues. Regarding the physical limitation of the device itself, the general opinion was that it is too small to accommodate groups of 4-6 persons which directly interact with it, rather groups of 2-3 persons. Also the relative small resolution of the device’s screen (1024×768 pixels) made the participants to feel it “small”.
Judging these results, the H1 hypothesis may be considered partially true. Considering users’ feedback, the only part that can be improved and thus make the hypothesis fully true is not in the scope of this thesis, as it regards only to the hardware used – would be more useful that the tabletop’s screen with a bigger resolution, allowing users to better use the space available.
The next set of questions focused on the usability of input possibilities offered by prototype – using the software keyboard, drawing on the surface and using the microphone. From all users, the overall utility of the editing feature was considered very useful by 10 %, 60 percent – useful and 30% – moderated, concluding that this overall feature is important for the users. In the author’s opinion, it is possible that the users which rated it as moderated to be the ones more interested in the coordination, rather than in editing the items. For the questions particularly focused on the specific input device, there is a clear opinion that the users are more comfortable with the keyboard and the drawing than with the microphone. While all of them tried at least once the first two methods, only 2 of 6 tried the microphone, for the second User Evaluation the microphone was disabled. The feedback gathered from the last question in this set is pointing to the conclusion that it was difficult to use the microphone, probably because of the noise in the room. Other participants mentioned that the tabletop’s Operating System offers just one software keyboard at one time, such that no two users can type at the same time. During the editing process, this is an important limitation, however outside the scope of the prototype built.
Following these results, clearly demonstrates that more than one input method is preferred and the H2 hypothesis is true. If the implementation would consider only the default mode to insert text, the software keyboard, the participants would have led to “keyboard fight”.
Following in the User Evaluation were questions regarding “Relating ideas” by color grouping and relating through graphs. More than half of the participants considered it useful and 33% very useful. The relating process was considered relatively simple or manageable by all users and the mean value was almost “Relatively simple”, 2.33 on a scale from 1 to 5, which can be considered a good value, given the fact that some of participants never interacted before with Microsoft Surface, everything being new for them in the proposed setup. All the participants used this features, 67% for “Few times” and the rest for “4-5 times”.
Given the fact that the brainstorming process was quick and the amount of ideas generated were 2-3 per participant and considering the previous paragraph statements, the author considers that also H3 hypothesis can be considered true.
The User Evaluation Questionnaire continued with “Generating ideas from external sources” which started with the question regarding the utility of this feature (the question was not focused on one mash, but considered together Flickr and Twitter interaction). It was considered very useful by 11% and Useful by 67%. From all participants, 33% used it few times and 22% “Probably one time”. This feature was considered relatively easy to use (on a scale from 1 to 5, one being the very easy) by half of the users and “Very easy” or “Manageable” by one participant, respectively. With a mean of value 2,33 we can concludes that this feature is easy to use by all users, especially that some of the participants never interacted with the tabletop device, the entire system being completely new for them. The users suggested many new sources to mash with, denoting their interest for this specific feature. One user remarked that even if the feature was “Very easy to use, the result retrieved wasn’t useful”, proving the fact that more alternative sources would have been improved the Brainstorming Session.
Following the above particular conclusions, it is proven that the fourth hypothesis is true and the system provides a useful and easy way to refresh the Brainstorming Session.
For the second group, the interaction with other mobile devices was enabled. There were two devices available to be used by the four participants to the session. All four had the opportunity to use them; one participant created 4-5 ideas and other two few ideas. From all, 75% considered that it is easier to create ideas on the iPhone than on Microsoft Surface, but 100% agreed that it is easier to interact with them on the Surface. Interestingly, half of the group felt excluded from the group when using the mobile device, being more concentrated to write the idea on it than to interact with the rest of the group. Another question revealed the fact that nobody felt uncomfortable when using the mobile device, as opposed with one of six people from the first group.
Based on the facts exposed in the previous paragraph, the author concludes that hypothesis H5 is true and the Brainstorming process is improved by using additional devices. Even if the users agreed that is easier to create ideas on their own small mobile device, having full access to keyboard (no more “keyboard fight” as already presented in the evaluation of the “Multiple input possibilities”), they also fully agreed that the interaction with the already created content is enhanced on the tabletop.
The last feature to evaluate was the sharing of the Brainstorming Session, where half of the participants considered it important and 10% essential, while 40% considered it useful. When asked about if they would like to have the possibility to later review the session, 50% considered it “Very useful” and the rest “Useful”. Regarding the willingness to review it with other colleagues, again all replyied yes, from which 20% “For sure YES”, 50% “YES, most of the time” and 30% “Probably YES”.
Based on previous data, also the H6 hypothesis is confirmed true, users indeed desire to share data, have the possibility to review it and for sure reviewing it with their colleagues.
The final part of the Questionnaire was related to the usability of the prototype. Most of the participants replied that the consider it 3/10, 1/10 being the best grade, the mean of 3.2 proving that even if the prototype is usable, it still requires some improvements; some of them have been implemented and the rest are proposed as future work. From the Utility point of view, most of the users considered it helpful and again considered it 3/10, 1/10 being the best grade, the mean of 2.8; I conclude that once the usability of the prototype will be improved, also its utility will increase. The overall User experience was graded as 2/10 by almost half of the participants, 1/10 being the best grade, the mean of 3.1 meaning that the prototype is engaging and pleasant to use. Another impressive observation is that even if the proposed application is still a prototype, all the participants would recommend it to their colleagues as a Brainstorming tool.
In conclusion, the evaluation sessions revealed that almost all hypotheses were true, with only one partial true because of the limitations of hardware used. We also found some problems occurred only on multi-user environment, such as the microphone issue – because of the noise it proved not to be a very useful feature, . Also, a flaw was detected when users tried to activate and deactivate the microphone several times in a very short time span, the application crashed. These were useful feedbacks and were taken into consideration and fixed. The author is very happy with the outcome and the feedback of the User Evaluation session of the Breiny prototype and believes that this prototype is close to become a concrete application.
