Roadmap for Simulation & Game-based Learning Research

Panel on Simulation and
Game-based Learning Research

October 18, 2006
Online Workshop

> Workshop Overview <

> Program <

> Evaluation <

> Attendee Map <

Evaluation Summary

This “virtual” workshop was conducted entirely via the Internet. There were 51 “seats” allocated to the event. 61 people participated. Of those who participated, 15 responded to our follow-up email requesting feedback on the content and quality of the session. Those responses are summarized below.

CONTENT
What were your reasons for participating?

Interest in the content: 9
Interest in how the technology worked: 4
Research area/Funding/networking: 3
Was asked to participate: 2

What did you get out of this conference?

Finding out what’s happening in this field; what the challenges and opportunities are: 6
Information on virtual patients, simulation and educational games: 4
Validation for many of many ideas: 3
Interesting talks from people that you don't get to meet often: 2
Great sense of community: l
Reflected over my own projects; got new ideas; will probably initiate contacts with one or two of the panelists: l

What additional follow up information would you like to have?

Copy of slides: 5
Copy of, or links to transcripts or play back: 2
Find a way to strengthen this community of practice: l
It would be great to also have bibliographies available: l
If possible, to build international version for those with a non-English Speaking Background: l
Q&A: l
More of same: l

CONFERENCE FORMAT
How comfortable were you with this online conference format?

Yes, comfortable: 11
OK: 2 ( if the session does not exceed 1.5-2 hours)
Comment: there was very little time for discussion/interaction. The comments people wrote were fine, but we could not discuss them. Also, whilst giving the presentation, the comments were distracting.

What was your level of interaction?

Listened: 12
Asked questions: 10
Made comments in text chat: 9
Changed slides: 4
Spoke as a panelist: 2

If we had clustered the talks and followed each cluster with a long question and answer session (questions through text chat), would you have used the chat format to ask questions and make comments about the content?

Yes/Probably: 10
I think that it would have been useful to have allotted more time for Q&A sessions. I think I may have asked questions, although I primarily joined the conference to listen and learn.
This might have proved difficult as I think people post simultaneously and responses to posts are presented out of context to the original question (personal feeling on all such chat-based group discussions). If there was a way to moderate or regulate postings, perhaps this would not be too chaotic.
An audio-based method may be better.
I would have used it, but perhaps it would be more difficult

TECHNOLOGY
Did you like the Marratech technology? What features did you like or dislike?
Positive Comments:

Great way to bring together fascinating people from all over the world, at the same time.
Extremely valuable method to organize workshops, project meetings for distributed participants.

Liked the concept of integrating ppt and chat and video used by Marratech, Breeze and others.
Nice to see the panelists on video while their slides were shown on the ‘whiteboard.’
I enjoyed the ability to ask questions.
I thought the underlying software was good.
Liked that I could control the power point slides if I wanted to (without taking control of the whole thing - I think!)
I liked the ability to mouse-over the slides, although I suspect these extra functionalities (not generally used) add to the bandwidth demands.

Negative Comments:
Audio: Audio had issues; audio was a huge distracter to the quality of the presentation; audio cut in and out… missed words; audio performance varied; lots of audio drop out. No, the performance was terrible. I was on a high speed data connection and the audio was terrible. The slides were running very slowly as well. I can understand the video being rough, but the audio quality was the worst I have heard in years.

Video: Quality of video was poor; video was not helpful running at still frame most of the time; occasional choppiness of sound and video; the transmission was garbled, picture was digital boxes much of the time.

Slides: The synchronization with the powerpoint slides was an issue; resolution was poor; slides were difficult to navigate at times; took a long time to download the slides. (Maybe people should be instructed to log in well in advance. More reliable would be to have all of the slides available on a standard web page as back up.)

Comments:

Bandwidth limitations were prominent from the user perspective.

You will never be able to completely eliminate problems on the client-side - my recommendation is to focus on the panelists and moderators.
What may be important to check is that the server can cope with the load which, in this case, may have been the problem. Other alternatives should also be evaluated.

Didn’t like the fact that you cannot move automatically with the speaker’s slides.
I didn’t like the fact that any participant could drive the slides (“lead” vs. “follow”); I think this is why some of the slides jumped around.
There were some bandwidth limitations and technical difficulties, but I felt those were to be expected. However, I wished there was an easier way to capture what was being shown on the whiteboard; I only found a way to save as mwb files, with which I am unfamiliar.

Did you test the technology ahead of the event?

Yes: 9
No: 3
I run into firewall problems at the University. UDP is blocked by default. The set up instructions should include information on which ports are used by the software.
I tested the technology the evening before the event. I accessed Marratech expecting to just see the SUMMIT sign and hear music, but someone from SUMMIT happened to be there during the test. I was able to hear audio fine after some adjustments during the test.
Yes, I downloaded it but could not test it. I had to download at home as our computers at work have firewalls that prevent anything from coming in. I had to be at work, during the test time.

Did you test with SUMMIT?

No: 8
Yes: 6
No, I wasn’t able to because I didn’t receive the pdf with instructions. You sent me again the event’s day, and I tried it an hour before.

If not, what prevented you from doing this test?

No time: 3
Received access info late
Time set up for testing were not convenient.
I was late to the testing time.
It probably would not have been a valid test anyway-unless you loaded 40-50 people on the server at the same time for load testing.

Where did you connect from? Home, Campus or Office?

Home: 4
Office: 11

What bandwidth did you have?

Campus network: 7
DSL and variants: 3
T1 and above: 3

GENERAL COMMENTS

Thank you. Your efforts are appreciated!
This was wonderful. Thank you so much for putting this together I hope to hear of other panels soon!
It was my pleasure to participate and this was the second such event (WebEx) I participated in within the last three weeks – none prior to that. Thanks to everyone and SUMMIT and to those not of SUMMIT that helped put this together. My answers to your questions are below. Two minor details when considering the travel savings. However, one of the benefits I receive when attending these events in person is developing the relationships that eventually lead to collaboration. We recently submitted a proposal partnering with someone I met at last October’s event at Stanford.
Thanks for taking the risk and leading the way. It will only get better. BTW, I used WebEx before - the audio was handled via conference call. No audio drop out, the moderator can mute or open the ‘mic’ of anyone on the call. Everything else looked very similar.
Good job for first performance!
Could I have a certificate of attendance? It is enough if you send me it by email (pdf file). Thanks, and congratulations again
Despite the technical difficulties I had, please put me on your mailing list for notification of future events.