User study on 360 videos
The second user study is concerned with the evaluation of user studies using categorical data, more specifically the use analysis of categorical information that has been collected using the Pair Comparison method. To overcome the problem of collecting data for a user study, everybody must collect some data and share it with the rest of the course.
We are using a 360 video that we downloaded from YouTube, showing a drive through Sao Paolo. We have started with a 2K video, extracted 1 minute and created several resolutions. We have created versions of the 360 video that are rotated by a 90, 180 and 270 degrees starting angle.
Your preparation task for the assignment is that each of you finds at least 5 participants who are willing to watch all combinations of the videos in a pair comparison study, and rate which video is the better in each pair. There are 6 different videos, leading to a total 15 combinations. This means that each participant must watch 15 pairs of videos. You must show these to you viewers in random order to ensure that exhaustion affects of pair similarly.
These comparisons should be made on a Desktop computer using vlc, which can reliably present these 360 videos.
This preparation task should be completed by Tuesday, October 12 (midnight). Send the results as a CSV to me, so that I can merge the CSVs from everybody and share them with all of you on Wednesday through he course web page. The file will enable everybody to conduct a statistical analysis of the data.
As usual, we use Mattermost to discuss problems and solution.
It is desirable that we collect a relevant amount of information that can expose statistical differences between our assessors: ideal are users who assessed the quality on a mobile phone versus a computer screen, male/female users, children/grown-ups, and many other options.
From the structure of the logfile, it is obvious that it is impossible to conduct an ANOVA study of the data directly based on the binary ratings collected from the pair comparisons. Every line of the logfile samples a user's decision between two video qualities. This data has to be transformed to ensure that every alternative (column) of the ANOVA evaluation refers to a sample (or group of sample) that you want to compare with all the other samples (or groups of samples). It is important to transform the data in such a way that the statistical meaning of the pair comparison is retained.
The task for the first round is to create a presentation (about 10-20 slides) that explains your user study. The presentation should explain the setup and the dataset that has been collected including some basic statistics about the data that has been collected. You should explain the question(s) that you intend to answer with your analysis of the data, the transformation of the logged data that you have performed (including an explanation of what that means), and the results of your ANOVA study/studies. Finally, you should interpret the results.
The presentation is supposed to be given to the class in October 19. It is expected that you listen to and comment on the presentations given by others.
This work is meant to be performed in teams of 2 students.
There is an opportunity to improve the presentation slides after the presentation. The deadline for the improved version is November 2, using Devilry. Please make sure that the presentation cover page contains names of all group members.
FAQ for the home exam
Links to the videos
- high quality, backward facing: video
- medium quality, backward facing: video
- low quality, backward facing: video
- high quality, sideways facing: video
- medium quality, sideways facing: video
- low quality, sideways facing: video
Data file
The data collection: log.csv