You will be provided a dataset collected using MONROE testbed. MONROE is the first European open access platform for independent, multihomed, and large-scale mobile measurements on commercial providers. This dataset is composed of different fields including the timestamp of the measurement, website visited, protocol, operator and QoE metric evaluated such as FirstPaint(FP), SpeedIndex(SI) and PageLoadTime(PLT).
To evaluate the performance of different protocols, you will use a methodology that illustrates how two different metrics compare. The goal is to provide a sound comparison between, in this case, the two protocols under as much similar conditions as possible. Assume the performance of H2 and QUIC will be compared. Let M(i) be the i-th visit of metric M ∈ {FP, SI , PLT }. The protocol gain, ΔM(i), can be computed as:
ΔM ∈ [−1, 1] – with negative (positive) values when H2 performs better (worse) than QUIC on the i-th visit. Next, the distribution of the ΔM(i) over all experiments is computed, and results are quantized into 9 bins centered around zero. The rationale is to identify ranges for which performance is equivalent (center bin), slightly better/worse (the first bin on the left or right), moderately better, much better, extremely better.
In the delivery of this home exam, you will conduct the analysis of protocol comparison using the above methodology. More specifically, you will analyze whether there any significant differences between H2 and QUIC for different performance metrics.
Background
We are witnessing two major changes in the Internet. On the one hand, smartphones and mobile broadband (MBB) networks have revolutionized the way people consume web content. On the other hand, web protocols are undergoing a deep rethinking, with the advent of HTTP/2 and QUIC, that are designed to improve performance and overcome HTTP(S) and TCP bottlenecks. In such a dynamic environment, it is critical to understand the major factors that affect web performance in MBB, especially the impact of the protocols on the web performance.
There are different metrics to evaluate the web performance. A common metric used is the page load time (PLT), a metric primarily based on OnLoad event fired by the browser. Unfortunately, PLT does not tell much of the user experience. An alternative metric is First Paint, which tells how long it takes to see something on the screen. SpeedIndex captures the user perceived experience, however, it requires to film the page loading process. The RUMSpeedIndex method was then proposed to estimate the SpeedIndex using “resource timing” data. For the datasets that are provided part of this assignment, Page Load Time, First Paint and RUMSpeedIndex are considered as the performance metrics.
System-under-test
You will be provided two datasets that were collected using MONROE via the open source web performance tool. The starting point for the analysis is the www paper, and the dataset provided there, which was collected between April and May 2018. You are further provided with a new dataset that was recently collected in October 2019: 2019 dataset. Your task is to evaluate whether there have been any significant changes in time in terms of web performance (PLT, FP and SI) for different protocols (H2 and QUIC).
Task
First, you will use the 2018 dataset and evaluate the performance comparison of H2 and QUIC using 3 web metrics: PLT, FP and SI, using the above defined method.
Second, you will use the 2019 dataset and repeat the same evaluation above.
The evaluation should contain not only the protocol comparison for the datasets given but also the evolution of the protocols in time. In other words, whether there are significant changes in terms of the comparison of the protocols in time.
Note: The evaluation should only be carried out for stationary nodes (node-type=FALSE/stationary) and for MBB connections (no eth).
Bonus: You can consider different websites and carry out the evaluation for a specific website to see whether there are any changes on the website level in time.