Session Date/Time: 22 Mar 2022 13:30

irtfopen

Summary

The IRTF Open meeting provided an overview of IRTF activities and featured two Applied Networking Research Prize (ANRP) talks. Colin Perkins, IRTF Chair, opened the session with standard IETF/IRTF policy reminders. The bulk of the meeting consisted of two highly technical presentations: Sangeeta Abdu Jyoti discussed the vulnerability of internet infrastructure to solar superstorms, and Bruce Spang presented on the biases inherent in A/B testing of networking algorithms in congested networks. Both talks highlighted critical challenges and called for re-evaluation of current practices in internet resilience and experimentation.

Key Discussion Points

IRTF Overview and Updates:
- The IRTF focuses on longer-term research issues related to the Internet, operating in parallel with the IETF's engineering and standards-making. It serves as a discussion forum for academic and industry researchers to interact with the IETF community.
- Updates were provided on active Research Groups (RGs), including Crypto Forum, Path Aware Networking, Information-Centric Networking, Measurement and Analysis of Protocols, Network Management, and Human Rights.
- Sophia Jelly was welcomed as the new co-chair of the Human Rights Protocol Considerations RG.
Applied Networking Research Prize (ANRP):
- The ANRP, awarded in conjunction with the Internet Society and sponsored by Comcast and NBCUniversal, recognizes impactful applied networking research.
- Talk 1: "Storms Planning for an Internet Apocalypse" by Sangeeta Abdu Jyoti (UC Irvine, VMware Research):
  - Threat: Coronal Mass Ejections (CMEs) from the sun induce Geomagnetically Induced Currents (GICs) in long cables with ground connections.
  - Vulnerability: Higher latitudes (above 40 degrees N/S) are at significantly higher risk. Historical events like the Carrington Event (1859) caused widespread telegraph damage; a similar event narrowly missed Earth in 2012. Solar activity is entering a more active phase (Solar Cycle 25).
  - Internet Components Affected: Long-distance cables (land and submarine) are most vulnerable due to electrically powered repeaters. Satellites are also vulnerable to radiation and orbital drag. Localized infrastructure (data centers, routers with suppressors) and optical fiber itself are generally resilient.
  - Analysis: Internet infrastructure, particularly submarine cable endpoints, routers, IXPs, and data centers, is disproportionately concentrated in vulnerable high-latitude regions. Submarine cables are more susceptible than land cables due to a higher percentage requiring repeaters.
  - Open Challenges:
    - Understanding the end-to-end impact on applications and the resiliency of top-level domain (TLD) and authoritative DNS servers.
    - Developing solutions like new cable routes in less vulnerable regions and inter-domain protocols for improved path diversity.
    - Utilizing the 13-hour to 5-day lead time from CME detection for pre-impact preparation (e.g., caching strategies, DNS cache time adjustments).
    - Rethinking failure analysis models to include large-scale, cascading failures.
    - Exploring temporary connectivity solutions post-impact (e.g., high-altitude platforms, balloons) and studying the interdependent failure characteristics of the Internet and power grids.
  - Discussion: Questions arose about the specific impact on DNS islands (root servers are distributed, TLD servers less known), satellite resilience in strong storms, and how IETF/IRTF protocol design might incorporate these resilience considerations (e.g., BGP path diversity, DNS caching strategy).
- Talk 2: "Unbiased Experiments in Congested Networks" by Bruce Spang (Stanford, Netflix Research Fellow):
  - Problem: A/B testing, a common methodology for evaluating networking algorithms, can be significantly biased in congested networks due to "interference" between treatment and control groups sharing network resources (links, queues).
  - Experiment (Netflix Bitrate Capping): During COVID-19, Netflix capped video bitrates to reduce internet load. A simulated A/B test (comparing 95% capped traffic to 5% capped traffic on separate but similar links) showed that A/B tests can not only misrepresent the magnitude of improvement but also the direction of the effect.
  - Results: While A/B tests showed a ~5% decrease in throughput for capped traffic, the "total treatment effect" (representing widespread deployment) actually showed a ~12% improvement in throughput by alleviating congestion. Similar misleading results were observed for Round Trip Time and Play Delay.
  - Risks: Biased A/B test results can lead to poor development decisions, such as abandoning good ideas or deploying ineffective solutions.
  - Mitigation Strategies:
    - Event Studies: Compare metrics before and after a significant deployment, though susceptible to seasonality.
    - Switchback Experiments: Alternate between treatment and control over time to reduce seasonality bias, though carryover effects need consideration.
  - Conclusion: There is a critical need for better experiment methodology in networking, leveraging network-specific knowledge to design more sophisticated experiments that accurately measure total treatment effects.
  - Discussion: Participants acknowledged the crucial insight, with questions on the impact of bottleneck location and user stickiness in A/B tests. The idea of using network knowledge to design "multiple access" metaphors for experimentation was raised. Guidance for IETF/IRTF was requested, emphasizing that designers should consider potential biases in validation experiments.
Applied Networking Research Workshop (ANRW): The 2022 ANRW will co-locate with IETF-114 in July (Philadelphia, hybrid). The Call for Papers is open, with submissions due at the end of April.
Travel Grant Program: Netflix and Comcast are sponsoring travel grants for early career academics and PhD students from underrepresented groups to attend IRTF meetings and the ANRW. Details for future grants will be posted online.

Decisions and Action Items

For Researchers and Protocol Designers:
- ACTION: When designing new algorithms and evaluating protocols, consider the potential for A/B test bias in congested networks.
- ACTION: Explore and adopt more robust experiment methodologies (e.g., event studies, switchback experiments, or network-aware designs) to accurately measure "total treatment effects" and improve confidence in evaluation results.
- ACTION: Consider incorporating resilience to large-scale, non-standard failure scenarios (like solar superstorms) into protocol design and infrastructure planning. This includes rethinking assumptions about global connectivity and exploring adaptive strategies.

Next Steps

ANRW: Consider submitting research results to the Applied Networking Research Workshop, with papers due at the end of April for the July IETF-114 co-located event.
Travel Grants: Keep an eye on IRTF announcements for future travel grant opportunities for IETF-114 and ANRW.
Further Research: The open questions and challenges identified in the ANRP talks present significant avenues for future networking research within the IRTF community and beyond.