Session Date/Time: 20 Mar 2024 05:00

rasprg

Summary

The RASP Research Group meeting featured presentations on various topics, including the use of Large Language Models (LLMs) for RFC analysis, tracking internet standards proposals, computationally understanding the IETF consensus process, and the time it takes to produce an RFC. The discussions were rich with technical details and suggestions for future research.

Key Discussion Points

RFCGPT (Nick):
- Presented RFCGPT, a tool using LLMs to interact with and summarize RFCs.
- Discussed the challenges of data preparation, including character encoding issues and file size limitations.
- Future directions include building an external Retrieval Augmented Generation (RAG), expanding the training data to include mailing list archives, and using open-source models.
- Concerns were raised about the accuracy and potential misuse of the tool, as well as the need for a quality measurement system.
Internet Standards Tracker (Carolina):
- Presented the Internet Standards Tracker, a tool for identifying and tracking standards proposals that could potentially challenge interoperability.
- Expanded the tracker's coverage to include IETF drafts.
- Described the methodology used to generate a score based on keywords in standards proposals.
- Feedback from Prague IETF meeting led to improvements like open dictionaries and search terms, a new section explaining the methodology and added the status of the proposal.
- Future steps will improve the search functionality and incorporate AI-powered semantic analysis.
- Address a bug that made the IETF tracker section appear blank
Computational Understanding of Consensus (Priyanka):
- Presented research on using computational methods to understand the IETF consensus process.
- Used tools like IETF data, bird topics, and empaths to analyze mailing list data.
- Showcased temporal topic activity and psycho-linguistic aspects of working group discussions.
- Future work might extend the model to capture specific topics in greater detail.
Making RFCs Nowadays (Jeff):
- Presented data on the length of time it takes to produce an RFC, the success rate of drafts, and the publication rate of RFCs.
- Observed that the average time to cook a draft is now 2 years 4 months, and the success rate is 21%.
- Discussed factors contributing to the increasing complexity and delays in the RFC process, including the need to maintain backward compatibility.
- Raised questions about the relevance of RFCs by the time they are published.
- Raised concerns that the massive amount of review process doesn't improve quality, but results in massive IRATA.
- Susan indicated the presentation needed to take into account ISG process.

Decisions and Action Items

Carolina (Internet Standards Tracker):
- Investigate and fix the issue where the IETF tracker appears blank.
- Consider making the Internet Standards Tracker's data available through an API.
- Follow up on the possibility of publishing an RFC with the dictionaries of search terms used.
Jeff (Making RFCs Nowadays):
- Consider feedback about the presentation needed to take into account ISG process.

Next Steps

Nick (RFCGPT): Continue development of RFCGPT, focusing on the future directions outlined in the presentation.
Carolina (Internet Standards Tracker): Implement the next steps outlined in the presentation, including improving the search functionality and testing AI-powered semantic analysis.
Priyanka (Computational Understanding of Consensus): Continue research on computationally understanding the consensus process, focusing on refining the models and gathering more meaningful results.