NetLLMBench: A Benchmark Framework for Large Language Models in Network Configuration Tasks

NetLLMBench: A Benchmark Framework for Large Language Models in Network Configuration Tasks

연구 분야: Networking

학회: 2024 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN)

Traditional network management techniques often struggle with the scale and dynamism of modern networks, requiring significant human oversight and being prone to high error rates. Large Language Models (LLMs) present a promising alternative to conventional approaches by automating network configuration and management. However, a systematic way to evaluate their performance is lacking in the literature.This paper introduces NetLLMBench, a novel framework designed to rigorously assess the performance of LLMs in managing computer networks. By integrating prompt engineering and network emulation in a closed loop, NetLLMBench benchmarks and validates LLMs’ responses in various configuration scenarios. The findings establish foundational benchmarks to guide future applications of LLMs in enhancing network management efficiency.