Knowledge Graph-Based Credibility Evaluation Method for Electric Grid Large Language Model Knowledge Question-Answering

In the field of electricity, specialized terminology is often intricate and complex, making it challenging for non-experts to comprehend. However, with the advancement of artificial intelligence technology, the emergence of large language models provides a new technological solution to address this issue. Large language models, based on deep learning techniques, have the capability to quickly understand and interpret specialized terminology in the electricity domain through learning from a vast corpus of professional literature and data. They can then be applied to various domains, including question-answering systems. However, existing large language models still face issues of unreliable outputs, necessitating a method to evaluate their results and improve the quality of their applications. We propose a knowledge graph-based credibility evaluation method for electric grid large language model knowledge question-answering. This method aligns the answers generated by large language models with the knowledge graph of a local knowledge base and calculates their cosine similarity and Pearson correlation coefficient. We batch-process the answers from the large language model into an electricity dataset and validate them using this method. Experimental results demonstrate that this method can accurately and efficiently reflect the relevance between texts, providing a reliable scoring basis for question-answering by large models in vertical domains. Future research can focus on exploring other embedding methods that can better extract semantic relationships between texts and validating the feasibility of this method in vertical domains other than electricity.