Privacy-Preserving Federated Learning through Clustered Sampling on Fine-Tuning Distributed non-iid Large Language Models

Recently, Large Language Models (LLMs) have been a phenomenal trend in the Artificial intelligence field. However, training and fine-tuning can be challenging because of privacy concerns and limited computing resources. Federated Learning (FL) has emerged as a novel machine learning framework offering privacy protection. The challenges in applying FL to real-world applications include dealing with heterogeneous data, poor client updates, and client selection. This paper introduces Privacy-preserving Federated Learning through Clustered Sampling on LLMs (FCLM), a framework that clusters models by their distribution similarity. It helps the model group similar models to improve text data heterogeneity handling and privacy concerns in distributed machine-learning environments. The FCLM framework is implemented and evaluated using popular Language models and text data. The framework shows a robust performance over the heterogeneous text data, which can further extend to the use of more complex LLMs.