Text this: Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs