Text this: Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation on Large Language Models