Finetuning an LLM on a Celebrity’s Content + Voice Clone Integration

A client approached All-In Consulting to build an AI-generated 24/7 radio show of celebrities speaking on modern day topics. 

They had singled out one particular celebrity whose prolific library of past content gave us enough data to finetune an LLM on. Their popularity would also help this product spread among their existing audience.

The client was aiming to present this during investor meetings after. 

Solution 

We followed our 4 Phase AI Project Methodology to complete this project:

  • Discovery (45%) – where we scraped, cleansed, and prepared examples of the celebrity’s tone, and unique ways of speaking to finetune the LLM with. We also tested different prompts to generate the right type of structure for the show
  • Implementation (25%) – when we integrated the voice clones and built out a tool for the client to interact with the fine-tuned model
  • Enhancements (25%) – when we improved the accuracy, performance, and aesthetics of the tool
  • Go-Live (5%) – when we deployed the tool and handed it off 

When the client presented the initial pitch deck and sketch of the tool they needed, we suggested that we bring on one of our designers to improve the aesthetic appeal of both the presentation and the tool, which greatly enhanced the presentation.

We also ran double-blind testing with our client to verify that the fine-tuned model performed better than the vanilla model. We would show our client two pieces of AI-generated content from the tool, and have the client guess which output was from the finetuned model, and which was from the baseline model. 

We were able to see noticeable improvement in tone, style, and opinions from the finetuned model such that the client was able to detect which content was from the finetuned vs baseline model.

Impact

We finished the initial prototype within 1 month, and the entire project in 7 weeks. We’ve incorporated the pitch deck into the tool, and the client is now preparing for the investor meeting. We plan on continuing accuracy improvements of this tool post investor meeting.

The client also left us a 5-star review.

Key Learnings

  • When finetuning models, ensure that you are comparing results from the same baseline model. For example a finetuned ChatGPT-3.5 should be compared with the baseline ChatGPT-3.5. This is an easy mistake to make because more advanced models have been released, but finetuning is only supported on certain models. 
  • Designers play a key role in projects where aesthetics matter as they helped clean up the UI/UX and bring the client’s vision to life faster
  • Fine-tuning an LLM indeed leads to noticeable improvements, and we saw that performance improvements correlate with the amount of data we fine-tune the model with
  • When comparing fine-tuned results with vanilla model results, running double-blind testing to ensure customers can actually tell the difference was an effective way to measure accuracy improvements
  • The Discovery phase on AI projects is the most important phase of AI projects, and also the most time-consuming, as getting AI ready (ie deciding between API’s and technologies, preparing the data, testing different prompts) can be the majority of the work.

Download RAG System Design Guide

“Complete the form below to access our guide on how to design RAG chatbots and optimize it for performance and accuracy.

    See our 2024 Capabilities Brochure

      Download Our Success Measurement Guide

      Complete the form below to access our guide on how we measure AI success. Learn about the KPIs we track and how we ensure measurable returns on your AI investments, so you can maximize the impact of your AI projects.

        Download Our Methodologies Deck

        Fill out the form below to get our Methodologies Deck, where we walk you through our systematic approach to delivering AI projects. Learn how we align our solutions with your business objectives and ensure a seamless project execution process.

          Download Our Capabilities Deck

          Please fill out the form below to access our Capabilities Deck and discover how our AI solutions can drive transformation in your organization. Gain insights into our services, real-world case studies, and how we can help your business leverage the power of AI.