Simplify your LLM Chatbot Demos with SAP BTP and Gradio

NOTE: The views and opinions expressed in this blog are my own

In this blog I will demonstrate how you can quickly build a custom LLM Chatbot Demo with Gradio in under 100 lines of code, running on SAP BTP. Gradio’s agility makes it a quick solution to present your machine learning models and share with colleagues.

Gradio is commonly used to demonstrate open source models on Hugging Face.

For an example see Meta’s recent opensource Llama2 chatbot.

If you get to the end of this blog you will be able to build the following LLM Chatbot demo on BTP, using OpenAI API’s:

Click to Enlarge

The pre-requistes are:

• BTP Account..  (even a free trial account works)
• Cloud foundry (cf) cli installed
• OpenAI API key
• Basic Python knowledge

The build includes the steps to make the following:

NOTE: in this example I’ve focussed on deploying on the BTP CF runtime environment, but with some minor tweaks to the steps the Kyma (K8S) runtime could also be used.

openai_service

Create the following:

#Command line (Mac flavour)

cf cups openai_service -p '{"username":"api_key","password":"<your OpenAI API Key goes here>"}'

In BTP you should see something like the following.

Click to Enlarge

manifest.yaml

In your local machine create a file with the following:

---
applications:
- name: llm-chatbot
  random-route: true
  domain: cfapps.ap21.hana.ondemand.com
  path: ./
  memory: 128M
  buildpack: python_buildpack
  command: python server.py
services:
 - openai_service

requirements.txt

In your local machine create a file with the following:

gradio==3.38.0
cfenv
openai

server.py

In your local machine create a file with the following:

import os, cfenv
import gradio as gr 
import openai

port = int(os.environ.get('PORT', 7860))
print("PORT:", port)

# Create a Cloud Foundry environment object
env = cfenv.AppEnv()

# Check if the app is running in Cloud Foundry
if env.app:
    try:
        # Get the specified user-provided service by name
        service_name = "openai_service"
        service = env.get_service(name=service_name)
        
        # Access the service credentials
        credentials = service.credentials
        
        # Access the "openai_api_key" 
        openai_api_key = credentials['password']
        os.environ['OPENAI_API_KEY'] = openai_api_key

        openai.api_key = openai_api_key

        print("OpenAI API Key assigned")
    except cfenv.AppEnvError:
        print(f"The service '{service_name}' is not found.")
else:
    print("The app is not running in Cloud Foundry.")


prompt = " "

#Call the LLM Api's with a prompt/question and return a response
def generate_response(llm, prompt):
   if llm == 'OpenAI':
      completion = openai.Completion.create(
            model = "text-davinci-003",
            prompt = prompt,
            temperature = 0,
            max_tokens= 500, 
            top_p=1,
            frequency_penalty=0, 
            presence_penalty=0, 
            stop=[" Human:", " AI:"]
        ) 
      return completion.choices[0].text
   else:
      return "Unknown LLM, please choose another and retry."

#Handle the Chatbox call back
def llm_chatbot_function(llm, input, history):
    history = history or []
    my_history = list(sum(history, ()))
    my_history.append(input)
    my_input = ' '.join(my_history)
    output = generate_response(llm, my_input)
    history.append((input, output))
    return history, history 

#Define the ChatBot using Gradio Elements
def create_llm_chatbot():
    with gr.Blocks(analytics_enabled=False) as interface:
        with gr.Column():
            top = gr.Markdown("""<h1><center>LLM Chatbot</center></h1>""")
            llm = gr.Dropdown(
                    ["OpenAI", "Llama-2-7b-chat-hf"], label="LLM Choice", info="Which LLM should be used?" , value="OpenAI"
                )
        
            chatbot = gr.Chatbot()
            state = gr.State()
            question = gr.Textbox(show_label=False, placeholder="Ask me a question and press enter.") #.style(container=False)
            with gr.Row():
              summbit_btn = gr.Button("Submit")
              clear = gr.ClearButton([question, chatbot, state ])

        question.submit(llm_chatbot_function, inputs=[llm, question, state], outputs=[chatbot, state])
        summbit_btn.click(llm_chatbot_function, inputs=[llm, question, state], outputs=[chatbot, state])

    return interface

llm_chatbot = create_llm_chatbot()
    
if __name__ == '__main__': 
    llm_chatbot.launch(server_name='0.0.0.0',server_port=port)   

Deploy the App to SAP BTP

With the above service and files created your are ready to deploy the app.

From the same directory you created the files, run the following:

cf push

In BTP you should now see your app running:

Click to Enlarge

You can find  the url (application route) to your app when you navigate further on the llm-chatbot:

Click to Enlarge

Where to Next?

As you can see it only takes a few simple steps to deploy a machine learning demo app onto the BTP platform. Perfect for iterative testing and development.

So where do you you think it make sense to explore this further?

Ideas I have are:

• Add additional Open Source LLM’s
• Add Human Feedback to the Chatbot, to be used with fine tuning of open source models
• Add user level security (application routing)
• Integrating with Corporate Data

I welcome your additional ideas in the comments below.

SAP notes that posts about potential uses of generative AI and large language models are merely the individual poster’s ideas and opinions, and do not represent SAP’s official position or future development roadmap. SAP has no legal obligation or other commitment to pursue any course of business, or develop or release any functionality, mentioned in any post or related content on this website.