mirror of https://github.com/microsoft/autogen.git
315 lines
17 KiB
Plaintext
315 lines
17 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ec5c71a8",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Tutorial: Utilizing Open Source Language Models with AutoGen\n",
|
|
"\n",
|
|
"AutoGen is compatible with the OpenAI API library in Python for executing language models. Consequently, it can work with any models employing a similar API without the necessity to modify your AutoGen code.\n",
|
|
"\n",
|
|
"\n",
|
|
"In this guide, we will utilize the [modelz-llm](https://github.com/tensorchord/modelz-llm) package to illustrate how to locally serve a model and integrate AutoGen with the served model.\n",
|
|
"Actually there are multiple ways to serve the local model in Open AI API compatible way. At this point in time [modelz-llm](https://github.com/tensorchord/modelz-llm) is not usable on Unix.\n",
|
|
"For Windows, [LM Studio](https://lmstudio.ai/) works. It allows downloading, checking and serving the local Open Source models. More details of how to use are [here](https://medium.com/p/97cba96b0f75).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a18e29d5",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Instructions\n",
|
|
"\n",
|
|
"1. First, install the modelz-llm and related packages using the command: ```modelz-llm huggingface_hub transformers accelerate```\n",
|
|
"2. Next, initiate the serving of the model; for instance, executing ```modelz-llm -m bigscience/bloomz-560m --device auto``` will locally serve the bloomz-560m small model.\n",
|
|
"3. Lastly, you will need to modify the `OAI_CONFIG_LIST` file or the `config_list` variable to point to the local model. In this scenario, we are using the default port 8000.\n",
|
|
"\n",
|
|
"\n",
|
|
"Here, we utilize the Bloomz-560M model, which can run on a CPU for demonstration purposes. It's anticipated that the quality of responses will be low."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "92ca0c4a",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Collecting pyautogen\n",
|
|
" Downloading pyautogen-0.1.6-py3-none-any.whl (69 kB)\n",
|
|
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m69.3/69.3 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
|
|
"\u001b[?25hRequirement already satisfied: modelz-llm in /home/beibinli/anaconda3/lib/python3.9/site-packages (23.7.4)\n",
|
|
"Requirement already satisfied: huggingface_hub in /home/beibinli/anaconda3/lib/python3.9/site-packages (0.17.2)\n",
|
|
"Requirement already satisfied: transformers in /home/beibinli/anaconda3/lib/python3.9/site-packages (4.33.2)\n",
|
|
"Requirement already satisfied: accelerate in /home/beibinli/anaconda3/lib/python3.9/site-packages (0.21.0)\n",
|
|
"Requirement already satisfied: diskcache in /home/beibinli/anaconda3/lib/python3.9/site-packages (from pyautogen) (5.6.1)\n",
|
|
"Requirement already satisfied: flaml in /home/beibinli/anaconda3/lib/python3.9/site-packages (from pyautogen) (2.0.2)\n",
|
|
"Requirement already satisfied: openai in /home/beibinli/anaconda3/lib/python3.9/site-packages (from pyautogen) (0.27.8)\n",
|
|
"Requirement already satisfied: termcolor in /home/beibinli/anaconda3/lib/python3.9/site-packages (from pyautogen) (2.3.0)\n",
|
|
"Requirement already satisfied: llmspec in /home/beibinli/anaconda3/lib/python3.9/site-packages (from modelz-llm) (0.3.16)\n",
|
|
"Requirement already satisfied: falcon>=3 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from modelz-llm) (3.1.1)\n",
|
|
"Requirement already satisfied: uvicorn>=0.22 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from modelz-llm) (0.23.2)\n",
|
|
"Requirement already satisfied: mosec>0.6 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from modelz-llm) (0.8.0)\n",
|
|
"Requirement already satisfied: filelock in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (3.9.0)\n",
|
|
"Requirement already satisfied: fsspec in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (2023.4.0)\n",
|
|
"Requirement already satisfied: requests in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (2.28.2)\n",
|
|
"Requirement already satisfied: tqdm>=4.42.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (4.65.0)\n",
|
|
"Requirement already satisfied: pyyaml>=5.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (6.0)\n",
|
|
"Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (4.4.0)\n",
|
|
"Requirement already satisfied: packaging>=20.9 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from huggingface_hub) (21.3)\n",
|
|
"Requirement already satisfied: numpy>=1.17 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from transformers) (1.24.1)\n",
|
|
"Requirement already satisfied: regex!=2019.12.17 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from transformers) (2022.7.9)\n",
|
|
"Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from transformers) (0.13.3)\n",
|
|
"Requirement already satisfied: safetensors>=0.3.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from transformers) (0.3.1)\n",
|
|
"Requirement already satisfied: psutil in /home/beibinli/anaconda3/lib/python3.9/site-packages (from accelerate) (5.9.0)\n",
|
|
"Requirement already satisfied: torch>=1.10.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from accelerate) (2.1.0)\n",
|
|
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from packaging>=20.9->huggingface_hub) (3.0.9)\n",
|
|
"Requirement already satisfied: sympy in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (1.11.1)\n",
|
|
"Requirement already satisfied: networkx in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (3.0rc1)\n",
|
|
"Requirement already satisfied: jinja2 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (3.1.2)\n",
|
|
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
|
|
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
|
|
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
|
|
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (8.9.2.26)\n",
|
|
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (12.1.3.1)\n",
|
|
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (11.0.2.54)\n",
|
|
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (10.3.2.106)\n",
|
|
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (11.4.5.107)\n",
|
|
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (12.1.0.106)\n",
|
|
"Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (2.18.1)\n",
|
|
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (12.1.105)\n",
|
|
"Requirement already satisfied: triton==2.1.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from torch>=1.10.0->accelerate) (2.1.0)\n",
|
|
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate) (12.2.140)\n",
|
|
"Requirement already satisfied: click>=7.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from uvicorn>=0.22->modelz-llm) (8.0.4)\n",
|
|
"Requirement already satisfied: h11>=0.8 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from uvicorn>=0.22->modelz-llm) (0.14.0)\n",
|
|
"Requirement already satisfied: msgspec>=0.15.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from llmspec->modelz-llm) (0.18.3)\n",
|
|
"Requirement already satisfied: aiohttp in /home/beibinli/anaconda3/lib/python3.9/site-packages (from openai->pyautogen) (3.8.3)\n",
|
|
"Requirement already satisfied: charset-normalizer<4,>=2 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from requests->huggingface_hub) (2.0.4)\n",
|
|
"Requirement already satisfied: idna<4,>=2.5 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from requests->huggingface_hub) (3.4)\n",
|
|
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from requests->huggingface_hub) (1.26.16)\n",
|
|
"Requirement already satisfied: certifi>=2017.4.17 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from requests->huggingface_hub) (2023.5.7)\n",
|
|
"Requirement already satisfied: attrs>=17.3.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from aiohttp->openai->pyautogen) (19.3.0)\n",
|
|
"Requirement already satisfied: multidict<7.0,>=4.5 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from aiohttp->openai->pyautogen) (6.0.2)\n",
|
|
"Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from aiohttp->openai->pyautogen) (4.0.2)\n",
|
|
"Requirement already satisfied: yarl<2.0,>=1.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from aiohttp->openai->pyautogen) (1.8.1)\n",
|
|
"Requirement already satisfied: frozenlist>=1.1.1 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from aiohttp->openai->pyautogen) (1.3.3)\n",
|
|
"Requirement already satisfied: aiosignal>=1.1.2 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from aiohttp->openai->pyautogen) (1.2.0)\n",
|
|
"Requirement already satisfied: MarkupSafe>=2.0 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.2)\n",
|
|
"Requirement already satisfied: mpmath>=0.19 in /home/beibinli/anaconda3/lib/python3.9/site-packages (from sympy->torch>=1.10.0->accelerate) (1.2.1)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Installing collected packages: pyautogen\n",
|
|
"Successfully installed pyautogen-0.1.6\n",
|
|
"Note: you may need to restart the kernel to use updated packages.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%pip install pyautogen modelz-llm huggingface_hub transformers accelerate"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7b6362a8",
|
|
"metadata": {},
|
|
"source": [
|
|
"## The AutoGen Code\n",
|
|
"\n",
|
|
"Below, we demonstrate how to configure the `config_list` variable, and subsequently invoke the Completion method.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "e3048aa9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setup autogen with the correct API\n",
|
|
"import autogen\n",
|
|
"\n",
|
|
"autogen.oai.ChatCompletion.start_logging()\n",
|
|
"\n",
|
|
"config_list = [\n",
|
|
" {\n",
|
|
" 'model': 'bloomz-560m-my-small-model',\n",
|
|
" 'api_key': 'any string here is fine',\n",
|
|
" 'api_type': 'openai',\n",
|
|
" 'api_base': \"http://localhost:8000\",\n",
|
|
" 'api_version': '2023-03-15-preview'\n",
|
|
" }\n",
|
|
"]\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "ea2bf8d7",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The small model's answer is: a human being\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Perform Completion\n",
|
|
"question = \"Who are you?\"\n",
|
|
"response = autogen.oai.Completion.create(config_list=config_list, prompt=question, temperature=0)\n",
|
|
"ans = autogen.oai.Completion.extract_text(response)[0]\n",
|
|
"\n",
|
|
"print(\"The small model's answer is:\", ans)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "09a498e9",
|
|
"metadata": {},
|
|
"source": [
|
|
"## A More Sophisticated Example Featuring Multiple Agents\n",
|
|
"\n",
|
|
"It's noteworthy that various LLMs can be employed for different agents. In this example, we showcase two agents; one utilizes the bloomz model while the other employs GPT models (as derived from the `OAI_CONFIG_LIST`. It's presumed that you have an environment variable or a local file named \"OAI_CONFIG_LIST\" set up)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "a82d5cb9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from autogen import AssistantAgent, UserProxyAgent\n",
|
|
"\n",
|
|
"local_config_list = [\n",
|
|
" {\n",
|
|
" 'model': 'bloomz-560m-my-small-model',\n",
|
|
" 'api_key': 'any string here is fine',\n",
|
|
" 'api_type': 'openai',\n",
|
|
" 'api_base': \"http://localhost:8000\",\n",
|
|
" 'api_version': '2023-03-15-preview'\n",
|
|
" }\n",
|
|
"]\n",
|
|
"\n",
|
|
"\n",
|
|
"gpt_config_list = autogen.config_list_from_json(\n",
|
|
" \"OAI_CONFIG_LIST\",\n",
|
|
" filter_dict={\n",
|
|
" \"model\": {\n",
|
|
" \"gpt-4\",\n",
|
|
" \"gpt4\",\n",
|
|
" \"gpt-4-32k\",\n",
|
|
" \"gpt-4-32k-0314\",\n",
|
|
" \"gpt-4-32k-v0314\",\n",
|
|
" \"gpt-3.5-turbo\",\n",
|
|
" \"gpt-3.5-turbo-16k\",\n",
|
|
" \"gpt-3.5-turbo-0301\",\n",
|
|
" \"chatgpt-35-turbo-0301\",\n",
|
|
" \"gpt-35-turbo-v0301\",\n",
|
|
" \"gpt\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "a159dd60",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"small = AssistantAgent(name=\"small model\",\n",
|
|
" max_consecutive_auto_reply=2,\n",
|
|
" system_message=\"You should act as a student!\",\n",
|
|
" llm_config={\n",
|
|
" \"config_list\": local_config_list,\n",
|
|
" \"temperature\": 1,\n",
|
|
" })\n",
|
|
"\n",
|
|
"big = AssistantAgent(name=\"big model\",\n",
|
|
" max_consecutive_auto_reply=2,\n",
|
|
" system_message=\"Act as a teacher.\",\n",
|
|
" llm_config={\n",
|
|
" \"config_list\": gpt_config_list,\n",
|
|
" \"temperature\": 1,\n",
|
|
" })\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "78ce6584",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[33mbig model\u001b[0m (to small model):\n",
|
|
"\n",
|
|
"Who are you?\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[33msmall model\u001b[0m (to big model):\n",
|
|
"\n",
|
|
" Who are you?\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[33mbig model\u001b[0m (to small model):\n",
|
|
"\n",
|
|
"I am an AI language model, programmed to provide information, answer questions, and assist with various tasks. As a teacher, I can help explain various topics and concepts. What would you like to learn about today?\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[33msmall model\u001b[0m (to big model):\n",
|
|
"\n",
|
|
" What is the topic of the day? User\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"\u001b[33mbig model\u001b[0m (to small model):\n",
|
|
"\n",
|
|
"As an AI teacher, I don't have a predetermined topic of the day. However, I'm more than happy to discuss or teach any topic you'd like to learn about or explore. Please let me know your area of interest, and we can dive into it right away.\n",
|
|
"\n",
|
|
"--------------------------------------------------------------------------------\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"big.initiate_chat(small, message=\"Who are you?\")\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.16"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|