Vicuna is an open source LLaMA model fine-tuned on ~70k ShareGPT conversations with performance that rivals ChatGPT. We show that it can be easily run in LangChain at least two ways: (1) you can use the existing bindings for Llama-cpp or (2) you can use an API, such as Replicate. To benchmark performance of Vicuna relative to other popular models, we integrate it
We integrate Vicuna into auto-evaluator and benchmark its performance on two test sets relative to OpenAI GPT3.5-turbo and GPT4. We show that its performance in comparable to GPT-3.5 / 4 on one challenge, but lags on the larger test set (particularly in cases where the limited context length of 2048 constrains Vicuna performance). There are several opportunities to improve the developer experience with Vicuna, such as model hosting and fine-tuning the answer style.
Vicuna is an open source model that reports ~90% the quality of OpenAI ChatGPT and Google Bard, which is achieved through fine-tuning on 70K user-shared ChatGPT conversations. There is high interest in this model (here) due to the fact that it is open-source and can (impressively) run on consumer laptop CPUs thanks to the excellent work here on llama.cpp. Below, we show how Vicuna can be run within LangChain easily using the existing llama.cpp integration and we show that auto-evaluator, a framework for evaluating question+answering (QA) chains, can be used to benchmark Vicuna performance on any user-specified QA test set.
It is possible to run LLaMA series models on consumer laptop CPUs thanks to llama.cpp:
git clone [<https://github.com/ggerganov/llama.cpp>](<https://github.com/ggerganov/llama.cpp>)
make
python3 -m pip install -r requirements.txt
The Vicuna weights can be download for 7B and 13B model variants from HuggingFace. Simply put the model binary files in the /models subdirectory in your llama.cpp project:
models/vicuna_13B/ggml-vicuna-13b-4bit.bin
You can test the model(s) locally using the commands below:
./main -m ./models/vicuna_7B/ggml-vicuna-7b-q4_0.bin -n 256 --repeat_penalty 1.0 --mlock --color -i -t xxx -r "User:" -f prompts/chat-with-bob.txt
Here is a helpful thread on tuning latency. In short, these flags are important to consider:
-mlock force system to keep model in RAM and never resort to a swap file-t the number of CPU threads to use