1
0
Fork 0

add CPU version of the deployment

This commit is contained in:
Massaki Archambault 2024-02-21 22:01:24 -05:00
parent 2f7813c591
commit 3bb1117361
2 changed files with 27 additions and 1 deletions

View File

@ -38,7 +38,16 @@ A quick prototype to self-host [LibreChat](https://github.com/danny-avila/LibreC
6. Browse http://localhost:3080/
7. Create an admin account and start chatting!
The API along with the APIDoc will be available at http://localhost:8000/
### Steps for NO GPU (use CPU)
**Warning: This may be very slow depending on your CPU and may us a lot of RAM depending on the model**
1. Make sure your drivers are up to date.
2. Clone the repo.
3. Copy the CPU compose spec to select it. `cp docker-compose.cpu.yml docker.compose.yml`
4. Run `docker compose up`. Wait for a few minutes for the model to be downloaded and served.
5. Browse http://localhost:3080/
6. Create an admin account and start chatting!
## Configuring additional models

17
docker-compose.cpu.yml Normal file
View File

@ -0,0 +1,17 @@
include:
- docker-compose.base.yml
services:
# Begin Ollama service
ollama:
image: ollama/ollama:0.1.23
restart: unless-stopped
entrypoint: /bootstrap.sh
command: mistral
env_file:
- .env
ports:
- 11434:11434
volumes:
- ./ollama/bootstrap.sh:/bootstrap.sh:ro
- ./ollama:/root/.ollama