llama.cpp icon
llama.cpp icon

llama.cpp

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

Open and start using the WebUI in your browser

Cost / License

  • Free
  • Open Source (MIT)

Application type

Platforms

  • Windows
  • Mac
  • Linux
  • Docker
  • Homebrew  brew install llama.cpp
  • Nix Package Manager  nix profile install nixpkgs#llama-cpp
  • MacPorts  sudo port install llama.cpp
  • Self-Hosted
Add multiple text files from disk or from the clipboard to the context of your conversation
Attach one or multiple PDFs to your conversation. By default, the contents of the PDFs will be converted to RAW text, excluding any visuals.
+10
Optionally, the WebUI can process the PDFs as images when the AI model supports it.
llama.cpp alternatives  
8likes
36alternatives
0articles

Features

Properties

  1.  Lightweight
  2.  Privacy focused
  3.  AI-Powered
  4.  Minimalistic

Features

  1.  Works Offline
  2.  Hardware Accelerated
  3.  No Tracking
  4.  No registration required
  5.  GPU Acceleration
  6.  AI Chatbot
  7.  Local AI
  8.  Support for NVIDIA CUDA acceleration
  9.  Apple Metal support

 Tags

llama.cpp News & Activities

Highlights All activities

Recent activities

llama.cpp information

AlternativeTo Categories

AI Tools & Services, System & Hardware

GitHub repository

  •  128,124 Stars
  •  23,132 Forks
  •  2470 Open Issues
  •   Updated  
View on GitHub

Popular alternatives

View all

Our users have written 1 comments and reviews about llama.cpp, and it has gotten 8 likes

llama.cpp was added to AlternativeTo by bugmenot on and this page was last updated .

Comments and Reviews

   
David
0

I like llama.cpp because it is the first and best open source application for LLM. It is not only the core engine of GPT4ALL, LM Studio, Jan, Ollama and so on, but also a friendly application with WebUI and router mode.

Featured in Lists

NPU-powered apps

Wake up the NPU on your device

List by bugmenot with 46 apps, updated

Official Links

AppStores & Other Links

What is llama.cpp?

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.