JabRef
v6
v6
  • JabRef Bibliography Management
  • Installation
  • Getting started
  • Collect
    • Add entry manually
    • Add entry using an ID
    • Add entry using reference text
    • Searching externally using Online Services
    • Add entry using PDFs
    • Add PDFs to an entry
    • Browser Extension
    • Import
      • Custom import filters
      • Import inspection window
  • Organize
    • Edit an entry
    • Groups
    • Keywords
    • Mark and grade
    • Comment on an entry
    • Searching within the library
    • Complete information using online databases
    • Manage associated files
    • Manage field names and their content
    • Best practices
    • Check consistency
    • Cleanup entries
    • Check integrity
    • Find duplicates
    • Merge entries
    • Save actions
  • Cite
    • BibTeX and biblatex
    • Pushing to external editor application
    • Export to Microsoft Word
    • OpenOffice/LibreOffice integration
  • Share
    • Sharing a Bib(la)TeX Library
    • Shared SQL Database
      • Migration of pre-3.6 SQL databases into a shared SQL database
    • Export
      • Custom export filters
    • Send as email
  • AI functionality
    • AI providers and API keys
    • AI preferences
    • AI troubleshooting
    • Running a local LLM model
  • Configuration
    • Customize the citation key generator
    • Customize entry types
    • Customize general fields
    • Customize key bindings
    • Library properties
    • Entry preview setup
    • Manage external file types
    • Manage protected terms
    • The string editor
  • Advanced information
    • Main Window
    • Entry Editor
      • Links to other entries
      • The Bibtex / Biblatex source tab
      • The 'owner' field
      • Time stamp field
      • LaTeX Citations Tab
    • About BibTeX and its fields
    • Strings
    • Field content selector
    • URL and DOI links in JabRef
    • Command line use and options
    • Automatic Backup (.sav and .bak) and Autosave
    • XMP metadata support in JabRef
    • Remote operation
    • Custom themes
    • Journal abbreviations
    • New subdatabase based on AUX file
    • How to expand first names of an entry
    • Debugging your library file
    • Resources
    • License
    • Knowledge
      • MS Office Bibliography XML format
      • Comparison of the Medline (txt), Medline (xml), and RIS format
      • EndNote Export Filter
  • Frequently Asked Questions
    • Linux
    • Mac OS X
    • Windows
  • JabKit
  • Discussion Forum
  • Contribute to JabRef
    • How to Improve the Help Page
    • How to translate the JabRef User Interface
Powered by GitBook
On this page
  • High-level explanation
  • Step-by-step guide for ollama
  • Step-by-step guide for GPT4All

Was this helpful?

Edit on GitHub
Export as PDF
  1. AI functionality

Running a local LLM model

PreviousAI troubleshootingNextConfiguration

Last updated 16 days ago

Was this helpful?

Notice:

  1. LLMs require a lot of computational power and therefore lots of electricity.

  2. Smaller models typically respond qualitatively worse than bigger ones, but they are faster, need less memory and might already be sufficient for your use case.

  3. The size of a model can be measured in number of parameters in its neural network. The "b" in the model name typically stands for billion parameters. It also can be measured in terms of gigabytes required to load the model into your devices RAM/VRAM.

  4. The model should always completely fit into VRAM (fast), otherwise layers will be offloaded to RAM (slower) and if it doesn't fit in there either, it will use SSD (abysmally slow).

  5. Hardware recommendation for maximize prompt processing and token generation speed: A device with high bandwidth. A modern GPU with lots of VRAM will satisfy this requirement best.

High-level explanation

You can use any program that creates a server with OpenAI-compatible API.

After you started your service, you can do this:

  1. The "Chat Model" field in AI preferences is editable, so you can enter any model you have downloaded

  2. There is a field called "API base URL" in "Expert Settings" where you need to provide the address of an OpenAI-compatible API server

VoilĂ ! You can use a local LLM right away in JabRef.

Step-by-step guide for ollama

The following steps guide you on how to use ollama to download and run local LLMs.

  1. Install ollama from

  2. Select a model that you want to run. The ollama provides to choose from. Some popular models are for instance , , , or .

  3. When you have selected your model, type ollama pull <MODEL>:<PARAMETERS> in your terminal. <MODEL> refers to the model name like gemma2 or mistral, and <PARAMETERS> refers to parameters count like 2b or 9b.

  4. ollama will download the model for you

  5. After that, you can run ollama serve to start a local web server. This server will accept requests and respond with LLM output. Note: The ollama server may already be running, so do not be alarmed by a cannot bind error. If it is not yet running, use the following command: ollama run <MODEL>:<PARAMETERS>

  6. Go to JabRef Preferences -> AI

  7. Set the "AI provider" to "OpenAI"

  8. Set the "Chat Model" to the model you have downloaded in the format <MODEL>:<PARAMETERS>

  9. Set the "API base URL" in "Expert Settings" to http://localhost:11434/v1/

Now, you are all set and can chat "locally".

Step-by-step guide for GPT4All

The following steps guide you on how to use GPT4Allto download and run local LLMs.

  1. Open JabRef, go to "File" > "Preferences" > "AI"

  2. Set the "AI provider" to "GPT4All"

  3. Set the "Chat model" to the name (including the .ggufpart) of the model you have downloaded in GPT4All.

  4. Set the "API base URL" in "Expert Settings" to http://localhost:4891/v1/chat/completions.

Install GPT4Allfrom their .

Open GPT4All, , configure it in the and .

their website
a large list of models
qwen3:30b-a3b
granite3.1-moe:3b
devkit/L1-Qwen-1.5B-Max
mistral:7b
mistral-small3.1:24b
website
download a model
settings
run it as a server