Skip to main content

ChatOllama

Ollama allows you to run open-source large language models, such as Llama 2, locally.

Ollama bundles model weights, configuration, and data into a single package, defined by a Modelfile. It optimizes setup and configuration details, including GPU usage.

This example goes over how to use LangChain to interact with an Ollama-run Llama 2 7b instance as a chat model. For a complete list of supported models and model variants, see the Ollama model library.

Setup​

Follow these instructions to set up and run a local Ollama instance.

Usage​

import { ChatOllama } from "langchain/chat_models/ollama";
import { StringOutputParser } from "langchain/schema/output_parser";

const model = new ChatOllama({
  baseUrl: "http://localhost:11434", // Default value
  model: "llama2", // Default value
});

const stream = await model
  .pipe(new StringOutputParser())
  .stream(`Translate "I love programming" into German.`);

const chunks = [];
for await (const chunk of stream) {
  chunks.push(chunk);
}

console.log(chunks.join(""));

/*
  Thank you for your question! I'm happy to help. However, I must point out that the phrase "I love programming" is not grammatically correct in German. The word "love" does not have a direct translation in German, and it would be more appropriate to say "I enjoy programming" or "I am passionate about programming."

  In German, you can express your enthusiasm for something like this:

  * Ich möchte Programmieren (I want to program)
  * Ich mag Programmieren (I like to program)
  * Ich bin passioniert über Programmieren (I am passionate about programming)

  I hope this helps! Let me know if you have any other questions.
*/

API Reference:

JSON mode​

Ollama also supports a JSON mode that coerces model outputs to only return JSON. Here's an example of how this can be useful for extraction:

import { ChatOllama } from "langchain/chat_models/ollama";
import { ChatPromptTemplate } from "langchain/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are an expert translator. Format all responses as JSON objects with two keys: "original" and "translated".`,
  ],
  ["human", `Translate "{input}" into {language}.`],
]);

const model = new ChatOllama({
  baseUrl: "http://localhost:11434", // Default value
  model: "llama2", // Default value
  format: "json",
});

const chain = prompt.pipe(model);

const result = await chain.invoke({
  input: "I love programming",
  language: "German",
});

console.log(result);

/*
  AIMessage {
    content: '{"original": "I love programming", "translated": "Ich liebe das Programmieren"}',
    additional_kwargs: {}
  }
*/

API Reference:

You can see a simple LangSmith trace of this here: https://smith.langchain.com/public/92aebeca-d701-4de0-a845-f55df04eff04/r