Local LLM: run AI on your own computer

Find out what a local LLM is, what you can run on your computer, the best local LLM models and how to set one up to keep business data in-house.

A local LLM runs a large language model on your computer, so you can draft emails and work with documents without sending their contents to a hosted model. Start with a small downloaded model on the hardware you own. A graphics card helps with speed, but you don’t need a dedicated AI workstation to try it.

For a store handling customer messages, the privacy benefit depends on the whole setup. The model, document processing and any connected tools must all stay local.

What a local LLM is

A local LLM is a language model whose calculations run on your device. You download its weights, the files containing the model’s learned parameters, then load them into a program that accepts prompts.

A self-hosted LLM runs on infrastructure your business manages. That can be an office server used by several staff members. It can also be a rented cloud server, so “self-hosted” alone doesn’t mean customer data stays in your office. For an in-house setup, both the model and its supporting services run on hardware you control at your premises.

A local model can turn a supplier’s terms into a summary or draft a reply from a customer’s email. The application sends the prompt to the model on your machine and displays its answer. You still need to check the answer against the original document before using it.

Ollama says it doesn’t see prompts or data when you run its models locally. Its cloud models are a separate option. Selecting a downloaded local model is part of keeping the work on your computer.

What local LLM can I run?

Available memory sets the first limit on which local LLM you can run. A graphics processing unit (GPU) has dedicated video memory, usually called VRAM. On an Apple silicon Mac, the CPU and GPU share unified memory, which must also accommodate the operating system and other applications.

Model labels such as 8B mean roughly eight billion parameters. Quantisation stores those parameters at lower precision to reduce memory use. The table assumes a roughly four-bit model, one active conversation and a modest context window. Context is the text the model holds while answering.

Hardware starting points

Hardware you haveSensible starting model sizeExampleWhat limits it
Laptop with 8 GB system memory1B to 4BA small text modelLittle memory remains for applications and long documents
Laptop with 16 GB system memory4B to 8BQwen3 8BCPU-only replies can be slow
Apple silicon Mac with 24 GB unified memory8B to 14BQwen3 14BThe operating system shares this memory
Apple silicon Mac with 32 GB or more unified memory12B to 27BGemma 3 27B at the larger endLong context needs additional memory
PC with 24 GB VRAM and adequate system memory14B to 27BGemma 3 27BModel weights must leave room for context and runtime memory

These are planning estimates, not guaranteed loading limits. LM Studio’s system requirements recommend at least 16 GB of system memory. A model’s download size is smaller than the total memory its running session needs.

Which GPU suits your budget?

For a first trial, spend nothing on a GPU. Use a small model on your existing computer and find out whether it handles your documents.

For a lower-cost new desktop build, our memory-focused pick is the RTX 5060 Ti with 16 GB. NVIDIA also sells an 8 GB version, so select the 16 GB specification. It gives 8B to 14B models more working room than an 8 GB card.

For a used-card budget aimed at larger models, consider an RTX 3090 with 24 GB. Allow for the card’s condition, cooling and power supply requirements. For a premium new build, the RTX 5090 has 32 GB, giving more memory headroom for context and larger models.

These choices prioritise memory capacity. They aren’t a speed ranking. A 24 GB card you already own is a sensible place to try Gemma 3 27B before buying a replacement.

The best local LLM models

Qwen3 8B is our starting choice for business writing on a modest computer. For a 24 GB graphics card, try Gemma 3 27B when you want to compare a larger model’s answers. Choose between them using your actual tasks, including the time you spend correcting each reply.

Local models to try

ModelOllama tagListed download sizeTask to assess
Qwen3 8Bqwen3:8b5.2 GBShort email drafts and rewriting
Qwen3 14Bqwen3:14b9.3 GBMore detailed drafting and instruction following
Gemma 3 12Bgemma3:12b8.2 GBDocument questions on a smaller setup
Gemma 3 27Bgemma3:27b17 GBDocument summaries with more memory available

The Qwen sizes come from Ollama’s listings for Qwen3 8B and Qwen3 14B. Its listings for Gemma 3 12B and Gemma 3 27B give the Gemma sizes. They describe the listed downloads, not a guarantee that a session fits in that much memory.

For a useful comparison, give each model the same anonymised supplier document. Ask it to extract the minimum order quantity and delivery terms, then quote the supporting passage. Reject answers that invent a term or miss an exception. Compare a customer reply too, checking that it follows your returns policy without promising an unauthorised refund.

How to run a local LLM

Install a local runner, download a model that fits your memory and make it answer a harmless prompt before adding business documents. Use the vendor’s installation instructions for your operating system. The walkthroughs below follow the official guides.

Set up Ollama

  1. Download Ollama using its quickstart guide. Install the version for your operating system and open the application.
  2. Open a terminal and run ollama run qwen3:8b. This downloads the model if needed, then opens a conversation. Wait for the input prompt before continuing.
  3. Ask, “Write a two-sentence reply thanking a customer for asking about delivery. Don’t invent a delivery estimate.” Confirm that an answer appears.
  4. For local-only use, set "disable_ollama_cloud": true in the JSON configuration file ~/.ollama/server.json, preserving any existing settings. Restart Ollama. Its logs must show Ollama cloud disabled: true.
  5. With the model downloaded, disconnect the internet and ask a new harmless question. A fresh answer confirms that this conversation can run offline.

Ollama’s command reference documents model commands. Run ollama ps in another terminal while a reply is generating to see whether the model uses the CPU, GPU or both.

Set up LM Studio

  1. Follow LM Studio’s setup guide to install the application on a supported computer.
  2. Open Discover and search for a model from the table. Choose a compatible quantised download that leaves spare memory, then wait for the download to finish.
  3. Open Chat, open the model loader and select the downloaded model. Wait for it to load before sending your first message.
  4. Send the same delivery-reply prompt used above. Check that you receive a response without an invented delivery promise.
  5. Attach a non-sensitive document and ask a question whose answer you know. Check the response against the file, then repeat a harmless question with the internet disconnected.

LM Studio’s offline documentation says downloaded models and document chat work locally. Finding and downloading new models still needs internet access. Keep external tools disconnected when your goal is an entirely local workflow.

When the setup fails

If a model fails to load or the computer starts using swap memory heavily, close other applications and select a smaller model. Shorten the context setting if the failure appears after adding a long document.

If replies are slow, use ollama ps to check whether generation is using system memory instead of the graphics card. Compare your card and driver with Ollama’s supported hardware. A smaller model that fits fully in GPU memory can be more useful than a larger one that spills into system memory.

If document answers miss a clause, ask a narrower question and include the relevant section. A scanned PDF might need optical character recognition before its text is usable. Compare the extracted text with the original before trusting an answer about quantities or payment terms.

Training and customising a local model

Most stores don’t need to train a local LLM to answer questions about supplier files or store policies. Start by giving an existing model the relevant documents and clear instructions.

For a short document, the application can place the text directly in the conversation. For longer collections, retrieval-augmented generation (RAG) finds relevant passages and gives them to the model with your question. LM Studio’s document guide explains how its document chat chooses between these approaches.

Retrieval is usually the better starting point for changing business facts because you can replace the source document when a policy changes. Fine-tuning changes a model using training examples. It can help with a repeated format or writing style, but it doesn’t create a dependable store of current stock or policy details.

For example, attach an anonymised returns policy and ask the model to draft a reply using only that file. Require it to flag missing information. Review the relevant policy passage yourself, especially where a customer’s rights or refund amount are involved.

Keep retrieval local too. A document index or embedding service that runs elsewhere can send your files outside the business even when the final language model runs on your laptop.

Local LLMs and customer data

A fully local setup keeps the prompts and documents used for inference off an outside model provider’s servers. A customer email can stay on the office computer throughout drafting if the runner and every connected tool process it locally.

That protection ends wherever another service receives the content. A cloud model selection, external document processor or networked tool changes the data path. Synced folders and cloud backups can also copy local chat files outside the office.

Local execution leaves device security with your business. A shared staff account, stolen laptop or malicious application can expose saved chats and source files. Use individual accounts, disk encryption and restricted document access. Keep the operating system and model runner updated, and include chat logs in your retention policy.

Ollama listens on the local loopback address by default. Keep that default for a single-user setup. An office-wide self-hosted service needs access controls before staff connect, and it must stay off the public internet unless a qualified administrator secures it.

Before introducing real customer information, run one complete task with synthetic data and the network disconnected. Confirm that the document loads, the model answers and the saved chat stays in the location you expect. Then decide which customer information the task actually needs and remove the rest before using it.

Give your store its own shopping agent

Customers ask in their own words. Your agent finds, compares and checks out from your catalogue, right on your site.

See it in actionFree trial · Simple setup