Use an API model by default. Run a local model only when the data can't leave your machine, you have steady high volume on one simple task, or you work offline. By the end you'll have a task-by-task decision table and a cost formula you can run on last month's token bill.
A hosted API is usually cheaper to start, faster to change and less work to keep running. The Bootstrapped Founder puts the local cases at three: data that legally can't leave your infrastructure, cost at sustained and predictable volume, or an offline environment. Outside those, skip the hardware.
Labels, so you know what to trust: the prices and VRAM numbers below are published figures, the break-even rule is plain arithmetic, and whether a local model is good enough for your task is untested until you run the 20-input check in step 7.
Pick the model by the task, not by the brand
| Task | Data it touches | Pick |
|---|---|---|
| Hard reasoning, long documents, big-context coding | Yours or public | Frontier API model |
| Tagging, extracting fields, short summaries, first drafts, thousands a month | Yours or public | Local 7B to 8B model, API as fallback |
| Anything with client records or private code | A client's | Local, or an API plan whose written terms say prompts aren't used for training |
| A few requests a week, or bursts | Any | API. Idle weeks cost nothing |
| Work with no internet | Any | Local |
Frontier hosted models still lead on hard reasoning and large context, while open-weights models like Llama, DeepSeek, Qwen and Mistral are often enough for focused tasks. So the mistake is not choosing local. The mistake is judging local on a hard task it was never meant for, then writing it off for the easy ones.
Some customers with strict privacy or compliance needs won't allow data to go to outside systems. Read that contract line before you paste a client file into any API. If you give an agent access to client files, add tools the safe way first.
Run the cost formula on last month's tokens
A local model costs a fixed amount whatever your volume. An API charges per token. Heavy, predictable work favors local. Light or bursty work favors the API. Here is the math to find out which one you are.
API cost = (input tokens / 1,000,000 x input price) + (output tokens / 1,000,000 x output price)
Local cost per month = (hardware price / months you'll use it) + power
Price input and output separately. Claude Haiku 4.5 is listed at $1.00 per million input tokens and $5.00 per million output tokens, with cached input at $0.10. Output costs five times input, so the output side is where a bill grows. Prices change, so check your provider's own page before you plug a number in.
Example: say a task uses 3,000,000 input tokens and 1,000,000 output tokens a month on Haiku 4.5. Input is 3 x $1.00 = $3.00. Output is 1 x $5.00 = $5.00. Your API bill is $8.00. Now say a used GPU costs $600 and you'll use it for 36 months, with $5 a month in power. That's $16.67 + $5 = $21.67 a month.
| Monthly cost | |
|---|---|
| API at 3M in, 1M out | $8.00 |
| Local, $600 GPU over 36 months plus $5 power | $21.67 |
| Volume where they break even | about 2.7 times yours, or roughly 8.1M in and 2.7M out |
At that volume the API is cheaper by $13.67 a month. Local only wins once your bill climbs past $21.67, and only if the local model passes your spot check. A cheaper bill from a worse answer isn't a saving.
Turn on caching before you buy hardware
If a long system prompt repeats on every call, cached input at $0.10 per million replaces the $1.00 rate. Say 2,000,000 of your 3,000,000 input tokens are that repeated prompt. You save 2 x ($1.00 - $0.10) = $1.80, and the bill drops from $8.00 to $6.20 with no new machine.
Payback rule for hardware
Payback months = hardware price / (monthly API bill avoided minus monthly power)
In the example above the bill avoided is $8.00 and power is $5, so $600 / $3 = 200 months. Nobody keeps a GPU that long, so stay on the API. If the number is longer than the time you'll own the hardware, the hardware loses.
Fit the model to your graphics card before you buy anything
The model size is set by your GPU memory (VRAM). At Q4_K_M quantization on Ollama, a 7B model needs about 6 GB, a 13B about 11 GB and a 70B about 42 GB.
| Your VRAM | Biggest model that fits | Use it for |
|---|---|---|
| 8 GB | 7B to 8B | Tagging, extraction, short rewrites |
| 12 GB | 13B | The same, with a little more room |
| 48 GB | 70B | Rarely worth buying for a solo business |
Don't overflow. Ollama runs fastest when the whole model sits in VRAM: typically 50 to 80 tokens per second for 7B to 32B models on an RTX 5090. Once layers spill into system RAM it drops to 3 to 8 tokens per second. That's not usable for work you wait on. The same page recommends at least 16 GB of system RAM and an NVMe SSD.
When the model you want needs more VRAM than you have, don't buy a bigger card. Pay for the API call, or step down one model size.
Seven steps to decide task by task
-
- List every task an AI does in your business, one line each.
-
- Next to each, write what data it touches: public, yours, or a client's.
-
- Mark client or regulated data "stays local or needs written API terms".
-
- Estimate monthly tokens: count one typical request, multiply by requests per month.
-
- Run the API formula for the cheapest model that does the job.
-
- Run the local formula only for tasks with steady volume.
-
- Test the local model on 20 real inputs and compare against the API output. Keep local only for the tasks that pass, and keep an API key as the fallback.
Re-run this once a quarter. If you run several agents across products, per-task cost adds up fast, and the portfolio setup is where I'd look first. If you're still on an API-based coding setup, this one stays safe.
Four mistakes that cost you money or hours
- Buying hardware before measuring tokens. The machine sits idle at low volume.
- Comparing a small local model to a frontier model on a hard task. It looks bad, and you drop a local option that would have been fine for simple work.
- Adding input and output at one price. You'll underestimate any output-heavy task by a wide margin.
- Forgetting upkeep. A local setup is yours to update, monitor and fix, and your hours are the scarce resource.
Do this in the next ten minutes
Open your provider's usage page and write down last month's input and output tokens for your biggest task. Run the API formula on them. If the answer is under $20 a month, you have your decision: stay on the API.
Send me the task that eats your hours
If you have a manual task you think a tool should already handle, email me with the task and what it touches. Email Me.