Running Local LLMs in Python

Want to build an app that uses an LLM for processing and don't know how to get started? Interested in the idea of using these neural networks in your projects but don't know how to get started? Well, I'll show you how you can implement AI into your apps to make them have semantic understanding with your requests.

What is ollama?

Ollama is a platform built on top of llama.cpp which allows you to easily run LLMs locally and have access to an API so you can send requests to the LLMs with code. You can run LLMs similar to chatGPT and Gemini but with your own hardware. Ollama provides us a way to explore models on their page ollama and run them locally or even through their machines at a cost.

What LLM to choose?

In the local LLM space, you have a plethora of options; however, you're severely limited by the VRAM and TFLOPS. My current hardware is a Ryzen 5600x as the CPU, an RTX 3090 for the GPU and 48gb of ddr4 ram. They key part of my machine is the RTX 3090 which will both hold and run the model. The RTX 3090 has 24GB vram and what ever model you choose should fit in your VRAM

When running AI models, you want to make sure that the whole model fits into your GPUs vram, if you don't have enough memory and some of it is loaded to the RAM, you will have a severe speed penalty since RAM is slower than VRAM for the GPU. Some options include:

• gpt-oss - openAI's answer to local LLMs, it pretty rapid but doesn't have the ability for deeper thinking
• qwen3 - My personal favourite, developed by Alibaba
• gemma3 - Googles answer to local LLMs

My personal preference is Qwen3 as it seems to be the most capable for my hardware limits.

Accessing ollama API

Once you've installed ollama to your PC, you can send an API call to it either through the ollama library in python or using regular HTTP requests since it exposes a standard rest API. The code below shows a simple implementation of calling ollama with python and simply asking the LLM if the user input is a food item or not.

import ollama
import json

# Qwen:3-30b quantised
MODEL_NAME : str = "hf.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q3_K_M" 

# My ollama runs at a different url
client = ollama.Client(host="http://192.168.1.100:11435") 

def is_food(item : str) -> json:
    """Ask the LLM if the item is food.
    Returns a dict like {'answer': True}."""

    try:
        response :json = client.generate(
            model = MODEL_NAME,
            prompt=f"is {item} considered edible food?",
            system="""Respond ONLY with valid JSON in the format
              {\"answer\": true} or {\"answer\": false}.""",
            format="json"
        )
        return response["response"]

    except ConnectionError:
        print("Cannot connect to ollama instance")

if __name__ == "__main__":
    query_item = input("Enter a food or non food item: ")
    print(is_food(query_item))
    

Trying to manually code or coming up with a way to deem each individual item as a food item would be time consuming, however since LLMs take into account semantics, it can easily tell apart a food item and something else. Lets look at an example here.

python .\food_checker.py                                                                                                                           
Enter a food or non food item: apple
{"answer": true}

python .\food_checker.py                                                                                                                           
Enter a food or non food item: rotten apple
{"answer": false}

In the example above, we can see that when asking if "apple" is considered an edible food, it returns as a json "true" and when we add context to say the apple is rotten, it returns false. It has semantic understanding that the apple is no longer edible.

python .\food_checker.py                                                                                                                           
Enter a food or non food item: small green round apple that is in a fridge but has partly browned skin
{"answer": true}

Now, choosing to add more context, the apple makes it its answer change to true.

Local LLMs are the future of private, cost-effective app development. By leveraging Ollama and models like Qwen, you can add complex logic to your Python scripts in minutes.