Using edge computing and Large Language Models (LLM) for constructing a local agent

Abstract

The growing demand for artificial intelligence systems that are responsive, privacy-preserving, and capable of operating autonomously is driving the adoption of Large Language Models (LLMs) on edge devices. However, deploying LLMs as the core of agents on devices with limited resources still faces computational and memory constraints. This research develops an LLM-based local agent architecture that runs locally on a Raspberry Pi without relying on cloud services. The system integrates local LLM inference with agentic functions, including intent processing, reasoning, tool invocation, and interaction with hardware via GPIO. Several quantized Small Language Models (SLMs) were evaluated on a Raspberry Pi 5 with 16 GB of RAM using CPU-based inference. The evaluation included total latency, Time to First Token (TTFT), token generation rate, peak RAM usage, peak CPU utilization, and peak temperature. The results showed significant performance differences among the models. Qwen 2.5 0.5B Q3 produced the lowest total latency of 23.9 seconds, the lowest TTFT of 269.0 ms, and the highest generation rate of 12.1 tokens/s. However, the results indicate that inference speed alone is not sufficient to determine the most suitable model as the core of a local agent. Most models reached peak CPU utilization above 95%, while RAM usage and thermal characteristics varied across configurations. Based on a combination of inference performance and resource usage, models in the 1B–1.7B range specifically LLaMA 3.2 1B Q4, Qwen 2.5 1.5B Q4, and SmolLM 1.7B Q4 show potential as a balanced set of candidates between responsiveness and model capacity. The research results demonstrate the feasibility of deploying LLMs as local agents on low-cost edge devices and provide a basis for selecting models based on a balance between inference efficiency, resource constraints, and agentic capabilities.