LLM Performance Test Between Raspberry Pi 5 and Orange Pi 5

I am very interested in running LLMs on more efficient hardware (at least, in terms of power consumption). Given this, I have always enjoyed testing the latest SBC performance with whatever small models are currently the most popular. I recently got a new 8GB Raspberry Pi 5 and wanted to see how it did running some small parameter LLMs. Seeing that I also have a 16GB Orange Pi 5B, which can theoretically run some small models very, very well (at least in terms of generation speed), I decided to test the two Pi's in an LLM comparison.

To ensure that the testing was as fair as possible, I wanted to run as similar a model as I could find on either board. The Orange Pi is a bit restricted in terms of actual software support, and there are far fewer available "off the shelf" models to run for it. I ended up using Phi-3 mini 4k instruct for the testing. While the model I used for the Orange Pi has a "w8a8 quantization", I had to use the q4_0 quant for the Pi 5, as the 8 wouldn't fit into memory, so this was a bit of a difference in terms of the comparison, but given the lack of widespread support for the Orange Pi, this was as close as I was going to come for a simple test like this.

For the test, I simply asked the model: "What is a laptop", and allowed it to respond. First up, the RPi 5. Because of the popularity of the RPi, running an LLM was as simple as installing Ollama and downloading the model. The RPi 5 was acceptable speed-wise, and the output was lucid and of reasonable quality. Overall it was definitely a step up compared to any previous RPi and would be a fine way for someone to experience the world of local LLMs, especially when factoring in the benefit of not needing to do anything crazy in-depth or technical to run a model.

Now for the Orange Pi 5B, which will from here on out be referred to as the OPi 5. Unfortunately, the steps required to get an LLM running on the OPi 5 were a bit more in-depth than what I had experienced with the RPi. The board required a special version of Ubuntu and had a very slim selection of models ready to run. Once everything was set up, I asked the same laptop question I had previously, and it responded. While the OPi 5 was far, far faster, the output was a bit chaotic at times and it would begin to respond to itself on occasion. Apparently this is a known issue and has some potential workarounds. The big negative here for the OPi 5, is that none of this was possible while running the desktop environment. In order to properly run the model, I had to connect remotely through SSH. Trying to run the model from the desktop environment caused errors and was not possible.

Something that I would like to give praise to, is this library, which allowed me to run the model on the OPi 5 without needing to do a bunch of extra work. The party responsible for this library also maintains a HF list of models ready to be run on the OPi 5.

Overall, my conclusion is as follows. I think for those among us who are keen to tinker with things and not afraid to troubleshoot, the Orange Pi 5B is a pretty potent option if you have a niche use case that requires the most power for running an LLM. I understand this is likely a very, very niche percentage of users, but I wanted to share as it is nice to have other options in this space.

As a final note, the Orange Pi 5B is just one of a few different boards that use the RK3588(S), the Rock Pi does as well IIRC, and a few others. Here is a small subreddit related to utilizing the NPU on these boards.

You can view the video for this article on my YouTube Channel