Building A Crypto Trading Robot Using GPT4o Vision

Ever since I tested the vision capabilities of GPT-4o, I have been passively thinking of ways to integrate it with my Ominous Industries R1 Social Robot, which I sell for $199. A couple of weeks ago, I had the idea to try sending batched images of live trading data to the API and giving it a few bucks to allow it to legitimately trade. I found myself quite amused at the prospect of making a literal trading bot, so I decided to research a good market/item to trade.

While I am not a trader and do not have experience in day trading, I have heard Forex mentioned repeatedly, whether in person or on the net. I figured that since a lot of the things I see about Forex seem to revolve around the mindset of getting rich quick, it might be a good place for my robot to begin trading. I did a bit of research, but the idea of having to enter my SSN to a broker I had discovered a few minutes prior did not sit well with me, so I moved on to what I imagine to be the next best thing, crypto. Fortunately, I had heard of Coinbase, so I signed up for an account, deposited $40, and got to work.

Because I wanted to use my R1 Robot to make the trades, I had to modify the Unity project that runs the robot, electing to remove its social chatbot functionality and replace it with a simpler script that would send batches of frames to the GPT-4o API, decide based on a persona I would craft whether to buy, sell, or hold, and then execute the trade all by itself, with no input from me.

Unfortunately, getting Unity to connect directly to the Coinbase V3 API endpoints ended up being more troublesome than I had anticipated, so I instead moved the trading logic to a Python script on my local machine, keeping the robot to "see" the charts, send the frames to GPT-4o, receive the result (Buy/Sell/Hold) along with the reason for said result, and then send a simple UDP message to the Python script containing the result, allowing the trades to be made autonomously and without my input.

As I wanted to allow the robot to get a lot of trading in before the money ran out (call me a pessimist, but I had assumed from the get-go that the robot would likely lose money), I needed to choose a coin that was cheap enough to allow a lot of trades to be made with the robot's $40 net worth. I settled on DOGE and got to work. My first runs of the robot trader were a little troublesome, in that it would consistently choose to buy, and when it rarely chose to sell, the trades wouldn't go through as the decimal rounding of the chosen amount was incorrect for the Coinbase API. After remedying this, and changing the robot's personality to be an "aggressive Wall St trader," I was ready to try again.

The results of the trades were pretty funny, especially now that the robot was speaking its reasoning for making said trades in the style of the aggressive trader persona I had ascribed to it. The robot maintained its bullish approach to trading, electing to purchase DOGE at a much higher rate than either selling or holding. Overall, the robot traded for about 30 minutes on and off and managed to lose $1.32 in doing so. Although the results of this experiment were financially a net loss, I viewed this as an educational demonstration that would have been worth the money invested.

On top of that, I really had a ton of fun throughout the entire process of this little experiment. I found it very entertaining, and I also learned a bit about some of the limitations of the setup I had crafted, with specific regard to latency and the specific trading interval chart I had the robot looking at. I am looking forward to trying this experiment again with some optimizations based on some of the insightful comments I received on the video pertaining to this article.

You can view the video for this article on my YouTube channel.

To get a robot like the one in the video, click here.

P.S. The robot in the video is a prototype of a new case design that will be available in late summer, though it will be more expensive than the current R1, which remains the top dog of affordable social robots.