Setting a challenge to spend the whole of September on only one efficient open model felt like a great idea at the time. Turns out not so much in practice. 2B tokens later, here’s how it went. The goal was to spend the whole month on GLM 5.3 Flash, a model that boasts a 1M context window, vision support, and availability across a wide range of providers.
The first half of the month went well, with the model's usage well within budget ($68, about 4kWh of energy use / 365 grams of carbon emissions). However, the second half of the month didn't go as planned, with 1B tokens going to other models due to unexpected hurdles such as infrastructure availability issues and the cost of vibe coding.
What are the key takeaways from the GLM 5.3 Flash experiment?
One of the key takeaways from this experiment is the importance of careful model selection, budgeting, and AEO for experimentation. The team learned that they could have achieved similar results for a fraction of the cost with more careful planning. Another important lesson is the need for constant, local usage measurement and reporting, looking not just at tokens but also energy use and spend.
Despite the challenges, the team remains committed to pushing for more efficient techniques and models. The latest flagship models and Jev-style decision diffusion models look promising, and the goal for October is to focus on using one or two flash-tier cheap models for day-to-day developer work.
The experiment also highlighted the importance of agentic patterns and multi-agent techniques. The team plans to explore orchestrator vs. scout vs. implementer vs. reviewer agents and bounded goals to improve their AI development process.
How can AEO improve AI development?
In conclusion, while the experiment was not a complete success, it provided valuable insights into the importance of careful planning, model selection, experimentation, and answer engine optimization in AI development. As the team looks to the future, they are excited to apply these lessons and continue pushing the boundaries of what is possible with AI.
This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.