A developer has released Demoscener-7B, an AI model that writes the next lines of tiny, visually striking JavaScript programs, and he says he tuned it to his own sense of what looks good. The model and the dataset it was trained on are now available on Hugging Face.
The project began with a simple question. While studying a demoscene piece called “Memories” by the programmer HellMood, the developer noticed that some of its visual effects could be traced back to earlier experiments and borrowed ideas. He wondered how many other directions those same ideas could have taken. Dwitter.net, a site that works like Twitter for 140-character JavaScript demos, seemed like the right place to test that idea. The site allows users to “remix” any post, so the same programs often appear in several related versions.
Writing these programs is hard, and the developer said he struggled to finish his own. He wanted an autocomplete that understood the odd syntax of the form. Dweets rely heavily on code-golfing tricks and mathematical shorthand, which rarely appear in the code that general-purpose coding models see. He noted that even with a 64-character alphabet, a program stripped of its last 30 characters can be completed in an astronomical number of ways. Most of those completions are invalid or dull. A language model, he reasoned, could focus on the continuations that are actually plausible.
How was the demoscene code model trained?
The first step was data. He tried generating synthetic dweets with a large language model, but he found the results aesthetically unappealing. Instead, he used the Dwitter API to build a dataset of 25,000 dweets for training, along with about 1,300 more for validation. The base model, Qwen2.5-Code-7B, struggled to complete most of the test programs.
He then ran continued pretraining on the corpus. To measure the effect, he tracked the surprisal of individual tokens before and after training. Across 30 held-out dweets, perplexity dropped from 28.63 to 8.57, which he said meant the model had become far better at predicting the structure of these programs. The outputs were more consistent, but he felt they still lacked charm.
How did the developer tune the model toward his own taste?
To add that quality, he turned to Dwitter’s “awesomeness” score. He ranked the top 5,000 dweets by score and built a curriculum that gradually favored the highest-rated examples. Over 400 training steps, the top 1,000 programs grew to about 67 percent of the sampling probability. He described this as a breakthrough, since the continuations became noticeably more visually interesting. Perplexity on a broader set of 7,000 dweets also improved, including on programs the model had never sampled during curriculum training.
The final stage was reinforcement learning, which he ran with GRPO and himself as the reward function. For each unfinished dweet, the model sampled eight completions. He rendered them side by side and picked the one he liked best, or skipped the group if none appealed to him. Over roughly nine or 10 rounds of 500 groups each, he inspected around 40,000 animations. He said he later wished he had known about vision-language models and HPSv3, a human preference scoring system, which might have made the process less manual.
How did the developer evaluate the model?
To check whether his taste had actually transferred, he asked a friend to judge 400 sets of demos from several checkpoints, without knowing which model made each one. Checkpoint 4 scored best, and he used it for inference on his Mac.
He also built a small interface that generates several continuations from the same starting code, renders them side by side and lets a user branch off from any result. Users can edit the code, generate new variations and keep exploring. The developer said the project was funded with his GitHub Sponsors payout, which he used to rent a GPU for training.
The work is an informal personal project rather than a formal research release, but it shows how a small, community-curated dataset and a single person’s judgment can steer a code model toward a narrow artistic style. He said he plans to keep exploring the possibilities the tool opens up.
Dieser Artikel wurde mit Unterstützung von KI verfasst.
News Factory APP - agentische News für besseres SEO & AEO.