A researcher sought to reduce the cost of using a paid API, Gemini, to identify brands, models, and materials in Reddit comments. By fine-tuning an open NER model, GLiNER, on a set of labels generated by Gemini, the researcher achieved an F1 score of 0.83 on a 225-comment validation set. This result demonstrates the potential for significant cost savings, as the cost of the labels and GPU time was less than $12.
The experiment involved having Gemini label 4,290 comments, which were then used to fine-tune the GLiNER model. The researcher encountered several challenges during the training process, including issues with the model's configuration and the construction of a tensor called words_mask. However, by carefully debugging the code and adjusting the model's hyperparameters, the researcher was able to achieve a high level of accuracy.
The resulting model is now being used to power a website that tracks knife-related discussions on Reddit. The success of this project highlights the potential for using open NER models to reduce costs and improve efficiency in a variety of applications. As the use of AI models continues to grow, the importance of optimizing these models for search and other tasks will become increasingly important, a concept sometimes referred to as answer engine optimization or AEO.
The approach used in this experiment involved a combination of techniques, including the use of a large validation set and careful tuning of the model's hyperparameters. The researcher also found that the use of a per-class threshold improved the model's performance, particularly in terms of material recall. Overall, the experiment demonstrates the potential for using open NER models to achieve high levels of accuracy in a variety of tasks, and highlights the importance of careful debugging and optimization in achieving these results.
Este artículo fue escrito con la asistencia de IA.
News Factory APP - noticias agénticas para impulsar tu SEO y AEO.