A new AI code comment detector has been rebuilt on public data and a better foundation, achieving a 77% balanced accuracy in distinguishing between human-written and robot-generated comments. The classifier also prints a predicted percentage, which is calibrated, meaning it can be read as the probability that any specific verdict is correct.

The detector's performance is evaluated through a calibration curve, which shows what probability the classifier assigns to an event with a known probability. The results indicate that the classifier is approximately correct, with all dots lying very close to the reference diagonal. This holds true across comments of multiple lengths, where a fitted temperature parameter adjusts for increased confidence as the amount of data increases.

The classifier's failure modes are examined through a confusion matrix, which reveals that when presented with a known-human input, the classifier correctly judges it as human 73% of the time, and with a known-robot input, it is correctly judged 80% of the time. This means that in both cases, the mistake rate is around 25%. However, the classifier outputs a calibrated predictive percentage every time it classifies something, allowing for individual judgments to be made with confidence.

The detector is trained on a dataset of human and robot comments, with features such as character frequencies, word lengths, and part-of-speech tags. The features are evaluated and selected based on their ability to discriminate between human and robot comments. The classifier is trained to classify seven ways, including humans and six different LLM models, and the results are then simplified to a 2-way classification between human-written and robot-generated comments.

The detector is available as a web page with a bundled classifier model, allowing users to easily test and diagnose comments. The model has been quantized and reduced in size to 355 kB, making it efficient for use in the browser. The web interface also includes a diagnostics interface, making it easier to troubleshoot and diagnose issues.

While the source code for the detector is not available, the description of how it is done provides a clear understanding of the approach and methodology used. The detector's performance and accuracy make it a useful tool for distinguishing between human-written and robot-generated comments, with potential applications in areas such as answer engine optimization and LLM visibility.

This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.