Detecting GPT-Assisted Writing Using Interpretable Stylometric Features
Random Forest using stylometric features detected ChatGPT-assisted student writing with ROC-AUC 0.87.
The paper tests whether interpretable stylometric features taken only from submitted text can separate independently written student work from ChatGPT-assisted writing. Ninety participants produced both conditions, and eight classifiers were evaluated with same-participant data kept together in validation. On the held-out test set, Random Forest reached an ROC-AUC of 0.87 and an F1-score of 0.84, with a 22.2% false positive rate and an 11.1% false negative rate. SHAP analysis attributed predictions mainly to lexical and grammatical features.
- Study used writing from 90 participants, both independent and ChatGPT-assisted.
- Random Forest reached ROC-AUC 0.87 and F1 0.84 on a held-out test set.
- False positive rate was 22.2% and false negative rate was 11.1%.
- SHAP analysis points to lexical and grammatical features as main drivers.
- Same-participant data stayed together during validation to limit leakage.
Full article113 words · extracted from arxiv.org · click to collapse
Distinguishing GPT-assisted from independently authored student writing has become a critical challenge in academia. This paper evaluates the discriminative capability of interpretable stylometric features extracted solely from submitted text. Using data from 90 participants who wrote both independently and with ChatGPT assistance, we evaluate eight machine learning classifiers while keeping data from the same participant together during validation. On the held-out test set, Random Forest achieved an ROC-AUC of 0.87 and an F1-score of 0.84, with False Positive and False Negative rates of 22.2% and 11.1%, respectively. SHAP analysis shows that lexical and grammatical characteristics drive the resulting predictions. The findings suggest that transparent, text-intrinsic features provide measurable signal for detecting GPT-assisted writing.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.26687