Abstract

This study investigates differentiating human-written text from AI-generated text. The rise of AI-generated writing raises concerns for academic integrity, authorship attribution, and plagiarism detection. This work addresses the growing need for detection methods that are accurate, interpretable, and grounded in measurable linguistic evidence by asking a central question: what linguistic and representational patterns most reliably distinguish AI-generated writing from human writing across different modeling approaches? This study evaluates four supervised modeling pipelines: stylometric features with machine learning models (logistic regression, support vector machines, random forests, and gradient-boosted trees); word- and BPE-level TF-IDF representations with linear, kernel, and neural classifiers; fine-tuned transformer models; and a tokenizer-controlled TF-IDF pipeline using logistic regression and support vector machines to isolate vocabulary effects. A balanced dataset of 10,000 English texts was compiled from public sources, covering essays, creative writing, QA, paraphrasing, and summarization. The fine-tuned RoBERTa model performed best, achieving 0.925 accuracy. Stylometric models reached 0.77 to 0.85, while TF-IDF pipelines achieved 0.76 to 0.80. The tokenizer-controlled TF-IDF experiment improved model performance to 0.81, demonstrating the influence of representation. These pipelines enable examination of whether detection accuracy is driven primarily by model choice or by the underlying feature representation used to encode text. Interpretability analyses, including SHAP, LIME, PDPs, TF-IDF contribution maps, and transformer attributions like attention and integrated gradients, showed that stylometric and TF-IDF features capture only broad linguistic tendencies without clearly separating human and AI texts, a pattern confirmed by unsupervised clustering. In contrast, t-SNE visualizations of RoBERTa embeddings revealed distinct separation driven by deeper semantic and structural signals. Moving beyond recent work that treats AI-text detection primarily as a classification problem, this study emphasizes the importance of understanding the representational signals behind predictions, underscoring the value of interpretability and evidence-based reasoning for developing more transparent and reliable detection systems.

Advisor

Musgrave, John

Department

Computer Science

Keywords

AI-generated text detection, authorship classification, stylometric analysis, TF-IDF, transformer models, interpretability, Explainable AI, natural language processing, text representation, machine learning

Publication Date

2026

Degree Granted

Bachelor of Arts

Document Type

Senior Independent Study Thesis

Download

Share

COinS
 

© Copyright 2026 Aditi Jha