Enterprise AI Analysis

Our Novel GenAI Detection Methodology

Our detector identifies GenAI-assisted submissions through two primary mechanisms: over-uniqueness and sample matching. GenAI-generated code often exhibits a unique style, distinct from typical student submissions, and produces similar solutions for common prompts. By analyzing these characteristics, we can effectively flag potentially assisted code. A key advantage is its independence from training data and language-specific rules, making it highly adaptable across various programming contexts.

The detector integrates seamlessly with existing code similarity analysis tools like SSTRANGE, employing five robust similarity measurements—Cosine, Jaccard, MinHash, Super-Bit, and RKRGST—each with standard and sensitive modes. This comprehensive approach ensures high detection accuracy while maintaining efficiency.

The process begins with preprocessing submissions to normalize code and generalize identifiers. Next, similarity scores are calculated. These scores then feed into our over-uniqueness and sample matching algorithms. Finally, a detailed report highlights suspicious submissions, enabling educators to make informed decisions about academic integrity.

Robust Evaluation & Performance Metrics

Our detector was evaluated across four diverse Python and Java datasets, encompassing thousands of student submissions and GenAI-assisted examples. The Mean Average Precision (MAP) metric was used to assess effectiveness, focusing on the ranked position of identified GenAI-assisted submissions. Processing time measured efficiency.

The over-uniqueness mechanism showed strong performance, especially on assessments expecting longer solutions (Exam dataset: 82% average MAP). The sensitive mode, which accounts for identifier names and constants, often improved detection, particularly for distinct GenAI styles. However, its effectiveness was significantly reduced when GenAI-generated code was explicitly aligned to student styles (Weekly Align dataset: 26% average MAP).

The sample matching mechanism proved highly effective, achieving an 80% average MAP on the Weekly dataset. Its strength lies in identifying code segments similar to known GenAI samples, with sensitive mode often yielding statistically significant improvements. This mechanism is particularly strong when students are less fluent in disguising GenAI output.

Combining both mechanisms yielded an overall average of 72% MAP. While sometimes outperforming individual mechanisms, the combination did not consistently result in higher effectiveness than sample matching alone, suggesting that not all GenAI-assisted submissions are simultaneously unique and similar to a GenAI sample. Efficiency analysis consistently showed MinHash and Super-Bit as the fastest measurements due to their locality-sensitive hashing and binning mechanisms, while RKRGST was the slowest due to its quadratic complexity.

Limitations & Future Research Directions

While effective, our current GenAI detector has certain limitations. It was primarily evaluated on introductory programming courses and Python/Java submissions, suggesting a need for replication across diverse programming languages, course levels, and institutional settings. The chosen metrics (MAP, processing time) provide strong indicators, but exploring precision, recall, and ROC curves could offer a more nuanced understanding of performance trade-offs.

Future work will involve testing additional similarity measurements like Winnowing or local alignment, and comparing our detector against existing text-based and programming-specific GenAI detectors in controlled environments. Investigating the impact of submission length, content variation, assessment design, and the proportion of aligned content on detection performance is crucial. We also plan to develop a script to leverage results from popular plagiarism detectors (MOSS, JPlag, Sherlock) and explore additional mechanisms for identifying heavily disguised or aligned GenAI-assisted submissions, potentially by monitoring the creation process for anomalous behaviors. Enhancements to token weighting and the combination of over-uniqueness and sample matching mechanisms are also planned to maximize effectiveness.

Ultimately, this research serves as a stepping stone towards more robust and adaptive tools for maintaining academic integrity in the evolving landscape of AI-assisted learning.