trueparagraph.ai-DistilBERT

This model is a fine-tuned version of distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:

• Accuracy: 0.9427
• F1: 0.9429
• Precision: 0.9352
• Recall: 0.9506
• Mcc: 0.8854
• Roc Auc: 0.9427
• Pr Auc: 0.9136
• Log Loss: 0.9232
• Loss: 0.3017

Model description

DistilBERT is a smaller, faster, cheaper version of BERT, achieved through knowledge distillation. It retains 97% of BERT’s language understanding while being 60% faster and smaller. This fine-tuned version of DistilBERT is trained to detect AI-generated text in paragraphs from the STEM domain.

Key characteristics:

• Architecture: Transformer-based model
• Pre-training objective: Masked Language Modeling (MLM)
• Fine-tuning objective: Binary classification (Human-written vs AI-generated)

Intended uses & limitations

Intended uses

• AI Text Detection: Identifying paragraphs in the STEM domain that are generated by AI versus those written by humans.
• Educational Tools: Assisting educators in detecting AI-generated content in academic submissions.
• Research: Analyzing the effectiveness of AI-generated content detection in STEM-related texts.

Limitations

• Domain Specificity: The model is fine-tuned specifically on STEM paragraphs and may not perform as well on texts from other domains.
• Generalization: While the model is effective at detecting AI-generated text in STEM, it may not generalize well to other types of AI-generated content outside of its training data.
• Biases: The model may inherit biases present in the training data, which could affect its performance and fairness.

Training and evaluation data

The model was fine-tuned on the "16K-trueparagraph-STEM" dataset, which consists of 16,000 paragraphs from various STEM domains. The dataset includes both human-written and AI-generated paragraphs to provide a balanced training set for the model.

Dataset Details

• Size: 16,000 paragraphs
• Sources: Academic papers, research articles, and other STEM-related documents.
• Balance: Approximately 50% human-written paragraphs and 50% AI-generated paragraphs.

Training procedure

Preprocessing

• Tokenization: Texts were tokenized using the DistilBERT tokenizer.
• Truncation/Padding: All inputs were truncated or padded to a maximum length of 512 tokens.

Hyperparameters

• Optimizer: AdamW
• Learning Rate: 5e-5
• Batch Size: 16
• Number of Epochs: 3

Training

• Loss Function: Binary Cross-Entropy Loss
• Evaluation Metrics: Accuracy, Precision, Recall, F1-Score, ROC-AUC

Hardware

• Environment: Training was conducted on a single NVIDIA Tesla V100 GPU.
• Training Time: Approximately 4 hours.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Accuracy	F1	Precision	Recall	Mcc	Roc Auc	Pr Auc	Log Loss	Validation Loss
0.5806	0.6297	500	0.8207	0.8349	0.7708	0.9108	0.6525	0.8211	0.7464	3.1049	0.4137
0.3015	1.2594	1000	0.8919	0.8885	0.9137	0.8646	0.7849	0.8918	0.8574	1.7818	0.3298
0.2287	1.8892	1500	0.9175	0.9155	0.9330	0.8987	0.8354	0.9174	0.8889	1.3631	0.2585
0.1444	2.5189	2000	0.9310	0.9312	0.9240	0.9386	0.8621	0.9310	0.8978	1.1225	0.2439
0.1149	3.1486	2500	0.9272	0.9304	0.8874	0.9778	0.8589	0.9274	0.8788	1.1773	0.3574
0.0716	3.7783	3000	0.9401	0.9405	0.9311	0.95	0.8805	0.9402	0.9095	0.9662	0.2655
0.0411	4.4081	3500	0.9427	0.9429	0.9352	0.9506	0.8854	0.9427	0.9136	0.9232	0.3017

Framework versions

• Transformers 4.42.4
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1