| --- |
| language: |
| - en |
| - es |
| pipeline_tag: text-classification |
| tags: |
| - ai-generated-text-detection |
| - text-classification |
| - ensemble |
| - deberta |
| - lightgbm |
| - tf-idf |
| - pytorch |
| - scikit-learn |
| --- |
| |
| # EC_MODELS (AI vs Human Detection) |
| |
| This repository contains **model artifacts** for AI‑generated text detection experiments, including a HOPE classifier and a stacked ensemble (DeBERTa + LightGBM + TF‑IDF/SGD). It is **not a single model**, but a collection of checkpoints and meta‑models produced during experiments. |
| |
| ## Model inventory |
| |
| Artifacts are stored in `src/ai_vs_human/models/`: |
| |
| - `hope/` |
| HOPE checkpoints (`fold_*/best_model.pt`) for a transformer‑based classifier with memory modules. |
| - `deberta_v3_base/` |
| DeBERTa base checkpoints used in the ensemble. |
| - `lightgbm/`, `lightgbm_numeric/` |
| LightGBM models on engineered features. |
| - `tfidf_sgd/` |
| TF‑IDF + SGD models. |
| - `stack_meta/meta_learner.joblib` |
| Logistic‑regression meta‑learner used for stacking. |
| |
| Out‑of‑fold predictions used to compute metrics and train the stacker are in: |
| |
| - `src/ai_vs_human/oof/` (e.g., `oof_stack.csv`, `oof_deberta.csv`, `oof_lgb.csv`, `oof_sgd.csv`) |
| |
| ## Intended use |
| |
| These models are intended for **research and evaluation** of AI‑generated text detection. They can be used to: |
| |
| - compare HOPE vs. ensemble baselines, |
| - reproduce experiments from the notebooks, |
| - evaluate domain‑shift and multilingual robustness. |
| |
| ## How to evaluate (metrics only) |
| |
| Use the metrics‑only notebook to compute standard metrics without retraining: |
| |
| - `src/ai_vs_human/metrics_only.ipynb` |
| |
| This notebook loads `oof_stack.csv` and prints AUC‑ROC, PR‑AUC, Accuracy, Precision, Recall, F1, Brier, and ECE, plus a best‑F1 threshold. |
| |
| ## Training data |
| |
| Training data and features were produced from the project’s datasets under `src/ai_vs_human/`, including: |
| |
| - `merged_ai_human_multisocial_features*.csv` |
|
|
| See the dataset‑building and training notebooks for details: |
|
|
| - `src/ai_vs_human/ai_generated_text_detection.ipynb` |
| - `src/ai_vs_human/hope_train_distributed.py` |
|
|
| ## Evaluation notes |
|
|
| Metrics depend on threshold selection and domain. This repo includes tools for: |
|
|
| - internal test evaluation, |
| - external/ood evaluation, |
| - calibration and threshold selection. |
|
|
| See: |
|
|
| - `src/ai_vs_human/evaluation_suite.ipynb` |
|
|
| ## Limitations and biases |
|
|
| - Performance can degrade under **domain shift** (new sources or languages). |
| - False‑positive rates for **human multilingual text** can be high without careful calibration. |
| - These models are **not** a definitive AI‑detection system and should not be used for high‑stakes decisions without additional validation. |
|
|
| ## License |
|
|
| No explicit license is included in this repo. Please add a license if you intend to distribute or reuse the models. |
|
|
