IMDb情感分析全流程：DistilBERT LoRA与TF-IDF基线对比
IMDb Sentiment Analysis with DistilBERT LoRA, TF-IDF Baselines, Calibration, Interpretability, Robustness Testing, and Semi-Supervised Learning
做NLP实战的同学必看，这份教程把数据审计、LoRA微调、校准和半监督学习串成完整可复用的流程，直接照着跑就能落地。
In this tutorial, we develop an end-to-end sentiment analysis workflow using the Stanford NLP IMDb Large Movie Review Dataset and compare classical machine learning with parameter-efficient transformer fine-tuning. We begin by establishing a reproducible environment and auditing the dataset for class ordering, review-length skew, duplicate leakage, and preprocessing artifacts before training a strong TF-IDF and Logistic Regression baseline. We then fine-tune DistilBERT with LoRA through PEFT, evaluate it using accuracy, macro-F1, ROC-AUC, confusion matrices, and ROC curves, and examine threshold selection and probability calibration through Expected Calibration Error and reliability analysis. Beyond headline metrics, we investigate confident errors, performance across review lengths, word-level occlusion saliency, and head-versus-tail truncation to understand how the model reaches its predictions and where long-context limitations affect performance. Finally, we use the unlabeled IMDb split for confidence-based pseudo-labeling, compare the resulting semi-supervised model against our baseline, and save the merged transformer for reusable sentiment inference.
在本教程中，我们使用斯坦福NLP IMDb大型电影评论数据集开发一个端到端的情感分析工作流，并比较经典机器学习与参数高效Transformer微调。我们首先建立一个可复现的环境，并在训练强TF-IDF和逻辑回归基线之前，审计数据集的类别顺序、评论长度偏斜、重复泄漏和预处理伪影。然后，我们通过PEFT使用LoRA微调DistilBERT，使用准确率、宏F1、ROC-AUC、混淆矩阵和ROC曲线进行评估，并通过预期校准误差和可靠性分析检查阈值选择和概率校准。除了总体指标外，我们研究自信错误、不同评论长度下的性能、词级遮挡显著性以及头部与尾部截断，以了解模型如何做出预测以及长上下文限制在何处影响性能。最后，我们使用未标记的IMDb分割进行基于置信度的伪标签，将得到的半监督模型与我们的基线进行比较，并保存合并后的Transformer以供可重用的情感推理。
import importlib.util, subprocess, sys, os, time, random, warnings, inspect, hashlib
warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["WANDB_DISABLED"] = "true"
_REQUIRED = {
"transformers": "transformers",
"datasets": "datasets",
"peft": "peft",
"accelerate": "accelerate",
"sklearn": "scikit-learn",
}
_missing = [pkg for mod, pkg in _REQUIRED.items() if importlib.util.find_spec(mod) is None]
if _missing:
print(f"Installing: {', '.join(_missing)} ...")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", *_missing], check=True)
print("Done. (If imports fail below, restart the runtime and re-run.)\n")
import numpy as np
import pandas as pd
import torch
import matplotlib.pyplot as plt
from datasets import load_dataset
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (accuracy_score, f1_score, roc_auc_score,
classification_report, confusion_matrix, roc_curve)
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
TrainingArguments, Trainer, DataCollatorWithPadding,
EarlyStoppingCallback, set_seed)
from peft import LoraConfig, get_peft_model, TaskType
def _disable_torchao_probe():
patched = []
try:
import peft.import_utils as _piu
_piu.is_torchao_available = lambda: False
patched.append("peft.import_utils")
except Exception:
pass
for _name, _mod in list(sys.modules.items()):
if _name.startswith("peft") and hasattr(_mod, "is_torchao_available"):
_mod.is_torchao_available = lambda: False
patched.append(_name)
return patched
try:
import torchao as _tao
_v = getattr(_tao, "__version__", "?")
if tuple(int(x) for x in _v.split(".")[:2]) < (0, 16):
print(f"[compat] torchao {_v} < 0.16 -> disabling PEFT's torchao probe: "
f"{', '.join(_disable_torchao_probe())}")
except Exception:
_disable_torchao_probe()
SEED = 42
MODEL_NAME = "distilbert-base-uncased"
MAX_LEN = 256
N_TRAIN = 5000
N_EVAL = 2000
N_UNSUP = 3000
EPOCHS = 2
BATCH = 16
LR = 3e-4
FULL_RUN = False
if FULL_RUN:
N_TRAIN, N_EVAL, EPOCHS = 25000, 25000, 3
set_seed(SEED); random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("=" * 79)
print(f"device={DEVICE} | torch={torch.__version__} | "
f"gpu={torch.cuda.get_device_name(0) if DEVICE=='cuda' else 'n/a'}")
print("=" * 79)
t0 = time.time()
raw = load_dataset("stanfordnlp/imdb")
print(raw, f"\nloaded in {time.time()-t0:.1f}s\n")
print("--- example (truncated) ---")
print("label:", raw["train"][0]["label"], "|", raw["train"][0]["text"][:300], "...\n")
first_labels = np.array(raw["train"]["label"][:5])
last_labels = np.array(raw["train"]["label"][-5:])
print(f"TRAP #1 - split ordering: first 5 labels {first_labels}, "
f"last 5 labels {last_labels} -> ALWAYS shuffle before subsampling.")
train_full = raw["train"].shuffle(seed=SEED)
test_full = raw["test"].shuffle(seed=SEED)
train_ds = train_full.select(range(min(N_TRAIN, len(train_full))))
eval_ds = test_full.select(range(min(N_EVAL, len(test_full))))
print(f" after shuffle+subsample: train balance = "
f"{np.bincount(train_ds['label'])}, eval balance = {np.bincount(eval_ds['label'])}")
lens = np.array([len(t.split()) for t in train_full["text"]])
q = np.percentile(lens, [50, 75, 90, 95, 99])
print(f"\nTRAP #2 - length (words): median={q[0]:.0f} p75={q[1]:.0f} p90={q[2]:.0f} "
f"p95={q[3]:.0f} p99={q[4]:.0f} max={lens.max()}")
print(f" ~{(lens > MAX_LEN*0.75).mean()*100:.1f}% of reviews exceed MAX_LEN={MAX_LEN} "
f"tokens (rough words->tokens factor 1.3). Section 9 measures what that costs.")
h_tr = {hashlib.md5(t.encode()).hexdigest() for t in raw["train"]["text"]}
h_te = {hashlib.md5(t.encode()).hexdigest() for t in raw["test"]["text"]}
print(f"\nTRAP #3 - leakage: {len(h_tr & h_te)} exact duplicate reviews across "
f"train/test; {len(raw['train'])-len(h_tr)} dupes inside train itself.")
def clean(t):
return t.replace("<br />", " ").replace("<br/>", " ").strip()
plt.figure(figsize=(11, 3.2))
plt.subplot(1, 2, 1)
plt.hist(np.clip(lens, 0, 1000), bins=60)
plt.axvline(MAX_LEN, ls="--", color="k", label=f"MAX_LEN={MAX_LEN}")
plt.title("Review length (words, clipped at 1000)"); plt.legend()
plt.subplot(1, 2, 2)
plt.bar(["neg", "pos"], np.bincount(raw["train"]["label"]))
plt.title("Train class balance (perfectly balanced)")
plt.tight_layout(); plt.show()We configure the Colab environment, install the required libraries, apply the PEFT–torchao compatibility fix, and set deterministic seeds for reproducible experiments. We load the Stanford IMDb dataset, shuffle and subsample the train and test splits, and inspect class balance, review-length distributions, duplicate leakage, and HTML artifacts. We also visualize review lengths and label frequencies so we understand the dataset structure before building any models.
我们配置Colab环境，安装所需的库，应用PEFT–torchao兼容性修复，并设置确定性种子以实现可复现的实验。我们加载斯坦福IMDb数据集，对训练和测试分割进行洗牌和子采样，并检查类别平衡、评论长度分布、重复泄漏和HTML伪影。我们还可视化评论长度和标签频率，以便在构建任何模型之前了解数据集结构。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力