huangfu/photonAI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve split strategy for more reliable training evaluation

Browse Source 
Group samples by identical inputs before splitting, add target-aware stratification options, and cover the behavior with tests so repeated-input rows no longer leak across train, validation, and test sets.

Made-with: Cursor
This commit is contained in:
huangfu
2026-04-19 16:43:00 +08:00
parent 4981df0c02
commit 745868a456
7 changed files with 352 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -45,7 +45,7 @@
`v_pi_min <= V_pi <= v_pi_max`（默认 **`[0, 500]`**）。**区间之外整行剔除**。
该步骤专门针对「以最后一列 `V_pi` 为正常范围」的需求。
4. **（可选）严格正电压**：`remove_nonpositive_vpi: true` 时，在区间过滤之后再删除 `V_pi <= 0`（若需保留 `V_pi = 0` 且仍在 `[0,500]` 内，请保持为 `false`）。
5. **后续步骤**：train/val/test 划分、（可选）训练集离群策略、仅在训练集上拟合 `StandardScaler` 等，与原先一致。
5. **后续步骤**：默认采用**按 8 个输入字段分组**的 train/val/test 切分，避免「同输入异输出」同时落入不同集合；再按 `split_stratify_target`（默认 `V_pi`）做组级近似分层；之后才做（可选）训练集离群策略与仅在训练集上拟合 `StandardScaler`。
清洗前会在日志与 `data_report.md` 中报告：给定 `[v_pi_min, v_pi_max]` 下 **`V_pi` 越界行数**、重复样本、同输入异输出等统计，便于核对。
@@ -138,7 +138,7 @@ pytest -q tests/test_smoke.py
主要字段：
- **数据与清洗**：`data_path`、`remove_duplicate_rows`、**`filter_v_pi_range` / `v_pi_min` / `v_pi_max`**（默认按 **`V_pi ∈ [0, 500]`** 剔除越界行，对应 txt **第 11 列**）、`remove_nonpositive_vpi`、`outlier_strategy`（`none` / `iqr` / `zscore` / `quantile_clip`）及 `outlier_apply_to`（`targets` / `all`）。
- **划分**：`split_ratios`、`random_seed`；先 shuffle 再切分；**仅在训练子集**上拟合标准化器；离群阈值（若启用）也在训练子集上统计。
- **划分**：`split_ratios`、`random_seed`、`split_mode`、`split_stratify_target`、`split_stratify_bins`。默认 `grouped_stratified`：先按 8 维输入分组，再按指定目标（默认 `V_pi`）做组级近似分层；也可切回 `random`。**仅在训练子集**上拟合标准化器；离群阈值（若启用）也在训练子集上统计。
- **模型**：`hidden_dims`、`batchnorm`、`dropout`、`residual`。
- **训练**：`AdamW`、`lr`、`weight_decay`、`batch_size`、`epochs`、早停 `early_stopping_patience`。
- **调度器**：`cosine`（默认）或 `plateau`。

10
configs/default.yaml
View File

@@ -5,6 +5,10 @@ data_path: data/dataset.txt
split_ratios: [0.7, 0.15, 0.15] # train, val, test；可改为 [0.8, 0.1, 0.1]
random_seed: 42
# 切分策略：按 8 维输入分组，避免“同输入异输出”跨集合泄漏；再按目标分桶近似分层
split_mode: grouped_stratified # grouped_stratified | random
split_stratify_target: V_pi
split_stratify_bins: 10
remove_duplicate_rows: true
@@ -32,7 +36,8 @@ model:
hidden_dims: [200, 300, 350, 300, 200]
output_dim: 3
batchnorm: false
dropout: 0.0
# 温和 dropout，实测略优于全 0（见 results/run_20260419_163305）
dropout: 0.05
residual: false
optimizer:
@@ -55,7 +60,8 @@ training:
loss:
type: huber # huber | weighted_mse
huber_delta: 1.0
target_weights: [1.0, 1.0, 1.0]
# BW_3dB, IL, V_pi；略加重 V_pi 以小幅提升其测试 R²
target_weights: [1.0, 1.0, 1.2]
# 总输出目录；每次训练会在其下创建 run_时间戳/
output_dir: results

59
configs/mild_reg.yaml Normal file
View File

@@ -0,0 +1,59 @@
# 温和正则 + 略提高 V_pi 权重（在 default 基线上小幅改动，便于对比）
# 使用: python -m src.main train --config configs/mild_reg.yaml
data_path: data/dataset.txt
split_ratios: [0.7, 0.15, 0.15]
random_seed: 42
split_mode: grouped_stratified
split_stratify_target: V_pi
split_stratify_bins: 10
remove_duplicate_rows: true
outlier_strategy: none
outlier_apply_to: targets
outlier_config:
iqr_k: 1.5
zscore_threshold: 4.0
quantile_lower: 0.001
quantile_upper: 0.999
filter_v_pi_range: true
v_pi_min: 0.0
v_pi_max: 500.0
remove_nonpositive_vpi: false
model:
input_dim: 8
hidden_dims: [200, 300, 350, 300, 200]
output_dim: 3
batchnorm: false
dropout: 0.05
residual: false
optimizer:
name: adamw
lr: 0.001
weight_decay: 0.0001
scheduler:
type: cosine
plateau_factor: 0.5
plateau_patience: 10
plateau_min_lr: 1.0e-6
training:
batch_size: 128
epochs: 300
early_stopping_patience: 30
num_workers: 0
loss:
type: huber
huber_delta: 1.0
target_weights: [1.0, 1.0, 1.2]
output_dir: results
last_run_dir: null

59
configs/reg_vpi_try1.yaml Normal file
View File

@@ -0,0 +1,59 @@
# 实验：较强正则 + 提高 V_pi 损失权重（2026-04-19 试跑）
# 结果：早停偏早，test 整体差于 default 基线；仅作记录，日常训练请用 default.yaml
data_path: data/dataset.txt
split_ratios: [0.7, 0.15, 0.15]
random_seed: 42
split_mode: grouped_stratified
split_stratify_target: V_pi
split_stratify_bins: 10
remove_duplicate_rows: true
outlier_strategy: none
outlier_apply_to: targets
outlier_config:
iqr_k: 1.5
zscore_threshold: 4.0
quantile_lower: 0.001
quantile_upper: 0.999
filter_v_pi_range: true
v_pi_min: 0.0
v_pi_max: 500.0
remove_nonpositive_vpi: false
model:
input_dim: 8
hidden_dims: [200, 300, 350, 300, 200]
output_dim: 3
batchnorm: false
dropout: 0.15
residual: false
optimizer:
name: adamw
lr: 0.001
weight_decay: 0.0002
scheduler:
type: cosine
plateau_factor: 0.5
plateau_patience: 10
plateau_min_lr: 1.0e-6
training:
batch_size: 128
epochs: 300
early_stopping_patience: 30
num_workers: 0
loss:
type: huber
huber_delta: 1.0
target_weights: [1.0, 1.0, 1.75]
output_dir: results
last_run_dir: null

12
src/config.py
View File

@@ -62,6 +62,9 @@ class AppConfig:
data_path: str
split_ratios: List[float]
random_seed: int
split_mode: str
split_stratify_target: str
split_stratify_bins: int
remove_duplicate_rows: bool
outlier_strategy: str
outlier_config: OutlierConfig
@@ -90,6 +93,9 @@ class AppConfig:
data_path=str(raw["data_path"]),
split_ratios=list(raw["split_ratios"]),
random_seed=int(raw["random_seed"]),
split_mode=str(raw.get("split_mode", "grouped_stratified")),
split_stratify_target=str(raw.get("split_stratify_target", "V_pi")),
split_stratify_bins=int(raw.get("split_stratify_bins", 10)),
remove_duplicate_rows=bool(raw["remove_duplicate_rows"]),
outlier_strategy=str(raw.get("outlier_strategy", "none")),
outlier_config=OutlierConfig(
@@ -153,6 +159,12 @@ def load_config(path: str | Path) -> AppConfig:
raise ValueError("split_ratios 必须为长度为 3 的列表 [train, val, test]")
if abs(sum(sr) - 1.0) > 1e-6:
raise ValueError(f"split_ratios 之和必须为 1，当前为 {sum(sr)}")
if cfg.split_mode not in ("random", "grouped_stratified"):
raise ValueError("split_mode 必须为 random 或 grouped_stratified")
if cfg.split_stratify_target not in ("BW_3dB", "IL", "V_pi"):
raise ValueError("split_stratify_target 必须为 BW_3dB、IL 或 V_pi")
if cfg.split_stratify_bins < 2:
raise ValueError("split_stratify_bins 必须 >= 2")
if cfg.outlier_strategy not in ("none", "iqr", "zscore", "quantile_clip"):
raise ValueError(f"未知 outlier_strategy: {cfg.outlier_strategy}")
if cfg.outlier_apply_to not in ("targets", "all"):

129
src/preprocess.py
View File

@@ -12,6 +12,7 @@ from typing import Dict, List, Optional, Tuple
import numpy as np
import pandas as pd
import torch
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch.utils.data import DataLoader, TensorDataset
@@ -120,7 +121,7 @@ def clean_dataframe(
return out
def stratified_split_indices(
def _random_split_indices(
n: int,
ratios: List[float],
seed: int,
@@ -143,6 +144,130 @@ def stratified_split_indices(
return i_train, i_val, i_test
def _quantile_bin_labels(values: np.ndarray, n_bins: int) -> np.ndarray | None:
"""
基于秩做近似等频分桶，避免重复值导致的 qcut 退化。
返回每个样本所属桶标签；若样本过少则返回 None。
"""
if len(values) < 2:
return None
q = min(int(n_bins), len(values))
if q < 2:
return None
ranks = pd.Series(values).rank(method="first")
labels = pd.qcut(ranks, q=q, labels=False, duplicates="drop")
if labels is None:
return None
arr = np.asarray(labels, dtype=int)
if len(np.unique(arr)) < 2:
return None
return arr
def _grouped_split_indices(
df: pd.DataFrame,
ratios: List[float],
seed: int,
stratify_target: str,
stratify_bins: int,
report_lines: List[str],
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
先按输入 8 维分组，再在组级别按目标统计量近似分层切分。
这样可避免“同输入异输出”同时落在 train/val/test，提升评估稳定性。
若分层条件不足，则退化为组级随机切分。
"""
if len(df) < 3:
raise ValueError("样本数过少，无法做 train/val/test 切分。")
group_ids = df.groupby(INPUT_COLUMNS, sort=False, dropna=False).ngroup().to_numpy()
n_groups = int(group_ids.max()) + 1
group_df = df.copy()
group_df["_group_id"] = group_ids
group_stat = (
group_df.groupby("_group_id", sort=True)
.agg(group_size=("V_pi", "size"), strat_value=(stratify_target, "median"))
.reset_index()
)
group_id_arr = group_stat["_group_id"].to_numpy(dtype=int)
labels = _quantile_bin_labels(
group_stat["strat_value"].to_numpy(dtype=np.float64),
stratify_bins,
)
tr, va, te = ratios
holdout_ratio = va + te
val_ratio_in_holdout = va / holdout_ratio
def _split_groups(use_stratify: bool) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
strat = labels if use_stratify and labels is not None else None
train_groups, holdout_groups = train_test_split(
group_id_arr,
train_size=tr,
test_size=holdout_ratio,
random_state=seed,
shuffle=True,
stratify=strat,
)
holdout_strat = None
if strat is not None:
label_map = dict(zip(group_id_arr.tolist(), labels.tolist()))
holdout_labels = np.asarray(
[label_map[int(g)] for g in holdout_groups],
dtype=int,
)
if len(np.unique(holdout_labels)) >= 2:
holdout_strat = holdout_labels
val_groups, test_groups = train_test_split(
holdout_groups,
train_size=val_ratio_in_holdout,
test_size=1.0 - val_ratio_in_holdout,
random_state=seed + 1,
shuffle=True,
stratify=holdout_strat,
)
return (
np.asarray(train_groups, dtype=int),
np.asarray(val_groups, dtype=int),
np.asarray(test_groups, dtype=int),
)
split_note = (
f"按输入分组切分，共 {n_groups} 个唯一输入组；"
f"组级按 {stratify_target} 中位数分 {min(stratify_bins, n_groups)} 桶近似分层。"
)
try:
train_groups, val_groups, test_groups = _split_groups(use_stratify=True)
report_lines.append(split_note)
except ValueError as e:
train_groups, val_groups, test_groups = _split_groups(use_stratify=False)
report_lines.append(f"{split_note} 但分层条件不足，退化为组级随机切分：{e}")
i_train = np.flatnonzero(np.isin(group_ids, train_groups))
i_val = np.flatnonzero(np.isin(group_ids, val_groups))
i_test = np.flatnonzero(np.isin(group_ids, test_groups))
return i_train, i_val, i_test
def build_split_indices(
df: pd.DataFrame,
cfg: AppConfig,
report_lines: List[str],
) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""根据配置生成 train/val/test 行索引。"""
if cfg.split_mode == "random":
report_lines.append("切分策略：随机打乱后按比例切分。")
return _random_split_indices(len(df), cfg.split_ratios, cfg.random_seed)
return _grouped_split_indices(
df=df,
ratios=cfg.split_ratios,
seed=cfg.random_seed,
stratify_target=cfg.split_stratify_target,
stratify_bins=cfg.split_stratify_bins,
report_lines=report_lines,
)
def apply_train_only_outliers(
X_train: np.ndarray,
y_train: np.ndarray,
@@ -404,7 +529,7 @@ def prepare_training_data(
X = cleaned[INPUT_COLUMNS].to_numpy(dtype=np.float64)
y = cleaned[TARGET_COLUMNS].to_numpy(dtype=np.float64)
i_tr, i_va, i_te = stratified_split_indices(len(X), cfg.split_ratios, cfg.random_seed)
i_tr, i_va, i_te = build_split_indices(cleaned, cfg, report_lines)
save_split_indices(run_dir, i_tr, i_va, i_te)
X_train, y_train = X[i_tr], y[i_tr]
X_val, y_val = X[i_va], y[i_va]

85
tests/test_smoke.py
View File

@@ -10,6 +10,7 @@ import torch
import yaml
from src.config import load_config
from src.data import INPUT_COLUMNS
from src.data import load_raw_txt
from src.model import MLPRegressor
from src.preprocess import prepare_training_data
@@ -106,3 +107,87 @@ def test_one_epoch_training_pipeline(tmp_path: Path) -> None:
assert (run_dir / "checkpoints" / "best.pt").is_file()
meta = json.loads((run_dir / "cleaning_meta.json").read_text(encoding="utf-8"))
assert meta["n_train"] > 0
def test_grouped_split_keeps_same_inputs_together(tmp_path: Path) -> None:
data_txt = tmp_path / "grouped_data.txt"
rng = np.random.default_rng(7)
rows = []
base_inputs = rng.normal(size=(24, 8))
for x in base_inputs:
for _ in range(3):
y0 = float(x[0] * 2.0 + rng.normal(scale=0.01))
y1 = float(x[1] * -1.5 + rng.normal(scale=0.01))
y2 = float(abs(x[2]) * 20.0 + 10.0 + rng.normal(scale=0.1))
rows.append(np.concatenate([x, [y0, y1, y2]]))
mat = np.asarray(rows, dtype=float)
data_txt.write_text(
"\n".join(",".join(str(v) for v in row) for row in mat),
encoding="utf-8",
)
cfg_dict = {
"data_path": str(data_txt),
"split_ratios": [0.7, 0.15, 0.15],
"random_seed": 3,
"split_mode": "grouped_stratified",
"split_stratify_target": "V_pi",
"split_stratify_bins": 6,
"remove_duplicate_rows": False,
"outlier_strategy": "none",
"outlier_apply_to": "targets",
"outlier_config": {
"iqr_k": 1.5,
"zscore_threshold": 4.0,
"quantile_lower": 0.001,
"quantile_upper": 0.999,
},
"remove_nonpositive_vpi": False,
"filter_v_pi_range": True,
"v_pi_min": 0.0,
"v_pi_max": 500.0,
"model": {
"input_dim": 8,
"hidden_dims": [16, 16],
"output_dim": 3,
"batchnorm": False,
"dropout": 0.0,
"residual": False,
},
"optimizer": {"name": "adamw", "lr": 0.01, "weight_decay": 0.0},
"scheduler": {
"type": "cosine",
"plateau_factor": 0.5,
"plateau_patience": 10,
"plateau_min_lr": 1e-6,
},
"training": {
"batch_size": 16,
"epochs": 1,
"early_stopping_patience": 1,
"num_workers": 0,
},
"loss": {"type": "huber", "huber_delta": 1.0, "target_weights": [1.0, 1.0, 1.0]},
"output_dir": str(tmp_path / "results"),
}
cfg_path = tmp_path / "cfg_grouped.yaml"
cfg_path.write_text(yaml.safe_dump(cfg_dict), encoding="utf-8")
cfg = load_config(cfg_path)
df = load_raw_txt(data_txt)
run_dir = tmp_path / "run_grouped"
run_dir.mkdir()
bundle = prepare_training_data(df, cfg, run_dir)
split_data = json.loads((run_dir / "split_indices.json").read_text(encoding="utf-8"))
split_name_by_row = {}
for split_name, indices in split_data.items():
for idx in indices:
split_name_by_row[int(idx)] = split_name
cleaned = df.reset_index(drop=True)
for _, sub in cleaned.groupby(INPUT_COLUMNS, dropna=False):
assigned = {split_name_by_row[int(i)] for i in sub.index.to_list()}
assert len(assigned) == 1
assert len(bundle.X_train) > 0