{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Proact-VL 验证 (Validation) Notebook\n", "\n", "基于 [microsoft/AnthropomorphicIntelligence/Proact-VL](https://github.com/microsoft/AnthropomorphicIntelligence/tree/main/Proact-VL) 的评估流程, 可直接在 Colab 运行。\n", "\n", "**三种模式 (在配置区改 `MODE`)**\n", "\n", "- **`baseline` (默认)**: 使用官方发布的 baseline 预测结果 (`oaaoaa/proactvl_results`), 无需 GPU 推理; 配置 OpenAI 兼容 API key 后追加 LLM Score / CC / PAUC。\n", "- **`generate`**: 在 Colab GPU 上用开源 VLM (默认 Qwen2.5-VL-3B) 对 LiveGamingBenchmark 逐秒流式生成 Proact-VL 风格主动解说, 再用官方指标评估。(官方 Proact-VL 权重目前受限制, 此模式为行为近似)\n", "- **`custom`**: **上传你自己的视频** (如衣物投放视频), 逐秒流式分析。`task=count` 时统计\"用户投了几件衣服\": 模型每秒判断是否有**新的**衣物投入, 客户端做去重合并 (投放动作跨 1-2 秒), 输出事件时间线 + 总数, 可填 `MY_GT_COUNT` 自动核对。\n", "\n", "**前置条件**\n", "\n", "1. HF token: 数据集为 auto-gated, 需先在 HuggingFace 网页接受条款, 用任意有效 token 登录 (Mode A/B 需要; 纯 `custom` 模式上传本地视频时可跳过 HF 下载, 但代码区仍会 clone 仓库)。\n", "2. LLM 指标需要 OpenAI 兼容 API (`OPENAI_API_KEY`, 可选 `OPENAI_BASE_URL`), F1/TimeDiff 不需要。\n", "3. Mode B / C 需要 GPU (T4/P100 16GB 即可跑 3B; CPU 也能跑但逐秒很慢, 建议视频 ≤30s)。" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 依赖安装 (Colab 自带 torch/transformers)\n", "!pip install -q openai huggingface_hub datasets tqdm\n", "!pip install -q qwen-vl-utils decord # Mode B / C 需要\n", "\n", "import torch, transformers\n", "print('torch', torch.__version__, '| gpu', torch.cuda.is_available(), '| transformers', transformers.__version__)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ 配置区 ============\n", "import os\n", "\n", "# --- 1) 运行模式 ---\n", "MODE = 'custom' # 'baseline' | 'generate' | 'custom'\n", "BASELINE_FILE = 'qwen2_5_vl_7b_instruct.jsonl' # baseline 模式的预测文件\n", "BASELINE_ID = 'qwen2_5_vl_7b_instruct'\n", "\n", "# --- 2) HuggingFace (baseline / generate 模式需要; custom 模式若跳过下载可留空) ---\n", "HF_TOKEN = os.environ.get('HF_TOKEN') or input('HF token (skip: 回车, 仅 baseline/generate 需要): ')\n", "if HF_TOKEN:\n", " os.environ['HF_TOKEN'] = HF_TOKEN\n", " from huggingface_hub import login\n", " login(token=HF_TOKEN)\n", "\n", "# --- 3) LLM judge (API 指标; 不需要可留空跳过) ---\n", "OPENAI_API_KEY = os.environ.get('OPENAI_API_KEY') or input('OpenAI-compatible API key (empty to skip LLM metrics): ')\n", "OPENAI_BASE_URL = os.environ.get('OPENAI_BASE_URL') or input('API base url (empty = default openai.com): ')\n", "JUDGE_MODEL = os.environ.get('OPENAI_JUDGE_MODEL', 'gpt-4o')\n", "JUDGE_LIMIT = 50\n", "\n", "os.environ['OPENAI_AUTH_MODE'] = 'api_key'\n", "if OPENAI_API_KEY: os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY\n", "if OPENAI_BASE_URL: os.environ['OPENAI_BASE_URL'] = OPENAI_BASE_URL\n", "os.environ['OPENAI_JUDGE_MODEL'] = JUDGE_MODEL\n", "\n", "# --- 4) generate / custom 模式的模型与视频参数 ---\n", "VL_MODEL = 'Qwen/Qwen2.5-VL-3B-Instruct' # 16GB GPU; 7B/8B 需 LOAD_IN_4BIT=True\n", "LOAD_IN_4BIT = False\n", "NUM_SAMPLES = 5 # generate 模式评估前 N 条\n", "VIDEO_INCLUDE = ['DATA/anns/*',\n", " 'DATA/game_commentary/LOL/**/*.mp4',\n", " 'DATA/game_commentary/CSGO/**/*.mp4']\n", "\n", "# --- 5) custom 模式: 自己的视频 (衣物投放计数) ---\n", "MY_TASK = 'count' # 'count' (投了几件) | 'commentary' (逐秒解说)\n", "MY_SCENE = '衣物投放 (clothing items being dropped into a washing machine)'\n", "MY_BEGIN = 0 # 分析区间 [begin, end) 秒\n", "MY_END = 60\n", "MY_GT_COUNT = -1 # 你知道实际投了几件就填数字, 用于自动核对; 不知道填 -1\n", "MY_MERGE_GAP = 1 # 相邻投放动作合并阈值 (秒)\n", "\n", "print('MODE=', MODE, '| judge=', JUDGE_MODEL or '(skip)', '| vl=', VL_MODEL,\n", " '| custom:', MY_TASK, MY_SCENE, f'[{MY_BEGIN},{MY_END})', 'gt=', MY_GT_COUNT)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ 拉取官方评估代码 (MIT), 下载数据 ============\n", "!git clone --depth 1 https://github.com/microsoft/AnthropomorphicIntelligence ./anim 2>/dev/null || echo already-cloned\n", "os.chdir('./anim/Proact-VL')\n", "import pathlib\n", "for f in ['evaluation/gaming/llm_score.py', 'evaluation/gaming/llm_judge.py']:\n", " p = pathlib.Path(f)\n", " s = p.read_text()\n", " s = s.replace('deployment_name = \"gpt-5.1_2025-11-13\"',\n", " 'deployment_name = os.getenv(\"OPENAI_JUDGE_MODEL\", \"gpt-5.1_2025-11-13\")')\n", " p.write_text(s)\n", "print('judge-model patch applied')\n", "\n", "if MODE in ('baseline', 'generate') and os.environ.get('HF_TOKEN'):\n", " from huggingface_hub import snapshot_download\n", " snapshot_download(repo_id='oaaoaa/proactvl_results', repo_type='dataset',\n", " local_dir='./data/proactvl_results', token=os.environ['HF_TOKEN'])\n", " RES = pathlib.Path('./data/proactvl_results/results')\n", " print('ann files:', sorted(x.name for x in (RES/'anns').iterdir()))\n", " print('baselines:', sorted(x.name for x in (RES/'baseline').iterdir()))\n", "else:\n", " RES = None\n", " print('HF data download skipped (custom mode). F1 类指标将不可用, count 任务不依赖标注.')" ] }, { "cell_type": "code", "execution_count": null, "id": "849e06bb", "metadata": {}, "outputs": [], "source": [ "# ============ 写入流式推理模块 (generate / custom 模式使用) ============\n", "import pathlib\n", "pathlib.Path('proactvl_stream_infer.py').write_text(r'''\n", "#!/usr/bin/env python3\n", "\"\"\"Proact-VL-style streaming inference with off-the-shelf VLMs (Colab friendly).\n", "\n", "The official Proact-VL checkpoint is currently restricted, so this module\n", "approximates its proactive streaming behavior (a per-second SPEAK/SILENCE\n", "decision plus a short commentary line, like `MultiAssistantStreamInference.infer_one_chunk`)\n", "using a Qwen2.5-VL / Qwen3-VL backbone.\n", "\n", "Two tasks:\n", " --task commentary (default)\n", " Per-second proactive commentary. Silent seconds are omitted from `pred`\n", " (matching the official quickstart), which `f1_only` / `pauc` treat as\n", " silence. Output follows the official evaluation JSONL format so the\n", " results can be scored with `proactvl_eval` / the official scripts.\n", " --task count\n", " Per-second event detection: the VLM decides each second whether a NEW\n", " item (e.g. a piece of clothing) is being dropped into the machine,\n", " with client-side merge dedup. Output record adds `events`, `count`,\n", " and optionally `gt_count` for immediate accuracy check.\n", " Use this for 衣物投放视频 (\"how many clothes did the user drop\").\n", "\n", "Output record (one line per video segment):\n", " {\"video_id\": \"myclip.mp4\", \"dataset_name\": \"custom\", \"begin\": 0, \"end\": 60,\n", " \"idx\": 0, \"tag\": \"custom\", \"model_id\": \"...\", \"task\": \"count\",\n", " \"scene\": \"衣物投放\", \"pred\": {\"5\": \"DROP: red shirt\", \"12\": \"DROP: blue jeans\"},\n", " \"events\": [{\"sec\": 5, \"desc\": \"red shirt\"}, {\"sec\": 12, \"desc\": \"blue jeans\"}],\n", " \"count\": 2, \"gt_count\": 3}\n", "\n", "Examples:\n", " # single video, count dropped clothing items\n", " python proactvl_stream_infer.py --video clip.mp4 --task count \\\n", " --begin 0 --end 60 --gt_count 3 --out count.jsonl\n", " # Proact-VL-style commentary (official eval pipeline input)\n", " python proactvl_stream_infer.py --video clip.mp4 --task commentary \\\n", " --begin 0 --end 30 --tag \"Solo commentators\" --out pred.jsonl\n", " # LiveGamingBenchmark segments\n", " python proactvl_stream_infer.py --dataset oaaoaa/LiveGamingBenchmark \\\n", " --test_name all_in_one --video_dir ./data/LiveGamingBenchmark/DATA \\\n", " --num_samples 5 --out pred.jsonl\n", "\n", "Hardware note: the 3B model fits a 16GB GPU in fp16/bf16; 7B/8B models need\n", "`--load_in_4bit` on Colab T4/P100.\n", "\"\"\"\n", "\n", "import argparse\n", "import json\n", "import os\n", "from dataclasses import dataclass\n", "from pathlib import Path\n", "\n", "COMMENTARY_SYSTEM_PROMPT = (\n", " \"You are a live commentator for a game video. \"\n", " \"Your role is to independently analyze and narrate the game, delivering \"\n", " \"insightful, engaging, and natural commentary just like a human expert. \"\n", " \"It is not necessary to speak continuously; during uneventful or \"\n", " \"transitional parts, remain silent.\"\n", ")\n", "\n", "COUNT_SYSTEM_PROMPT = (\n", " \"You are a careful video analyst counting physical objects being placed \"\n", " \"into a machine or container in real time. You watch a live stream \"\n", " \"second by second and detect the exact moment each NEW object is dropped. \"\n", " \"An object already in progress from the previous second is not a new one.\"\n", ")\n", "\n", "DEFAULT_COUNT_SCENE = \"衣物投放 (clothing items being dropped into a washing machine / drop bin)\"\n", "\n", "\n", "@dataclass\n", "class InferenceConfig:\n", " model: str = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n", " device: str = \"cuda\"\n", " dtype: str = \"auto\" # auto | float16 | bfloat16\n", " load_in_4bit: bool = False\n", " task: str = \"commentary\" # commentary | count\n", " scene: str = \"\" # used for count task, e.g. 衣物投放\n", " merge_gap: int = 1 # event seconds within this gap are merged (dedup)\n", " gt_count: int = -1 # -1 = unknown; else used for accuracy check\n", " frames_per_sec: int = 2\n", " max_new_tokens: int = 12 # mirrors the official generate_config\n", " temperature: float = 0.7\n", " top_p: float = 0.9\n", " repetition_penalty: float = 1.15\n", " do_sample: bool = True\n", " history_lines: int = 8 # spoken lines kept in context (commentary)\n", " model_id: str = \"stream_infer\"\n", "\n", "\n", "def sample_frames(video_path: str, begin: int, end: int, fps: int,\n", " out_hw=(288, 512)) -> dict:\n", " \"\"\"Return {second: [PIL.Image, ...]} for t in [begin, end).\"\"\"\n", " from PIL import Image\n", "\n", " try:\n", " import decord\n", " vr = decord.VideoReader(video_path, num_threads=1)\n", " fps_video = vr.get_avg_fps()\n", " frames = {}\n", " for t in range(begin, end):\n", " i0 = int(t * fps_video)\n", " idxs = [min(i, len(vr) - 1) for i in range(i0, i0 + fps)]\n", " frames[t] = [Image.fromarray(vr[i].asnumpy()).convert(\"RGB\").resize(out_hw)\n", " for i in idxs]\n", " return frames\n", " except ImportError:\n", " import cv2\n", " cap = cv2.VideoCapture(video_path)\n", " fps_video = cap.get(cv2.CAP_PROP_FPS) or 30.0\n", " frames = {}\n", " for t in range(begin, end):\n", " imgs = []\n", " for k in range(fps):\n", " cap.set(cv2.CAP_PROP_POS_FRAMES, int((t + (k + 0.5) / fps) * fps_video))\n", " ok, fr = cap.read()\n", " if ok:\n", " fr = cv2.resize(fr, out_hw)\n", " imgs.append(Image.fromarray(fr[:, :, ::-1]).convert(\"RGB\"))\n", " frames[t] = imgs\n", " cap.release()\n", " return frames\n", "\n", "\n", "class OpenVLStreamInfer:\n", " \"\"\"Per-second proactive decisions using an open VLM as the decision head.\"\"\"\n", "\n", " def __init__(self, cfg: InferenceConfig):\n", " self.cfg = cfg\n", " import torch\n", " from transformers import AutoProcessor, AutoModelForVision2Seq\n", "\n", " self.torch = torch\n", " kwargs = {}\n", " if not cfg.load_in_4bit:\n", " kwargs[\"torch_dtype\"] = cfg.dtype\n", " kwargs[\"attn_implementation\"] = \"sdpa\"\n", " else:\n", " from transformers import BitsAndBytesConfig\n", " kwargs[\"quantization_config\"] = BitsAndBytesConfig(\n", " load_in_4bit=True, bnb_4bit_compute_dtype=\"float16\")\n", " self.model = AutoModelForVision2Seq.from_pretrained(cfg.model, **kwargs)\n", " self.model = self.model.to(cfg.device).eval()\n", " self.processor = AutoProcessor.from_pretrained(cfg.model)\n", "\n", " # ---------- commentary task ----------\n", "\n", " def _history_text(self, spoken: list) -> str:\n", " if not spoken:\n", " return \"(no prior commentary yet)\"\n", " return \"\\n\".join(f\"[t={t}] {txt}\" for t, txt in spoken[-self.cfg.history_lines:])\n", "\n", " def chunk(self, second: int, imgs, spoken: list) -> str:\n", " \"\"\"Return commentary text, or '' for silence.\"\"\"\n", " cfg = self.cfg\n", " prompt = (\n", " f\"Video stream up to second {second}:\\n\"\n", " f\"Prior commentary:\\n{self._history_text(spoken)}\\n\"\n", " f\"Current frames for second {second} are attached below.\\n\"\n", " \"Decide now: if something notable is happening in these frames \"\n", " \"and you have not just said it, reply with one short commentary \"\n", " \"line (max ~12 words). Otherwise reply exactly SILENCE.\"\n", " )\n", " messages = [\n", " {\"role\": \"system\", \"content\": COMMENTARY_SYSTEM_PROMPT},\n", " {\"role\": \"user\", \"content\": [\n", " *[{\"type\": \"image\", \"image\": im} for im in imgs],\n", " {\"type\": \"text\", \"text\": prompt},\n", " ]},\n", " ]\n", " inputs = self._generate(messages)\n", " text = self._decode(inputs)\n", " if text.upper().startswith(\"SILENCE\") or not text:\n", " return \"\"\n", " return text\n", "\n", " # ---------- count task ----------\n", "\n", " def chunk_event(self, second: int, imgs, events: list) -> str:\n", " \"\"\"Return event description ('DROP: ...' style), or '' for no new event.\"\"\"\n", " cfg = self.cfg\n", " scene = cfg.scene or DEFAULT_COUNT_SCENE\n", " secs = \", \".join(str(e[\"sec\"]) for e in events) if events else \"none\"\n", " prompt = (\n", " f\"You are watching a live video of {scene}.\\n\"\n", " f\"New items confirmed so far: {len(events)} (at seconds: {secs}).\\n\"\n", " f\"Attached frames show second {second} of the stream.\\n\"\n", " \"If a NEW item is clearly being dropped/placed in during second \"\n", " f\"{second} (it was not already in progress in the previous second), \"\n", " \"respond in exactly this format:\\n\"\n", " \"DROP: \\n\"\n", " \"If no new item is dropped in this second, respond exactly:\\n\"\n", " \"NONE\"\n", " )\n", " messages = [\n", " {\"role\": \"system\", \"content\": COUNT_SYSTEM_PROMPT},\n", " {\"role\": \"user\", \"content\": [\n", " *[{\"type\": \"image\", \"image\": im} for im in imgs],\n", " {\"type\": \"text\", \"text\": prompt},\n", " ]},\n", " ]\n", " inputs = self._generate(messages)\n", " text = self._decode(inputs).strip()\n", " if \"DROP\" in text.upper():\n", " desc = text.split(\":\", 1)[1].strip() if \":\" in text else text\n", " return desc\n", " return \"\"\n", "\n", " # ---------- shared ----------\n", "\n", " def _generate(self, messages: list):\n", " cfg = self.cfg\n", " inputs = self.processor.apply_chat_template(\n", " messages, add_generation_prompt=True, tokenize=True,\n", " return_dict=True, return_tensors=\"pt\")\n", " inputs = {k: v.to(cfg.device) for k, v in inputs.items()}\n", " with self.torch.inference_mode():\n", " out = self.model.generate(\n", " **inputs, max_new_tokens=cfg.max_new_tokens,\n", " do_sample=cfg.do_sample, temperature=cfg.temperature,\n", " top_p=cfg.top_p, repetition_penalty=cfg.repetition_penalty)\n", " return inputs, out\n", "\n", " def _decode(self, generated: tuple) -> str:\n", " inputs, out = generated\n", " gen = out[0][inputs[\"input_ids\"].shape[1]:]\n", " return self.processor.decode(gen, skip_special_tokens=True).strip()\n", "\n", " def run_video(self, video_path: str, begin: int, end: int) -> dict:\n", " \"\"\"Run the configured task over [begin, end).\n", "\n", " commentary -> {\"pred\": {sec: text}}\n", " count -> {\"pred\": {sec: \"DROP: ...\"}, \"events\": [...], \"count\": int}\n", " \"\"\"\n", " frames = sample_frames(video_path, begin, end, self.cfg.frames_per_sec)\n", " out: dict = {}\n", " if self.cfg.task == \"count\":\n", " events = []\n", " pred = {}\n", " for t in range(begin, end):\n", " desc = self.chunk_event(t, frames[t], events)\n", " if desc.upper().startswith('DROP:'):\n", " desc = desc.split(':', 1)[1].strip()\n", " if desc:\n", " last = events[-1][\"sec\"] if events else -10**9\n", " if t - last <= self.cfg.merge_gap:\n", " # continuation of the same drop motion: merge\n", " events[-1][\"desc\"] = desc\n", " pred[str(t)] = f\"DROP: {desc}\"\n", " print(f\"[Sec {t}]: (merged) DROP: {desc}\")\n", " continue\n", " events.append({\"sec\": t, \"desc\": desc})\n", " pred[str(t)] = f\"DROP: {desc}\"\n", " print(f\"[Sec {t}]: DROP: {desc} (total {len(events)})\")\n", " else:\n", " print(f\"[Sec {t}]: \")\n", " out[\"pred\"] = pred\n", " out[\"events\"] = events\n", " out[\"count\"] = len(events)\n", " if self.cfg.gt_count >= 0:\n", " out[\"gt_count\"] = self.cfg.gt_count\n", " out[\"count_correct\"] = len(events) == self.cfg.gt_count\n", " print(f\"Predicted {len(events)}, ground truth {self.cfg.gt_count} -> \"\n", " f\"{'CORRECT' if out['count_correct'] else 'MISMATCH'}\")\n", " else:\n", " pred = {}\n", " spoken = []\n", " for t in range(begin, end):\n", " txt = self.chunk(t, frames[t], spoken)\n", " if txt:\n", " pred[str(t)] = txt\n", " spoken.append((t, txt))\n", " print(f\"[Sec {t}]: {txt if txt else ''}\")\n", " out[\"pred\"] = pred\n", " return out\n", "\n", "\n", "def iter_records(args, infer: OpenVLStreamInfer):\n", " \"\"\"Yield prediction/analysis records (official eval JSONL compatible).\"\"\"\n", " if args.dataset:\n", " from datasets import load_dataset\n", " ds = load_dataset(args.dataset, name=args.test_name, split=\"test\")\n", " n = min(args.num_samples, len(ds)) if args.num_samples > 0 else len(ds)\n", " for idx in range(n):\n", " rec = ds[idx]\n", " video_path = os.path.join(args.video_dir, rec[\"video_path\"])\n", " result = infer.run_video(video_path, rec[\"video_begin\"], rec[\"video_end\"])\n", " yield {\n", " \"video_id\": os.path.basename(video_path),\n", " \"dataset_name\": rec[\"dataset_name\"],\n", " \"begin\": rec[\"video_begin\"],\n", " \"end\": rec[\"video_end\"],\n", " \"idx\": rec.get(\"idx\", idx),\n", " \"tag\": rec.get(\"tag\", \"\"),\n", " \"model_id\": args.model_id,\n", " \"task\": args.task,\n", " **result,\n", " }\n", " else:\n", " result = infer.run_video(args.video, args.begin, args.end)\n", " yield {\n", " \"video_id\": os.path.basename(args.video),\n", " \"dataset_name\": args.dataset_name,\n", " \"begin\": args.begin,\n", " \"end\": args.end,\n", " \"idx\": 0,\n", " \"tag\": args.tag,\n", " \"model_id\": args.model_id,\n", " \"task\": args.task,\n", " **result,\n", " }\n", "\n", "\n", "def _cuda_available() -> bool:\n", " try:\n", " import torch\n", " return torch.cuda.is_available()\n", " except ImportError:\n", " return False\n", "\n", "\n", "def parse_args():\n", " p = argparse.ArgumentParser(description=__doc__,\n", " formatter_class=argparse.RawDescriptionHelpFormatter)\n", " p.add_argument(\"--model\", default=\"Qwen/Qwen2.5-VL-3B-Instruct\")\n", " p.add_argument(\"--model_id\", default=None, help=\"Label written into outputs\")\n", " p.add_argument(\"--device\", default=None)\n", " p.add_argument(\"--dtype\", default=\"auto\", choices=[\"auto\", \"float16\", \"bfloat16\"])\n", " p.add_argument(\"--load_in_4bit\", action=\"store_true\")\n", " p.add_argument(\"--fps\", type=int, default=2, dest=\"frames_per_sec\")\n", " p.add_argument(\"--history_lines\", type=int, default=8)\n", " p.add_argument(\"--task\", default=\"commentary\", choices=[\"commentary\", \"count\"])\n", " p.add_argument(\"--scene\", default=\"\", help=\"Scene description for count task, e.g. 衣物投放\")\n", " p.add_argument(\"--merge_gap\", type=int, default=1,\n", " help=\"Drops within this many seconds are merged as one event\")\n", " p.add_argument(\"--gt_count\", type=int, default=-1,\n", " help=\"Ground-truth count; -1 to skip comparison\")\n", " p.add_argument(\"--out\", required=True)\n", " g = p.add_argument_group(\"input: single video\")\n", " g.add_argument(\"--video\")\n", " g.add_argument(\"--begin\", type=int, default=0)\n", " g.add_argument(\"--end\", type=int, default=30)\n", " g.add_argument(\"--tag\", default=\"Solo commentators\")\n", " g.add_argument(\"--dataset_name\", default=\"custom\")\n", " g = p.add_argument_group(\"input: LiveGamingBenchmark segments\")\n", " g.add_argument(\"--dataset\", default=None, help=\"HF dataset, e.g. oaaoaa/LiveGamingBenchmark\")\n", " g.add_argument(\"--test_name\", default=\"all_in_one\")\n", " g.add_argument(\"--video_dir\", default=None)\n", " g.add_argument(\"--num_samples\", type=int, default=5)\n", " return p.parse_args()\n", "\n", "\n", "def main():\n", " args = parse_args()\n", " if not args.video and not args.dataset:\n", " raise SystemExit(\"provide --video ... or --dataset ...\")\n", " if args.model_id is None:\n", " args.model_id = os.path.basename(args.model) + \"_stream\"\n", "\n", " cfg = InferenceConfig(\n", " model=args.model,\n", " device=args.device or (\"cuda\" if _cuda_available() else \"cpu\"),\n", " dtype=args.dtype,\n", " load_in_4bit=args.load_in_4bit,\n", " task=args.task,\n", " scene=args.scene,\n", " merge_gap=args.merge_gap,\n", " gt_count=args.gt_count,\n", " frames_per_sec=args.frames_per_sec,\n", " history_lines=args.history_lines,\n", " model_id=args.model_id,\n", " )\n", " infer = OpenVLStreamInfer(cfg)\n", " records = list(iter_records(args, infer))\n", " out = Path(args.out)\n", " out.parent.mkdir(parents=True, exist_ok=True)\n", " with open(out, \"w\", encoding=\"utf-8\") as f:\n", " for rec in records:\n", " f.write(json.dumps(rec, ensure_ascii=False) + \"\\n\")\n", " print(f\"Wrote {len(records)} record(s) to {out}\")\n", " for rec in records:\n", " if rec.get(\"task\") == \"count\":\n", " print(f\" {rec['video_id']}: predicted count = {rec['count']}\"\n", " + (f\" (gt={rec['gt_count']}, correct={rec['count_correct']})\"\n", " if \"gt_count\" in rec else \"\"))\n", "\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "\n", "'''.lstrip())\n", "print('proactvl_stream_infer.py written:', pathlib.Path('proactvl_stream_infer.py').stat().st_size, 'bytes')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ Mode A: 确定性指标 (F1 / F1+TimeDiff), 无需 API ============\n", "if MODE == 'baseline':\n", " REF = str(RES / 'anns/all_in_one_val_proactive.jsonl')\n", " PRED = str(RES / 'baseline' / BASELINE_FILE)\n", " os.makedirs('eval_out', exist_ok=True)\n", " !python -m evaluation.gaming.f1_only {PRED} --reference {REF} --output eval_out/f1.json --verbose\n", " !python -m evaluation.gaming.f1_timediff {PRED} --reference {REF} --output eval_out/f1_timediff.json --alpha 0.2 --verbose\n", "else:\n", " print('skipped (MODE =', MODE, ')')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ Mode A: LLM 指标 (需要 OPENAI_API_KEY; 无 key 则跳过) ============\n", "if MODE == 'baseline' and not OPENAI_API_KEY:\n", " print('skip LLM metrics (no API key)')\n", "elif MODE == 'baseline':\n", " ASR = str(RES / 'anns/all_in_one.jsonl')\n", " BASE_CC = str(RES / 'baseline/captions/gemini-2.5-pro.jsonl')\n", " # LLM Score: LiveU / FinalQ (每样本 2 次 judge 调用)\n", " !python -m evaluation.gaming.llm_score --model_id {BASELINE_ID} --prediction_jsonl {PRED} --asr_jsonl {ASR} --output_dir eval_out/llm_score --num_workers 8\n", " # CC win-rate: 与 gemini-2.5-pro baseline 比较 (每样本 4 次调用)\n", " !python -m evaluation.gaming.llm_judge --model_id {BASELINE_ID} --prediction_jsonl {PRED} --num_workers 8 --baseline_id gemini2.5-pro --baseline_jsonl {BASE_CC} --asr_jsonl {ASR} --output_dir eval_out/cc\n", " # PAUC (请求数受 JUDGE_LIMIT 约束)\n", " !python -m evaluation.gaming.pauc --func one_step --pred_fname {PRED} --gold_fname {REF} --output_fname eval_out/pauc/pauc_{BASELINE_ID}.json --openai_model {JUDGE_MODEL} --concurrency 8 --start_score 0 --judge_limit {JUDGE_LIMIT} --resume\n", " print('LLM metrics done')\n", "else:\n", " print('skipped (MODE =', MODE, ')')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ Mode B: 下载基准集 (标注 + 选定视频) + 生成 + 评估 ============\n", "if MODE == 'generate':\n", " from huggingface_hub import snapshot_download\n", " snapshot_download(repo_id='oaaoaa/LiveGamingBenchmark', repo_type='dataset',\n", " local_dir='./data/LiveGamingBenchmark',\n", " allow_patterns=VIDEO_INCLUDE, token=os.environ['HF_TOKEN'])\n", " extra = ' --load_in_4bit' if LOAD_IN_4BIT else ''\n", " # 逐秒流式解说生成 (T4/P100 可跑 3B; 60s 视频约 3-8 分钟)\n", " !python proactvl_stream_infer.py --model {VL_MODEL}{extra} --dataset oaaoaa/LiveGamingBenchmark --test_name all_in_one --video_dir ./data/LiveGamingBenchmark/DATA --num_samples {NUM_SAMPLES} --model_id {os.path.basename(VL_MODEL)}_stream --out pred_stream.jsonl\n", " PRED_GEN = 'pred_stream.jsonl'\n", " REF = str(RES / 'anns/all_in_one_val_proactive.jsonl')\n", " os.makedirs('eval_out', exist_ok=True)\n", " !python -m evaluation.gaming.f1_only {PRED_GEN} --reference {REF} --output eval_out/f1_gen.json --verbose\n", " !python -m evaluation.gaming.f1_timediff {PRED_GEN} --reference {REF} --output eval_out/f1_timediff_gen.json --alpha 0.2 --verbose\n", " if OPENAI_API_KEY:\n", " ASR_GEN = str(RES / 'anns/all_in_one.jsonl')\n", " !python -m evaluation.gaming.llm_score --model_id stream_gen --prediction_jsonl {PRED_GEN} --asr_jsonl {ASR_GEN} --output_dir eval_out/llm_score_gen --num_workers 8\n", " print('Mode B done')\n", "else:\n", " print('skipped (MODE =', MODE, ')')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ Mode C: 上传自己的视频 (衣物投放计数) ============\n", "if MODE == 'custom':\n", " from google.colab import files\n", " up = files.upload() # 弹出文件选择框, 拖入视频 (mp4/mov)\n", " if up:\n", " MY_VIDEO = list(up.keys())[0]\n", " print('uploaded:', MY_VIDEO, '-> /content/' + MY_VIDEO)\n", " else:\n", " MY_VIDEO = input('或输入已存在的视频路径 (如 /content/myvideo.mp4): ')\n", " print('using local video:', MY_VIDEO)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Mode C: 逐秒流式分析 (count: 投了几件; commentary: 逐秒解说)\n", "if MODE == 'custom':\n", " extra = ' --load_in_4bit' if LOAD_IN_4BIT else ''\n", " gt = f' --gt_count {MY_GT_COUNT}' if MY_GT_COUNT >= 0 else ''\n", " !python proactvl_stream_infer.py --model {VL_MODEL}{extra} --video \"{MY_VIDEO}\" --task {MY_TASK} --scene \"{MY_SCENE}\" --begin {MY_BEGIN} --end {MY_END} --merge_gap {MY_MERGE_GAP}{gt} --model_id {os.path.basename(VL_MODEL)}_custom --out custom_result.jsonl\n", " import json\n", " rec = json.loads(open('custom_result.jsonl').readline())\n", " if MY_TASK == 'count':\n", " print('\\n===== 衣物投放计数结果 =====')\n", " print('预测件数:', rec['count'])\n", " for e in rec['events']:\n", " print(f\" [t={e['sec']}s] {e['desc']}\")\n", " if 'gt_count' in rec:\n", " print('实际件数:', rec['gt_count'], '->', '正确' if rec['count_correct'] else '不符')\n", " print('(完整结果见 custom_result.jsonl; pred 字段为逐秒事件, 可接官方 F1/PAUC 流程)')\n", " else:\n", " print('逐秒解说:', rec['pred'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ============ 汇总 ============\n", "import json, statistics, pandas as pd\n", "\n", "def f1line(path):\n", " try:\n", " d = json.load(open(path))\n", " m = d['overall']['overlap_micro']; t = d['overall'].get('timediff')\n", " row = {'F1_micro%': round(m['f1'] * 100, 2), 'P%': round(m['precision'] * 100, 1),\n", " 'R%': round(m['recall'] * 100, 1), 'n': m['sample_count']}\n", " if t: row['TimeDiff_avg'] = round(t['avg'], 3)\n", " return row\n", " except FileNotFoundError:\n", " return None\n", "\n", "rows = []\n", "for name, path in [('f1', 'eval_out/f1.json'), ('f1_timediff', 'eval_out/f1_timediff.json'),\n", " ('f1_gen', 'eval_out/f1_gen.json'), ('f1_timediff_gen', 'eval_out/f1_timediff_gen.json')]:\n", " r = f1line(path)\n", " if r:\n", " rows.append({'metric': name, **r})\n", "F = {'liveu': ['Time', 'Rate', 'TextU'], 'finalq': ['Fidelity', 'Continuity', 'Substance']}\n", "for name, fields in F.items():\n", " for suffix, p in [(BASELINE_ID, f'eval_out/llm_score/llm_score_{name}_{BASELINE_ID}.jsonl'),\n", " ('stream_gen', f'eval_out/llm_score_gen/llm_score_{name}_stream_gen.jsonl')]:\n", " try:\n", " vals = [json.loads(l) for l in open(p) if l.strip()]\n", " for k in fields:\n", " v = [x[f'{name}_score'][k] for x in vals if k in x.get(f'{name}_score', {})]\n", " rows.append({'metric': f'{name}:{k}({suffix})', 'mean': round(statistics.mean(v), 2) if v else None, 'n': len(v)})\n", " except FileNotFoundError:\n", " pass\n", "try:\n", " crec = json.loads(open('custom_result.jsonl').readline())\n", " if crec.get('task') == 'count':\n", " rows.append({'metric': 'custom_count', 'predicted': crec['count'],\n", " 'gt': crec.get('gt_count', None),\n", " 'correct': crec.get('count_correct')})\n", "except FileNotFoundError:\n", " pass\n", "df = pd.DataFrame(rows)\n", "if len(df):\n", " print(df.to_string(index=False))\n", "else:\n", " print('No results found yet.')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 指标说明\n", "\n", "| 指标 | 含义 | 脚本 | 需要 API |\n", "|---|---|---|---|\n", "| F1 (micro) | 逐秒\"该说话时刻\"的说话/沉默判定重合度 | `f1_only.py` | 否 |\n", "| TimeDiff | 说话时刻相对标注的时序偏差 (k=2, alpha=0.2) | `f1_timediff.py` | 否 |\n", "| LLM Score (LiveU) | 流式交付动态: Time / Rate / TextU, 1-10 分 | `llm_score.py` | 是 |\n", "| LLM Score (FinalQ) | 最终文案质量: Fidelity / Continuity / Substance | `llm_score.py` | 是 |\n", "| CC win-rate | 与 baseline 的 A/B 偏好胜率 | `llm_judge.py` | 是 |\n", "| PAUC | 主动预测的渐进式 AUC | `pauc.py` | 是 |\n", "\n", "**Mode C (自己的视频 / 衣物投放计数) 说明**\n", "\n", "- 上传方式三选一: notebook 内 `files.upload()` (本 notebook 已内置, 弹框选择); Colab 左侧 \"Files\" 面板拖入 `/content`; 或放 Drive 后 `!gdrive mount` 取路径。长视频建议先剪辑到 ≤60s。\n", "- `count` 任务: VLM 每秒判断是否有**新的**衣物投入 (帧率 2fps/秒), 输出 `DROP: 描述` 或 `NONE`; 客户端按 `MY_MERGE_GAP` 合并相邻秒的同一动作, 得到 `events` (时间线) 和 `count`。填 `MY_GT_COUNT` 会直接告诉你预测是否正确。\n", "- 计数是小目标 + 遮挡场景, 3B 模型可能漏计; 提高准确率的顺序: 调 `MY_MERGE_GAP=0` (不合并) 或 `--fps 4` (更密集采样, 更慢) 或换 7B (加 `LOAD_IN_4BIT=True`)。\n", "- 结果 jsonl 的 `pred` 字段是逐秒事件, 若你有人工标注 (每秒该不该有事件), 可照 Mode B 的流程接 F1/PAUC 评估。\n", "\n", "**其他注意**\n", "\n", "- 官方 Proact-VL 权重 (`oaaoaa/proactvl_base_*`) 目前受合规审查限制, 需邮件作者获取; Mode B/C 为开源 VLM 近似。\n", "- LLM 指标与 judge 模型强相关; 复现官方数字需其指定 judge (gpt-5.1 系列)。\n", "- 成本: `llm_score` 每样本 2 次 judge 调用, `cc` 4 次, `pauc` 受 `--judge_limit` 约束。\n", "- 流式长视频可继续用 `label_streaming2standard.py --mode pred` 切成 30s 段再评估 (官方 streaming 流程)。" ] } ], "metadata": { "accelerator": "GPU", "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }