zen-guard/eval

Evaluation Instructions for Qwen3Guard on QwenGuardTest

This repo contains scripts to reproduce the evaluation results of Qwen3Guard-Gen and Qwen3Guard-Stream on the Qwen3GuardTest dataset.

The Qwen3GuardTest dataset is a benchmark used in evaluating Qwen3Guard. Distinct from existing safety guardrails benchmarks, it focuses on two emerging and underexplored scenarios:

1. Safety classification of reasoning-model outputs: As model architectures increasingly incorporate explicit long reasoning process, safety evaluation must extend beyond final answers to the reasoning process itself. Yet, benchmarks targeting the safety of intermediate reasoning steps remain scarce. To bridge this gap, we manually annotated responses, including internal reasoning traces, from open-source reasoning models, enabling guard models to assess the safety of the entire reasoning trajectory.

2. Streaming moderation evaluation: Qwen3Guard-stream introduces real-time, token-level moderation, empowering proactive intervention during generation. To evaluate streaming moderation performance, we provide human-annotated, sentence-level safety labels, supporting comprehensive assessment of both detection accuracy and timeliness (e.g., latency to first unsafe segment identification).

Results of Safety Classification on Responses with Thinking Content

Model Name	Type	F1
Qwen3Guard-Gen-0.6B	Generative	83.6
Qwen3Guard-Gen-4B	Generative	84.0
Qwen3Guard-Gen-8B	Generative	84.0
Qwen3Guard-Stream-0.6B	Stream	81.6
Qwen3Guard-Stream-4B	Stream	85.4
Qwen3Guard-Stream-8B	Stream	83.6

Results of Stream Response Detection Latency with Thinking Content

Model Name	exact hit (%)	hit in first 128 token (%)
Qwen3Guard-Stream-0.6B	20.91	78.38
Qwen3Guard-Stream-4B	21.62	66.78
Qwen3Guard-Stream-8B	4.7	64.67

Note that exact hit calculates the percentage of instances where the model's detection of unsafe content falls precisely within the start range of the content flagged as unsafe by human annotators. The hit in first 128 token metric calculates the percentage of samples where the model detects unsafe content at any point before the annotator-marked start of unsafe content, or within the 128 tokens after it.

Results of Stream Response Detection Latency without Thinking Content

Model Name	exact hit (%)	hit in first 128 token (%)
Qwen3Guard-Stream-0.6B	83.52	90.41
Qwen3Guard-Stream-4B	85.98	93.73
Qwen3Guard-Stream-8B	85.36	92.12

Python Environment

conda create -n eval python=3.10
conda activate eval
pip install transformers torch datasets accelerate

Download Dataset

The QwenGuardTest dataset can be downloaded from:

• 🤗 Hugging Face
• 🤖 ModelScope

The dataset is organized into three distinct splits:

• thinking: This split comprises 1,059 samples that include the responses with thinking. These were generated by prompting various "thinking" models with harmful prompts from the Beavertails test set.
• thinking_loc: A subset of the thinking split, this contains 569 samples, all of which are labeled as unsafe. Each sample is annotated with the precise start and end indices of the first unsafe sentence.
• response_loc: This split consists of 813 samples that contain only the final response, without the thinking process. Every sample in this split is labeled as unsafe and includes the start and end indices of the first unsafe sentence.

Evaluation for Qwen3Guard-Gen

Evaluation of Safety Classification on thinking subset

python eval_gen.py \
    --model_path your_local_model_path \
    --input_file your_local_dataset_dir \
    --output_file result_gen_4b.jsonl

# You can also directly specify the huggingface repository:

python eval_gen.py \
    --model_path Qwen/Qwen3Guard-Gen-4B \
    --input_file Qwen/Qwen3GuardTest \
    --output_file result_gen_4b.jsonl

Expected Output

After execution, the model predictions will be saved to result_gen_4b.jsonl. The script will also print Strict F1 to the console:

Recall: 0.7364
Precision: 0.9767
F1 Score: 0.8397

Evaluation for Qwen3Guard-Stream

1. Evaluation of Safety Classification on thinking subset

python eval_stream.py  \
   --model_path Qwen/Qwen3Guard-Stream-4B  \
   --input_path Qwen/Qwen3GuardTest     \
   --output_path result_stream_thinking_4b.jsonl     \
   --data_type response     \
   --split thinking     \
   --thinking

Expected Output

Calculating F1 score...
Unsafe F1 Score(strict): 0.8522. precision(strict): 0.9464. recall(strict): 0.7750
Unsafe F1 Score(loose): 0.8536. precision(loose): 0.9446. recall(loose): 0.7786

2. Evaluation of Stream Moderation on thinking_loc subset

python eval_stream.py \
    --model_path Qwen/Qwen3Guard-Stream-4B \
    --input_path Qwen/Qwen3GuardTest \
    --output_path result_stream_thinking_latency_4b.jsonl \
    --split thinking_loc \
    --data_type response \
    --eval_unsafe_latency \
    --thinking

Expected Output

Calculating F1 score...
Unsafe F1 Score(strict): 0.8733. precision(strict): 1.0000. recall(strict): 0.7750
Unsafe F1 Score(loose): 0.8755. precision(loose): 1.0000. recall(loose): 0.7786

Calculating unsafe latency...
Processed 569 unsafe samples.
Bins Count:  {'Ahead': 98, 'Hit': 123, '1-32': 96, '33-64': 41, '65-128': 22, '129-256': 32, '>256': 31, 'Safe': 126}
First 128 tokens stop rate:  0.6678383128295254
Exact hit rate:  0.21616871704745166

3. Evaluation of Stream Moderation on response_loc subset

python eval_stream.py \
    --model_path Qwen/Qwen3Guard-Stream-4B \
    --input_path Qwen/Qwen3GuardTest \
    --output_path result_stream_response_latency_4b.jsonl \
    --split response_loc \
    --data_type response \
    --eval_unsafe_latency

Expected Output

Calculating F1 score...
Unsafe F1 Score(strict): 0.9489. precision(strict): 1.0000. recall(strict): 0.9028
Unsafe F1 Score(loose): 0.9676. precision(loose): 1.0000. recall(loose): 0.9373

Calculating unsafe latency...
Processed 813 unsafe samples.
Bins Count:  {'Ahead': 39, 'Hit': 699, '1-32': 21, '33-64': 2, '65-128': 1, '129-256': 0, '>256': 0, 'Safe': 51}
First 128 tokens stop rate:  0.9372693726937269
Exact hit rate:  0.8597785977859779