1University of Cincinnati · 2Voxel51 · 3KAUST · *Work done during an internship at KAUST
Figure 1. Framework Overview. X-CASE evaluates generative agents across three dimensions: safety improvement over time via iterative plan revision, detection of unsafe activities across social contexts, and social dynamics through interaction and acceptance rates. Five agents (PR, KS, JS, CH, AV) operate in a shared virtual environment, each managing hourly social activity plans.
Unsafe activities are accepted when paired with misleading visual cues; agents over-rely on image context even when it contradicts stated safety risks.
GPT-4o-mini averages 55% unsafe-to-safe conversion. The lowest of the three models. Qwen-VL (58%) outperforms it despite being an open-source model.
Claude 3.5 Sonnet achieves the highest average unsafe-to-safe conversion rate (75%) across all scenarios, reducing unsafe activities per plan from 7.8 to 2.1.
Fire/Heat scenarios with Claude 3.5 Sonnet achieve a 98% correction rate, the highest observed across all model-scenario combinations.
Qwen-VL-2B-Instruct, the open-source model, averages 58% conversion but delays corrections until late in the simulation (around step 450).
Multi-risk "Risk Mix" scenarios are the hardest: GPT-4o-mini achieves only 20% correction rate when multiple risk categories overlap.
1,000 social activity scenarios spanning 21 hazard categories and 192 subcategories. Every scenario includes a paired unsafe/safe hourly plan (7 PM – 5 AM, 11 steps) with CLIP-verified images.
Figure 2. Dataset Construction Pipeline. (1) An LLM generates social scenario descriptions from 192 subcategories. (2) Hourly unsafe plans are generated and rewritten as safe counterparts. (3) Images are retrieved via Pexels API and verified with CLIP (ViT-L/14, cosine ≥ 0.35). (4) All steps are human-verified by 4 annotators.
Each scenario comes with a real image, an unsafe hourly plan, and its safe rewrite; paired step by step.
1 / 8 · use arrows or swipe
GPT-4o-mini generates social activity descriptions across 21 hazard categories and 192 subcategories with an unsafe hourly plan.
Each unsafe activity is rewritten activity-by-activity to eliminate risks while preserving the social event's character and temporal structure.
Images are retrieved via Pexels API and scored with CLIP ViT-L/14 cosine similarity (threshold 0.35). Low-scoring images trigger up to 3 additional searches.
Four annotators (1 SE + 1 Master's student + 2 PhD researchers) verify every plan step. 100% of entries are human-reviewed.
5 generative agents run for 600 steps (10 hours). Every 50 steps, each agent enters a 3-stage plan revision cycle with a Judge Agent.
Unsafe-to-safe conversion rate, plan revision count, interaction counts, and acceptance rates are logged every 10 steps.
Simulation Demo. Formal Party Environment. Five generative agents navigate the party location over 600 steps (10 simulated hours, 7 PM – 5 AM). At every 50-step interval each agent's unsafe activities are flagged, revised, and verified by a Judge Agent before the plan is updated. The location is restricted to the party setting to ensure all 21 hazard categories appear in an ecologically valid social context.
Figure 3. Agent Architecture. Each agent maintains a memory stream (associative, spatial, scratch), and a Plan Revision Layer. Every 50 steps: (1) the agent assesses each activity against its current multimodal context, (2) proposes safe alternatives, and (3) a separate Judge Agent verifies safety before accepting revisions.
Successful Revision. Agent detects "jumping from rooftop into pool" as unsafe and revises to "relaxing by poolside." Judge Agent confirms.
Failed Revision. Agent incorrectly judges "sitting on rooftop edge" as safe. Judge Agent overrides and flags the activity, demonstrating local vs. global alignment failure.
Figure 4. Safety Improvement Over Time. Average unsafe activities per plan across 600 simulation steps (averaged over 100 runs). Claude 3.5 Sonnet achieves the largest reduction (7.8 → 2.1), stabilizing early. GPT-4o-mini shows gradual, consistent improvement (8.9 → 5.1). Qwen-VL-2B remains flat until step 450, then corrects sharply (9.8 → 5.2). None of the models fully eliminates unsafe actions.
Figure 5. Conversion Rate Heatmap. Percentage of unsafe-to-safe plan conversions across 8 risk scenarios × 5 agents × 3 models. Average conversion: Claude 75%, Qwen-VL 58%, GPT-4o-mini 55%; Qwen-VL outperforms GPT-4o-mini overall. Risk Mix (multi-risk) is the hardest scenario (GPT-4o-mini: 20%, Qwen-VL: 35%). Fire/Heat and Unsafe Sports achieve the highest rates across all models.
| Scenario | Claude 3.5 | GPT-4o-mini | Qwen-VL-2B |
|---|---|---|---|
| Fire / Heat | 98% | 49% | 64% |
| Unsafe Sports | 95% | 84% | 84% |
| Collapse | 91% | 67% | 58% |
| Food / Drink | 85% | 53% | 60% |
| Risk Mix (multi-risk) | 87% | 20% | 35% |
| Sound / Vibration | 82% | 60% | 65% |
| Animals / Nature | 73% | 45% | 53% |
| Crowd Dynamics | 65% | 62% | 49% |
| Average | 75% | 55% | 58% |
Figure 6. Text-Only Ablation. Without visual context, agents show minimal safety improvement across 600 steps. Claude text-only partially improves early but plateaus at ~3 unsafe activities/plan. GPT-4o-mini and Qwen remain nearly flat throughout. Multimodal consistently outperforms text-only across all models, confirming the importance of visual grounding.