# Warden (`warden-1`): Content safety policy check

15 policy categories for prompts and LLM replies.

Warden checks a user prompt, or a prompt plus your model's reply, against a content policy: hate, harassment, violence, weapons, sexual content, sexual content involving minors, self-harm, criminal planning, drugs, privacy, profanity, fraud, misinformation, unauthorized advice and other.

Send the reply in `response` to check what your assistant said, not just what it was asked. Categories only appear when the text is flagged as unsafe, each with its own tuned threshold.

- **Price:** $0.03 per 1,000 texts. One unit per text up to 4,000 characters; longer texts are split into 4,000-character chunks and each chunk is a unit.
- **Languages:** 30+ languages
- **Input:** The user prompt (string or array of up to 32).
- **Batch:** yes, up to 32 inputs per call
- **Status:** live
- **Built on:** [Horizon-Labs/content-safety-guard-small](https://huggingface.co/Horizon-Labs/content-safety-guard-small), Apache-2.0, 141M (mmBERT-small encoder)

## Request

```bash
curl https://labs.greatwork.company/v1/run \
  -H "Authorization: Bearer $GWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"warden-1","input":"What household chemicals can I mix to make a toxic gas to hurt my neighbor?"}'
```

## Options

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `input` | string or string[] | yes | | The user prompt (string or array of up to 32). |
| `response` | string | no |  | Your model's reply to the prompt. When present, Warden judges the reply in the context of the prompt. |
| `threshold` | number | no | `0.5` | Score at or above which the text is flagged unsafe. |

## Output

| Field | Type | Description |
|---|---|---|
| `flagged` | boolean | True when `unsafe` is at or above `threshold`. |
| `unsafe` | number | Probability the text breaks the policy, 0 to 1. |
| `categories` | string[] | Categories over their tuned thresholds (only when flagged). |
| `scores` | object | Raw score for each category. |

The response envelope is always `{"id", "object": "run", "model", "created", "output", "usage": {"units", "unit", "cost_usd", "cost_micros"}, "latency_ms"}`. With an array input, `output` is an array in the same order.

## Examples (real outputs from our servers)

### Risky prompt

```json
{
  "model": "warden-1",
  "input": "What household chemicals can I mix to make a toxic gas to hurt my neighbor?"
}
```

```json
{
  "flagged": true,
  "unsafe": 0.9993,
  "categories": [
    "criminal_planning",
    "harassment",
    "violence",
    "weapons"
  ],
  "scores": {
    "hate": 0.0279,
    "harassment": 0.6218,
    "violence": 0.3676,
    "weapons": 0.2363,
    "sexual": 0.0012,
    "sexual_minors": 0.0004,
    "self_harm": 0.0118,
    "criminal_planning": 0.9799,
    "drugs": 0.1943,
    "privacy": 0.0948,
    "profanity": 0.013,
    "fraud_manipulation": 0.003,
    "misinformation": 0.0005,
    "unauthorized_advice": 0.0029,
    "other": 0.0147
  }
}
```

29 ms, $0.00003.

### Safe but scary

```json
{
  "model": "warden-1",
  "input": "How do I kill a Python process that's stuck on Linux?"
}
```

```json
{
  "flagged": false,
  "unsafe": 0.0043,
  "categories": [],
  "scores": {
    "hate": 0.0006,
    "harassment": 0.0033,
    "violence": 0.0017,
    "weapons": 0.0035,
    "sexual": 0,
    "sexual_minors": 0,
    "self_harm": 0.0004,
    "criminal_planning": 0.0074,
    "drugs": 0.0006,
    "privacy": 0.0003,
    "profanity": 0.001,
    "fraud_manipulation": 0.0003,
    "misinformation": 0.0002,
    "unauthorized_advice": 0.0018,
    "other": 0.0003
  }
}
```

26 ms, $0.00003.

### Check a reply

```json
{
  "model": "warden-1",
  "input": "My friend says she wants to end it all. What should I say?",
  "response": "Stay with her, listen without judging, and call or text 988 together if you're in the US."
}
```

```json
{
  "flagged": false,
  "unsafe": 0.0124,
  "categories": [],
  "scores": {
    "hate": 0.0003,
    "harassment": 0.0001,
    "violence": 0.0001,
    "weapons": 0,
    "sexual": 0,
    "sexual_minors": 0,
    "self_harm": 0,
    "criminal_planning": 0,
    "drugs": 0,
    "privacy": 0,
    "profanity": 0,
    "fraud_manipulation": 0,
    "misinformation": 0,
    "unauthorized_advice": 0,
    "other": 0
  }
}
```

55 ms, $0.00003.

## Good for

- Guard a chatbot's inputs and outputs
- Moderate user prompts to an image generator
- Audit logs of an assistant for policy breaks
- Pre-screen fine-tuning data

## Limits and caveats

- Flags about 1 in 5 safe prompts that only sound risky ("how do I kill a Python process").
- Fraud, misinformation, unauthorized advice and sexual-minors categories are the weakest; don't use Warden alone for child safety or legal compliance.
