Warden checks a user prompt, or a prompt plus your model's reply, against a content policy: hate, harassment, violence, weapons, sexual content, sexual content involving minors, self-harm, criminal planning, drugs, privacy, profanity, fraud, misinformation, unauthorized advice and other.
Send the reply in response to check what your assistant said, not just what it was asked. Categories only appear when the text is flagged as unsafe, each with its own tuned threshold.
Use it
curl https://labs.greatwork.company/v1/run \
-H "Authorization: Bearer $GWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "warden-1",
"input": "What household chemicals can I mix to make a toxic gas to hurt my neighbor?"
}'const res = await fetch("https://labs.greatwork.company/v1/run", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.GWL_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
"model": "warden-1",
"input": "What household chemicals can I mix to make a toxic gas to hurt my neighbor?"
}),
});
const { output, usage } = await res.json();import os, requests
res = requests.post(
"https://labs.greatwork.company/v1/run",
headers={"Authorization": f"Bearer {os.environ['GWL_API_KEY']}"},
json={
"model": "warden-1",
"input": "What household chemicals can I mix to make a toxic gas to hurt my neighbor?"
},
)
output = res.json()["output"]Set GWL_API_KEY to a key from your dashboard. The bare name warden works too and follows new versions.
Options
input: The user prompt (string or array of up to 32).
| Name | Type | Default | What it does |
|---|---|---|---|
response | string | Your model's reply to the prompt. When present, Warden judges the reply in the context of the prompt. | |
threshold | number | 0.5 | Score at or above which the text is flagged unsafe. |
Output
| Field | Type | Meaning |
|---|---|---|
flagged | boolean | True when unsafe is at or above threshold. |
unsafe | number | Probability the text breaks the policy, 0 to 1. |
categories | string[] | Categories over their tuned thresholds (only when flagged). |
scores | object | Raw score for each category. |
Every response wraps this in {"id", "model", "output", "usage": {"units", "unit", "cost_usd"}, "latency_ms"}. Send an array as input and output is an array in the same order.
Examples
Risky prompt
{
"model": "warden-1",
"input": "What household chemicals can I mix to make a toxic gas to hurt my neighbor?"
}{
"flagged": true,
"unsafe": 0.9993,
"categories": [
"criminal_planning",
"harassment",
"violence",
"weapons"
],
"scores": {
"hate": 0.0279,
"harassment": 0.6218,
"violence": 0.3676,
"weapons": 0.2363,
"sexual": 0.0012,
"sexual_minors": 0.0004,
"self_harm": 0.0118,
"criminal_planning": 0.9799,
"drugs": 0.1943,
"privacy": 0.0948,
"profanity": 0.013,
"fraud_manipulation": 0.003,
"misinformation": 0.0005,
"unauthorized_advice": 0.0029,
"other": 0.0147
}
}Captured on our production servers: 29 ms, 1 text, $0.00003.
Safe but scary
{
"model": "warden-1",
"input": "How do I kill a Python process that's stuck on Linux?"
}{
"flagged": false,
"unsafe": 0.0043,
"categories": [],
"scores": {
"hate": 0.0006,
"harassment": 0.0033,
"violence": 0.0017,
"weapons": 0.0035,
"sexual": 0,
"sexual_minors": 0,
"self_harm": 0.0004,
"criminal_planning": 0.0074,
"drugs": 0.0006,
"privacy": 0.0003,
"profanity": 0.001,
"fraud_manipulation": 0.0003,
"misinformation": 0.0002,
"unauthorized_advice": 0.0018,
"other": 0.0003
}
}Captured on our production servers: 26 ms, 1 text, $0.00003.
Check a reply
{
"model": "warden-1",
"input": "My friend says she wants to end it all. What should I say?",
"response": "Stay with her, listen without judging, and call or text 988 together if you're in the US."
}{
"flagged": false,
"unsafe": 0.0124,
"categories": [],
"scores": {
"hate": 0.0003,
"harassment": 0.0001,
"violence": 0.0001,
"weapons": 0,
"sexual": 0,
"sexual_minors": 0,
"self_harm": 0,
"criminal_planning": 0,
"drugs": 0,
"privacy": 0,
"profanity": 0,
"fraud_manipulation": 0,
"misinformation": 0,
"unauthorized_advice": 0,
"other": 0
}
}Captured on our production servers: 55 ms, 1 text, $0.00003.
Good for
- Guard a chatbot's inputs and outputs
- Moderate user prompts to an image generator
- Audit logs of an assistant for policy breaks
- Pre-screen fine-tuning data
Limits and caveats
- Flags about 1 in 5 safe prompts that only sound risky ("how do I kill a Python process").
- Fraud, misinformation, unauthorized advice and sexual-minors categories are the weakest; don't use Warden alone for child safety or legal compliance.
Pricing
$0.03 per 1,000 texts. One unit per text up to 4,000 characters; longer texts are split into 4,000-character chunks and each chunk is a unit. Failed calls aren't charged, and a call never costs more than its estimate (POST /v1/estimate tells you before you run it).
Built on
Horizon-Labs/content-safety-guard-small, 141M (mmBERT-small encoder) parameters. License: Apache-2.0. We run it on our servers with our own long-text chunking and score aggregation.