AI Testing · Prompt Regression · LLM QA

assert(prompt_output == expected)

PromptAssert catches prompt regressions before they reach production. Write test assertions for your LLM outputs — run them in CI, on every model update, on every prompt change.

Start Free →View on GitHub →

test_support.pypython

from promptassert import PromptAssert

pa = PromptAssert(model="gpt-4o")

@pa.test
def test_customer_support_tone():
    response = pa.call("You are a helpful assistant. Be professional.")
    pa.assert_contains(response, ["help", "assist"])
    pa.assert_not_contains(response, ["sorry", "unfortunately"])
    pa.assert_sentiment(response, "positive")
    pa.assert_max_tokens(response, 200)

✓$ promptassert run tests/

PASS test_customer_support_tone

PASS test_refund_policy_accuracy

FAIL test_escalation_tone — assert_not_contains failed: found "unfortunately"

3 tests · 2 passed · 1 failed · 4.2s

How it works

Same mental model as unit tests. Applied to prompts.

Write assertions

Define expected behaviors for your prompts using familiar test syntax. If it should always be professional, assert it. If it should never say sorry, assert that too.

Run in CI

Execute your prompt test suite on every commit, every model update, every prompt change. Integrate with GitHub Actions in two lines of YAML.

Catch regressions

Get instant alerts when LLM behavior drifts from what you expect. See exactly which assertion failed, why, and what the model returned.

Everything you need to test AI behavior

Rich assertion primitives, provider-agnostic runtime, built for the CI pipelines you already use.

✓

Provider agnostic

Works with OpenAI, Anthropic, Google, Cohere, or any LLM with a compatible API.

⚙

CI/CD ready

GitHub Actions, GitLab CI, Jenkins, CircleCI — run your prompt suite anywhere in the pipeline.

∿

Rich assertions

Semantic similarity, sentiment scoring, regex matching, contains/not-contains, token limits.

⬡

Snapshot testing

Baseline your outputs and automatically alert on meaningful behavioral drift across versions.

⇄

Model comparison

Run the same test suite against gpt-4o, claude-3-5-sonnet, gemini-pro — see which model passes.

⚡

Zero-config start

pip install promptassert. Write your first test in under 5 minutes. No infrastructure needed.

Simple, honest pricing

Start free. Scale when you need it.

Free

For solo developers and side projects testing prompt quality.

Start free →

✓1,000 assertions / month
✓1 project
✓All assertion types
✓CLI + Python SDK
✓Community support
✗Slack / webhook alerts
✗Team dashboard
✗SSO

Your prompts deserve tests too.

Start with PromptAssert free — no credit card required. Up and running in 5 minutes.

Start Free →Compare plans