How I work with AI agents, before you work with me.

providers:
  - id: openai:gpt-4o-mini
  - id: anthropic:claude-3-5-sonnet-20241022

tests:
  - description: Critical Incident
    vars:
      subject: Critical error in checkout flow
      body: Customer cannot complete order. Payment gateway returning errors. Urgent assistance required.
    prompt: |
      Classify the following support request by priority and category.
      Subject: {{subject}}
      Message: {{body}}
      Output: Priority (CRITICAL | HIGH | NORMAL) and Category (TECHNICAL | BILLING | FEATURE).
    assert:
      - type: llm-rubric
        value: Reliably identifies genuine production incidents (payment errors, system outages, data loss). Only returns CRITICAL for those, never for usability requests.
        threshold: 0.85
      - type: cost
        threshold: 0.003

  - description: Feature Request misclassified as urgent
    vars:
      subject: Idea for a mobile app
      body: It would be convenient if I could use your website from my mobile phone. Can you offer that?
    prompt: |
      Classify the following support request by priority and category.
      Subject: {{subject}}
      Message: {{body}}
      Output: Priority (CRITICAL | HIGH | NORMAL) and Category (TECHNICAL | BILLING | FEATURE).
    assert:
      - type: llm-rubric
        value: Distinguishes product wishes from real incidents. Returns NORMAL or HIGH (never CRITICAL) for feature requests.
        threshold: 0.85
      - type: cost
        threshold: 0.003

  - description: Billing Dispute at Medium Priority
    vars:
      subject: Incorrect invoice for last month
      body: I was double-charged. Please review my billing history and correct it.
    prompt: |
      Classify the following support request by priority and category.
      Subject: {{subject}}
      Message: {{body}}
      Output: Priority (CRITICAL | HIGH | NORMAL) and Category (TECHNICAL | BILLING | FEATURE).
    assert:
      - type: llm-rubric
        value: Recognizes billing issues and rates them HIGH (not CRITICAL) so genuine emergencies do not get blocked in the queue.
        threshold: 0.80
      - type: cost
        threshold: 0.003

A real Promptfoo run from a champion pilot. Support-email classification with LLM rubric (quality) and cost budget per test. Distinguishes genuine incidents from feature requests.