PromptKit

PromptKit

Multi-provider prompt testing and deployment. Test prompts before they fail in production.

PromptKit is the open-source toolkit for PromptPack — test prompts across Claude, OpenAI, Gemini, Azure, and local models, run red-team scenarios in CI, and compile portable packs for runtime. The spec lives upstream so your config isn't locked to a vendor.

How it fits together

PromptPack  ── open spec for portable prompts (JSON, vendor-neutral)
    │
    └── PromptKit  ── this repo
         ├── promptarena  ── test, red-team, evaluate (CLI)
         ├── packc        ── compile config → portable pack
         └── SDK          ── embed in your Go application

Why PromptKit

_{¹ Promptfoo has a text-only simulated user and separate audio testing, but doesn't combine them — it can't drive a persona-driven caller through a realtime voice agent.}

Install

npm install -g @altairalabs/promptarena @altairalabs/packc

Building from source: see CONTRIBUTING.md.

Quick Start

1. Create a project

promptarena init my-project --template iot-maintenance-demo
cd my-project

2. Inspect configuration

promptarena config-inspect

3. Run a test scenario

promptarena run --scenario scenarios/hardware-faults.scenario.yaml

4. Red-team security testing

promptarena run --scenario scenarios/redteam-selfplay.scenario.yaml

5. Review results

promptarena view

6. Deploy with the SDK

Compile prompts and run in your Go application:

packc compile -c config.arena.yaml -o app.pack.json

Voice-agent self-play

You can't unit-test a voice agent — so PromptKit has AI personas call it. Synthetic, personality-driven callers (hostile, evasive, anxious) are driven through TTS into your realtime agent (Gemini Live, OpenAI Realtime), and structured assertions score whether it holds policy under pressure — never issuing an unauthorized refund, escalating when it should. It even checks the caller sounds angry (speech-emotion recognition), not just says angry words.

Try it in one command — keyless, runs green out of the box:

promptarena init my-refund-demo --template voice-refund-demo
cd my-refund-demo
promptarena run --provider mock-duplex --ci   # no API keys needed

Swap in --provider gemini-2-flash or openai-gpt4o-realtime (plus TTS keys) to run it against a live voice agent — pass rates vary, and that variation is the test. Full breakdown: voice-refund walkthrough.

Features

GitHub Actions

Integrate PromptKit into your CI/CD pipelines with our GitHub Actions:

PromptArena Action

Run prompt tests in your workflows:

- name: Run prompt tests
  uses: AltairaLabs/PromptKit/.github/actions/promptarena-action@v1
  with:
    config-file: config.arena.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

PackC Action

Compile and publish prompt packs to OCI registries:

- name: Build and publish pack
  uses: AltairaLabs/PromptKit/.github/actions/packc-action@v1
  with:
    config-file: config.arena.yaml
    registry: ghcr.io
    repository: ${{ github.repository }}/prompts
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}

See the GitHub Actions documentation for full usage details.

Repository Structure

promptkit/
├── tools/arena/     # PromptKit Arena CLI (testing, red-team, deploy)
├── tools/packc/     # Pack Compiler CLI
├── sdk/             # Production SDK (conversations, workflows, A2A, skills)
├── runtime/         # Shared runtime (providers, pipeline, tools, skills, a2a, deploy)
├── examples/        # Example projects
└── docs/            # Documentation

Contributing

See CONTRIBUTING.md.

AI Development

For AI coding assistants working on this repository, see AGENTS.md for critical development rules and pre-commit requirements.

License

Apache 2.0 - See LICENSE.

Built by AltairaLabs.ai