PickYourTechPickYourTech home
Shortest logo

Shortest

Write and run AI-driven natural language E2E tests

QA & End-to-End Testing

Shortest lets developers author end-to-end tests in plain English, leveraging Anthropic Claude to execute them via Playwright, with built-in GitHub 2FA and email validation support.

Shortest banner

Overview

Overview

Shortest is a testing framework that lets developers describe end‑to‑end scenarios in natural language. By sending those descriptions to Anthropic Claude, the AI translates them into Playwright actions, handling browser automation, authentication flows, and API calls without hand‑written scripts.

Capabilities & Deployment

The CLI (shortest init) bootstraps configuration, installs the package, and creates environment placeholders for required keys. Tests can include callbacks, lifecycle hooks, and chaining for complex user journeys. Built‑in support for GitHub two‑factor authentication and Mailosaur email validation expands coverage to security‑critical flows. Running pnpm shortest --headless integrates smoothly into CI/CD pipelines, provided the Anthropic API key is stored securely. The framework is ideal for teams that want rapid test authoring while retaining the power and reliability of Playwright.

Highlights

Write tests as plain English sentences
AI translates descriptions to Playwright actions via Anthropic Claude
Built‑in GitHub 2FA login and Mailosaur email validation
CLI supports CI/CD integration and headless execution

Pros

  • Reduces boilerplate by using natural language
  • Leverages powerful LLM for dynamic test generation
  • Seamless Playwright integration ensures reliable automation
  • Supports lifecycle hooks and test chaining for complex flows

Considerations

  • Requires Anthropic API key, incurring usage cost
  • AI interpretation can introduce nondeterministic behavior
  • Limited to environments where Playwright runs
  • Debugging AI‑generated steps may be less transparent

Managed products teams compare with

When teams consider Shortest, these hosted platforms usually appear on the same shortlist.

BrowserStack Automate logo

BrowserStack Automate

Cross-browser test automation on 3500+ real browser/OS combinations

GHO

Ghost Inspector

Automated browser tests for websites and web apps

LambdaTest logo

LambdaTest

Cross-browser & mobile app testing cloud with real devices and automation

Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.

Fit guide

Great for

  • Teams seeking rapid test authoring without deep Playwright expertise
  • Projects with frequent UI changes needing adaptable tests
  • CI pipelines that can store Anthropic credentials securely
  • Applications that require GitHub 2FA login testing

Not ideal when

  • Environments with strict offline or no‑API policies
  • Tests demanding pixel‑perfect visual verification
  • Teams preferring fully deterministic, hand‑coded scripts
  • Projects lacking Playwright compatibility

How teams use it

Login flow with GitHub 2FA

Validate end‑to‑end authentication including OTP handling without writing Playwright code

API endpoint validation

Assert that a GET request returns only active users using a single natural language description

Sequential user actions

Chain multiple business‑logic steps (login, invoice creation, view) in a readable list

Database verification after UI actions

Run custom callbacks to query the database and assert state after UI interactions

Tech snapshot

TypeScript97%
JavaScript2%
Ruby1%
CSS1%

Tags

automationtestingplaywrighte2e-teststesting-frameworkend-to-end-testingtest-automatione2e-testingnextjsanthropictesting-toolchromiumjavascript

Frequently asked questions

Do I need an Anthropic account to use Shortest?

Yes, an Anthropic API key is required; the key is supplied via ANTHROPIC_API_KEY or SHORTEST_ANTHROPIC_API_KEY environment variables.

Can Shortest run in headless CI environments?

Yes, use the --headless flag or set headless:true in the config to execute tests in CI pipelines.

How does Shortest handle GitHub two-factor authentication?

It supports GitHub 2FA by accepting a TOTP secret; the CLI can prompt for the OTP code during login.

Is Playwright the only browser engine supported?

Currently Shortest is built on Playwright, so tests run using Playwright’s supported browsers.

Can I add custom assertions beyond the AI-generated steps?

Yes, you can provide callback functions or lifecycle hooks to run additional code and assertions.

Project at a glance

Stable
Visit siteView repo
Stars
5,488
Watchers
5,488
Forks
328
LicenseMIT
Repo age1 year old
Last commit5 months ago
Primary languageTypeScript

Last synced yesterday