Devarshi Ojha
Lead SDET · AI Quality & GenAI Evaluation
I build the test systems that let teams ship AI and web products with confidence, from Playwright frameworks to evaluation gates for LLM pipelines.
- Jira specs
- AI agents generate tests
- Web · API · Mobile
- AI eval gates
- Release
- years in quality engineering
- 9+9+
- regression after Playwright migration
- 8h → 3h8h → 3h
- test cases from agentic generation
- ~500~500
- AI evaluation run time
- 5–6h → ~3h5–6h → ~3h
- business verticals on my CoE framework
- 1616
- QA engineers trained
- 20+20+
/ Case studies
KlearNow.AI
Evaluating production GenAI document processing
Problem
AI was classifying and extracting data from trade documents (invoices, bills of lading, customs filings), and errors flowed straight into customer operations.
Solution
I owned and strengthened our evaluation framework: a golden dataset of ~2,000 real and 400–500 synthetic documents across 8 classes and ~100 fields, scored with per-class precision/recall/F1, field-level exact-match and schema checks as CI quality gates. I piloted DeepEval, RAGAS and Promptfoo, and integrated Arize for observability.
Impact
- ▸Run time cut from 5–6 hours to ~3
- ▸Region-specific misclassification caught before release
- Documents
- Classify
- Extract
- Eval gates (F1 · exact-match · schema)
- Arize dashboards
- Python
- DeepEval
- RAGAS
- Promptfoo
- Arize
- Jenkins
Avis Budget Group, via StatusNeo
Testing a mainframe-to-AWS migration
Problem
A car-rental reservation platform was moving from a mainframe to AWS microservices, with 6,000+ legacy test cases and a slow, flaky Selenium suite.
Solution
I architected the UI framework and led the migration to Playwright (TypeScript) with Docker-based CI, and owned testing of the Kafka reservation flows, PostgreSQL and DB2 data replication.
Impact
- ▸Regression went from 8 hours to 3
- ▸Flakiness dropped by ~60%
- ▸6,000+ legacy cases became ~1,200 layered tests
- DB2 mainframe
- Transform
- Kafka
- PostgreSQL
- APIs
- New website
- Playwright
- TypeScript
- Selenium
- REST Assured
- Kafka
- PostgreSQL
- Docker
- AWS
KlearNow.AI
Spec-driven agentic test generation
Problem
Writing test cases and scripts took the QA team 2–3 days every sprint.
Solution
Playwright Test Agents with an Atlassian MCP server read Jira stories and Confluence specs, then generated test cases and scripts. Healing was limited to locator fixes, and a person reviewed every one.
Impact
- ▸~500 test cases generated
- ▸The team's time moved from writing tests to reviewing them
- Jira + Confluence
- Planner agent
- Generator agent
- Human review
- CI
- Playwright Test Agents
- MCP
- TypeScript
- Jira
- Confluence
StatusNeo
A Quality CoE framework that became a product
Problem
Each business vertical was building its own automation from scratch.
Solution
I built shared framework modules (Maven/npm), onboarded 7 verticals myself and trained 20+ engineers.
Impact
- ▸Adopted by 16 verticals
- ▸Later ported into StatusNeo's TestCraft product
- Shared modules (Maven/npm)
- Vertical onboarding
- Engineer training
- 16 verticals
- TestCraft product
Side project
Solar business platform
A multilingual site and CMS built with AI assistance (Next.js, TypeScript) and tested with Playwright.
- Next.js
- TypeScript
- Playwright
Visit site ↗(opens in a new tab)/ What I do
AI & GenAI Evaluation
Golden datasets, LLM metrics and CI gates that catch AI regressions before users do
Automation Architecture
Playwright, Selenium and Appium frameworks built to scale across teams
Agentic QA
AI agents that turn specs into tests, with a person reviewing every change
API & Integration
REST, GraphQL, Kafka and data-replication testing for microservices
Performance & NFR
Load testing with JMeter and k6, monitoring with Grafana, Datadog and Dynatrace
QA Leadership
Test strategy, release sign-off, CoE standards and mentoring
/ Experience
06/2026 – 09/2026 · Gurugram
KlearNow.AI
Lead Quality Assurance Engineer (AI Ops)
- ▸Led a QA team of ~6: test strategy, risk-based testing and release sign-off for production GenAI document processing
- ▸Owned and strengthened the AI evaluation framework; cut run time from 5–6 hours to ~3
- ▸Introduced agentic test generation (~500 test cases from Jira and Confluence specs)
- GenAI evaluation
- Agentic QA
- Playwright (TypeScript)
- DeepEval
- Arize
- REST Assured
02/2025 – 06/2026
Career Break
Family care, upskilling & freelance
- ▸Family medical emergency, then upskilling in GenAI evaluation and agentic QA
- ▸4 freelance Playwright and API testing engagements for Southeast Asian clients
07/2022 – 01/2025 · Gurugram
StatusNeo Consulting
Senior Consultant
- ▸Quality CoE framework adopted by 16 verticals; trained 20+ engineers
- ▸Led Selenium → Playwright migration for Avis Budget Group: regression 8h → 3h
- ▸Owned microservices, Kafka and data-replication testing in a mainframe-to-AWS modernization
- Playwright
- Selenium
- REST Assured
- Kafka
- AWS
- Docker
06/2019 – 06/2022 · Noida
Publicis Sapient
Senior QE Associate
- ▸Sole QA for a headless AEM CMS; authored its test framework from scratch
- ▸Contributed to Appium mobile automation on BrowserStack and real devices
- AEM
- Selenium
- Appium
- BrowserStack
04/2016 – 06/2019 · Noida
I3infosoft
QA Consultant
- ▸Selenium (Java) + TestNG automation for e-commerce web apps
/ Skills
AI/ML & GenAI Testing
- Golden datasets
- Precision/Recall/F1
- Field-level validation
- JSON schema validation
- LLM-as-judge
- Prompt regression
- CI evaluation gates
- DeepEval
- RAGAS
- Promptfoo
- Arize
Automation
- Playwright (TypeScript)
- Selenium (Java)
- REST Assured
- Appium
- Cucumber/BDD
- TestNG
- Pytest
- GraphQL
- Kafka
Agentic QA
- Playwright Test Agents
- MCP (Playwright, Atlassian)
- Claude Code
- Cursor
NFR
- JMeter
- k6
- OWASP ZAP
- Applitools
- Lighthouse
- Datadog
- Grafana
- Dynatrace
CI/CD & Cloud
- GitHub Actions
- Jenkins
- Azure DevOps
- Docker
- Kubernetes
- AWS
- Azure
/ About
I've spent 9+ years making sure software works before users find out it doesn't, from e-commerce sites to mainframe migrations to GenAI pipelines. Lately my focus is the hardest version of that problem: testing systems whose answers aren't fixed. I believe a good test suite is a product in its own right: fast, trusted and easy for the next engineer to extend.