SDET - Software Eng Test, Alexa AI Developer Tech Alexa+: An AI assistant that gets things done, she's smarter, more conversational, and more capable. With 600 million Alexa devices now out in the world, the latest advancements in generative AI have unlocked new possibilities enabling us to reimagine the experience in our pursuit of making customers' lives better and easier every day. Experts enables Alexa+ to complete complex tasks using Amazon and third‑party (3P) APIs and websites. Some of these include playing your favorite music, controlling your smart devices, reserving a table at a restaurant, setting your morning alarms, kitchen timers, booking event tickets, scheduling an appointment and planning a trip. No more jumping between apps—just say what you want, and it's done. Developers and partners have been core to our vision since the beginning. With Alexa+, we reimagined how developers can build for Alexa. The Alexa AI developer technology charter is to enable the creation of experts at scale and make it seamless for 1P and 3P developers to integrate their businesses with Alexa+. We are measurably making Alexa+ smarter, and we need your help to define and build the next generation of capabilities using Generative AI.
Key Job Responsibilities As an SDET driving the creation of next‑generation LLM‑based evaluation systems for Alexa+, you’ll design and build the frameworks that define how conversational intelligence is measured, determining whether millions of daily interactions feel accurate, natural, and human‑centered. Your work goes far beyond binary pass/fail tests: you’ll engineer automated systems that assess accuracy, reasoning depth, tone, and responsiveness across multimodal, context‑rich conversations.
In this role, traditional testing boundaries dissolve. You’ll evaluate not just functional correctness, but whether the AI’s responses are contextually relevant, emotionally aligned, and conversationally fluent. Your systems will measure everything from factual accuracy and task completion to subtler attributes such as dialog flow, personality coherence, and graceful curtailment. Partnering closely with scientists and engineers, you’ll automate the detection of conversational regressions, identifying hallucinations, degraded reasoning, or misaligned tones before they reach customers. You’ll leverage prompt‑driven evaluation pipelines, LLM‑as‑a‑Judge (LLMaaJ) frameworks, and reference‑based validation to ensure assessments remain consistent, explainable, and scalable across model versions and releases.
You’ll also collaborate with prompt engineers, model developers, and product teams to establish robust, category‑specific testing methodologies—from quick one‑shot actions and task fulfillment to multi‑turn dialogues and creative, open‑ended interactions. Through your work, evaluation evolves from a quality gate into a self‑improving assessment framework—one that learns, adapts, and ensures every voice interaction feels naturally conversational.
Qualifications
Knowledge of overall system architecture, scalability, reliability, and performance in a database environment
Experience with security in service‑oriented architectures and web services
Master’s degree in Computer Science or a related field
Experience with AWS or cloud technologies
Experience with LLM‑based evaluation, prompt‑driven testing, or LLM‑as‑a‑Judge frameworks
Hands‑on experience using AI and LLM‑based frameworks to automate quality testing and build intelligent evaluation systems
Ability to design semantic and behavioral validation beyond functional correctness
Proven record of collaborating with scientists and engineers to define measurable AI quality metrics
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
#J-18808-Ljbffr