Encora

Senior AI QA Automation Engineer

Colombia · Work style not stated

Apply on the employer’s site ↗See how well you match this job

You apply directly with the employer. ROLIVA does not submit applications from this page and is not the employer.

About the role

Description from the employer’s posting. Check the employer’s page for the current version.

EmployerEncora
LocationColombia
Work styleWork style not stated
Employer posted1 day ago

Job Title: Senior AI QA Automation Engineer

Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring

Experience: 5+ YOE.

Location: Costa Rica, Peru, Colombia, Bolivia.

Mode: Remote

We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.

Key Responsibilities

Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.

Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.

Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.

Show full description

Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Required Skills & Qualifications

5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.

Strong Python skills with experience building reusable automated testing frameworks.

Hands-on experience with Playwright and/or Selenium.

Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.

Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.

Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.

Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.

Experience implementing CI/CD quality gates and production monitoring.

Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.

Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Skills:

Experience testing Generative AI, RAG solutions, copilots, or AI agents.

Experience with LLM evaluation frameworks and advanced prompt testing.

Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.

Experience with adversarial testing and Responsible AI testing.

Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.

Posted On: 12-10-2026

At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.

See how well you match this job Apply on the employer’s site ↗

Check your resume against this job Free, no sign-up. Nothing you paste is stored.

Staying safe while you apply

Never pay to apply, interview or accept an offer. ROLIVA never asks you for payment to apply to a job.
Apply only through the official “Apply on the employer’s site” button, on the employer’s own site, and check that a recruiter’s email uses that employer’s own domain rather than a free email address.
Be cautious of an offer made with no real interview, or pressure to decide quickly or share financial details early.

Does something look wrong with this listing? Use “Report broken link” in the role details, or read ROLIVA’s full job-safety guidance ↗.

Preparing for this role with ROLIVA

Check the fit.
Compare the requirements with the experience you have confirmed, including gaps.
Prepare honestly.
Build interview stories and application drafts from your own evidence.
Keep track.
Save the role and record your own application status and follow-ups.

Create a free workspace or log in.