DevScout
Open job

AI Conversation QA Specialist

Mountain View (MTV), Oakland, CA or San Diego, CA

Artificial Intelligence · On-site

Bachelor's degree in EnglishCommunicationsLinguisticsBusiness AnalyticsData AnalyticsComputer ScienceMicrosoft ExcelGoogle SheetsSQLPythondata filteringPivot Tableschartsgraphsbasic statistical analysis

About the job

AI Conversation QA Specialist for Sibitalent. Evaluate AI-powered conversations, agentic workflows, and LLM outputs to improve response quality, accuracy, and overall user experience. Requirements: Bachelor's degree in English, Communications, Linguistics, Business Analytics, Data Analytics, Computer Science, or related fields. 1–3 years of experience in Customer Support, Customer Service, Content Design QA, or similar roles. 📄 POST ORIGINAL COMPLETO: Ritik Singhal • 3º e + Sr. Technical Recruiter @ Sibi Talent | MNC | 12.2K+ Followers Family | Technical Sourcing Expertise | Talent Acquisition Specialist | Recruiterment | Ex - Skyper Business Solution | Ex - Jconnect 🚀 We're Hiring: AI Conversation QA Specialist 📍 Location: Mountain View (MTV), Oakland, CA or San Diego, CA 💼 Experience: 8+ Years 🎓 Education: Bachelor's Degree 🏢 Work Location: As per client requirements We are looking for an AI Conversation QA Specialist to evaluate AI-powered conversations, agentic workflows, and LLM outputs to improve response quality, accuracy, and overall user experience. 🔹 Key Responsibilities: Evaluate AI conversations, agent traces, and identify quality issues. Identify trends, tool-calling errors, intent issues, and member frustration. Perform data labeling and generate synthetic datasets for AI evaluations. Develop LLM-as-a-Judge evaluators, rubrics, and prompts. Analyze AI model outputs and recommend improvements. Query logs, analyze conversation traces, and identify recurring patterns. Collaborate with AI Scientists, Product Managers, and Developers. Perform quality audits, calibration, and data analysis. 🔹 Mandatory Skills & Qualifications: Bachelor's degree in English, Communications, Linguistics, Business Analytics, Data Analytics, Computer Science, or related fields. 1–3 years of experience in Customer Support, Customer Service, Content Design QA, or similar roles. Understanding of LLMs, AI conversations, and prompts. Strong analytical and problem-solving skills. Experience with Microsoft Excel or Google Sheets. Knowledge of SQL and Python. Experience with data filtering, Pivot Tables, charts, graphs, and basic statistical analysis. Excellent communication, attention to detail, and organizational skills. Ability to evaluate AI responses with accuracy, empathy, and an unbiased approach. 🔹 Preferred Qualifications: Experience in financial institutions or regulated environments. Familiarity with CRM and QA platforms. Experience with AI evaluation, prompt engineering, and LLM-based tools. Spanish fluency is a plus (not mandatory). 🎯 Ideal Candidate: Someone with a background in customer support, content QA, or data analytics who understands LLMs and has a strong interest in improving AI response quality. 📩 Interested candidates: Share your updated resume at [email removido] Subject: AI Conversation QA Specialist – Location 1 📋 Requisitos: • Bachelor's degree in English • Communications • Linguistics • Business Analytics • Data Analytics • Computer Science • Microsoft Excel • Google Sheets • SQL • Python • data filtering • Pivot Tables • charts • graphs • basic statistical analysis Requisitos: - Bachelor's degree in English - Communications - Linguistics - Business Analytics - Data Analytics - Computer Science - Microsoft Excel - Google Sheets - SQL - Python - data filtering - Pivot Tables - charts - graphs - basic statistical analysis Responsabilidades: - Evaluate AI conversations, agent traces, and identify quality issues - Identify trends, tool-calling errors, intent issues, and member frustration - Perform data labeling and generate synthetic datasets for AI evaluations - Develop LLM-as-a-Judge evaluators, rubrics, and prompts - Analyze AI model outputs and recommend improvements - Query logs, analyze conversation traces, and identify recurring patterns - Collaborate with AI Scientists, Product Managers, and Developers - Perform quality audits, calibration, and data analysis Diferenciais: - Spanish fluency - Experience in financial institutions or regulated environments - Familiarity with CRM and QA platforms - Experience with AI evaluation - prompt engineering - LLM-based tools - prompt-engineering

Job posted by a recruiter registered on DevScout.