Avant is a Language Proficiency Company that unites every stage of the language journey - learning, assessment, and certification. Serving students, educators, administrators, professionals, and employers, Avant’s product and services deliver meaningful and measurable outcomes that drive real proficiency.

The Director of Rating is an advanced-level position.The Director of Rating provides strategic leadership for all aspects of rating operations, quality assurance, calibration, training systems, and rater performance across language assessment types. This role is responsible for building and maintaining standardized, scalable, and data-informed systems that ensure accurate rubric application, operational efficiency, and consistent rater development across all languages.

The Director leads the integration of operational metrics, quality monitoring, calibration systems, learning design, and AI-supported scoring tools to create a unified rating ecosystem that minimizes rubric drift and supports high-quality assessment outcomes.

This position oversees the rating team and collaborates closely with Learning Design personnel, psychometric teams, product/technology teams, and executive leadership to ensure scoring reliability, continuous improvement, and organizational alignment.

Responsibilities:

Strategic Leadership

• Develop and implement a unified vision for rating operations, calibration, training, and quality assurance across all language programs.
• Lead organizational restructuring initiatives that improve consistency, scalability, and operational effectiveness.
• Establish standardized workflows, communication protocols, and quality assurance systems across rating teams.
• Serve as the organizational authority on rating operations, calibration systems, and scoring quality.

Rating Operations Management

• Oversee operational performance metrics including:
• Items per hour (IPH)
• Rater productivity
• Scheduling and availability
• Scoring turnaround times
• Quality indicators
• Develop scalable operational systems that balance efficiency with scoring accuracy.
• Monitor and analyze operational data to identify trends, risks, bottlenecks, and improvement opportunities.
• Partner with technology and AI teams to optimize the use of internal scoring tools (e.g., OTTO) and workflow systems.

Quality Assurance & Calibration

• Lead the design and implementation of standardized calibration systems across all languages.
• Oversee quality assurance processes including:
• Anchor item programs
• Double-blind scoring
• Override review procedures
• Drift detection systems
• Inter-rater reliability monitoring
• Establish consistent protocols for rubric interpretation and adjudication.
• Ensure alignment and consistency in scoring practices across all rating teams and languages.
• Monitor feedback quality and ensure coaching practices reinforce calibrated rubric application.

Training & Learning Systems

• Oversee the development and maintenance of standardized rater training and certification programs.
• Partner with Learning Design personnel to:
• Develop scalable e-learning modules
• Maintain a centralized learning management system
• Track training completion and certification
• Ensure all training materials reflect current rubric standards
• Establish systems for ongoing professional learning, recalibration, and remediation.
• Ensure training systems function as the organization’s “single source of truth” for rubric interpretation and scoring guidance.

Leadership & Team Management

• Lead and support Rating Managers, Language Specialists, and related operational staff.
• Redefine and clarify role expectations, accountability structures, and decision-making authority across teams.
• Foster a collaborative, cross-language culture focused on consistency, continuous improvement, and shared accountability.
• Develop leadership capacity within rating management teams.

Communication & Change Management

• Design and maintain structured communication systems for rubric updates, calibration changes, and operational initiatives.
• Ensure timely, consistent dissemination of scoring guidance and policy changes across all rating personnel.
• Lead change management efforts related to process improvements, technology integration, and organizational restructuring.

Data, Reporting & Continuous Improvement

• Use operational and quality data to drive strategic decision-making.
• Develop dashboards and reporting systems related to rater performance, calibration outcomes, and operational effectiveness.
• Identify systemic scoring trends and partner with relevant teams to implement targeted interventions.
• Lead continuous improvement initiatives that strengthen scoring reliability and organizational efficiency.

Qualifications:

Required

• Bachelor’s degree or higher in Applied Linguistics, Second Language Education, Language Assessment, Educational Leadership, Measurement, Curriculum & Instruction, or related field.
• Significant experience in language assessment, rating operations, or large-scale scoring systems.
• Demonstrated leadership experience managing teams and complex operational systems.
• Experience with quality assurance, calibration, and rater training systems.
• Strong understanding of rubric-based scoring methodologies and inter-rater reliability principles.
• Experience using data and analytics to improve operational performance and quality outcomes.
• Excellent organizational, communication, and change management skills.

Preferred

• Experience leading multilingual or large-scale assessment programs.
• Experience designing online professional learning or certification systems.
• Familiarity with AI-supported scoring systems and assessment technologies.
• Experience with learning management systems (LMS), workflow tools, or rating platforms.
• Background in process improvement, systems design, or operational transformation.

All Avant Employees Are Expected To:

• Maintain discretion when working with sensitive and confidential information.
• Demonstrate a commitment to continuous learning and cross-functional collaboration.
• Support Avant’s mission, values, and the learners and educators we serve.

Avant Assessment is proud to be an equal opportunity employer.