Create Automated Scoring Questions for Video Interviews

Updated by Amy Thomas

Create custom questions in your Interview Content Library to ask your candidates during the video interview.

Automated Scoring is not available for live interviews.
If the questions you are creating will be asked during a live interview, or you plan to manually evaluate your candidates' video interviews, you will need to create your interview questions in the classic format.
If you are not certain whether a question will be rated by your evaluators or by Criteria's IIQ, we recommend creating it as an Automated Scoring question. The scoring type can be changed to Human Scoring in the interview template.

Start the question

You may add questions in video or text format. If you do not want to record yourself asking a question, you can include written questions for your candidates to respond to instead.

  1. Click Manage and select Interview Content Library from the dropdown list.
  2. In the Custom Library tab, click Create New Question and select Automated for the Scoring Type.

Step 1: What Competency Will This Question Evaluate?

  1. You can choose an option from the list or add one of your own.
    1. If you add your own competency, this will be validated to make sure it is suitable for automatic scoring. Click Validate after typing this in.
  1. Click Proceed to Next Step.

Step 2: Question Creation

Using a pre-loaded competency

If you have selected one of the preloaded competencies, some example questions are presented. You may select one of these or write your own question.

  1. If you select an example question, click Proceed to Next Step.

or

  1. To add your own question, click Write your own Question.
    1. Type in your question. Please ensure that it aligns with your selected competency and is structured as a behavioral, experience-based, or situational type of question. Click + next to each question type to learn more.
    2. Click Validate. Criteria's AI will check that the question can be automatically scored.
    3. Click Proceed to Next Step.

Using a custom competency

  1. If you have created your own competency, you will need to create your own question as well. Please structure your question using one of these question types: Behavioral, Experience-based, or Situational.
    1. Behavioral interview questions are designed to understand how a candidate has responded to situations in the past. These questions focus on actions and outcomes from real experiences.
    Example Behavioral questions
    • Describe a time where you identified a risk or hazard in your work environment. What steps did you take to mitigate that risk?
    • Talk about a decision you needed to make at work for which there was no clear best option. How did you decide what to do?
    1. Experience-based interview questions are used to learn more about the candidate's past job roles, responsibilities, and achievements. These questions should focus on specific aspects of their work history.
    Example Experience-based questions
    • Describe your experience working in sales for technology or software.
    • Can you describe your previous call center work experience?
    1. Situational interview questions are hypothetical and assess how a candidate would handle challenges or situations that they may not have experienced.
    Example Situational questions
    • Imagine you had to write a detailed report at work. What steps would you take to ensure every detail is accounted for?
    • Imagine you had to evaluate someone else's work or project plan. What would you be looking for in this evaluation?
  1. Click Validate. Criteria's AI will check that the question can be automatically scored.
  1. Click Proceed to Next Step.

Step 3: Approve Response Rating Scales

Using a pre-loaded competency and question

If you have selected one of the preloaded competencies and a suggested question, the rating scale will load immediately.

  1. If you want to make any changes to the Indicators that the AI will score your candidates on, click Edit Rating Scale.
    1. Make the required changes, then click Validate.
    2. Once this has been validated, click Save Changes.
  2. Tick the box: I Approve this Rating Scale.
  1. Click Approve Rating Scale.

Using a custom question

If you have written your own question (whether you selected a preloaded competency or chose your own), Criteria's AI will need to generate the rating scale. This could take up to a minute to create.

  1. Review the Rating Scale once it has been generated.
  2. If you want to make any changes to the Indicators that the AI will score your candidates on, click Edit Rating Scale.
    1. Make the required changes, then click Validate.
    2. Once this has been validated, click Save Changes.
  3. Tick the box: I Approve this Rating Scale.
  1. Click Approve Rating Scale.

Step 4: Record Video & Adjust Settings

  1. Complete the required information. Fields that are not required for Automated Scoring or cannot be altered will appear disabled (greyed out).
    1. Category: Select an available category from the dropdown list or create your own, making your content easier to find in your Interview Content Library. 
      1. To create your own category, click General Questions and start typing in the Enter new category here field. Click Enter to create the new category.
    2. Response Type: The response type is Video, as the Criteria AI only evaluates video responses.
    3. Preparation Time: the amount of time candidates have to consider how to respond to the question before the recording begins.
    4. Answer Time: the amount of time candidates have to respond to the question.
    5. Record or Upload Video: Here you can record your question directly in the platform or upload a question recorded elsewhere.
  2. Click Save and Complete. You will be returned to your Interview Content Library.

Next steps

Once you have created all the questions and messages you need, you will be ready to create an interview template.

How did we do?