How CodeSignal’s Skills Evaluation Lab Builds Unbiased Assessments

Today, 93% of leaders agree that diversity, equity, and inclusion (DE&I) is a top priority for their organization, but only 34% view it as a current strength in their workplace. Most tech companies struggle to increase the diversity of their engineering hires, in particular.

Building unbiased technical assessments that empower companies to reach their DE&I hiring goals is the core mission driving CodeSignal’s Skills Evaluation Lab.

What is the Skills Evaluation Lab?

The Skills Evaluation Lab is made up of CodeSignal’s teams of PhD Industrial-Organizational (IO) Psychologists and Assessment Design Engineers. Together, these teams develop, validate, maintain, and ensure compliance for CodeSignal’s Tech Screen and Pre–Screen assessments, as well as coding questions for live interviews. 

The Skills Evaluation Lab is also responsible for developing the Skills Evaluation Frameworks that power CodeSignal Pre-Screen and Tech Screen. These frameworks are built from a research-backed set of requirements and guidelines that enable companies to consistently identify qualified candidates for their roles. Skills Evaluation Frameworks support question variations at scale, which mitigates the impact of leaked questions—a common and unavoidable occurrence in tech hiring—by ensuring that no two candidates see the same series of questions. IO Psychologists continue to monitor the assessments to ensure questions are of equal difficulty, accurately measure critical skills for the role, and don’t introduce adverse impact.

What is IO Psychology?

Industrial-organizational (IO) psychology is the scientific study of human behavior in the context of work. IO Psychologists work to improve all aspects of the workplace for both the organization and the employees—which is especially important as more companies strive to improve diversity, equity, and inclusion (DE&I) in their hiring practices.

“Society is moving towards higher standards of what DE&I looks like within companies,” said Sylvia Mol, IO Psychologist and Assessment Research Lead at CodeSignal. “The Uniform Guidelines on Employee Selection Procedures are a great starting point to anti-discrimination, but they were also established in the 1970s. The concept of fairness has changed a lot over the last 50 years, and it’s important to consider what that looks like within the context of recruiting today.”

“Society is moving towards higher standards of what DE&I looks like within companies.”

Sylvia Mol, IO Psychologist and CodeSignal Assessment Research Lead

What does the Skills Evaluation Lab do?

1. Performs job analyses to understand different engineering roles

Job analysis is a scientific approach to measuring and understanding the responsibilities required for a given role at an organization. By using science rather than gut instinct or opinion, teams can help reduce bias in job descriptions and make sure that qualified candidates aren’t deterred from applying. 

When CodeSignal’s IO psychologists conduct a job analysis for our customers, they think through the KSAOs: knowledge, skills, abilities, and other characteristics that allow people to succeed. They also seek input from a representative sample of a customer’s employees that can adequately reflect the diverse perspectives and experiences on the team. Then, they help the customer use the output of the job analysis to add structure and consistency to recruiting practices, from writing the job description to training the interviewers on what is job-relevant. 

The foundation of having a clear job analysis helps organizations create accurate job descriptions, offering transparency to candidates and helping recruiters and hiring managers ensure that they are focusing on only job-relevant criteria when evaluating candidates’ suitability for a role. 

2. Develops & validates coding assessments that predict on-the-job performance

Using pre-employment assessments can bea powerful DE&I strategy. When assessments replace something like a phone screen or human resume review, it’s much harder for human bias to be introduced at that stage. Here at CodeSignal, we know from decades of research that most people tend to be overconfident in their ability to make fair hiring decisions. Assessments help to ensure that all candidates, regardless of their background, are being provided the same opportunity to demonstrate their fit to the role. 

Assessments are a great strategy in general, but it all comes down to the execution. If done without thought and structure, assessments can be biased and unfair, too. That’s why our Skills Evaluation Lab always asks: is the assessment valid and fair for this purpose? The Lab’s Assessment Design Engineers and IO Psychologists collaborate to create Skills Evaluation Frameworks and coding questions with high validity and strong fairness metrics, ensuring that candidates are being accurately evaluated on only job-relevant criteria.

3. Builds structured interviews to minimize bias & increase diversity

Without a data-driven and structured approach, interviews are left open for human bias to creep in. It’s natural and expected for interviewers to want to form connections with the candidates. But as soon as an interviewer starts asking the question differently, providing more guidance towards one candidate over another, or giving a candidate the benefit of the doubt when scoring, they are changing the assessment and how candidates are impacted. 

CodeSignal’s Skills Evaluation Lab can help teams bring structure to coding interviews. Depending on a customer’s needs, this might include: 

• Creating and reviewing an approved set of questions and explaining how they tie back to important aspects of the role
• Explaining how to create and use rating scales for each question, or 
• Explaining common biases to be aware of and strategies for removing them from the interview and scoring process.

These principles and best practices from IO Psychology are valuable in setting a strong foundation for people processes that are backed by science, therefore reducing subjectivity and the potential for human bias in the interview process. 

Learn more

Want to learn more about bringing validated and research-backed technical assessments to your organization’s hiring process? CodeSignal’s team of experts would love to talk with you. Reach us at [email protected]