Many of our clients started by using Survey Monkey, Gallop’s Q12, or another common engagement survey. The wonderful thing about these surveys is they are generalizable to a broad range of companies, which allows you to utilize benchmarks. The downside is they are generic and lack connections to an organization’s identity and culture.
Culture is a complex topic that is unique to each organization and depends on company size, industry, and stage of development. With our Values Scorecard and Culture & Engagement Insights, we offer surveys customized to a company’s unique needs while also ensuring that our surveys meet high thresholds for validity and reliability.
Validity and reliability are terms often thrown around when discussing data. Most of us know having valid and reliable data is important, but you might not clearly understand the difference between the two. In short, validity measures accuracy, or if your metrics truly and accurately represent what you say they do. Reliability measures precision, or the degree to which your metrics are consistent and minimize random error. If you want to get better data, you must start at the earliest stages of the project; designing your survey.
Here at illumyx, we have honed our survey design process over the years. This article will outline a few ways our research-based process leads to more valid and reliable data.
Why Successful Surveys Require Valid and Reliable Data
Surveys must produce accurate data so you can make informed decisions. Bad data can be misleading and potentially jeopardize your ability to meet your company goals by leading you to focus on the wrong things.
High-quality data allows you to trust the patterns and trends you uncover. This allows organizations to address the most critical areas of their culture such as accountability, teamwork, and leadership to best retain talent and foster overall organizational success.
There are many ways to assess both validity and reliability, but in this article, we will focus on how we address three of them: content validity, face validity, and internally consistent reliability.
Interviews & Focus Group to Develop Content Validity
Content validity assesses whether the content of your survey is fully representative of what you are trying to understand and measure.
How illumyx addresses Content Validity:
When we work with an organization, we combine qualitative and quantitative analysis to build the most relevant and actionable surveys possible.
Even doing a handful of interviews (qualitative data) helps us clarify or establish Content validity by understanding the specific behaviors and attitudes within an organization related to concepts such as communication, accountability, teamwork, and cross-functional collaboration. We start with broad questions and then explore the topics that come up to learn as much as possible about each concept. When developing a Values Scorecard for a company, this same approach is used to understand how their values are lived and experienced.
We then utilize our database of over 900 survey statements to dial in on what we believe are the most relevant survey statements to include in each construct. We reference not only our experience and the qualitative feedback we have collected, but also reference survey statements that have proven valid and reliable for other companies in a similar industry, company size, and those dealing with similar issues/symptoms. Understanding context is always critical.
Co-create Survey to Improve Face Validity
Face validity is an initial assessment of whether the survey is relevant and understandable to the people taking it.
How illumyx addresses Face Validity:
The next step is to run our assumptions and question selections past a small group of leaders and employees.
We will work through the initial draft with Human Resources and usually another leader, providing our rationale for question selection. This is a key time-saving step before bringing the survey draft to a Survey Team or Culture Team for input.
It is important to get feedback from a cross-section of employees, but adding more voices to the conversation requires more time. To maximize the benefit of feedback and minimize time commitment, we do not want a Survey or Culture Team to create the survey from scratch. We review the refined draft with them, and they help us pair the survey down even further and reword statements to best capture the employee experience in terms everyone will understand. This team may even help craft 1-5 completely customized survey statements to include, which helps content validity.
We act as a check and balance to ensure that key statements are kept and that any new statements are properly worded. This is usually done in 2-3 meetings and can be completed in as little as 2 weeks. This is a crucial step to ensure the survey is relevant to the specific company and context.
Evaluate Constructs for Internal Consistency
Internal consistency is a measure of reliability that evaluates correlations between survey questions to ensure they work together to measure a concept. If a group of questions measures a single concept, we expect respondents will choose similar answers for related questions.
How illumyx addresses Reliability:
Once the survey is finalized and signed off on by senior leadership, we deploy the survey and collect the quantitative data. Once all data is collected, we begin a process of reducing the number of variables needed to explain results. In other words, we take the 50+ survey questions and group them into 5-10 themes. Because we designed the survey to address specific themes, we already have a strong idea of how the questions will best be combined. Finally, we calculate a score for the internal consistency of each theme and ensure they meet an acceptable threshold. Throughout this process, some questions may be moved to themes other than we initially expected. When a question does not correlate to others as we expected, it provides an opportunity to examine that question before the next survey and continually improve our measures.
Every Company Is Different. Don’t Make Your Surveys Generic.
A one-size fits all approach does not work and we have found that a valid and reliable survey construct for one company is not necessarily valid and reliable for another because they are dealing with different circumstances.
It is important to develop relevant and actionable measures for each company we work with.
So, before you release your survey, make sure it covers what you are trying to measure, is understandable to the people taking it, and provides actionable data that you can be confident will positively impact your culture.
