Kristopher John leads product strategy at New Meridian, bringing almost two decades of experience to the job. He served for nearly 10 years at the College Board, rising to the position of Executive Director of Research and Development, and spent years developing state and national assessments for use in K-12 classrooms and higher education.
As New Meridian continues to design, prototype and pilot a new instructionally-aligned assessment system in partnership with Montana and Louisiana, insight from the community remains a core design principle. We caught up with John to discuss the role of community involvement in creating this next generation assessment. The conversation has been edited for length, style and clarity.
Why is it important to involve communities in assessment?
We’re working to develop products that will have a high degree of utility, and you are not going to be able to build something that has high utility without actually speaking to the audience and understanding what their goals and pain points are. That’s an important part of this stakeholder input process.
The process of testing has impacts that go beyond just the people who use the tests. Teachers and administrators are making decisions that have impact and that cascade to the entire community. So I think that’s a strong reason to have community input in the development of tasks and the reporting and analysis of test results.
We need to make sure that assessment is a process that’s open to a high degree of public scrutiny. So the more we can engage broader communities and engage with diverse sets of stakeholders, the better chance we have at making a positive impact and avoiding negative, unintended consequences. I think those are important things.
Can you give us some specific examples or use cases?
I think of teachers as a user of assessments and also as members of the community. In our discussions with teachers, we’ve tried to broaden the conversation. What would be useful to you? What would help you understand where students are? How does administering assessment affect your school? What kinds of things do you do in advance of administering an assessment? We looked not just at the assessment itself, but at the whole process and the broader impacts to the school environment.
Is it different talking to administrators than it is to teachers?
It is. They often have different agendas, different goals and different pain points. They obviously share a lot of goals. They’re primarily concerned with the education of students. But there are a lot of things they don’t have in common as well. For example, compliance with policies at the state level is something that’s really a concern of district administrators.
Why do you gather input from parents?
It’s important to talk to parents and caregivers because they are users of the results and it is valuable to inform them about what’s going on. Anytime you’re contacting an audience there’s an opportunity to give them information about what assessment actually means and potentially clear up misconceptions.
Some caregivers tend to be more active than others. Some are really looking for the school to answer the question, ‘what should I do now?’ Other parents want to know, ‘what can I do to supplement what the school is doing?’ So there’s a lot of variety. Some may understand assessments really well and some may not understand them at all. But you have to be able to communicate with the entire audience, and you don’t have the luxury of explaining everything or providing training on how to use the results. So the results have to be easy to understand. Talking to parents helps us do that.
New Meridian also talks to students. Isn’t that difficult?
Yes, it is challenging to get to a student audience. I think that’s one of the least involved audiences when it comes to assessment. But we definitely talk to students. We’re using an empathy interview approach, which is really designed to allow the person being interviewed to lead the conversation. We try to talk about how they feel and students have a lot of feelings about assessment. We’re trying to understand the specific emotional impact on students that you can only understand by talking to them. Otherwise, you’re only getting anecdotal information from teachers and parents. Nothing compares to talking directly with students. You can’t really understand the emotional impact and the value of your product without talking to that audience.
Tell us more about the process of empathy interviews.
The process is to really let the subject lead the conversation, and you do that less by asking direct questions and more by asking questions like, ‘what is the mood like in your school?’ or ‘how do you feel when you get the test results?’ By asking these kinds of open-ended questions, you’re asking the person to tell the story of using the test. That’s what’s different. This is really about trying to understand, trying to open up the conversation in a way that reveals their concerns and fears.
Once you have done interviews, what comes next?
We read through all of the transcripts and try to extract the most salient details. We also develop user personas, which are archetypical descriptions of potential users, and those are used to guide design decisions. They’re used as a way of testing assumptions that you might be making about design.
This work is only the beginning. New Meridian will continue to engage stakeholders and collect feedback throughout the pilot process to refine the design of this new system. Because people at all levels of the education system should benefit from the next generation of assessment.
