Literature Review

We reviewed literature broadly related to our topic in order to shed light on all of our research questions. Since we are interested in methodology of comparing students’ data and visualization we focused on literature regarding learning analytics methodologies.

Data Exploration

We also did some quantitative data analysis focused on data-mining of the MDR data (school level demographic data), to find correlations between demographic data and growth using some basic statistical analysis and visualization. For example, the graph to the left (below) tells us that there is a difference between how well a school is doing compared to averages, and how well a school is doing compared to demographically similar peers. The graph to the right tells us that some controllable factors such as teacher-to-student ratio have little effect on students’ performance and growth. 

Competitive Analysis

To gain an understanding for how data coordinators currently report data and how demographic comparison is currently done, we identified 11 competitive products and analyzed them on two dimensions: Dashboard Quality and Comparison Ability. Overall, these products vary in function and usability, but none of Renaissance’s major competitors provide demographic comparison as we define it. This creates an opportunity for Renaissance to provide more frequent data and create interventions before students take more high-stakes tests. 

We conducted 10+ one-hour structured interview with district administrators to understand their data usage and their expectations regarding demographic based comparisons. Overall we want to understand what data they need for demographic comparison, how data informs their decision making, and their general data-use process.

Then we used affinity diagramming to consolidate our interpretations and see how opinions clustered. The affinity diagram generated lots of insights. These insights informed our co-design sessions design, survey design and model building. 

Through our interview, we found administrators care certain demographic factors the most including: Ethnicity, Gender, English Language Learner, Free/Reduced Lunch, School Size, School Type, School Location

Not everyone is a data person. Teachers and school leaders usually prefer visuals to raw numbers. When administrators communicate with school leaders and teachers, using raw data can be overwhelming.

Through our interviews, we found some administrators connect to other schools through personal networks, and some take the time to research and reach out to similar schools.

We also sent a survey to a larger population to gather more quantitative data on which demographic factors administrators would like to use when looking at students’ performance and growth. As shown in the survey, overall administrators valued Free/Reduced Lunch student percentage the most, followed by school size and type. This result serves as a great guidance for us later on on designing the algorithm of the tool.

We also conducted two co-design sessions with district administrators to understand:

• What information they would need given a certain scenario of analyzing or presenting data and how do they prioritize the information.

• Their preferred way of visualizing the information in the given scenario.

• How they would use this information and communicate the data with others.

Our methods included card sorting and role-playing. Participants were asked to choose scenarios they are interested in, and then sort out cards with information they needed. With this information, they were prompted to describe or draw the visuals they wanted to use for communication. At last, they role-played with the interviewer to show how they would communicate the information with other educators in their school, using data as a reference. 

After our main insights were gained, we developed a Cultural Model to articulate our findings and better inform how administrators currently use data in their work-flow. More importantly, it reveals the value points, and motivations of different stakeholders. It helped us identify the breakdowns of the flow of information between different stakeholders, as well the opportunities we could target as intervention points to design a better product for Renaissance’s users. 

The key takeaways of creating this model are the identified opportunities as follows: 

• Comparing to other demographically similar schools 

• Connecting to other schools, learn from what other schools are doing 

• Providing an easier way to help other educators understand data 

• Providing manipulatable raw data 

• Automatically generate good visuals 

• Providing a better way to figure out which subgroup of students needs help 

• Providing data-driven suggestions on system level interventions 

After we synthesized our generative research results and identified the breakdowns and opportunities, we did several rounds of ideation and pitching, we then narrowed down the ideas to nine product concepts.

Speed-dating with target users for desirability evaluation

Our team and the client evaluate feasibility

After the speed-dating session, we summarized the result had the spring meeting to report the nine ideas and if the target users find them desirable or not. Another purpose for the spring meeting with our client was to discuss the feasibility of all the concepts in terms of implementation, especially the top-ranked ones by the target users.

Summary of Ideation Evaluation

Concept Description

A Similarity Score shows how similar schools are in number form. The score is calculated based on a set of variables such as poverty level, school size, ethnicity ratio.

After the spring meeting with our client, we have been prototyping and testing. We usually have several testing sessions each week and then iterate based on feedbacks we gathered from our potential users.

Parallel Prototyping - early white board sketches

Paper Prototyping

How does it work

Mid-Fi Wire-framing

Visual System

Examples of iteration

High-Fi Prototype

I have to say over the past few months, I’ve learned a lot and am proud of both what I learned and what we as a team achieved. Hopefully Renaissance Learning will release the product in a few month and I can’t wait to hear the feedbacks from our users for further evaluation and improvements.

This capstone project also offered me a great opportunity to learn how a product design should work in the real world. We went through the real-world process such as communicating with different departments including data team, legal team, management team as well as UX team. The success of releasing a product are the shard and common effort of all members in the whole journey.

There are a lot of pieces to making a great product. Looking into the details has made me realize there is more than just the looks. It’s about the usefulness, the experience, the problem it solves, and much more.

The Dream Team

Thanks for reading!