MyBlock App: Using AI to Synthesize Data and Rewrite a User Survey

Challenge:

We’re not collecting good data in our MyBlock user survey

Solution:

Use AI to synthesize the existing user survey results and rewrite the survey to help us more efficiently collect data and improve user experiences

Background:

MyBlock is an H&R Block app that clients use to store documents, communicate with tax pros, and complete their taxes. Under the Help section of the app, users can complete an optional survey to give feedback on their experience with MyBlock.

Originally, my PM asked me to do a quick check on the copy since no one had looked at the survey in years.

However, as I looked through the data, I found that the survey needed a complete overhaul so that we could more efficiently collect data and, ultimately, start to address some of the issues users were telling us about in the survey.

Before:

After:

Take a look at the process (below) to see how we got here…

Initial questions:

1) Why did we choose the copy that currently exists in the survey?

The folks who initially worked on this survey were no longer with the company, so I wasn’t able to get an answer to this.

2) What is the data from the current survey?

3) How can we use the current data to improve this survey and make the it easier to gather data (without having to sift through all the “other” answers)?

Going through existing data

I started by going through the data. How were users answering the survey as it currently exists?

Here’s some of the data that stood out to me:

30.9% of users — the 2nd most common answer — selected “other” for question #1
90.6% of users selected “no” for question #2

Getting to the bottom of “other”

My first goal was to figure out what why users were selecting “other” for question #1.

In the survey, any time a user answered “other”, we gave them an open-ended box to give an answer.

So I pulled that data (from the last year) to see what users were saying.

Well, there were thousands of responses.

I asked the research team for guidance, and they suggested I put the data through AI to find the most common themes users were mentioning.

What I found was that 49.6% of users mentioned software issues, which is a major plot twist because our tax software users don’t use our MyBlock app.

Understanding the plot twist

I read through some specific user responses and talked to my PM to try to better understand why folks would come to an app — that has nothing to do with our tax software — and give negative feedback about our tax software. Here are my conclusions:

Users tried to find help via other avenues (customer service hotline, etc.).
When they couldn’t get the help they needed, they looked for other ways to get in touch with someone.
Users saw our app (MyBlock) as a last resort to get in touch with someone who could help them with their software issues.

Unfortunately, my role has doesn’t have any involvement with the software team, so I wasn’t able to directly “fix” this issue. However, I passed these findings to my PM so that she could pass the intel to relevant folks.

What my role did allow, though, was to change the copy for the survey so that we could better track when users come to MyBlock to seek help for their software woes.

Writing new copy for question 1

In addition to software issues, the other common themes users mentioned in their feedback were: tax pro communication, account management, customer support, and payment.

So, I rewrote the copy to reflect what users are already coming to this forum to tell us.

NOTE: Even though our tax software LOB has nothing to do with MyBlock, I kept that as an option because users are coming to MyBlock to tell us about tax software. This is how we can meet users where they’re at until we are able to make broader improvements to our systems.

A small update to question 1

When I ran this new copy by my PM, she had one small request — add in an option that mentions issues with product upgrades.

While this was not an issue reflected in user feedback for this survey, it is a common theme that comes up in different rooms across the company. I agreed that was a strong enough reason to add it as an option.

Rewriting question 2

Instead of asking users a yes/no question, I wanted to give them options that allowed for more nuance:

Reworking the order

The flow needed a little work, so I changed how the first question was worded and then flipped question 1 and question 2.

Adding question 3

And finally, I added the third question back in but shortened it for more readability and scannability.

Next steps

Even though this project ended up taking significantly more time, I was still able to get final approval from my PM and mark this project complete by the end of the sprint.

However, due to contract hours with Medallia, we still have not been able to implement the new copy.

We are hoping the new copy will be implemented in the next contract cycle.