Join gaming leaders online at GamesBeat Summit Next this upcoming November 9-10. Learn more about what comes next. 

When Kristel Kruustük finished high school in Estonia, she went to London and became a waitress. Her friends turned her on to the tech scene, and she decided to go back to school to study software development. Kruustük fell in love with software testing and started her own company, Testlio, testing mobile apps with her husband Marko Kruustük.

Their company took first place at the AngelHack hackathon in London in 2012. And then it embarked on its rapid growth to become an expert in testing mobile apps. Kruustük has built out the headquarters team in San Francisco, while Marko Kruustük serves as chief technology officer with the team in Estonia. And now Testlio’s quality assurance work has touched apps that reach more than 650 million monthly users.

The company has 65 employees, and it has access to 1,000 freelancer testers. But Kruustük said that Testlio hires fewer than 3 percent of tester applicants because it looks for those who have a real passion for testing. Kruustük’s goal is to raise the store rating of her client’s apps once they start working with her company. Customers include Microsoft, Lyft, the NBA, and

At 28, Kruustük is a rare female CEO in Silicon Valley. She said 50 percent of her employees are women or minorities, and 40 percent of the company’s leadership team is female. A few of the things she has learned? Diversity makes her business stronger. Apps that are updated more frequently have higher app store ratings. Twenty-three percent of customers abandon an app after one use. And 67 percent of customers mention bad experiences as a reason for quitting an app.

Here’s an edited transcript of our interview.

Above: Kristel Kruustük grew up in Estonia and runs Testlio from San Francisco.

Image Credit: Testlio

VentureBeat: Tell me what your company does.

Kristel Kruustük: We’re making a global community of testers on our platform, to work with companies like Lyft and Microsoft, for example. We verify our testers, make sure they’re real experts, and match them with companies that need testing for their products. The idea of Testlio was born out of my own frustration, back in 2012. I’d been working as a tester for about three years. I’d just graduated college. As an Estonian, I wanted to move away from Estonia and see what could happen abroad. My cofounder, now my husband, Marko and I decided to move to London.

In the beginning of 2012 we ended up in London and I continued my full-time career as a tester. At the time, I was the only tester in my company, and I felt like I really wasn’t improving myself fast enough. I had developed a real passion for testing. For such a long time, testers and testing had been an afterthought for companies. It’s not something you think about when you’re starting to build a company. Testing was down the line, something you eventually had to think about.

I wanted to develop my skills further, so I reached out to different thought leaders in the industry and went to a lot of different meetups. I signed up for different crowdsourced testing platforms. The idea was simple — they just connected testers from all over the world on one platform and matched them with companies.

After a couple of months working actively on these platforms on the weekends, I realized a couple of things. One, they didn’t really care about quality. They paid testers per bug or per issue. That meant testers weren’t incentivized to dig deep in the product and figure out real issues. Customers ended up getting loads of low- or medium-priority issues, even though they wanted the biggest critical issues fixed so they could get into production.

The other aspect was, since the platform connected testers from all over the world, testers started competing with each other. The first one who reports the issue gets paid, so there’s no teamwork happening. In my opinion, quality always should have been part of a team’s responsibility. Everyone has to think about it – designers, developers, testers, it’s everyone’s responsibility. I felt like it couldn’t go any longer like this.

I ended up seeing an advertisement on Eventbrite for a hackathon called AngelHack. AngelHack was a global hackathon series, and one of the competitions happened in London. I signed up for that, and since then, the rest is history. Now we’re a team of 65 people. We have an office with about 20 people here in San Francisco. We have about 45 more in Estonia. Last year, we did a series A round of about $6.5 million. In total, our funding is about $7.5 million. I started the company when I was 23 and now I’m 28. I don’t know if that’s young or not. [Laughs]

Above: Marko Kruustük is chief technology officer at Testlio.

Image Credit: Testlio

VB: Can you tell me more about that hackathon?

Kruustük: Marko ended up going to it with me, because he started believing in this idea. He’d been building companies for almost 10 years at that point. We went to the hackathon and formed a team, just the two of us, because we didn’t have anybody else who wanted to join. I was pretty bad at presenting, so I couldn’t attract talent very easily to the team at that point. After 25 hours of work, we ended up getting third place out of 64 ideas, which was a big achievement for us.

The funny thing was, we didn’t mention anything about the team when I pitched the idea for Testlio in front of the judges. They thought, “Oh, this is a great idea, but your team is just two random people getting together at this event. It doesn’t seem like that will be super successful.” But Marko met the founder of the event the next day, after the London event, and told him the entire story — that we had been working together on different projects for a long time, that we’d been together for three years. They guy said, “Why didn’t you tell me that? If you’re starting a company, the team is the most important aspect of your success.” He offered us a ticket and two months later we were able to present at the finals in San Francisco.

We got to present to companies like Andreessen Horowitz and Google Ventures, and we ended up winning. We won $25,000, which was our first investment. At that point I also managed to bring in our first paying customer. That helped me quit my full-time job and focus on Testlio to see what was going to happen. It’s pretty crazy. Marko just happened to go to the Google campus in London that next day and run into the founder of AngelHack and tell him our story. That helped us tremendously.

VB: What kinds of products are you testing? Your competition could be all those testing companies that employ lots of people in India and places like that.

Kruustük: Right. Another aspect — I know you’re a big gamer. Gaming companies, for example, use their entire community to participate in testing, because those communities are so passionate. They’re already in the weeds with the platform, understanding the complexities and what can and can’t be done.

The types of companies we usually work with are bigger enterprises. We partner with internal QA teams. One of the benefits of using Testlio is we have a global reach. Companies that want to expand, for example, to Europe, they don’t necessarily want to build up a team in Europe. They can tap into the community at Testlio and we’ll take care of it.

The same thing happens when — you mentioned different software testing service providers that have a huge pool of testers in a particular place. The benefit we offer is the globalization aspect. We have testers in specific locations. There are some big media companies that use our services because we can tap into local communities. We source testers and make sure they’re experts and can provide good feedback. We can ensure quality in cases like that.

One of our customers is Lyft, the ride-sharing company. They have a huge internal QA team, but they use us to run regression tests and exploratory testing overnight. They give us test plans to execute in their evening, in San Francisco, and they get their results in the morning. That’s another benefit of having testers all over the world.

Above: Testlio’s offices

Image Credit: Testlio

VB: Do you have a broader range of testers you have access to, or are those 65 people all the testers you have?

Kruustük: Oh, no. The community is a different piece of Testlio. Our internal employees are all product managers and developers who build the platform to make our services scalable. We are a service company, but it’s enabled by the platform. That’s what’s going to help us scale very big. Our internal employees are mostly sales, marketing, and product development.

The community — today, we have a pretty huge community, about 300 expert testers. We also have a further community on standby. At the moment, as we verify testers who want to become part of our community, the acceptance rate is about 3 or 4 percent.

VB: How many people do you have access to as testers altogether, then?

Kruustük: I can say more than 1,000 today.

VB: Is there a particular project that you can talk more about as far as something you guys have tested or helped bring to market?

Kruustük: I can’t name too many companies, but Lyft has a great story as far as how they partnered up with Testlio. We’ve been working with them for quite a long time now. It’s helped them scale internally. They can focus more on high-level things, and we take care of the testing aspect.

When we talk about the Testlio platform — I try to make sure we’re perceived as a product company as well as a service company. It’s a service, but it’s enabled by a product. The platform itself connects the testers. It helps us to provide good service to our customers, because through the platform and the tools we’ve built for our testers, they can communicate with each other. They have a test plan management system built in. They can see our customers’ app store ratings and figure out what are the biggest frustrations for users so we can focus our testing on things that really matter.

Anyone can find issues in the apps that we work with on a daily basis. But it really comes down to understanding end users and figuring out what their priorities are. If 50 percent of customers are facing an issue on iPhone 6+, we should probably fix that issue as soon as possible before customers get frustrated and move to another app. That’s why a lot of the bigger companies we work with today have realized the importance of customer experience.

Software testing itself has been the fastest-growing IT segment. Companies used to spend about 12 or 13 percent of their budgets on testing. Now it’s growing to more like 25 or 26 percent, and it’s projected to grow to around 32 percent by the end of next year. Everyone realizes how much competition they have. You need to release fast and you need to maintain high quality. You can’t release products full of issues to your customers, because they can instantly turn to your competitors that offer a better customer experience. We’ve been very successful in partnering up with different QA teams in big organizations.

Above: Another view of Testlio’s offices.

Image Credit: Kristel Kruustük

VB: What would you say you’ve learned about product design or the right way to create tech products?

Kruustük: One of the biggest aspects for us since the beginning has been, are we a product company or are we a services company? We’re both, as I say. When you talk about product design, the biggest thing it comes down to is understanding your customers, listening to your customers, and building the product as they need it.

Back in the beginning of 2014, we had just graduated from the Techstars accelerator. They’re one of the best accelerators in the world, accepting just 1 percent of applicants into their program. It’s a three-month incentive program. After that, we wanted to start raising funding. I came to the U.S. and started talking with different investors. At the same time, I was looking for customers. One thing investors brought back to us was, “You don’t have a subscription model. You’re basically a services company. Customers come on and off. You don’t know their schedule. There’s nothing predictable in your model.”

I met the founder of a company called Acompli, which was acquired by Microsoft later on. He was very clear about what he wanted from us, and for the first three months we were basically competing with another services provider that had internal QA people working from India. We had test cycles happening every weekend.

The founder of this company was very clear on what he wanted to see from Testlio — pulling in app store reviews, categorizing them, and understanding them. “Fifty percent of our customers on this device experience issues with this feature. Fifty percent of the one-star reviews are because we’re missing this feature in this product.” Every week, what they got from us was a clear overview of what was happening in the product. “Here are priority issues we found in this cycle. Here’s customer feedback we have categorized for you. These are the top bugs and top feature requests.”

That’s one of the biggest things I learned about product design. You need to listen to your customers. You have to innovate, as well, but if you listen to your customers carefully, you can spot the diamonds that will help you envision where the product needs to go.

VB: It seems like the best way to do things is to have this combination of internal and external testing, as well as professional testing and community testing.

Kruustük: Exactly. In Lyft’s case, for example, they also have a huge community. They have thousands of people that are part of their beta testing group. Those people also provide feedback into their support channel. “I downloaded this preproduction build. Here are the issues I’m seeing on my device.” All that feedback has to be carefully organized so it can have an impact on the real product. Every customer has hundreds of issues in their backlog, but it comes down to prioritization. That’s where we come in and help do that through our expert testers and the platform we have.

VB: Where are you headed from here? Do you have a road map of interesting things you’re looking forward to?

Kruustük: Always. I don’t know that I can share too much at this point, but one of the biggest things for us is building a scalable service. All the workflow that our internal people are doing has to be automated. That’s the biggest thing at the moment. We want to build a platform that scales and automates as much as possible where it makes sense. Identifying insights and different trends can help our customers identify issues.