When Kristel Kruustük finished high school in Estonia, she went to London and became a waitress. Her friends turned her on to the tech scene, and she decided to go back to school to study software development. Kruustük fell in love with software testing and started her own company, Testlio, testing mobile apps with her husband Marko Kruustük.
Their company took first place at the AngelHack hackathon in London in 2012. And then it embarked on its rapid growth to become an expert in testing mobile apps. Kruustük has built out the headquarters team in San Francisco, while Marko Kruustük serves as chief technology officer with the team in Estonia. And now Testlio’s quality assurance work has touched apps that reach more than 650 million monthly users.
The company has 65 employees, and it has access to 1,000 freelancer testers. But Kruustük said that Testlio hires fewer than 3 percent of tester applicants because it looks for those who have a real passion for testing. Kruustük’s goal is to raise the store rating of her client’s apps once they start working with her company. Customers include Microsoft, Lyft, the NBA, and Hotels.com.
At 28, Kruustük is a rare female CEO in Silicon Valley. She said 50 percent of her employees are women or minorities, and 40 percent of the company’s leadership team is female. A few of the things she has learned? Diversity makes her business stronger. Apps that are updated more frequently have higher app store ratings. Twenty-three percent of customers abandon an app after one use. And 67 percent of customers mention bad experiences as a reason for quitting an app.
Here’s an edited transcript of our interview.
VentureBeat: Tell me what your company does.
Kristel Kruustük: We’re making a global community of testers on our platform, to work with companies like Lyft and Microsoft, for example. We verify our testers, make sure they’re real experts, and match them with companies that need testing for their products. The idea of Testlio was born out of my own frustration, back in 2012. I’d been working as a tester for about three years. I’d just graduated college. As an Estonian, I wanted to move away from Estonia and see what could happen abroad. My cofounder, now my husband, Marko and I decided to move to London.
In the beginning of 2012 we ended up in London and I continued my full-time career as a tester. At the time, I was the only tester in my company, and I felt like I really wasn’t improving myself fast enough. I had developed a real passion for testing. For such a long time, testers and testing had been an afterthought for companies. It’s not something you think about when you’re starting to build a company. Testing was down the line, something you eventually had to think about.
I wanted to develop my skills further, so I reached out to different thought leaders in the industry and went to a lot of different meetups. I signed up for different crowdsourced testing platforms. The idea was simple — they just connected testers from all over the world on one platform and matched them with companies.
After a couple of months working actively on these platforms on the weekends, I realized a couple of things. One, they didn’t really care about quality. They paid testers per bug or per issue. That meant testers weren’t incentivized to dig deep in the product and figure out real issues. Customers ended up getting loads of low- or medium-priority issues, even though they wanted the biggest critical issues fixed so they could get into production.
The other aspect was, since the platform connected testers from all over the world, testers started competing with each other. The first one who reports the issue gets paid, so there’s no teamwork happening. In my opinion, quality always should have been part of a team’s responsibility. Everyone has to think about it – designers, developers, testers, it’s everyone’s responsibility. I felt like it couldn’t go any longer like this.
I ended up seeing an advertisement on Eventbrite for a hackathon called AngelHack. AngelHack was a global hackathon series, and one of the competitions happened in London. I signed up for that, and since then, the rest is history. Now we’re a team of 65 people. We have an office with about 20 people here in San Francisco. We have about 45 more in Estonia. Last year, we did a series A round of about $6.5 million. In total, our funding is about $7.5 million. I started the company when I was 23 and now I’m 28. I don’t know if that’s young or not. [Laughs]
VB: Can you tell me more about that hackathon?
Kruustük: Marko ended up going to it with me, because he started believing in this idea. He’d been building companies for almost 10 years at that point. We went to the hackathon and formed a team, just the two of us, because we didn’t have anybody else who wanted to join. I was pretty bad at presenting, so I couldn’t attract talent very easily to the team at that point. After 25 hours of work, we ended up getting third place out of 64 ideas, which was a big achievement for us.
The funny thing was, we didn’t mention anything about the team when I pitched the idea for Testlio in front of the judges. They thought, “Oh, this is a great idea, but your team is just two random people getting together at this event. It doesn’t seem like that will be super successful.” But Marko met the founder of the event the next day, after the London event, and told him the entire story — that we had been working together on different projects for a long time, that we’d been together for three years. They guy said, “Why didn’t you tell me that? If you’re starting a company, the team is the most important aspect of your success.” He offered us a ticket and two months later we were able to present at the finals in San Francisco.
We got to present to companies like Andreessen Horowitz and Google Ventures, and we ended up winning. We won $25,000, which was our first investment. At that point I also managed to bring in our first paying customer. That helped me quit my full-time job and focus on Testlio to see what was going to happen. It’s pretty crazy. Marko just happened to go to the Google campus in London that next day and run into the founder of AngelHack and tell him our story. That helped us tremendously.
VB: What kinds of products are you testing? Your competition could be all those testing companies that employ lots of people in India and places like that.
Kruustük: Right. Another aspect — I know you’re a big gamer. Gaming companies, for example, use their entire community to participate in testing, because those communities are so passionate. They’re already in the weeds with the platform, understanding the complexities and what can and can’t be done.
The types of companies we usually work with are bigger enterprises. We partner with internal QA teams. One of the benefits of using Testlio is we have a global reach. Companies that want to expand, for example, to Europe, they don’t necessarily want to build up a team in Europe. They can tap into the community at Testlio and we’ll take care of it.
The same thing happens when — you mentioned different software testing service providers that have a huge pool of testers in a particular place. The benefit we offer is the globalization aspect. We have testers in specific locations. There are some big media companies that use our services because we can tap into local communities. We source testers and make sure they’re experts and can provide good feedback. We can ensure quality in cases like that.
One of our customers is Lyft, the ride-sharing company. They have a huge internal QA team, but they use us to run regression tests and exploratory testing overnight. They give us test plans to execute in their evening, in San Francisco, and they get their results in the morning. That’s another benefit of having testers all over the world.
VB: Do you have a broader range of testers you have access to, or are those 65 people all the testers you have?
Kruustük: Oh, no. The community is a different piece of Testlio. Our internal employees are all product managers and developers who build the platform to make our services scalable. We are a service company, but it’s enabled by the platform. That’s what’s going to help us scale very big. Our internal employees are mostly sales, marketing, and product development.
The community — today, we have a pretty huge community, about 300 expert testers. We also have a further community on standby. At the moment, as we verify testers who want to become part of our community, the acceptance rate is about 3 or 4 percent.
VB: How many people do you have access to as testers altogether, then?
Kruustük: I can say more than 1,000 today.
VB: Is there a particular project that you can talk more about as far as something you guys have tested or helped bring to market?
Kruustük: I can’t name too many companies, but Lyft has a great story as far as how they partnered up with Testlio. We’ve been working with them for quite a long time now. It’s helped them scale internally. They can focus more on high-level things, and we take care of the testing aspect.
When we talk about the Testlio platform — I try to make sure we’re perceived as a product company as well as a service company. It’s a service, but it’s enabled by a product. The platform itself connects the testers. It helps us to provide good service to our customers, because through the platform and the tools we’ve built for our testers, they can communicate with each other. They have a test plan management system built in. They can see our customers’ app store ratings and figure out what are the biggest frustrations for users so we can focus our testing on things that really matter.
Anyone can find issues in the apps that we work with on a daily basis. But it really comes down to understanding end users and figuring out what their priorities are. If 50 percent of customers are facing an issue on iPhone 6+, we should probably fix that issue as soon as possible before customers get frustrated and move to another app. That’s why a lot of the bigger companies we work with today have realized the importance of customer experience.
Software testing itself has been the fastest-growing IT segment. Companies used to spend about 12 or 13 percent of their budgets on testing. Now it’s growing to more like 25 or 26 percent, and it’s projected to grow to around 32 percent by the end of next year. Everyone realizes how much competition they have. You need to release fast and you need to maintain high quality. You can’t release products full of issues to your customers, because they can instantly turn to your competitors that offer a better customer experience. We’ve been very successful in partnering up with different QA teams in big organizations.
VB: What would you say you’ve learned about product design or the right way to create tech products?
Kruustük: One of the biggest aspects for us since the beginning has been, are we a product company or are we a services company? We’re both, as I say. When you talk about product design, the biggest thing it comes down to is understanding your customers, listening to your customers, and building the product as they need it.
Back in the beginning of 2014, we had just graduated from the Techstars accelerator. They’re one of the best accelerators in the world, accepting just 1 percent of applicants into their program. It’s a three-month incentive program. After that, we wanted to start raising funding. I came to the U.S. and started talking with different investors. At the same time, I was looking for customers. One thing investors brought back to us was, “You don’t have a subscription model. You’re basically a services company. Customers come on and off. You don’t know their schedule. There’s nothing predictable in your model.”
I met the founder of a company called Acompli, which was acquired by Microsoft later on. He was very clear about what he wanted from us, and for the first three months we were basically competing with another services provider that had internal QA people working from India. We had test cycles happening every weekend.
The founder of this company was very clear on what he wanted to see from Testlio — pulling in app store reviews, categorizing them, and understanding them. “Fifty percent of our customers on this device experience issues with this feature. Fifty percent of the one-star reviews are because we’re missing this feature in this product.” Every week, what they got from us was a clear overview of what was happening in the product. “Here are priority issues we found in this cycle. Here’s customer feedback we have categorized for you. These are the top bugs and top feature requests.”
That’s one of the biggest things I learned about product design. You need to listen to your customers. You have to innovate, as well, but if you listen to your customers carefully, you can spot the diamonds that will help you envision where the product needs to go.
VB: It seems like the best way to do things is to have this combination of internal and external testing, as well as professional testing and community testing.
Kruustük: Exactly. In Lyft’s case, for example, they also have a huge community. They have thousands of people that are part of their beta testing group. Those people also provide feedback into their support channel. “I downloaded this preproduction build. Here are the issues I’m seeing on my device.” All that feedback has to be carefully organized so it can have an impact on the real product. Every customer has hundreds of issues in their backlog, but it comes down to prioritization. That’s where we come in and help do that through our expert testers and the platform we have.
VB: Where are you headed from here? Do you have a road map of interesting things you’re looking forward to?
Kruustük: Always. I don’t know that I can share too much at this point, but one of the biggest things for us is building a scalable service. All the workflow that our internal people are doing has to be automated. That’s the biggest thing at the moment. We want to build a platform that scales and automates as much as possible where it makes sense. Identifying insights and different trends can help our customers identify issues.
VB: It seems like your field might be like a lot of fields, where automation can replace people. Is that something that worries you, or have you figured out how to stay ahead of that? We hear a lot now about how AI can replace people in their jobs, but it can also assist people.
Kruustük: That’s how we see it, as well. It’s necessary to automate repetitive tasks, but when it comes down to really grasping the context of the product and figuring out what customers really need, you need human assistance.
When you think about building a company, the product can change very quickly. The design can change. That’s where you can’t really automate the process, because the return on investment ends up being very low. Instead of writing new tests, you’re just maintaining the old ones. That’s what a lot of companies struggle with. But the bigger customers we work with have automation in place and we help them do that as well, or our community does.
VB: Are you self-sustaining at this point, or do you plan to raise more money?
Kruustük: We’re not raising at the moment. We have enough in the bank. We have customers. At this point, it’s really about figuring out the machine first. As I said before, we want this to be scalable and to be automated where it makes sense. That’s our focus right now.
VB: How did you decide on San Francisco as a good place to have your headquarters?
Kruustük: Back in 2012, when we started the company, we moved back to Estonia at first. We’d quickly realized we needed to lower our costs and figure out what we were going to do next. We moved in with our parents for a bit, and then one of the entrepreneurs we’d met in Estonia — the founder of Pipedrive, actually — told us to apply to an accelerator in the U.S.
We ended up going to Techstars in Austin, Texas. After a three-month program we were sure we wanted to build the team there in Austin. The U.S. is a great market and companies want innovation. They want better services. They’re very knowledgeable about different competitors in the market and they know exactly what they want. It was clear that we needed to stay in the U.S.
When we started looking for a salesperson, we found a very good one here in San Francisco, so we ended up coming here. She has scaled her team to about seven people. That’s how we ended up here. It’s better as well, though, because we have a lot of customers here on the West Coast.
VB: You’re relatively rare as a woman leading a tech startup. How has that been for you?
Kruustük: When I started my company, especially when we went to Techstars Austin, I really didn’t understand the importance of talking about the topic of women in tech. “Why do we have to force people to do something that we really don’t want to do?” I wanted to be in tech and I never let anyone stop me from doing that. But then I realized that there are so many different cultural issues. People grow up in different ways and it influences the way they think about possibilities in the world. That’s when I realized how important it is to talk about this.
I’ve personally never faced any issues of discrimination. There was only one incident, a couple of years back, where I met a local entrepreneur in Moldova and he told me I was only able to raise a seed round because I was a good-looking blond woman or something. That was a little surprising. But that’s the only incident I’ve ever had. Now the topic’s very close to my heart. That’s why I think it’s important for companies to make their organizations as diverse as possible. It’s proven that companies that have more diversity when it comes to culture and gender — it just makes you more successful.
At this point, we’re 40 percent women and 50 percent minorities in our organization, all told. I think we’ve done well. At any company, it’s something you need to think about from the beginning. For us, it came naturally, though, because between me and Marko — the fact that I’m a woman founder, I’ve already created the beginning of a more welcoming environment for people. Recently, we’ve had many candidates coming in here in the U.S., and they notice that the company’s so diverse. It makes them want to work here. I’m very happy about that.
VB: What about testing was interesting to you when you first got into it on your own?
Kruustük: You can’t actually go to school and learn testing. It’s something people stumble into. Whenever you meet another tester somewhere, you can ask them the same question, and 99 percent of the time they’ll say, “I just fell into it.” But like I said before, it’s a huge market.
The way it worked for me, in 2008 I graduated high school in Estonia. I didn’t really know what I wanted to do with my life, so I spent that summer working in London as a waitress and hanging out with my sister’s friends. They were all working tech, and they all told me, “Hey, you should be a part of the future with us. You should study computer science.”
I thought, “Okay, why not?” and put in my papers at a college in Estonia. I got in, and for the next two years I studied development. During that second year of school, although I hadn’t had too much practice at that point I wanted to start looking for a job. A lot of friends in class were telling me that if I wanted an entry-level job in tech, I should try testing. That’s an assumption a lot of people start with, that testing is an entry-level job.
For me, it sounded like a great opportunity, but quickly, after four months working as a tester — with that look into the industry I realized that there’s a huge opportunity. It’s something that a lot of people don’t talk about. I really enjoyed testing, so I decided not to continue my career path all the way into becoming a developer. I just wanted to become a really good tester.
VB: Were you just working with any kind of software, or were you working with anything in particular?
Kruustük: The first company I worked for was a small software development boutique in Estonia. It was maybe six developers building websites for different travel companies. I was basically just thrown into the project. “Hey, just find issues for us.” That was the only expectation set for me at that point. So I went in and started finding issues.
Later on in my career, obviously, I realized that testing is way more than just finding issues. It’s really about understanding what our customers need. As I said before, everyone has lots of issues, and it comes down to prioritization. That’s where testers need to grasp the entire workflow and how our customers think and put things into perspective.
VB: Should your testers be people with computer science degrees, or do they just need to be consumers?
Kruustük: At this point, most of our testers have a background. They’ve been working as testers for at least two or three years. That’s the type of people we currently have in our community. They’re not just saying, “Hey, here’s this issue.” They can explain exactly why something isn’t working, where the issue is coming from, and what you can do to reproduce it. Then they send it to the developer to investigate so they can fix it.
VB: Is there a quality of debugging scale in there, in addition to just being someone who tests things and spots things?
Kruustük: Right. We don’t go into code, but we debug in the sense of, “Okay, I’ve run into this issue. Now I need to think back on the steps I took to make it happen.” That’s what makes testing more complicated. You have to be very organized and structured in your process of exploring and learning about the product you’re testing. Critical issues can happen for so many reasons. Maybe your device just has low CPU. There are so many aspects that can affect the performance of the product. It’s a tester’s responsibility to figure out where and why and what is happening.
VB: On the product side, I’ve seen some companies that have talked about products that capture and record and document a bug, so they make it easier to send off to someone who’s supposed to fix it.
Kruustük: Right, there are SDKs that have been built into some products. Different companies out there do that. But you still need testers to figure out how to find an issue and track it down. It’s a different part of the process. If your product crashes, then you need to get the logs and all this other data. A streaming application, for example, you need to understand if the streaming is coming from the right funnel. You need to grab the network logs and figure out where things are coming from.
VB: Was there something in your background that led you to want to become an entrepreneur?
Kruustük: I didn’t necessarily aspire to be an entrepreneur. One thing, though, when I look at my parents, my father owned a company for some time in Estonia. I’ve always liked to organize things, to figure out things, to take charge. I love project management. It’s just something I was driven to, I think. But it’s not as if I dreamed about being an entrepreneur. I just wanted to be good at something I do, to find something I enjoyed.
Estonian culture definitely has that aspect. Estonia still has the most startups per capita in the world, I think, which is amazing. It comes out of the big success stories like Skype. Skype was founded a long time ago now, but it was the first company where people saw that you can really build a successful, global company from a place like Estonia.
The story of Skype has been a great inspiration for the entire ecosystem, and now there are so many other companies coming out of it — Pipedrive, TransferWise. Maybe you’ve heard of Starship Technologies, as well, with their robot that delivers food to your door. Or that’s the vision. There are so many great things happening. It’s very inspiring for Estonians. We have a very small population — only maybe 1.3 million — we’re innovative. We want to change the world.
If I’d known entrepreneurship was so hard maybe I wouldn’t have started, but I’ve enjoyed every minute, despite all the challenges. I’ve learned that the only way to go forward is to keep improving. I like what Microsoft’s CEO is always saying — that you have to be a “learn-it-all”, not a “know-it-all”. That’s a mantra I really like. When we’re hiring people, it’s something that I look out for. You’re never going to be perfect. You can always improve everything you do. When we develop the product, we can always improve it and make it better for our customers — faster, higher quality. There are all these possibilities. You have to keep an open mind.