A Year of Experiments At MailPoet

We close 2018 with a great sense of satisfaction. This has been our best year yet since we launched in 2011.

At the heart of it, we expanded our team, which brought in new talent and fun colleagues. The by-product of this expansion has been a better product and solid growth in revenue.

It was the year we transitioned from the original MailPoet plugin to a new and completely rewritten version. A different plugin, a fresh start and a daunting new path forward. The results have confirmed we made the right decision.

We listened intently to our users throughout the year to decide our priorities, but also to our own instincts. Outside resources also helped us find our way. We were guided by the expanding literature available online for product development and third party consultants, but also valuable experience from our own colleagues.

While we dwell on our successes, it’s important to learn from our failed assumptions as well. For transparency’s sake, we share in this post a few outcomes we experienced first-hand throughout the year.

Each of the projects we’ll explore in this post were initially conducted internally as experiments in order to measure and learn. The process in itself had its learning curves and for which we, as a team, are getting better at.

In this post, we’ll share the experiments we conducted in 2018:

1. Can we improve the new user experience?
2. Should we switch to Gutenberg for the email editor?
3. Can we reduce the number of sending issues for our users?
4. Can we reduce the number of emergency releases?
5. Can we add value to WooCommerce users?
6. Can we provide faster free support?
7. Starting a new plugin instead of updating the original one
8. What’s in store for 2019?

1. What is our new users’ first impression and can we improve it?

More than half of those who install MailPoet stop using it within a month. WordPress users like to try plugins and themes. This high rate might be alarming at first, but it’s in fact pretty normal for free plugins. In fact, MailPoet is better at retention than most plugins if you look at our growth in the plugin repository.

We did see our growth slow down in mid 2018, so we decided to focus on retaining these new users more than we’ve ever done previously.

Like any experiment, we first asked ourselves basic questions. For example, what happens to a user within the first hour of using MailPoet?

To answer this question and others, we went about:

• conducting polls and surveys
• scoring users with the Net Promoter Score
• interviewing dozens of users
• conducting user testing videos
• analyzing support conversations with new users
• enabling in-app chat
• tracking user behavior (for those who opt-in)

The data we gathered was then prioritized and converted into dozens of improvements that target new users. For example, we added a welcome wizard, or improved the template selection step.

Result: The majority of new users are generally satisfied or even impressed with our offering based on our Net Promoter Score (NPS):

With NPS, a score of 40 is good. A score of 50 is great and anything above that is amazing. With this in mind, one can conclude that we’ll keep on trying to improve this.

In regards to our user growth, we set ourselves the goal to end the year with 100,000 active websites. We fell a bit short with 90,000. Nothing to be ashamed of! We’re closer to our goal.

2. Should we switch to Gutenberg for the email editor?

Gutenberg, the new post and page editor for WordPress, created quite a stir in 2018. It is one of the biggest changes to WordPress well, ever.

We believe this upgrade was overdue. WordPress has lapsed compared to other website builders in the past few years, like Squarespace and Wix.

We used Gutenberg ourselves in the past year to write our blog posts. We even redesigned our entire website with Gutenberg in mind, stretching Gutenberg beyond its intended use.

More importantly, our engineering colleagues studied the code extensively to determine if we could replace our current email designer with Gutenberg.

Result: Gutenberg is still too constrained as a post editor. We need to wait for phase two of the Gutenberg project when it starts tackling menus, sidebars and footers. We hope to see its source code offering the kind of extended capabilities that MailPoet needs.

3. Can we reduce the number of sending issues for our users?

Email deliverability issues are quite common for any website owner trying to send emails from WordPress. In fact, a third of our support queries relate to sending problems:

• The plugin cannot send emails because the host is not configured to do so;
• Emails don’t arrive at all because the host has a bad reputation; and
• Emails don’t arrive at all because the email service provider rejects them.

This spurred us to launch our own MailPoet Sending Service, both as a paid offering but also as a free plan that could help the large majority of our users.

Our sending infrastructure is quite a feat: 98% of all emails get delivered, which is at the top of industry standards. Read more on our in-house sending service.

We thought that this would be the magic bullet needed to fix all sending issues.

Result: We were a little too optimistic. Only 25% of our users have signed up for the sending plan, free or paid.

Perhaps one of the reasons that this number is small is that a good chunk of WordPress users entirely ignore that there might be delivery problems until they run into them, if that ever happens.

Furthermore, we also know that many users who can’t send from day one stop using our plugin altogether. A large variety of causes can occur that relate to the WordPress installation or the host. We could do a better job at catching these scenarios and we’re working on it.

On the up side, our sending plans drive most of our business today. This will only grow over time thanks in large part to the high satisfaction rating of our own in-house sending service technology.

4. Can we reduce the number of emergency releases?

This is a typical experiment of any software development team and it gives an insight into our inner workings.

Developing a fully-fledged email plugin within WordPress brings its share of complexities and with them, a higher chance of bugs, failures, and sometimes websites that crash. When that happens – we go into emergency mode, publishing a fix as soon as possible, or unpublishing the faulty release.

When the unfortunate happens, a lot of anxiety is shared between our users and ourselves. This 1-star rating is a compelling example.

To put the above rating into context, it’s important to understand the minefield in which we operate:

• Hundreds of hosts with different configurations and varying performances;
• Websites with different themes and thousands of plugins, not to mention a plethora of WordPress versions;
• Emails are displayed in more than a dozen different email clients, desktop or mobile; and
• Spam filters that adapt their rules based on changing threats.

In 2017, we had 13 emergency releases. With MailPoet, we had even more.

Addressing the quality of our releases requires an ongoing multipronged approach, which is at the heart of our ambitions. Our plans this year included:

1. Hiring a full time Quality Assurance colleague;
2. Extending our automated test coverage. Think of a robot that simulates a human being using MailPoet with their browser;
3. Continuing what is called Test Driven Development, which means that our all written code is also tested automatically, but without a browser this time;
4. Stopping support for old and no longer maintained PHP language versions sequentially (5.3, 5.4 and 5.5) which represents around 15% of our user base. This means that we can leverage previously forbidden capabilities in PHP and third party libraries to provide simpler, faster, less error-prone and more powerful features for our customers;
5. Extending our suite of tools to automatically detect errors while we’re coding;
6. Shortening peer review times by developers thanks to more stringent standards and other tactics; and
7. Improving our “continuous delivery” metrics. This is how fast we add or change code, deploy, release and, if necessary, revert (undo!), which allows us to deliver new features and respond to existing issues more quickly.

Result: We reached our goals of limiting emergency releases to the bare minimum of the 47 releases in 2018. Only nine required an emergency release.

We strive to reduce emergency releases in 2019 even further. Ideally, we want to remove all stress our users might encounter when they update.

5. Can we add value for WooCommerce users?

It’s quite astonishing that a quarter of our users have WooCommerce installed. It’s become that ubiquitous. Which makes sense — WooCommerce is the world’s most popular e-commerce platform.

With this is in mind, we’ve wanted to add new features specifically for WooCommerce store owners:

• Default customer list;
• Segmentation of customers;
• Email marketing automation;
• Drag and drop products into emails; and
• Tracking revenue per email.

Result: We haven’t done half of what we’ve wanted to do— 2018 was not the “WooCommerce” year we set out to achieve.

We’re a small team and the reality is that we simply cannot do everything. We tackled priorities that were, in hindsight, more important:

• Making it easier for new users to get started with MailPoet;
• Satisfying the needs of our historical users;
• Improving quality of releases; and
• Creating tools to help adhere to the European directive on user privacy (GDPR).

6. Can we provide faster free support?

We have always offered free support, but with long waiting times of anything between 24 hours and a full week.

Most of the questions we get from free users happen during the installation and configuration stage. They’re evaluating MailPoet and trying to get it to work or simply don’t understand some feature.

For the sake of improving our onboarding experience, we decided to pay more attention to these free users by running these experiments:

• Improved our response times in the WordPress plugin forums;
• Tried a chat for first time users right within the plugin;
• Enabled free users to email us in-app; and
• Slashed our email response times to free users from 24 down to 6 hours.

Result: This proved a great success. We achieved it thanks to two prerequisites:

1. Our support team has been engaged in improving the plugin to remove friction, confusion and escalating bugs quickly; and
2. The support and development teams worked hand in hand to prioritize improvements that would alleviate the support workload.

7. Starting a new plugin instead of updating the original one

This was a unique experiment in many regards.

We decided to launch MailPoet version 3 as a new and separate plugin in late 2017. A choice we made for technological reasons.

There were 400,000 websites using the original MailPoet 2 plugin since 2011. So we were asking them to install a new plugin instead of just updating to a newer version.

We made this step more challenging for everyone because:

• Our automatic upgrade tool only partially upgraded users. They still had to recreate their templates from scratch, amongst other things;
• Some free features became Premium, such as Welcome Emails and detailed statistics. Free users rebelled against us.
• Not all features in the old MailPoet made it into the new one. Changing people’s habits, especially with a mission-critical plugin like ours, backfired; and
• Our pricing model got more complex since we added dozens of monthly plans that include a sending service.

Result: Around 50,000 websites switched to MailPoet 3, which is probably half our active users.

The others decided to stick to the old MailPoet 2 for the time being or just switched to another solution.

Because of this mixed result, we spent a considerable amount of time in 2018 listening to all our users. This helped us prioritize and communicate better. It also meant we added a lot more value in the free version and reinstated features that were previously in the old MailPoet 2.

Could we have done things fundamentally different? Yes, for sure. But who has the retrospective advantage of trying something new? In the end, we proved to ourselves that our team can adapt quickly to unforeseen scenarios.

Summary

We ran several experiments throughout 2018 and measured the results:

1. We found out that first time users had a mitigated first impression of MailPoet. We improved the new user experience and we shall continue throughout 2019 to improve user retention;
2. We’re going to wait a little longer before deciding to replace our email editor with Gutenberg;
3. Offering a sending service, even for free, does not resolve all sending issues. We could do a better job at helping users configure MailPoet properly;
4. We had nine emergency releases in 2018, which is a testament to our quality promise;
5. We wanted to expand WooCommerce features, but instead focused on other more pressing solutions;
6. Our free support has improved tremendously thanks to a team effort;
7. We launched the new version of MailPoet as a separate plugin. There were a few difficulties, which needed to be addressed throughout the year.

What’s in store for 2019?

Many of our efforts from last year will port into this year. Here’s an unordered list of items on our plate for 2019:

• Improvements to the email editor to make it a smoother experience;
• Improving and adding WooCommerce specific features;
• Improving our release quality track record;
• Maintaining our high levels of deliverability by:
  • Automatically unsubscribing disengaged subscribers automatically from your lists;
  • Progressively adopting stricter DMARC policies;
• We’ll be publishing a lot more tips and tricks on our blog too!

If you have any questions, leave them in the comments below!