We all use various services and can admit that they send us different transactional emails: user registration, password reset, notifications, and other types. It’s a part of the services’ core functionality.
But sometimes, we don’t get a receipt with order details, or a password reset link is broken. Something has gone wrong. Someone hasn’t paid the required attention to testing their emails.
To avoid such frustrations for users, DevOps engineers and dev teams must take email testing seriously and be aware of common issues that untested emails can bring.
An untested email was sent. So what?
Imagine you have sent an email to the wrong person or haven’t proofread it. You’d probably be upset about it. These are only personal cases.
As an illustration, your app sends 1,000 password reset emails per day on average, and they have a broken link. Users can’t update their passwords, they start reaching out to your support. A few teams are affected and need to fix the issue immediately, but most importantly, users got a poor CX.
In this way, sending untested emails can damage your company’s reputation and customers. Not mentioning the entire email sequence or campaign becomes pointless.
Okay, a broken link. And?
This is only one case. The issue might be more severe.
No matter your business size, you need to understand that email testing is a must. Email testing is a set of techniques to debug and polish many email parameters. And yes, it’s not only the duty of marketers or the product team.
Critical issues related to broken emails often come from a tech side, which is the area of software developers, testers, and DevOps engineers. Email testing is normally an automated process, a part of the general software testing cycle.
For DevOps engineers, the areas to handle emerge:
- what flows of email testing we can automate, and how
- how to secure the testing process
- how to establish an effective cross-team collaboration
Let’s explore why these points are so important to handle.
Is automation a key?
Emails have many parameters to be tested:
- CSS/HTML templates
- deliverability and spam scores
- infrastructure and authentication
- copy and links
Imagine manual testing each parameter in each group of emails that your app or website sends. It’s challenging, time-consuming, ineffective. Here’s automation is indeed a key. Many email testing tools have automated testing flows and APIs, which allow you to speed up a testing process.
However, not all tools can handle the testing of every element. For instance, you need to check email preview and responsiveness that requires the use of one tool; to ensure deliverability, you need to use another tool, and so on.
This makes you use a set of tools that may not work well with each other. Also, security concerns appear.
A DevOps engineer should care about security due to the involvement in all the stages of software development. Using a lot of various external services, vulnerabilities in the security and privacy of customers, employees, and issues with sensitive data may pop up.
Except for security, we should not forget about safety. It would be best if you guaranteed that all tested emails are in a safe and isolated environment. It’s a must. Otherwise, the risks of delivering untested emails to real users grow exponentially.
For example, HBO Max recently sent thousands of untested emails to customers:
Making your software send emails it’s not a task for one team. Devs and QA specialists are responsible for enabling sending, configuring infrastructure, improving email deliverability, and other technical aspects. Designers craft templates. Marketing teams organize the campaign, produce copy, and more.
It’s vital to establish convenient communication between the teams involved in the email testing process. Having dozens of accounts in dozens of tools, or showing examples, reporting bugs and fixes in separate environments doesn’t work well.
The task of the DevOps engineer to choose a collaborative-friendly service so the employees could perform and report their tasks in real-time.
What’s a full-cycle email testing
Full-cycle email testing should be a part of CI/CD.
At some point, you can set up email infrastructure by configuring outbound and inbound email protocols, choosing their ports, testing the load sending capabilities. It usually requires one period, and then you just monitor the process by using email infrastructure monitoring tools and approaches. At the same time, some email parameters, like layout, require testing before launching the campaigns.
It means that if your app sends emails, you can’t just do occasional email testings.
Under ‘full-cycle email testing,’ you should recognize the combination of two approaches:
Performance email testing. This is an area for marketers mainly. It relates to split testing (A/B tests) during the sending or based on the analysis of previous campaigns. The main focus here is experimenting with different copy elements (subject lines, headers, CTAs, messages, etc.) to improve engagement and email marketing metrics (open rate, CTR, and other). This type of testing directly depends on another approach.
Technical email testing. This one is at the core of software testing. As mentioned above, without technical email elements proper configuring and monitoring, there’s no chance to do performance testing. Simply because no single email will be sent and delivered.
Let’s explore the pillars of technical email testing below.
Technical email testing principles
Everything starts with email deliverability
On average, 2 of 10 emails sent aren’t delivered.
There’s no point in performing email testing if your team hasn’t ensured email deliverability. With low deliverability, users won’t get your email. One thing is if it’s some promotional or non-crucial transactional email. Still, another thing is non-ability to deliver an important email to a user, e.g., password reset email, account confirmation email, confirmation receipt. It leads not only to poor CX and frustration, but users may just leave your platform in favor of competitors.
To improve email deliverability, make sure to :
- set up email infrastructure (SMTP/IMAP/POP, feedback loops, shared/dedicated IP)
- configure correct email authentication records (DKIM, SPF, DMARC)
- do spam checks (avoid spam triggers, blacklisting check)
- maintain high domain health
- send on a schedule (relevant emails sent on the approximately same day/time)
- keep email lists clean (opt-in, segmentation, email validation)
To test infrastructure and email deliverability, you need to use special tools. Here are some that might be helpful: Mail-Tester, GlockApps, Wormly, GMass, SendForensics.
Here is a helpful tutorial to test SAP SMTP server.
Inspecting HTML templates is a must
You know that your emails will pass spam filters, and domain reputation is okay. It’s time to make them polished from a technical perspective. According to HubSpot research, HTML emails are dominant since 2014. It means that you should properly code HTML/CSS templates and debug them.
What to keep in mind:
- Test and debug each HTML template you use.
- Don’t use only HTML emails. Plain text is also required to not mess up with deliverability.
- Check how your email is rendered and displayed on each of the popular email clients. Usually, more problems with rendering are present in YahooMail and Outlook.
To make sure your emails are free of errors and look stunning, try using Mailtrap. For email preview, consider Litmus, PutsMail, or EmailOnAcid.
Correct links to right recipients
Many types of emails contain links and various CTA buttons. It’s convenient for both users and your product. Simultaneously, wrong, outdated, or broken links can harm your reputation and the experience of your customers. So don’t skip checking links at all costs!
The best practice for this is to use tools like Mailosaur. But here, a human factor is also a critical one. Setting up a collaborative environment for email testing is the thing you have to initiate. If a responsible developer, marketer, QA specialist check the correctness of links, you’ll minimize chances of failure in this parameter.
If you want to know more about tools and testing workflow in detail, check out this email testing checklist.
Email testing highlights
As a recap and for those who prefer the TL;DR approach, here’s what DevOps engineers must know about email testing:
- email testing is an unskippable part of the software testing cycle
- untested emails can ruin your product reputation
- the core of email testing is a technical, not a marketing one
- email infrastructure, deliverability, spam, layout, rendering, links must be inspected
- non-configured email infrastructure and poor email deliverability – no point in other tests
- you can initiate cross-platform automation testing
- without collaborative-friendly tools, email testing turns to mess
- you need to build a cross-functional email testing team
- email testing in a sandbox environment is a solution
- manual checks can be minimized but not avoided
Emails are still everywhere. Email testing is a critical thing to not mess up with. Its essence lies in debugging and configuring tech email aspects foremost, which are dev teams’ responsibilities.
DevOps experts can improve this process by choosing correct tools, embracing automation of testing processes, and advocating team collaboration.