Skip to main content

Meet the new Quorum PAC. Stay compliant, raise money, and report results.

Learn More
WP_Query Object ( [query] => Array ( [name] => guide-to-a-b-testing-public-affairs [post_type] => resources [resource-type] => blog ) [query_vars] => Array ( [name] => guide-to-a-b-testing-public-affairs [post_type] => resources [resource-type] => blog [error] => [m] => [p] => 0 [post_parent] => [subpost] => [subpost_id] => [attachment] => [attachment_id] => 0 [pagename] => [page_id] => 0 [second] => [minute] => [hour] => [day] => 0 [monthnum] => 0 [year] => 0 [w] => 0 [category_name] => [tag] => [cat] => [tag_id] => [author] => [author_name] => [feed] => [tb] => [paged] => 0 [meta_key] => [meta_value] => [preview] => [s] => [sentence] => [title] => [fields] => [menu_order] => [embed] => [category__in] => Array ( ) [category__not_in] => Array ( ) [category__and] => Array ( ) [post__in] => Array ( ) [post__not_in] => Array ( ) [post_name__in] => Array ( ) [tag__in] => Array ( ) [tag__not_in] => Array ( ) [tag__and] => Array ( ) [tag_slug__in] => Array ( ) [tag_slug__and] => Array ( ) [post_parent__in] => Array ( ) [post_parent__not_in] => Array ( ) [author__in] => Array ( ) [author__not_in] => Array ( ) [search_columns] => Array ( ) [ignore_sticky_posts] => [suppress_filters] => [cache_results] => 1 [update_post_term_cache] => 1 [update_menu_item_cache] => [lazy_load_term_meta] => 1 [update_post_meta_cache] => 1 [posts_per_page] => 10 [nopaging] => [comments_per_page] => 50 [no_found_rows] => [order] => DESC ) [tax_query] => [meta_query] => WP_Meta_Query Object ( [queries] => Array ( ) [relation] => [meta_table] => [meta_id_column] => [primary_table] => [primary_id_column] => [table_aliases:protected] => Array ( ) [clauses:protected] => Array ( ) [has_or_relation:protected] => ) [date_query] => [queried_object] => WP_Post Object ( [ID] => 6855 [post_author] => 43 [post_date] => 2022-05-26 20:53:04 [post_date_gmt] => 2022-05-26 20:53:04 [post_content] => If you're not A/B testing, you're missing opportunities to improve your email effectiveness. A/B testing is a strategy used by growth marketers that's used to help companies iterate and achieve their goals faster. Experts agree that A/B testing can be a critical lever for experimentation and developing a concrete path forward. Harvard professor Rembrand Koning has also observed, "We find that firms that adopt A/B testing both scale and fail faster." But testing isn't just for marketers. A/B testing is a strategy that public affairs teams can use to learn more about what drives their stakeholders to action, whether that action is getting advocates to sign up for their grassroots network, legislators to sponsor a key bill, stakeholders to attend an event, or voters to register for the next election. If you're looking for the best way to try out A/B testing for your organization, you've come to the right place. This guide walks through how you can use A/B testing in your lobbying and advocacy.

What is A/B testing? 

A/B testing, also commonly called "split testing," provides a quantifiable way to evaluate the effectiveness of different elements of your emails.  For example, let's say you have narrowed down two potential call to action (CTA) buttons for an email. Instead of simply choosing one of them and hoping for the best, A/B testing provides a scientific framework for testing which drives the best response from your audience. A grassroots call to action email, for example, might test button copy like "Take Action Now" (A) vs. "Write a Letter to Your Senator" (B). An email platform that offers A/B testing would send half of your audience option A and half option B, then tell you which drove more clicks. The A/B test provides a low-risk way to make sure you're achieving maximum impact.  In addition to testing text, you can use A/B testing to measure how changing email senders, adding images to email copy, and adjusting messaging, among other things.  The most common structure of sending an A/B test is to use a system that splits your list in half. The first half receive the test - so 25% of your list receives variation A and 25% get variation B. After a set amount of time (usually a few hours), the second 50% of the list receives the winning variation.

Why You Should A/B Test for Your Lobbying and Advocacy 

Email is a key strategy lobbying and advocacy teams use to communicate, whether it’s to schedule a meeting with a legislator or get an advocate to participate in a write-a-letter campaign. [post_title] => A/B Testing For Public Affairs Professionals [post_excerpt] => A/B testing is the process of testing the effectiveness of two potential elements in emails. A/B tests can be done on the subject line, sender, and content. [post_status] => publish [comment_status] => closed [ping_status] => closed [post_password] => [post_name] => guide-to-a-b-testing-public-affairs [to_ping] => [pinged] => [post_modified] => 2022-11-14 21:48:51 [post_modified_gmt] => 2022-11-14 21:48:51 [post_content_filtered] => [post_parent] => 0 [guid] => https://www.quorum.us/?post_type=resources&p=6855 [menu_order] => 0 [post_type] => resources [post_mime_type] => [comment_count] => 0 [filter] => raw ) [queried_object_id] => 6855 [request] => SELECT wp_posts.* FROM wp_posts WHERE 1=1 AND wp_posts.post_name = 'guide-to-a-b-testing-public-affairs' AND wp_posts.post_type = 'resources' ORDER BY wp_posts.post_date DESC [posts] => Array ( [0] => WP_Post Object ( [ID] => 6855 [post_author] => 43 [post_date] => 2022-05-26 20:53:04 [post_date_gmt] => 2022-05-26 20:53:04 [post_content] => If you're not A/B testing, you're missing opportunities to improve your email effectiveness. A/B testing is a strategy used by growth marketers that's used to help companies iterate and achieve their goals faster. Experts agree that A/B testing can be a critical lever for experimentation and developing a concrete path forward. Harvard professor Rembrand Koning has also observed, "We find that firms that adopt A/B testing both scale and fail faster." But testing isn't just for marketers. A/B testing is a strategy that public affairs teams can use to learn more about what drives their stakeholders to action, whether that action is getting advocates to sign up for their grassroots network, legislators to sponsor a key bill, stakeholders to attend an event, or voters to register for the next election. If you're looking for the best way to try out A/B testing for your organization, you've come to the right place. This guide walks through how you can use A/B testing in your lobbying and advocacy.

What is A/B testing? 

A/B testing, also commonly called "split testing," provides a quantifiable way to evaluate the effectiveness of different elements of your emails.  For example, let's say you have narrowed down two potential call to action (CTA) buttons for an email. Instead of simply choosing one of them and hoping for the best, A/B testing provides a scientific framework for testing which drives the best response from your audience. A grassroots call to action email, for example, might test button copy like "Take Action Now" (A) vs. "Write a Letter to Your Senator" (B). An email platform that offers A/B testing would send half of your audience option A and half option B, then tell you which drove more clicks. The A/B test provides a low-risk way to make sure you're achieving maximum impact.  In addition to testing text, you can use A/B testing to measure how changing email senders, adding images to email copy, and adjusting messaging, among other things.  The most common structure of sending an A/B test is to use a system that splits your list in half. The first half receive the test - so 25% of your list receives variation A and 25% get variation B. After a set amount of time (usually a few hours), the second 50% of the list receives the winning variation.

Why You Should A/B Test for Your Lobbying and Advocacy 

Email is a key strategy lobbying and advocacy teams use to communicate, whether it’s to schedule a meeting with a legislator or get an advocate to participate in a write-a-letter campaign. [post_title] => A/B Testing For Public Affairs Professionals [post_excerpt] => A/B testing is the process of testing the effectiveness of two potential elements in emails. A/B tests can be done on the subject line, sender, and content. [post_status] => publish [comment_status] => closed [ping_status] => closed [post_password] => [post_name] => guide-to-a-b-testing-public-affairs [to_ping] => [pinged] => [post_modified] => 2022-11-14 21:48:51 [post_modified_gmt] => 2022-11-14 21:48:51 [post_content_filtered] => [post_parent] => 0 [guid] => https://www.quorum.us/?post_type=resources&p=6855 [menu_order] => 0 [post_type] => resources [post_mime_type] => [comment_count] => 0 [filter] => raw ) ) [post_count] => 1 [current_post] => -1 [before_loop] => 1 [in_the_loop] => [post] => WP_Post Object ( [ID] => 6855 [post_author] => 43 [post_date] => 2022-05-26 20:53:04 [post_date_gmt] => 2022-05-26 20:53:04 [post_content] => If you're not A/B testing, you're missing opportunities to improve your email effectiveness. A/B testing is a strategy used by growth marketers that's used to help companies iterate and achieve their goals faster. Experts agree that A/B testing can be a critical lever for experimentation and developing a concrete path forward. Harvard professor Rembrand Koning has also observed, "We find that firms that adopt A/B testing both scale and fail faster." But testing isn't just for marketers. A/B testing is a strategy that public affairs teams can use to learn more about what drives their stakeholders to action, whether that action is getting advocates to sign up for their grassroots network, legislators to sponsor a key bill, stakeholders to attend an event, or voters to register for the next election. If you're looking for the best way to try out A/B testing for your organization, you've come to the right place. This guide walks through how you can use A/B testing in your lobbying and advocacy.

What is A/B testing? 

A/B testing, also commonly called "split testing," provides a quantifiable way to evaluate the effectiveness of different elements of your emails.  For example, let's say you have narrowed down two potential call to action (CTA) buttons for an email. Instead of simply choosing one of them and hoping for the best, A/B testing provides a scientific framework for testing which drives the best response from your audience. A grassroots call to action email, for example, might test button copy like "Take Action Now" (A) vs. "Write a Letter to Your Senator" (B). An email platform that offers A/B testing would send half of your audience option A and half option B, then tell you which drove more clicks. The A/B test provides a low-risk way to make sure you're achieving maximum impact.  In addition to testing text, you can use A/B testing to measure how changing email senders, adding images to email copy, and adjusting messaging, among other things.  The most common structure of sending an A/B test is to use a system that splits your list in half. The first half receive the test - so 25% of your list receives variation A and 25% get variation B. After a set amount of time (usually a few hours), the second 50% of the list receives the winning variation.

Why You Should A/B Test for Your Lobbying and Advocacy 

Email is a key strategy lobbying and advocacy teams use to communicate, whether it’s to schedule a meeting with a legislator or get an advocate to participate in a write-a-letter campaign. [post_title] => A/B Testing For Public Affairs Professionals [post_excerpt] => A/B testing is the process of testing the effectiveness of two potential elements in emails. A/B tests can be done on the subject line, sender, and content. [post_status] => publish [comment_status] => closed [ping_status] => closed [post_password] => [post_name] => guide-to-a-b-testing-public-affairs [to_ping] => [pinged] => [post_modified] => 2022-11-14 21:48:51 [post_modified_gmt] => 2022-11-14 21:48:51 [post_content_filtered] => [post_parent] => 0 [guid] => https://www.quorum.us/?post_type=resources&p=6855 [menu_order] => 0 [post_type] => resources [post_mime_type] => [comment_count] => 0 [filter] => raw ) [comment_count] => 0 [current_comment] => -1 [found_posts] => 1 [max_num_pages] => 0 [max_num_comment_pages] => 0 [is_single] => 1 [is_preview] => [is_page] => [is_archive] => [is_date] => [is_year] => [is_month] => [is_day] => [is_time] => [is_author] => [is_category] => [is_tag] => [is_tax] => [is_search] => [is_feed] => [is_comment_feed] => [is_trackback] => [is_home] => [is_privacy_policy] => [is_404] => [is_embed] => [is_paged] => [is_admin] => [is_attachment] => [is_singular] => 1 [is_robots] => [is_favicon] => [is_posts_page] => [is_post_type_archive] => [query_vars_hash:WP_Query:private] => 96b0f783a8c39651d85a75b4f55241fc [query_vars_changed:WP_Query:private] => [thumbnails_cached] => [allow_query_attachment_by_filename:protected] => [stopwords:WP_Query:private] => [compat_fields:WP_Query:private] => Array ( [0] => query_vars_hash [1] => query_vars_changed ) [compat_methods:WP_Query:private] => Array ( [0] => init_query_flags [1] => parse_tax_query ) )
!!! 6855
Blog

A/B Testing For Public Affairs Professionals

Theresa Hebert
May 26, 2022
A/B Testing For Public Affairs Professionals

If you’re not A/B testing, you’re missing opportunities to improve your email effectiveness.

A/B testing is a strategy used by growth marketers that’s used to help companies iterate and achieve their goals faster. Experts agree that A/B testing can be a critical lever for experimentation and developing a concrete path forward. Harvard professor Rembrand Koning has also observed, “We find that firms that adopt A/B testing both scale and fail faster.”

But testing isn’t just for marketers. A/B testing is a strategy that public affairs teams can use to learn more about what drives their stakeholders to action, whether that action is getting advocates to sign up for their grassroots network, legislators to sponsor a key bill, stakeholders to attend an event, or voters to register for the next election.

If you’re looking for the best way to try out A/B testing for your organization, you’ve come to the right place. This guide walks through how you can use A/B testing in your lobbying and advocacy.

What is A/B testing? 

A/B testing, also commonly called “split testing,” provides a quantifiable way to evaluate the effectiveness of different elements of your emails. 

For example, let’s say you have narrowed down two potential call to action (CTA) buttons for an email. Instead of simply choosing one of them and hoping for the best, A/B testing provides a scientific framework for testing which drives the best response from your audience. A grassroots call to action email, for example, might test button copy like “Take Action Now” (A) vs. “Write a Letter to Your Senator” (B). An email platform that offers A/B testing would send half of your audience option A and half option B, then tell you which drove more clicks. The A/B test provides a low-risk way to make sure you’re achieving maximum impact. 

In addition to testing text, you can use A/B testing to measure how changing email senders, adding images to email copy, and adjusting messaging, among other things. 

The most common structure of sending an A/B test is to use a system that splits your list in half. The first half receive the test – so 25% of your list receives variation A and 25% get variation B. After a set amount of time (usually a few hours), the second 50% of the list receives the winning variation.

Why You Should A/B Test for Your Lobbying and Advocacy 

Email is a key strategy lobbying and advocacy teams use to communicate, whether it’s to schedule a meeting with a legislator or get an advocate to participate in a write-a-letter campaign.

Defining Your Hypothesis

Your hypothesis is an educated guess regarding the outcome of your experiment. Defining a hypothesis helps do a gut check to tell if the outcome of your test will give you a repeatable best practice. Simply writing two different subject lines won’t give you an action item for your next email send, but testing a hypothesis that a time-based subject line will drive greater urgency, and thus clicks, gives you a takeaway you can repeat. 

As competitiveness for attention in a stakeholder’s inbox increases, you need to do as much as you can to get the most out of each email you send. 

A/B tests can help you optimize emails to:

  • Get a higher conversion on your fly-in outreach to schedulers
  • Get more eyes on your community impact reports
  • Increase the sponsors or votes on key bills amongst your champions
  • Drive more attendees to your events

Three Types of Tests to Run

Whatever your audience, A/B testing can help you optimize emails to raise awareness and encourage people to take action. This section will share different tests you can run and what metrics are impacted by each kind of test. 

  1. Subject Line Test
  2. Email Sender Test
  3. Email Content Test

Statistical significance

This metric tells you how likely the result is due to your changes versus due to chance. You can use free online statistical significance calculators to tell you if your result is worth using as a best practice moving forward in your emails. 

Subject Line Test

A common A/B testing element is subject line testing. No matter how engaging or helpful the content within the body of your email may be, if no one opens it, did it really matter? 

Here are a few A/B tests you can run on subject lines:

  1. Is a longer or shorter subject line more likely to drive opens?
  2. Will adding time-based messaging drive urgency?
  3. Does a question spark more interest in the recipient to open the email than a statement?
  4. Does adding personalization – like a person or office’s name in a subject line- improve open rates

A subject line test could be used in the public affairs industry in trying to schedule a meeting with legislators during your fly-in. In your emails to their schedulers, you could run these subject line tests: 

  1. Length: “Let’s talk about [Topic] at [Fly-in]” vs “We need to change [Topic]. Let’s discuss at your convenience during [Fly-in] DATE”
  2. Time-based: “Two weeks left to schedule a meeting about [Topic] at [Fly-in]!” vs “Schedule a meeting about [Topic] at [Fly-in]”
  3. Asking a question: “Can we count on you to discuss [Topic] at [Fly-in]?” vs “Let’s talk about [Topic] at [Fly-in]”
  4. Personalization: “‘Office Name’, see you at [Fly-in]?” vs “See you at [Fly-in]!”
  5. Framing: “Understand your constituents focus on [Topic] with 2022 [Topic] Research Report” vs “2022 [Topic] Research Report”

The main metric you’ll use to measure the success of a subject line test is open rate — of all people who received each variation, what percent opened the email?

However, subject line tests can also impact the engagement with the email after it’s opened based on how well your subject line set expectations for what readers would get when they opened the email. For example, a click-bait subject line may drive opens, but if the content within the email doesn’t deliver on the promise of the click-bait, then it won’t have an impact on your goals. 

Email Sender Test

How does the name of the email sender impact a recipient’s likelihood to open and respond to an email? 

Here are a few A/B tests you can run based on email sender:

    1. High-Profile Executive vs. Peer Employee – Are employees more likely to be driven to take action on a grassroots campaign when it comes from a high-profile name, like your CEO? Or are they more likely to respond to peer-to-peer outreach?
    2. Person’s Name vs. Organization Name – Are advocates more likely to react to an email that looks like it comes from a human, but they may not know who that person is? Or an email that is clearly marketed from an organization but that they can immediately identify?

Similar to a subject line test, open rate, and click-through rates are your measures of success for this test. 

Email Content Test

The test that leaves the most opportunity for creativity is in the actual body of the email. From the language you use on a button to whether you include an image or not, there are countless variables within the body of your email that can help determine what will drive greater engagement from your recipients.

What’s essential about A/B testing is only changing one variable and keeping all other variables the same. If you change the text on a button AND an image in the same email, you won’t know which drove the change you saw. 

  • Designed vs. plain text – are your stakeholders more likely to engage with content in a heavily-designed and branded email or a plain text one? Do certain types of emails work better with design versus others? 
  • Button CTA text – much like a subject line test, how can you label your buttons in a way that drives clicks? 
  • Methods of persuasion – If you’re looking to convince a legislator to take action on a key bill, you could test a data-based argument versus a constituent story to make your case. 

When running tests on the email copy, you’ll want to observe differences in click-through rate to determine a winner. The click-through rate measures the percentage of people who clicked something in your email out of everyone who opened the email. By looking only at the subset of recipients who opened the email, you know that they actually saw the test and were influenced by it. 


Take it to the next level: testing different list segments or email types

You ran an A/B test on a blast to your whole advocate network that told you your CEO’s name drives great open rates, congrats! That’s a great takeaway for future blasts. But would the same result hold true if you emailed just one department within your organization? Or just your new advocates who you’re trying to drive action from for the first time? 

Or maybe you found that designed emails aren’t great for your call-to-action emails asking employees to call their legislator because it’s clearly a blast and doesn’t feel personal. That doesn’t mean it won’t be a good fit for sharing community affairs updates to legislators — with design, they may better brand recall for the ways you’re engaging their constituents. 

You shouldn’t assume all audiences will react the same to your variations, so it’s worth continuing to test and seeing how you can get even more narrow in your best practices. 

Then, take it another step further and bring your insights to other channels. If you’ve found a certain button language performs really well in your emails to drive action to your campaigns, that same language may work well on your website buttons.

Conclusion

The possibilities for A/B testing are endless and they provide powerful insights for your team. As you are running tests, make sure your team has a way of keeping track of what’s working so you can continue to use the best practices you’ve learned from your winning variations. Over time, the incremental changes you make in your subject lines, email senders, and email content will build your emails into a powerful tool for moving the needle on the issues your team cares about.

See A/B Testing and Other Outbox Features with a Guided Walkthrough

Follow our guide to see the features Quorum offers with it’s integrated email tool, Outbox.

Click Through to See Outbox in Action