Advertising Effectiveness Measurement

How do I Know Which Half of my Advertising isn’t Working?

John Wanamaker wasn’t the only one to ask this question. In fact, with at least 60% of all ads under-performing, wasting millions of dollars in advertising investment, most marketers today are asking the same thing. Sadly, long-standing creative testing processes are broken, so answers are elusive. The standard practice of spending one’s full advertising effectiveness budget on pretests for 20% of their creative sheds no light on the remaining 80%. How many millions of dollars are at risk?

So, a solution was finally built. A creative testing system now measures every client ad, and every competitor ad, across all media types, with results in 24 hours, at a cost well below the old processes. Does this new process answer Wanamaker’s question?  You be the judge.

Advertising Effectiveness Measurement is Broken

Measuring advertising effectiveness began in the late 1800’s and has been the focus of countless studies ever since.  Methodologies have ranged from focus groups, readership studies, polling, Day-After-Recall, breakthrough, brain-wave tracking, galvanic skin response, eye-tracking, moment-by-moment, integrated market research, digital audience response and much more.  Given the size of big brand advertising budgets, market researchers have sought better ways to justify media and production spend and link to sales results.  

The predominant creative testing model has been the custom pretest, which determines an ad’s effectiveness based on consumer responses before it airs.  Another is custom post-testing, which provides periodic or continuous in-market research monitoring a brand’s performance, including brand awareness, brand preference, product usage and attitudes. 

Advertisers determine the top 10-20% of their ads in terms of planned media spend and have them pretested before they air.  Some advertisers send finished art to the pretest, and others send concepts, unfinished art or animatics.  All pretesting can be expensive and slow. Traditionally, a client will wait days, weeks or longer for results, meaning he may not have time to edit the pretested ads if problems are found.  Major advertisers like Unilever and Procter & Gamble have both expressed concern that the current timeline and cost for ad testing doesn’t fit today’s needs. See this startling article in Net Imperative, “The end of testing? Unilever shakes-up market research in real-time world.”

The reason advertisers only test 20% of their advertising, and usually only TV or Video, is because their budget goes only so far.  At the cost of pretesting and post-testing today, they can only test ads with the biggest planned media buys.  Still, that leaves 80% of ads that are not tested.  Since every ad, news story, social media post, etc. contributes to a company’s reputation, that’s a lot of risk.  

How much risk?  The study below, provided by ABX, shows ad scores for all ads tested during a 60-day period.  See how many fall beneath the horizontal yellow line, which stands for “Average.” Each tiny dot is a separate ad.  Each vertical cluster is a different media type.

The higher the scores, the more likely those ads will positively affect business results; the lower the score, the more likely the ad budget has been  wasted.  With at least 60% of ads underperforming in-market, how can an advertiser know which ones they are?  If they did know, they could adjust their media plan/spend. In addition, creative teams receive very little feedback, so how can they learn and constantly improve the creative process?

The Solution: Measuring Every Ad – and Every Competitor Ad

On the face of it, the idea that anyone would consider measuring “every ad” sounds ludicrous.  Certainly, using the custom pretest method, the cost alone would be impossible.  So how can one affordably test the lion’s share of their creative?

“There are two types of research, customized and syndicated. Customized research is conducted for a specific client to address that client’s needs. Only that client has access to the results of the research. Syndicated research is a single research study conducted by a research company with its results available, for sale, to multiple companies.”

Thanks to new technology and know-how in creative testing, it is now possible to measure all of one’s ads, and their competitors’ ads, at a cost well below custom. These solutions test a universe of ads for entire industry categories on an ongoing basis, with results provided to paying clients.

Some syndicated research firms focus on singular industries in their syndicated ad testing, and some focus on the entire market but limit their work to certain media only, like TV and video. One relatively new firm measures the entire market and all types of media through bespoke technology and market research know-how.  In fact, this firm (ABX) has now measured more than 365,000 ads and has the largest norms in the industry with hundreds more being added each day.

Major advertisers who elect to use a syndicated solution can understand the competitive picture and how their own ads stack up.  More importantly, because some of these firms deliver results very quickly, advertisers can identify an under-performing ad and replace it, or adjust the original ad.  Without the speed of syndicated creative testing, clients wouldn’t be as successful in moving media spend from losers to winners.

The best measurement programs utilize both custom and syndicated services. For some guidance on choices, please see,  “Should I Look at Faster, Lower Cost Options to Legacy Ad Testing?”

Syndicated Creative Testing and Links to Sales?

If one has relied on custom pretesting and post-analysis, one may have wondered why advertising results don’t correlate well to outcomes or perform well in market mix models.  Traditionally, models utilize media spend data, but not creative testing results.  The reason is simple: media spend data is readily available for every ad, but creative scores are not, since only 20% or so have been tested.  Thus, the modeler has no way to qualify the various ads and thus assumes they are equal.  Correlations are often poor and the true effect of advertising remains a mystery.

When all ads are tested in a syndicated system, creative scores are available for every one of them. The market mix modeler can merge media spend with creative scores, thus providing the needed qualitative weighting. In several case studies done by market mix modelers, advertising results show tremendous improvements in correlations to sales. Through further work, modelers have discovered that creative accounts for more than 60% of ad success.  A major study from Nielsen Catalina Solutions (NCS) in 2016-2017, “Five Keys to Advertising Effectiveness,” agrees that Creative is by far the most important element.  To see several exciting case studies, go to “Can Creative Testing Improve ROAS, ROI and Predictive Analytics.”

The New Creative Process

Historically, custom testing has provided the creative team with in-depth feedback on the top 20% of their ads.  But for the 80% of untested ads, the creative team has been in the dark.  Once the untested ads were placed in media, it was harder to know what worked and what didn’t.  Today’s digital media testing offers fast feedback on behavioral measures like clicks, likes, retweets and so on.  But even then, creatives don’t know what it was about their designs that yielded results.  What about results on Radio, Print, OOH, FSI’s, In-Store Signage and other media types beyond the most-tested, Television? 

Syndicated creative measurement provides answers readily. Most firms deliver in several days, and a few like ABX in 24 hours.  Now, the creative team can see for itself how its work is playing out, and whether certain ads need adjustment. For more on this topic, see “How is Speed Changing Advertising Effectiveness Measurement?

In addition, access to online dashboard delivery systems offer a treasure trove of creative ideas from one’s own past campaigns as well  as competitors’ and other industry campaigns.  If copywriters can see what types of ads yield a high Reputation or Purchase score, or a high Dislike score, they can start projects well-informed and ready to go.

To get a better idea of what makes a great ad across all media, see our Top Ads Of The Week Measured For Ad Effectiveness.

Comparing Ads Across Competitors, Categories, Media Types

The only way to really know how one’s creative stacks up is to compare the scores for the overall ads, and for their individual KPIs, against other ads within a common demographic target.  The only way to do this is through syndicated creative testing that uses at least one common demographic audience across the board.  Most ads are measured against very specific target audiences, and thus are not comparable horizontally. 

An interesting solution to this was devised by JJ Klein, Chairman of ABX, and his team.  If every ad is measured by a GenPop audience as well as a targeted demographic audience, ads could be compared across the entire database by KPI, Category, Media Type and more. Thus, a company gets a true measure of how ads are stacking up to competitors. Klein’s team has now measured more than 365,000 ads and international norms are being added in 14 countries.  

Measuring Local Advertising with National Creative Testing Panels

Many clients are anxious about using national ad testing panels for local ads.  After all, these national panels may not be aware of the local company or events. JJ Klein and his team looked into this by testing local ads using local panels and again using national panels.  They found no statistically significant difference in the creative scores between the two panels. 

Why is this?  A good ad is a good ad and a bad ad is a bad ad, no matter if its measured by a national or local panel resides.  In national creative testing, the panelists may not have heard of many of the brands whose advertising they will evaluate, so this is the same for local ads.  

Global in-Market Creative Testing

On the other end of the spectrum from local creative testing is global. Up until now, global testing had to be handled on a custom basis in pretests and post-market tests.  Several of the largest research firms have these capabilities across many countries.

There has been no syndicated creative testing process providing the in-market point of view across the globe until 2017, when methods used in the US successfully by ABX have been exported to 14 countries. Norms are available within and across countries by Category and Media Type.  Advertisers will have international advertising effectiveness data never before seen.  Countries to date include: Brazil, Canada, China, Australia, France, Germany, India, Italy, Japan, Mexico, Russia, Spain, the United Kingdom and the U.S. 

Gender Equality in Advertising

The Association of National Advertisers (ANA)  has created the #SeeHer program with a goal of improving the portrayal of women and girls in advertising by 20% by 2020.  Major advertisers including ad giants like P&G and Unilever are working to improve the way they portray women and girls in advertising and to associate their brands with programming that positively portrays females. 

This is not only the right thing to do socially, but data from Advertising Benchmark Index  shows that as gender portrayal improves, ads become more effective and both short- and long-term purchase intent improves significantly. Work is now being done with Multicultural groups to make sure people of all ethnicities are respectfully represented. 

For more information on this critical area in advertising click HERE

Jeff Schamel

Vice President, Operations

Jeff Schamel Directs Operations for ABX with Experience from IPSOS and P&G

Jeff has more than 25 years of market research experience including project management, client service, operations, production and sales.

Before joining the ABX Senior Management team, Jeff was a Senior Account Manager at Ipsos, working with Procter & Gamble and other client assignments. Prior to that, he was Senior Project Manager for Opinion One, where he was responsible for guiding the company’s online, mall and other initiatives for eight years.

Jeff began his market research career with Cooper Research while participating in the Professional Practice Program at the University of Cincinnati, Ohio. Upon graduating with his BS in Marketing and Management, Jeff led a team of data analysts as Group Manager at Matrixx Marketing Research (now Concentrix).

Jeff’s broad in-depth marketing research experiences enable him to bring a unique perspective to directing operations that creates effective and efficient environments to support a wide range of client activities.  Having come up in the industry alongside the advent and progression of online interviewing, he has a unique perspective on the benefits and challenges of conducting quality research resulting in meaningful results.

Jeff considers himself an amateur music historian and sports enthusiast, as well as a family man, who takes advantage of the many outdoor activities in his community including biking, hiking, kayaking and swimming.

Dr. Peter Moomaw

Co-Founder, Vice President, Development

Peter is the primary developer of operations technology and data analysis. As a chief visionary on the Research and Development team, Peter works to generate innovative ideas for the continued development of processes and protocols.

Peter brings 17+ years’ experience in software development in the market research industry. He was previously at Opinion One, where he was integral in the development of research software systems covering a number of areas.

Peter has a BA in Mathematics, a BS in Physics, and a PhD in Physics, all from the University Cincinnati.

Martin Geraghty
Martin Geraghty

Vice President, Client Services

Geraghty Brings Experience from Roper Starch, GfK, MRI and IPSOS to ABX

Martin has more than 20 years of experience in the market research space including business development, methodology, product design, operations, client service, analysis and sales. He is a member the ABX Management Group with both client-service and operational responsibilities. Before joining ABX, Martin held senior management positions at Roper Starch Worldwide, GfK, MRI (Mediamark Research) and Ipsos. At Ipsos, he managed client service covering all aspects of the research process including research design. He managed the development and implementation of several brand and advertising measurement programs across a broad range of business sectors, helping clients maximize marketing objectives and return on investment. Prior to joining Ipsos, Martin was Vice President at MRI and was involved in business development, day-to-day management and long-term planning and growth for syndicated and custom programs. He also developed and updated the organization’s operational infrastructure to bring new and improved measurements to the marketplace. Before joining MRI, Martin was a Vice President of Operations at GfK. He managed researchers developing programs to monitor and improve methodologies. He played a key role in overseeing profit responsibility of client programs. Prior to his role at GfK, he was Vice President of Research and Operations at Roper Starch Worldwide.
Michael Alf

Co-Founder Ocucom and ABX, Vice President, Software Development

Michael Alf Brings 12 Years’ Software Engineering in the Market Research Industry

Michael is one Founders of Ocucom, of which ABX is a division, and has over 12 years’ of software engineering experience in the market research industry. As a Lead Software Engineer with Ocucom, he brings a unique blend of a technological background along with hands-on market research experience. Michael specializes in providing custom software solutions utilizing Ocucom’s dynamic software platform to address unique client needs in an efficient and cost-effective manner.

Prior to starting with Ocucom, Michael was a senior software engineer at Opinion One in Cincinnati, where he was one of the principal architects of the OASIS 2 Survey Software Package. He also played a key role in developing a licensable solution to bring to the market place as well as handling the logistics of on-site training and support of end-client users.

Michael obtained his B.S. in Computer Science from the University of Cincinnati.

Greg Tunstill

Co-Founder, Chief Information Officer

Greg Tunstill Leads the Creation of ABX Communications Tools

Greg Tunstill, a founder of ABX, is the Chief Information Officer responsible for ABX’s technical infrastructure and maintenance. He also leads the design and development of ABX’s communications tools, a critical component of its technology services.

Prior to joining ABX, Greg was Director of Network Operations of Opinion One since its inception in 1992. He designed and developed the communications systems that were the core of its data collection networks and was responsible for continuing development and implementation of information technologies. Prior to his position at Opinion One, he designed inventory control and expert systems for BellSouth field offices and worked as a consultant and independent contractor. In 1991 he joined Americom Research as a systems engineer where he advanced to Director of Network Operations and met members of the team that formed Opinion One.

Greg is a graduate of Middle Tennessee State University where he obtained a Bachelor of Science in Computer Science.

Julian Brown

Chief Technology Officer

Julian Brown is an ABX Founder and Chief Visionary for Strategic Planning

Julian brings 20+ years of software engineering and market research experience to ABX. As one of the Founders and in his position as Chief Technology Officer, Julian is the chief visionary responsible for strategic planning and development.

His primary day to day roles include management of software development, oversight of survey production, and lead architect for custom software development projects. As the principal technology architect, Julian adapts emerging technologies for use in market research. He is responsible for maintaining the “continuous development” philosophy.

Before becoming one of the Founders, Julian was Vice President, Software Engineering at Opinion One. In this capacity, he was responsible for the design and development of Opinion One’s core technology and licensing products.

Prior to Opinion One, Julian worked for Americom Research, starting as a Software Engineer and progressing to Director, Software Development. While at Americom Research, Julian pioneered the design and creation of one of the market research industry’s first fully interactive computer-aided interviewing suites.

Julian received his Bachelor of Science in Computer Science from Middle Tennessee State University.

Gary Getto


A Pioneer in Measuring Across all Media Types

Gary has an extensive background in product development, strategic planning, and sales/marketing. He is a frequent speaker to the advertising, marketing, and public relations communities.

Gary is a pioneer in efforts to measure and analyze the combined impact of news coverage, social media, and advertising communications on key business metrics, such as sales or product preferences, with ground-breaking work linking communications activity to business outcomes. He was also one of the first to apply artificial intelligence to better understand how communications impacts consumer behavior.

As part of these pioneering integrated measurements, the need to benchmark every ad and build a comprehensive database of advertising performance metrics for evaluating all advertising creative became clear. Gary has been part of the ABX management group that developed the methodology, technology, and criteria for measuring advertising performance globally, and at scale, in an affordable manner. 

Now Gary and all ABX partners work with communications leaders across a diverse set of industries to expand that understanding by incorporating the first-ever Gender Equality Index™, creative measurement solutions for ESG, Diversity & Inclusion, Ethnicities (in and out-of-country), Product Placement/Integration, Brand Fit™, Gaming, Audio and more.

Gary holds a B.S., in Mechanical Engineering from Lehigh University and a Masters in Management Science from Stevens Institute of Technology.

J. J. Klein​

Chairman & CEO

Developing Valid and Reliable Information through Syndicated and Custom Research

JJ Klein has devoted his entire career to developing more valid and reliable information through syndicated and custom longitudinal research programs for Fortune 500 companies. This extensive knowledge base enabled him to create and develop ABX, which now offers the most accurate normative database for advertising effectiveness measurement in the marketplace.

JJ brings more than 35 years of marketing, research, consulting, and technology experience in designing and developing methodologies and programs for long-term growth. He has exceptional strength in domestic and international new business development, strategic planning, and identifying and tracking consumer, retail, and industry trends.

Leadership in Top Global Research Firms

Early in his career, after attending City University in New York as a math and science major, JJ joined Simmons Market Research Bureau as a Project Director. Over time, he rose to Associate Executive Vice President. Among his accomplishments, JJ created the first online interactive data analysis and delivery system for agencies, media, institutions, client, and service companies.  

JJ and Bill Simmons then founded a new venture …Three Sigma Research Center. It was here that JJ, Chief Operating Officer, created the first major syndicated rating system for Newspapers in the U.S., which later became the Scarborough Report. Three Sigma Research was eventually sold back to Simmons; Simmons sold it to Scarborough; and then Scarborough was acquired by Nielsen.

Following the sale of Three Sigma, JJ joined Audits & Surveys Worldwide as a principal leading the sales tracking and market share audits for a broad range of retail and service establishments both domestically and internationally. Following the acquisition of Audits & Surveys by GfK, JJ joined Roper Starch as Executive Director, reporting directly to the Chairman.

Creation and Development of Ocucom™ and ABX™

In 2004, as he continued his industry visionary travels, all of JJ’s experience came together as he tackled new solutions for market research in the Internet age. He and four partners founded Ocucom™ and invented a research-proven, respondent-oriented Ocular™ approach to internet sampling and multi-modal interviewing methodology. For the first time, respondents could answer text-based questions within a visual communication world where the computer screen was the interviewer/communicator. Inventions continued such as Ocucom’s PreCast™, the first long-form video viewing solution in the industry.

Also, during this time, JJ’s early leadership in internet research methodology caught the eye of top-tier global research competitors. He ended up designing methodologies for their online survey capabilities.

Then, in early 2012, he formed ABX™ as an outgrowth of Ocucom. JJ had conceived the idea for ABX some 15 years prior but had to wait for internet maturation and online panel/sampling companies to evolve. ABX partners focused on JJ’s ABX concept and methodology by building an infrastructure and a 5-year formative database before opening to the market in 2018. The new company offered the first high-quality Ocular research system for all ads in all media types with detailed, never-before analytics; the first syndicated, all-media type ad effectiveness platform; results in 24-to-48 hours on dashboard, and significantly lower costs. At ABX, as at Ocucom, inventions have continued such as the expansion of the ABX syndicated program into the first-ever global solution, and the first Gender Equality Index™. Solutions have expanded to include creative measurement for ESG, Diversity & Inclusion, Ethnicities (in and out of country), Product Placement/Integration, Brand Fit, Gaming, Audio, and more.

A Career Devoted to Validity and Reliability in Market Research

JJ’s background brings together an extremely unique blend of extensive experience in designing and developing global and regional longitudinal research programs. His accomplishments include the design and initiation of programs and methods that produce more valid and reliable information. They also include contributions to the market research profession through association membership, speaking, writing, and presenting.

Like Nielsen for TV, Arbitron for Radio, and MRI for Magazine, JJ’s original design for the newspaper program continues as the Scarborough Report ratings standard for Newspaper.  Today, JJ leads ABX as a syndicated and custom research service for ad effectiveness and gender equality, which have quickly become the gold standards of their own.

Angela Jeffrey, APR

Vice President, Brand Management

National Award-Winning Veteran of Public Relations, Advertising and Marketing

Angela is an internationally known speaker and national award-winning veteran of public relations, advertising and marketing.  She has been a teacher and leader in the use of metrics to measure communications impact for over 20 years, especially at the intersection of PR and Advertising.

Angela is primarily responsible for all ABX marketing efforts, and for projects that involve PR or social media measurement. She is the primary driver of our collaboration with the prestigious Institute for Public Relations as we develop methods to measure the impact of PR and social media on advertising effectiveness.

Prior to joining ABX, Angela was Strategy Director at Salience Insight, (now CARMA), a global provider of PR effectiveness measurement. Previously, she was a Vice President at Surveillance Data, Inc., an international analytics powerhouse that specializes in predictive modeling. Angela continued as a key leader upon the sale of her division to VMS and helped develop Vantage, with Gary Getto, the first platform to measure the combined impact of PR and Advertising.

Angela graduated summa cum laude from Southern Methodist University with a BFA in Journalism and BBA in Marketing.



Diane Light Waight

Vice President, Business Development

Diane Light Waight handles Sales, Business Development, and Marketing for ABX

Diane has 20+ years of experience helping Marketing, Advertising, and PR decision-makers at global brands, media companies, and agencies increase their ROI by connecting and authentically engaging their target customers through Sales, Business Development. MarTech, AdTech, Big Data, Market Research, SaaS, eCommerce, and Integrated Marketing.

She is passionate about her family, entertainment, health & wellness, empowering women, and environmental sustainability.