In 2015, we tested the products (with default settings) internally over a 6-month period, using spam mails provided by Abusix. Vendors received examples of isses, to check that our testing methods work, and to provide feedback. Several products had very low scores in the internal test run, and several bugs in the spam-filters and products were discovered and had to be fixed by the vendors. In some cases, poorly-performing third-party spam-filters were fixed or even replaced. In March 2016, we ran this public test.
With any detection test (including spam detection), it is important to test for false alarms. In this case, it should be considered that some programs automatically increase their sensitivity when spam mails make up a large percentage of total mails received. We conducted a short-term false alarm test for this report, by running each product for one week on a customer machine and inspecting afterwards if there were legitimate mails classified wrongly as spam (there were none for any of the products tested). A large-scale test with genuine emails would be impossible without breaching privacy; although this was not as statistically significant as we would like, we feel this was sufficient to demonstrate that none of the tested programs was prone to FPs.