Review Response and Analysis : Gizmo/TechSupportAlert.com

[SUPERAntiSpy]

We have been asked by many of our users to address and to provide a response and detailed analysis of Gizmo’s review that was presented in the Tech Support Alert news letter from www.techsupportalert.com dated April 12, 2007

Upon initial observation, we find it interesting that Gizmo used a VMWare (Virtual Computer) for testing. For those not familiar with VMWare, VMWare basically runs a copy of Windows (any version) in a virtual environment. This allows a user to run another “copy” of Windows on the system without risking damage to the main system. While this sounds ideal, anyone that is involved in real security analysis knows (or should know) that many malware infections will not infect a VMWare system as they know researchers use them to harvest samples. We observe this on a daily basis and the number is increasing. The infections detect the presence of VMWare and simply don’t install the full infection. Although it is convenient to test on VMWare instead of setting up a test system, to really test an application’s abilities it should be done on the same types of systems that a user would be running – which certainly is not on VMWare.

Gizmo himself states “SAS has developed an excellent reputation for the removal of an existing spyware infection. I have not tested this aspect myself…” Thus although he clearly indicates that he has knowledge of our “excellent reputation”, and why we have the excellent reputation, but fails to test that aspect of the product. Clearly, that speaks volumes about the validity of the testing and the methodology and motives behind the review..

Anti-spyware and anti-virus applications should be tested against real infections, as that is what real users encounter. Although simulators, may sound like a great idea and minimize the work involved, they do not remotely simulate actual malware infections that we find in the wild on a daily basis.

SUPERAntiSpyware is designed to detect and clean real infections found in the real world. SUPERAntiSpyware is NOT a behavioral/HIPS blocking product and examination of our literature and forum posts demonstrate that we have never claimed to be such a product. Those undertaking the testing of a product should understand the nature of the product, either by research or by contacting the company directly and then test the product against what it is designed to actually do - not what they think it should do. For instance, if you take a car and drive it into a lake and it sinks, does the reviewer make a category that says “Floats on water” with a “FAILED” result? Obviously that would be ridiculous and any user would be able to see that immediately. However, with the anti-spyware / anti-virus testing, users do not have the expertise to make their own analysis and thus they must rely on reviewers to present accurate and valid tests. Unfortunately reviews by unqualified individuals who fail to provide and to follow appropriate methodology result in inaccurate and unreliable results that do not represent the actual abilities and capabilities of the software. See our other detailed Blog on Testing Methodologies : http://forums.superantispyware.com/viewtopic.php?t=176

Reviews and actual testing of Anti-Malware/Spyware applications may tax the reviewer’s technical ability and require a level of actual understanding of malware/spyware infections. The question is ”do they really understand these infections and the technical intricacies involved in an actual infection in the wild and what are the qualifications of the reviewer. Are they actually qualified to review these types of products and provide useful reviews to the general public or are they providing opinions?

An appropriate question here is what are the qualifications of the author to respond to Gizmo’s Review? As the developer of a successful AntiSpyware product using and developing “state of the art” technology suggests that I am qualified to analyze and to review AntiSpyware in general and of the product which I have developed in particular. I deal with hundreds of malware infections daily. I have been developing software professionally for over 25 years. This includes both low-level driver and kernel level development. I have been creating Windows applications since the first version of Windows shipped over 20 years ago. I have been developing security based applications for the past 8 years. In the interest of brevity here, my professional credentials may be reviewed here. http://www.superantispyware.com/company.html

Below I address the specific tests that Gizmo has relied upon to test SUPERAntiSpyware. My analysis of each test Gizmo used to test SUPERAntiSpyware follows. The appropriateness of using simulators to test the efficacy of the product under test is also examined. I trust that you will find this analysis useful in reaching your own conclusions regarding the testing performed in the above referenced review.

1.1 Ghost Security Registry Test

This test is designed to quickly change registry values to see if they are detected. If SUPERAntiSpyware were a behavioral or HIPS product, this might be an accurate test. However, as SUPERAntiSpyware is not such a product, it really tests nothing. In addition, the test itself is flawed due to the fact that none of the values were changed to anything that was harmful – products should be designed to detect harmful items, not to detect simulators.

1.2 Regtick

This test is also designed to simply change various registry values. The review states “some of the registry values were deliberately chosen as relatively innocuous while others were more significant” The author fails to address why a security product would detect “innocuous” (not harmful) registry changes. Many of the items in this test are not even items we have ever seen changed by a spyware/malware infection, and certainly do not represent a “threat” to one’s system. This is one of the big problems with simulators, they often don’t simulate anything that really has to do with an actual spyware infection.

1.3 Scoundrel Simulator

The key word here again is “simulator” – the items changed were not harmful to the system, they were simulated results. A behavioral/HIPS blocker might flag these items – again SUPERAntiSpyware is not designed to be a behavioral/HIPS product.

1.4 ZapAss

This test is designed to test code injection (running) in Internet Explorer – the code injected was not harmful and the file downloaded also was not harmful – a product should not detect this as harmful, unless it is done by a spyware/malware process/application. Many legitimate products use these techniques to accomplish tasks that Windows may not easily allow. For example, download managers often trap Internet Explorer downloads and use their own systems to download the files to be able to handle them in their own application – they may accomplish this task by code injection and/or downloading a file from within Internet Explorer.

1.5 Trojan Simulator

Starting a server and adding a startup registry entry are common things done by literally tens of thousands of legitimate applications – again, this falls in the category of behavioral/HIPS products, and not a product designed to scan and remove harmful software. For example, if you install SUPERAntiSpyware it actually installs several services that are required for detecting and removing hard to remove spyware infections. In addition, SUPERAntiSpyware may modify the startup entries so SUPERAntiSpyware can optionally start when Windows starts, and perform our unique and powerful First Chance Prevention scanning which scans over 50 startup points when Windows starts to detect and remove infections before they take hold of your system.

1.6 Trojan Demo

This demo is provided by a company that produces a product that creates a virtual PC in which to operate in order to isolate the user from infection – this test shows nothing in the way of an actual spyware/malware infections. If SUPERAntiSpyware is not operating inside the virtual environment, then we would have no way of detecting the samples. This would be like telling someone to guess how many fingers you are holding up with the person guessing being blind folded.

1.7 Crash Test

This test is ridiculous – anti-spyware and anti-virus products are not designed to prevent a crash of the system. If a product wants to crash the system at kernel level there is little you can do to actually stop the crash – you can possible notify the user the system is going to crash, but certainly not prevent it.

1.8 Raw Memory Access Test

Again, this technique of raw memory access is used by many legitimate applications – a behavioral/HIPS application might detect this, but it certainly would not be detected by any typical scanning and removal products such as SUPERAntiSpyware. Applications often access the memory of other processes to monitor system activity, or perform their own protection of the system.

1.9 Program Termination Test

We have found few applications that can actually terminate SUPERAntiSpyware – Although we may add termination protection in a future version, products that block all termination though task manager can cause problems for users if they need to terminate the product due to a malfunction of a product or system instability. Proper termination protection should be limited to harmful software, not legitimate software designed to terminate processes. The author also used several developer oriented tools (DarkSpy, IceSword) that specialize in program termination – again something not found in actual infections, or on users installations.

1.10 Resource Usage Test

Although Gizmo did not indicate a “failure” or “pass” on the memory and resource usage, several things should be noted about this type of statistic. Firstly, Gizmo did not reveal what product was used to measure the resource results – the accuracy and efficiency of the resource meter is unknown. Secondly, you will note that the scale is set to “1 minute” – any product that does anything on your system must use resources to accomplish the task. As you note, the “time slice” (amount of time over the length of the graph) is very small for SUPERAntiSpyware indicating it used the CPU for only a fraction of a second while scanning the newly started process. This, of course, is a good thing, but was ignored and not mentioned by the reviewer as a positive attribute. Resource usage is also a very subjective measurement. The bottom line on resource usage, is does it effect your particular system, do you, the user see any slowdown or performance degradation. SUPERAntiSpyware is specifically designed to be light on resource and to co-exist with other anti-virus and anti-spyware applications, something most other “suites” do not do at this time.

2.1 Keylogger Detection Test

Although SUPERAntiSpyware detects many Password Stealing (PWS) keyloggers, we have made a conscious decision not to detect general keyloggers as many are used for legitimate purposes by businesses and agencies. The functions Gizmo mentions ( GetKeyState, GetASyncKeyState ) are all used by standard Windows applications to perform basic keyboard interaction tasks. An application should not be flagging these items as “harmful”. The Keyboard Hook and Journal Record Hook are also part of the standard Windows API/system and have many legitimate uses amongst commercial applications. Many of the hooks are used for accessibility interfaces where the user may use specialized devices to provide input to the computer. Simply flagging “hooks” really indicates nothing – many products do this, and users end up confused and may disable legitimate applications because a “hook” was flagged as potentially harmful. Reviewers need to understand the keylogging issue in much more depth before passing judgment on how applications detect, or do not detect these items.

2.2 DFK Threat Simulator

Again, the word “simulator” jumps out - all of the items presented here are simulators and rootkit sample applications, not actual threats. These are specifically not detected by SUPERAntiSpyware as many legitimate applications use these rootkit/kernel level samples as a template for their own applications. These code samples are often distributed as “proof of concept” and are used by legitimate applications as well. For a simulator to be “valid” it should provide a set of tests using a known sample that is specifically detected for the purpose of simulation testing, not random registry and system modifications that really don’t represent a threat.

2.3 Hostile Browsing Tests

Unfortunately, the author did not reveal the sites that were visited here. We understand that publishing harmful sites is to be avoided. From viewing the “WhatChanged Log” it appears that SUPERAntiSpyware would detect and remove many of the items installed on a scan of the computer. There is no explanation why Gizmo would not scan the system and let SUPERAntiSpyware do what it does best – detect and remove actual infections.

2.4 “Shoot in the Foot” Tests

As is evident from this set of tests, which actually tested against more real-world type infections, SUPERAntiSpyware did quite well at detecting the infections and blocking them before the PC became infected. The test bed used should be available for validation of the reviewers results.

2.5 Rootkit Detection Test

SUPERAntiSpyware properly detected the Hacker Defender root kit that we have see in the wild. The FuTo rootkit is distributed as an example/prototype rootkit from www.rootkit.com and we detect some of the variants of spyware/malware based upon this sample, but we consciously do not detect the sample itself as it is only a prototype/example and not harmful to a users system.

2.6 Archived Malware Detection Test

Archived files do not represent real-world situations. In our daily analysis, we do not see actual infections inside archives on real infected systems (except for samples submitted for analysis). Many products do scan inside archives, but we have elected to focus on the real infections and not just to scan archives to pass these types of tests. Scanning inside archives can greatly increase resource usage and unnecessarily slow scan speeds. Often when threats are detected inside archives, the infection cannot be removed, as the entire archive would need to be removed, which may render a system unstable or limit its functionality. Anti-Virus and Anti-Spyware applications that do detect threats inside archives typically cannot re-compress the archive removing only the infected sample, so the threat remains. To do this test properly, it should be noted if the applications detects AND removes the sample and leaves the archive intact.

2.7 Compressed Executable Malware Detection Test

Taking a sample that is non-packed, and packing it with the various packers does not show anything that we actually see in the real-world from dropped infections. SUPERAntiSpyware detects tens of thousands of packed samples that we have observed in the “wild”.

As of June 19, 2007 – Gizmo states that Windows Defender is his first choice for anti-spyware/malware protection and cleaning. Although Windows Defender may offer some proactive protection features, anyone who has used Windows Defender against actual infections knows that it certainly is not the “best” at detecting and removing infections. I believe this statement by Gizmo is a complete disservice to end-users and provides a completely false sense of security.

Conclusion

The bottom line on this review is this – if you are going to fly in a plane, would you rather have a pilot that has flown simulators (as in this test) and never ventured into a real plane or a pilot that has actually been flying real airplanes in the heat of battle (SUPERAntiSpyware).

Reviewing and testing anti-spyware and anti-virus applications is not an easy task as we have noted in our various blogs “Malware testing is certainly a daunting task and adequate documentation of methodology is the single most important element in validation of the results” Certainly individuals are entitled to express their own opinions. However, when the results are presented as performed by alleged experts and represented as such without disclaimers, they must be held to the highest standards. See also: “What are the Incentives for Malware Testing“ ( http://forums.superantispyware.com/viewtopic.php?t=177 )

I believe reviewers need to present their credentials and qualifications to undertake the testing, to use proper testing methodology, to perform the testing following proper and accurate testing procedures detailed in the methodology and to test applications against actual real world infections. Absent following these procedures, reviews should be labeled as opinions so as to not misinform the general public.