See Also

Patch Manager

Business challenges

Smartech QA team had a task to set up and implement testing process of one of the leading customer products - Patch Manager. Dedicated QA center managed to create all necessary documentation, tools and scripts to set up testing process and perform testing of the product to maintain its quality across several versions of the software. QA team was working in close touch with customer management team, outsource QA and development teams to provide the best quality of the product and improve efficiency of the entire production process.

Application description

Patch Manager is a distributed multi-tier application developed for analyzing critical software updates installed and missing from Windows and Unix systems and automated updates deployment.
The entire application consists of several modules.
  • Main module, Patch Manager, is C++ application. It shows main software UI and allows user to manage systems list, system analysis and remediation. It uses Microsoft SQL server 2000 to store data.
  • Agent Manager, C# application consisting of ASP.NET web interface for agent communication and service monitoring agent information. It uses Microsoft SQL server 2000 to store data.
  • Agent Job Manager, C++ service is a middle tier between Patch Manager and Agent Manager that provides integration of these logically separate modules.
  • Agent Service, C++ custom HTTP/HTTPS client that executes jobs provided by Agent Manager.

Testing approaches


Patch Manager testing
Since the application has UI and serves as main console allowing a user to perform all actions provided by the entire system, stores user data, the following major testing approaches have been used to test the application:
  • Functional testing provides assurance that all applications features are operable, work as expected and fully functional. Functional testing procedures are completely documented with test plans and test cases. Most common parts of functional test plan are tested with automated test scripts.
  • UI testing is performed to validate that user interface of the application meets general UI standards and basic usability requirements.
  • Documentation testing was made to verify that product help system, guides and information provided on the company web site is correct and sufficient to describe all application features and requirements.
  • Load testing was made to check the application operability in huge environments (over 10000 analyzed systems) and robustness after working for a long time. Load testing scenarios are a part of functional test plan and are fully automated.
  • Security analysis was performed since the application stores and uses credentials to access web site and target systems. Security analysis included verification of strong password encryption for storing and transition, binary files spoofing protection, potential usage of the specified credentials from different accounts, secure communication with web site.
  • Negative testing included fault tolerance and application recovery test scenarios when some critical resources become unreachable for the application such as missing or corrupted configuration files, wrong passwords, unavailable database, insufficient account permissions, etc.
  • Tight application profiling was made to check usage of critical resources by the application. Profiling included monitoring of memory usage, handle count, utilized GDI objects, usage of SQL connections pull and verification of non-user exceptions (like access violation) thrown and handled by the application.
Agent framework testing
Agent framework is a distributed set of applications that may interact with user only during installation, though each part is intended to be deployed in unattended mode, and thus general test approaches were adopted. General test approaches are:
  • Functional testing was made using Patch Manager console, and the same tests were run using agents as Patch Manager provides similar functionality using agents and not using them.
  • Load testing was made using Silk Performer that emulated numerous agent requests for jobs and uploading job results and was run against server system for a long time.
  • Stress testing was performed in using the same scripts that were used for load testing, but configured to simulate more simultaneous agent requests and ‘weak’ hardware configuration was used on the server side.
  • Security analysis included verification of possible ways to make agent run custom executable using file spoofing, usage of credentials from different account, http response spoofing.
  • Server side application profiling contained monitoring of critical resources used by the application, World Wide Web Publishing service and IIS worker process such as memory utilization, handle count, thread count, usage of SQL connections pull.

Summary

During testing of Patch Manager Smartech QA team found over 4000 bugs in the application. Patch Manager 4.0 was named Product of the Year in the Patch Management category in the SearchWindowsSecurity.com Products of the Year’ awards. Established QA process and prepared documentation is still being utilized by QA teams involved in testing the product.

Tools used for testing

  • TrackIt and Aqdevteam bug trackers;
  • TestComplete automated testing tool;
  • Squish testing automated testing tool;
  • Delphi 7 development tool for creating applications for testing purposes;
  • PerfMon and SysInternals (www.sysinternals.com) monitoring tools;
  • Visual Studio 2003 development tool for creating applications for testing purposes (C#/C++);
  • Silk Performer load/stress testing tool.