Hello All:
We have used HP Load Runner for conducting the web services testing. We have done fairly well in running tests for functional and load tests through HP Load Runner. I want to know that what are the unique features, which are being carried over by the HP Service test. How buying this tool will add value to the testing of web services exercise?
Also, any comparison with iTKO LISA tool on this front will be appreciated.
http://www.infoq.com/news/2009/03/soa-testing-tools
There are quite a few SOA testing tools to choose from now. I am often asked which is the best tool?
The answer of course depends on requirements. Here are the requirement RFI/RFP categories I often used to help with test tool selection:
* Test-aware interfaces: standardized interface and messages for testing
* Message-based test automation: record, replay and management of test scripts
* Virtualization: the ability simulate virtual service providers and consumers
* Simulation: the ability to simulate applications as part of a regression test
* Load Test: the ability to stress test applications
* Validation: pass/fail verification at the component (messaging) and application (data store) levels.
* Components: Java and/or .NET component visibility
* Introspection: WSDL and XML support to generate data and operation tests
* Management: Manage test cases, scripts, data and results
* Security: SSL, WS security federation, digital signatures
* Industry Specific Data Formats: e.g. EDI, HL7
* Continual Test: Automated build, deploy and test
http://www.soatesting.com/
http://www.infoworld.com/d/architecture/three-open-source-web-service-testing-tools-get-high-marks-995
Thursday, August 26, 2010
Generate the Custom Graphs using LR Analysis Component
Please validate the below before confirming with the client… an interesting article to try..
__________________________________________________________________________________________________________________
Hi All,
Easy to Generate the Graphs for PerfMon/Nmon (Unix flavor Operating System) counter data: (Silk Performer – CSV to TSD option)
HP LoadRunner Analysis is a free tool right? By using LoadRunner Analysis component, we can use to generate the graph for external data file (CSV/XLS).
Usually, we are monitoring the server’s operating system and server specific counters using other 3rd party tools like PerfMon/Nmon/VMSTAT/Top commands (Due to some security reasons, we are monitoring the servers using 3rd party tools and will generate the measurements in CSV/XLS format). After completion of collecting the data (in CSV/XLS format), we need to write a macro or manually plot the graphs in Excel. Instead of drawing the graphs manually, we can use the LR analysis component effectively. We can certainly install the trial version of LoadRunner 9.x. Simply use the Import Data functionality built into Analysis…
Please find the below example.
1. Collecting performance metrics from remote Linux Operating System to get the raw metrics provided from commands like vmstat, iostat and Top to get memory, disk and cpu utilization respectively.
2. Then quickly parse that data using some custom Perl scripts to help format those results into the csv format that LoadRunner Analysis expects.
For example, vmstat info that comes in its raw format like this:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
0 0 0 40563488 10311984 4 21 1 0 0 0 0 0 0 0 0 413 280 280 0 0 99
0 0 0 31902256 1908536 1 20 0 0 0 0 0 0 0 0 0 637 793 831 1 0 99
Can be turned into LoadRunner Analysis friendly format like this:
date,time,r,b,w,swap,free,re,mf,pi,po,fr,de,sr,m0,m1,m2,m1,in,sy,cs,us,sy,id
13/4/2008,17:36:22,0,0,0,31792512,2202496,0,2,0,0,0,0,0,0,0,0,0,577,699,706,1,0,99
13/4/2008,17:36:17,0,0,0,31779792,2201592,0,2,0,0,0,0,0,0,0,0,0,568,699,695,0,0,99
Using the Import Data Tool in LR Analysis.
To use the Import Data tool:
1. Choose Tools > External Monitors > Import Data. The Import Data dialog box opens.
2. Browse the CSV file to generate the graph
3. Merge the custom graph with LR client side Running Vuser graph
4. Prepare the Analysis Report.
__________________________________________________________________________________________________________________
Hi All,
Easy to Generate the Graphs for PerfMon/Nmon (Unix flavor Operating System) counter data: (Silk Performer – CSV to TSD option)
HP LoadRunner Analysis is a free tool right? By using LoadRunner Analysis component, we can use to generate the graph for external data file (CSV/XLS).
Usually, we are monitoring the server’s operating system and server specific counters using other 3rd party tools like PerfMon/Nmon/VMSTAT/Top commands (Due to some security reasons, we are monitoring the servers using 3rd party tools and will generate the measurements in CSV/XLS format). After completion of collecting the data (in CSV/XLS format), we need to write a macro or manually plot the graphs in Excel. Instead of drawing the graphs manually, we can use the LR analysis component effectively. We can certainly install the trial version of LoadRunner 9.x. Simply use the Import Data functionality built into Analysis…
Please find the below example.
1. Collecting performance metrics from remote Linux Operating System to get the raw metrics provided from commands like vmstat, iostat and Top to get memory, disk and cpu utilization respectively.
2. Then quickly parse that data using some custom Perl scripts to help format those results into the csv format that LoadRunner Analysis expects.
For example, vmstat info that comes in its raw format like this:
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
0 0 0 40563488 10311984 4 21 1 0 0 0 0 0 0 0 0 413 280 280 0 0 99
0 0 0 31902256 1908536 1 20 0 0 0 0 0 0 0 0 0 637 793 831 1 0 99
Can be turned into LoadRunner Analysis friendly format like this:
date,time,r,b,w,swap,free,re,mf,pi,po,fr,de,sr,m0,m1,m2,m1,in,sy,cs,us,sy,id
13/4/2008,17:36:22,0,0,0,31792512,2202496,0,2,0,0,0,0,0,0,0,0,0,577,699,706,1,0,99
13/4/2008,17:36:17,0,0,0,31779792,2201592,0,2,0,0,0,0,0,0,0,0,0,568,699,695,0,0,99
Using the Import Data Tool in LR Analysis.
To use the Import Data tool:
1. Choose Tools > External Monitors > Import Data. The Import Data dialog box opens.
2. Browse the CSV file to generate the graph
3. Merge the custom graph with LR client side Running Vuser graph
4. Prepare the Analysis Report.
Wednesday, August 25, 2010
Visual Studio 2010 - Load Test Tool Details
Scripting(Web test):
- Do we have Initialization section and Termination section for a user.(Ex: A user log in and does some set of action and log out from the web page. when a user enters the login and logout is done only once,the set of actions must be Iterated).
http://msdn.microsoft.com/en-us/library/bb514192.aspx
http://blogs.msdn.com/b/chrispat/archive/2010/01/10/branching-and-looping-in-a-declarative-web-test.aspx
- Error Handling (Ex:If error occurs need to stop the iteration and continue with the next).
http://blogs.msdn.com/b/slumley/archive/2007/04/10/how-to-debug-a-web-test.aspx
- How to include the custom messages to debug the script and display in the log. (Enhanced the code using Console.Writeln() but unable to see in the log).
http://msdn.microsoft.com/en-us/library/bb514192.aspx
- To suppress a particular dependent request instead of all the requests which are not required(Ex: some .gif, blocked contents etc).
A top-level request can appear to have failed because one of its dependent requests could not be found. This might indicate a problem with the Web application's HTML. These errors can be suppressed by disabling the ParseDependentRequests property on the request.
http://msdn.microsoft.com/en-us/library/ms364082(VS.80).aspx
- Log window options such as to enable/disable during the test run.
http://blogs.msdn.com/b/edglas/archive/2007/12/02/web-test-authoring-and-debugging-techniques-for-vs-2008.aspx
http://teamtestplugins.codeplex.com/
- Functions related to build the dynamic form data for a request.
Please share more details on the above point.
- How to extract data when a response page consists of multiple frames.
http://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.webtesting.rules.extracthiddenfields.aspx
- Parameterization: If we select the random username from the data source file, then how to retrieve the corresponding password of the username.
If both username & password are picked up from same datasource then corresponding password will be select with random username automatically.
- If we enhance the code in .cs filethen how to .webtest should get reflected.
It is not supported, since both are different files.
Execution(Load test):
- Can we limit the number of users per Load Agent(Ex: To execute 1000 users, can we distribute 500users from each LA instead of 1000 from a single LA).
You can use the Test Agent property called weighting to distribute the load.
Test Agent Property Description
Weighting Used to distribute load when you use test agents with different performance levels. For example, a test agent with a weighting of 100 receives two times the load as a test agent with a weighting of 50.
Please find below the link for more information on changing the test agent settings:
Changing the Settings for a Test Agent
http://msdn.microsoft.com/en-us/library/dd695837.aspx#ChangeAgent
- Do we have any options to Ramp down the users?
There are no built-in support for this, but one can write custom load test plug-in for ramp down users, please refer below link for more information:
http://blogs.msdn.com/b/yutong/archive/2010/03/15/load-test-plug-in-simulate-multiple-and-repeated-user-load-patterns-step-2-1.aspx
http://blogs.msdn.com/b/slumley/archive/2009/04/10/load-test-plug-ins.aspx
- Can we dynamically increase/decrease the user load during the test run.
http://blogs.msdn.com/b/yutong/archive/2010/03/15/load-test-plug-in-simulate-multiple-and-repeated-user-load-patterns-step-2-1.aspx
Reporting:
- How can we generate the automated report with graphs.
http://blogs.msdn.com/b/slumley/archive/2009/05/22/dev10-feature-load-test-excel-report-integration.aspx
- Do we have Initialization section and Termination section for a user.(Ex: A user log in and does some set of action and log out from the web page. when a user enters the login and logout is done only once,the set of actions must be Iterated).
http://msdn.microsoft.com/en-us/library/bb514192.aspx
http://blogs.msdn.com/b/chrispat/archive/2010/01/10/branching-and-looping-in-a-declarative-web-test.aspx
- Error Handling (Ex:If error occurs need to stop the iteration and continue with the next).
http://blogs.msdn.com/b/slumley/archive/2007/04/10/how-to-debug-a-web-test.aspx
- How to include the custom messages to debug the script and display in the log. (Enhanced the code using Console.Writeln() but unable to see in the log).
http://msdn.microsoft.com/en-us/library/bb514192.aspx
- To suppress a particular dependent request instead of all the requests which are not required(Ex: some .gif, blocked contents etc).
A top-level request can appear to have failed because one of its dependent requests could not be found. This might indicate a problem with the Web application's HTML. These errors can be suppressed by disabling the ParseDependentRequests property on the request.
http://msdn.microsoft.com/en-us/library/ms364082(VS.80).aspx
- Log window options such as to enable/disable during the test run.
http://blogs.msdn.com/b/edglas/archive/2007/12/02/web-test-authoring-and-debugging-techniques-for-vs-2008.aspx
http://teamtestplugins.codeplex.com/
- Functions related to build the dynamic form data for a request.
Please share more details on the above point.
- How to extract data when a response page consists of multiple frames.
http://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.webtesting.rules.extracthiddenfields.aspx
- Parameterization: If we select the random username from the data source file, then how to retrieve the corresponding password of the username.
If both username & password are picked up from same datasource then corresponding password will be select with random username automatically.
- If we enhance the code in .cs filethen how to .webtest should get reflected.
It is not supported, since both are different files.
Execution(Load test):
- Can we limit the number of users per Load Agent(Ex: To execute 1000 users, can we distribute 500users from each LA instead of 1000 from a single LA).
You can use the Test Agent property called weighting to distribute the load.
Test Agent Property Description
Weighting Used to distribute load when you use test agents with different performance levels. For example, a test agent with a weighting of 100 receives two times the load as a test agent with a weighting of 50.
Please find below the link for more information on changing the test agent settings:
Changing the Settings for a Test Agent
http://msdn.microsoft.com/en-us/library/dd695837.aspx#ChangeAgent
- Do we have any options to Ramp down the users?
There are no built-in support for this, but one can write custom load test plug-in for ramp down users, please refer below link for more information:
http://blogs.msdn.com/b/yutong/archive/2010/03/15/load-test-plug-in-simulate-multiple-and-repeated-user-load-patterns-step-2-1.aspx
http://blogs.msdn.com/b/slumley/archive/2009/04/10/load-test-plug-ins.aspx
- Can we dynamically increase/decrease the user load during the test run.
http://blogs.msdn.com/b/yutong/archive/2010/03/15/load-test-plug-in-simulate-multiple-and-repeated-user-load-patterns-step-2-1.aspx
Reporting:
- How can we generate the automated report with graphs.
http://blogs.msdn.com/b/slumley/archive/2009/05/22/dev10-feature-load-test-excel-report-integration.aspx
Details of MS VSTS 2010
Hello MSFT Guys:)
Can you please send me some information about the Visual Studio Test Professional 2010? Also, if you know people who are really involved in development/maintainance of this tool, that can be a great help for me also. Below are some of the links found by me:
http://www.radiotfs.com/2008/10/23/RoadToRosarioSpecialVSTS2010Test.aspx
http://blogs.msdn.com/b/james_whittaker/default.aspx?PageIndex=1
http://www.dotnetcurry.com/ShowArticle.aspx?ID=464&AspxAutoDetectCookieSupport=1
Here is the landing page : http://msdn.microsoft.com/en-us/vstudio/default.aspx - start from here to get info on Test Pro and other ALM SKUs
http://blogs.msdn.com/b/anutthara/archive/2010/02/09/resources-for-visual-studio-2010-test-tools.aspx has more resources. For all qs related to Test Pro, write to the MSDN forum listed in the resources list
A very basic description about the Memory Allocation:
http://blogs.msdn.com/b/abhinaba/archive/2009/01/25/back-to-basic-series-on-dynamic-memory-management.aspx
http://blogs.msdn.com/b/abhinaba/
http://blogs.msdn.com/b/amit_chatterjee/archive/2010/02/12/vs-2010-rc-is-ready-for-general-download.aspx
Can you please send me some information about the Visual Studio Test Professional 2010? Also, if you know people who are really involved in development/maintainance of this tool, that can be a great help for me also. Below are some of the links found by me:
http://www.radiotfs.com/2008/10/23/RoadToRosarioSpecialVSTS2010Test.aspx
http://blogs.msdn.com/b/james_whittaker/default.aspx?PageIndex=1
http://www.dotnetcurry.com/ShowArticle.aspx?ID=464&AspxAutoDetectCookieSupport=1
Here is the landing page : http://msdn.microsoft.com/en-us/vstudio/default.aspx - start from here to get info on Test Pro and other ALM SKUs
http://blogs.msdn.com/b/anutthara/archive/2010/02/09/resources-for-visual-studio-2010-test-tools.aspx has more resources. For all qs related to Test Pro, write to the MSDN forum listed in the resources list
A very basic description about the Memory Allocation:
http://blogs.msdn.com/b/abhinaba/archive/2009/01/25/back-to-basic-series-on-dynamic-memory-management.aspx
http://blogs.msdn.com/b/abhinaba/
http://blogs.msdn.com/b/amit_chatterjee/archive/2010/02/12/vs-2010-rc-is-ready-for-general-download.aspx
Friday, July 23, 2010
Some thing to chew - Dell moves to AMD and leaves Intel - Business case?
Hi all -
For people who want to chew a little more than what is given in our study curricula. Check this site out -
http://www.icmrindia.org/business%20Updates/micro%20casestudies/Business%20Strategy/MCBS0005.htm
This has a good deal of cases and interestingly this talks about the move made by Dell in disowning Intel and going towards AMD. Interesting move considering they operate at a different levels and AMD processors were always considered better unless in the Gaming circuit. This would alter the growth of Intel and an excellent case for business schools.
FYI - this site in general talks more in detail about several such business cases around Asia.
Have a good one
Vik
--
For people who want to chew a little more than what is given in our study curricula. Check this site out -
http://www.icmrindia.org/business%20Updates/micro%20casestudies/Business%20Strategy/MCBS0005.htm
This has a good deal of cases and interestingly this talks about the move made by Dell in disowning Intel and going towards AMD. Interesting move considering they operate at a different levels and AMD processors were always considered better unless in the Gaming circuit. This would alter the growth of Intel and an excellent case for business schools.
FYI - this site in general talks more in detail about several such business cases around Asia.
Have a good one
Vik
--
Error: Decompression functionon
I suspect that you haven't recorded the script with the necessary headers. Go to the recording options>http properties>headers.
Ensure that your list of headers to record contains:
1. Accept encoding
2. Content encoding
3. Accept languages
4. Pragma
5. Transfer encoding
6. User agent
7. Accept- Charset
8. Accept
9. Cache control.
These are the most commonly used header tokens used in http 1.1.
Now, what happens if you don't record the accept encoding/content-encoding header is that if your server response/request is expected to be in compressed/deflated(which is defined in web server settings) form, but if Loadrunner/Client sends an uncompressed/inflated file/message, then the protocol goes out of sync. As a result, the Loadrunner compression function would expect the data to be in gzip format but the data sent from server would not be in gzip format and hence the error.
Add web_add_auto_header("Accept-Encoding", "gzip") on top of the request where your script stops and throws the error. But, I would suggest you to re-record the script with the aforemention headers in your recording list.
-Chaitanya M Bhatt
http://www.performancecompetence.com
2010/7/21 Anuj Sharma
Can any body guide me for this issue in load runner.
Error -26601: Decompression function (wgzMemDecompressBuffer) failed,
return code=-3 (Z_DATA_ERROR), inSize=1773, inUse=5, outUse=0,
Snapshot Info [MSH 0 13] [MsgId: MERR-26601]
This is application error or load-runner configuration issue?
Ensure that your list of headers to record contains:
1. Accept encoding
2. Content encoding
3. Accept languages
4. Pragma
5. Transfer encoding
6. User agent
7. Accept- Charset
8. Accept
9. Cache control.
These are the most commonly used header tokens used in http 1.1.
Now, what happens if you don't record the accept encoding/content-encoding header is that if your server response/request is expected to be in compressed/deflated(which is defined in web server settings) form, but if Loadrunner/Client sends an uncompressed/inflated file/message, then the protocol goes out of sync. As a result, the Loadrunner compression function would expect the data to be in gzip format but the data sent from server would not be in gzip format and hence the error.
Add web_add_auto_header("Accept-Encoding", "gzip") on top of the request where your script stops and throws the error. But, I would suggest you to re-record the script with the aforemention headers in your recording list.
-Chaitanya M Bhatt
http://www.performancecompetence.com
2010/7/21 Anuj Sharma
Can any body guide me for this issue in load runner.
Error -26601: Decompression function (wgzMemDecompressBuffer) failed,
return code=-3 (Z_DATA_ERROR), inSize=1773, inUse=5, outUse=0,
Snapshot Info [MSH 0 13] [MsgId: MERR-26601]
This is application error or load-runner configuration issue?
Monday, July 19, 2010
Failover Cluster Testing Methods
http://blogs.technet.com/vipulshah/archive/2009/06/17/failover-cluster-testing-methods.aspx
Failover Cluster Testing Methods
1.1 System Failover Testing
During system testing, we will gather as much information as possible about the potential outcomes of system failures. We will not test, however, many component failures in the system such as a motherboard going out, losing a processor or having a cooling fan going out. These represent outages that have been planned for and many will be protected through Cluster Server, other failures will be covered through the fault-tolerance of the systems themselves (such as redundant power supply modules and cooling fans). We will test certain events that can cause failover within the cluster. These tests are listed below along with an explanation of the procedure along with the expected result of each test.
1.1.1 Disk Failure
Purpose: The purpose of testing a disk failure is to ensure that the RAID configuration will continue without interruption. We will also look at hot-spares and ensure that in the event of a disk failure the hot-spare will pickup for the failed drive.
Test Procedure: The procedure for this test is to pull out one of the hard drives in the SAN array while that drive is operational and is currently used by one of the nodes within cluster.
Expected Result: Uninterrupted service; Windows should not discover any problems at all; RAID management software should report loss of a drive and complete the procedure of substituting the hot spare and rebuilding the drive array. Disk performance might be significantly reduced during this time
1.1.2 Power Failure
Purpose: This test will verify that in the event of a server losing power, the opposite node in the cluster will bring all resources in the cluster online and resume operations.
Test Procedure: The procedure for this test is to simply pull all power plugs from one node while that node is operational and is hosting groups within cluster.
Expected Result: Cluster group hosted by the “failed” node should automatically fail over to a passive node. Service interruption should be in the range of 0-2 minutes.
1.1.3 Network Adapters
Purpose: Testing network adapters will serve dual purposes. We will test the functionality of the heartbeat and the ability for cluster heartbeat communications to be routed over the public network. We will also test the failover scenario in the case of both public network adapters (members of the network team) losing connections to the network.
Test Procedure: First we will test the heartbeat interconnect and ensure that cluster communications are carried through the public network without interrupting cluster communications by disconnecting the private network adapter. Secondly, we will test network adapters to ensure that when one out of the two network adapters are unplugged, the other network adapter will communicate with the network as usual. Then we will unplug the remaining network adapter, which at this time is carrying all network communication, including the heartbeat. After this series of tests is complete, we will bring the system back to a normal configuration and test the public network adapters by removing their connection to the network. However, in this last test we will leave the heartbeat interconnect in place.
Expected Result: For a disconnected private adapter, Windows is expected to switch internal cluster communications to a public adapter automatically. There should be no service interruption. For disconnecting one of the public network cards, network team driver is expected to switch to using another network card automatically. There should be no service interruption. For disconnecting all network adapters, cluster is expected to initiate the failover once it discovers that active node is unavailable. Service interruption should be in the range of 0-2 minutes.
1.1.4 Fiber Channel Components
Purpose: These tests will provide a level of understanding and documentation on expectations with regards to the redundancy of the fiber channel components within the HBA cards, the servers, and the CLARiiON SAN. We will be looking for results of unplugging certain components, simulating power losses, and other failures that will affect the cluster.
Test Procedure: During this test, we will disconnect the redundant fiber connections.
Expected Result: Depending on which fiber connection was disconnected, system should automatically switch to the reserved path. There should be no service interruption. In case of both fiber cables disconnected, so that cluster node completely loses communication to the SAN storage, cluster failover should be initiated. Service interruption should be in the range of 0-2 minutes.
1.2 Windows 2003 and SQL Server Failover Testing
Microsoft Cluster Server will ensure that application services continue running within the cluster in the event that either there are failures in Windows 2003 that prevent the application from operating properly, or if SQL Server itself ceases to function properly. The cluster can detect these failures and fail the application over to a passive node. During these tests, it is important to note that our single point of failure within Windows 2003 and SQL Server is the database(s). If database itself becomes corrupt or experiences some other catastrophic failure, the only solution is to restore this database from a backup copy.
Expected result in all tests is for a cluster to initiate the failover. Service interruption should be in the range of 0-2 minutes.
1.2.1 SQL Server Services
Purpose: The purpose of simulating service failures is to ensure that failover will occur and to monitor the activity that occurs during failover. We will be looking for the time required for failover, proper failover and ensuring dependencies are being brought online properly.
Procedure: The best approach to testing a clustered service is to stop this service from the Services snap-in within the Management MMC. A service that has become a clustered resource can only be managed through the cluster administrator. Performing services operations through the Services snap-in will appear to the cluster as a failure and therefore will simulate a service failing. Following are the services we will attempt to fail.
• SQL Server service
• SQL Server Agent service
• MS DTC service
1.2.2 Windows 2003 Failure
Purpose: The purpose here is to simulate the failure of Windows 2003 to demonstrate the ability of Cluster Server to realize Windows 2003 is not functioning/running on one of the clustered nodes and to initiate failover.
Procedure: Testing Windows 2003 failure will be difficult to simulate by any other means than simply choosing Shut Down. This will stop all services on the node being shut down and this node will cease participation in the cluster. The cluster service will be notified on the opposite node and failover of the application will occur.
1.2.3 Cluster Service Failure
Purpose: Cluster Service is responsible for maintaining cluster membership, monitoring resources and managing the clustered node. If this service were to fail, all clustered groups of resources would be forced to move to another cluster node. We will simulate a Cluster Service failure in this test and monitor the failover activity.
Procedure: Stop the Cluster Service from the Services snap-in and record results.
1.2.4 Quorum Failure
Purpose: The Quorum serves as a log for changes that occur while one node of a cluster is offline and as a tiebreaker, in the event all heartbeat communications are lost. This test will observe the cluster behavior when the quorum is lost. To view the results of the quorum serving as a tiebreaker, view the results under “System Failover Testing” above.
Procedure: Utilize the SAN configuration utility and unpresent the quorum drive from the active node. This will provide us with the results that would be seen if the active node loses access to the quorum drive. Failover should be initiated.
Utilize the RAID configuration utility and mark the logical drive of the quorum as offline. This will provide us with the results that would be seen if the drive that contains the quorum were to fail.
Failover Cluster Testing Methods
1.1 System Failover Testing
During system testing, we will gather as much information as possible about the potential outcomes of system failures. We will not test, however, many component failures in the system such as a motherboard going out, losing a processor or having a cooling fan going out. These represent outages that have been planned for and many will be protected through Cluster Server, other failures will be covered through the fault-tolerance of the systems themselves (such as redundant power supply modules and cooling fans). We will test certain events that can cause failover within the cluster. These tests are listed below along with an explanation of the procedure along with the expected result of each test.
1.1.1 Disk Failure
Purpose: The purpose of testing a disk failure is to ensure that the RAID configuration will continue without interruption. We will also look at hot-spares and ensure that in the event of a disk failure the hot-spare will pickup for the failed drive.
Test Procedure: The procedure for this test is to pull out one of the hard drives in the SAN array while that drive is operational and is currently used by one of the nodes within cluster.
Expected Result: Uninterrupted service; Windows should not discover any problems at all; RAID management software should report loss of a drive and complete the procedure of substituting the hot spare and rebuilding the drive array. Disk performance might be significantly reduced during this time
1.1.2 Power Failure
Purpose: This test will verify that in the event of a server losing power, the opposite node in the cluster will bring all resources in the cluster online and resume operations.
Test Procedure: The procedure for this test is to simply pull all power plugs from one node while that node is operational and is hosting groups within cluster.
Expected Result: Cluster group hosted by the “failed” node should automatically fail over to a passive node. Service interruption should be in the range of 0-2 minutes.
1.1.3 Network Adapters
Purpose: Testing network adapters will serve dual purposes. We will test the functionality of the heartbeat and the ability for cluster heartbeat communications to be routed over the public network. We will also test the failover scenario in the case of both public network adapters (members of the network team) losing connections to the network.
Test Procedure: First we will test the heartbeat interconnect and ensure that cluster communications are carried through the public network without interrupting cluster communications by disconnecting the private network adapter. Secondly, we will test network adapters to ensure that when one out of the two network adapters are unplugged, the other network adapter will communicate with the network as usual. Then we will unplug the remaining network adapter, which at this time is carrying all network communication, including the heartbeat. After this series of tests is complete, we will bring the system back to a normal configuration and test the public network adapters by removing their connection to the network. However, in this last test we will leave the heartbeat interconnect in place.
Expected Result: For a disconnected private adapter, Windows is expected to switch internal cluster communications to a public adapter automatically. There should be no service interruption. For disconnecting one of the public network cards, network team driver is expected to switch to using another network card automatically. There should be no service interruption. For disconnecting all network adapters, cluster is expected to initiate the failover once it discovers that active node is unavailable. Service interruption should be in the range of 0-2 minutes.
1.1.4 Fiber Channel Components
Purpose: These tests will provide a level of understanding and documentation on expectations with regards to the redundancy of the fiber channel components within the HBA cards, the servers, and the CLARiiON SAN. We will be looking for results of unplugging certain components, simulating power losses, and other failures that will affect the cluster.
Test Procedure: During this test, we will disconnect the redundant fiber connections.
Expected Result: Depending on which fiber connection was disconnected, system should automatically switch to the reserved path. There should be no service interruption. In case of both fiber cables disconnected, so that cluster node completely loses communication to the SAN storage, cluster failover should be initiated. Service interruption should be in the range of 0-2 minutes.
1.2 Windows 2003 and SQL Server Failover Testing
Microsoft Cluster Server will ensure that application services continue running within the cluster in the event that either there are failures in Windows 2003 that prevent the application from operating properly, or if SQL Server itself ceases to function properly. The cluster can detect these failures and fail the application over to a passive node. During these tests, it is important to note that our single point of failure within Windows 2003 and SQL Server is the database(s). If database itself becomes corrupt or experiences some other catastrophic failure, the only solution is to restore this database from a backup copy.
Expected result in all tests is for a cluster to initiate the failover. Service interruption should be in the range of 0-2 minutes.
1.2.1 SQL Server Services
Purpose: The purpose of simulating service failures is to ensure that failover will occur and to monitor the activity that occurs during failover. We will be looking for the time required for failover, proper failover and ensuring dependencies are being brought online properly.
Procedure: The best approach to testing a clustered service is to stop this service from the Services snap-in within the Management MMC. A service that has become a clustered resource can only be managed through the cluster administrator. Performing services operations through the Services snap-in will appear to the cluster as a failure and therefore will simulate a service failing. Following are the services we will attempt to fail.
• SQL Server service
• SQL Server Agent service
• MS DTC service
1.2.2 Windows 2003 Failure
Purpose: The purpose here is to simulate the failure of Windows 2003 to demonstrate the ability of Cluster Server to realize Windows 2003 is not functioning/running on one of the clustered nodes and to initiate failover.
Procedure: Testing Windows 2003 failure will be difficult to simulate by any other means than simply choosing Shut Down. This will stop all services on the node being shut down and this node will cease participation in the cluster. The cluster service will be notified on the opposite node and failover of the application will occur.
1.2.3 Cluster Service Failure
Purpose: Cluster Service is responsible for maintaining cluster membership, monitoring resources and managing the clustered node. If this service were to fail, all clustered groups of resources would be forced to move to another cluster node. We will simulate a Cluster Service failure in this test and monitor the failover activity.
Procedure: Stop the Cluster Service from the Services snap-in and record results.
1.2.4 Quorum Failure
Purpose: The Quorum serves as a log for changes that occur while one node of a cluster is offline and as a tiebreaker, in the event all heartbeat communications are lost. This test will observe the cluster behavior when the quorum is lost. To view the results of the quorum serving as a tiebreaker, view the results under “System Failover Testing” above.
Procedure: Utilize the SAN configuration utility and unpresent the quorum drive from the active node. This will provide us with the results that would be seen if the active node loses access to the quorum drive. Failover should be initiated.
Utilize the RAID configuration utility and mark the logical drive of the quorum as offline. This will provide us with the results that would be seen if the drive that contains the quorum were to fail.
Subscribe to:
Posts (Atom)
