SOA and Software Testing
By Robin Bloor, Partner
Software testing is one of the areas of SOA that is not discussed a great deal. There is an assumption, that with SOA, software testing procedures don?t need to change much. Unfortunately it isn?t so. Testing procedures will need to change and there are issues that need to be thought through.
This becomes clear if we think just of web services. It?s a compelling idea to take some of the applications that we have and expose their business functions as web services. Indeed this is the first thing that some IT shops do when they catch the SOA bug. They create a library of web services. And if they choose the right business components it can be very useful indeed.
But how do you test these web services so that you know they are doing exactly what you expect them to do? Bear in mind that the original application was probably never written with the idea that it would be used as a collection of components. The programs code may be spaghetti. Consider, say, something that?s reasonably easy to understand like managing a customer database. Let?s say that our Order Processing application does this and therefore we simply expose its customer update capability (add, amend and delete customers) as a web service. We know the application so we know that the program logic does what we want it to do (add, amend and delete). But what else does it do? Therein lies a potential problem.
Unit Testing of Web Services
How are we going to find out? We could read the program code. It would be a good thing to do, to see if we can spot any logic that gets called that we might not want to get called when we add, amend or delete customers. But we need to do much more than that to be rigorous. We have no choice but to test our new web service in conjunction with the testing of the Order Processing application. We could use an existing regression testing capability on the Order Processing application while we test all the possible combinations of SOAP messages that the new Customer Update web service will accept and respond to. When we have completed this test, all that we have done is completed the unit testing of the web service. The truth is that we need to do a test of this kind when we expose any web service for use. Indeed procedures for doing so need to be laid down as part of IT governance.
Consider now the situation where we are building a composite application by threading a collection of web services together and adding some new logic of our own. Hopefully every one of the web services involved will have been unit tested in this manner. The testing procedure should follow the usual cycle: Creating Test Plans, Creating a Test Bed and Test Procedures (Test Design, Specific Test Cases, Specific Scripts, Test Execution, Reporting). The fact that the web services we are using have been unit tested should not make us overconfident of their behavior. Even if they didn?t go wrong when tested individually it?s quite possible for them to go wrong when they?re holding hands. Our testing approach needs to allow us to break down the end-to-end transactions to detect the point where failure occurs. This means being able to capture and analyze all the SOAP messages that are passed from one component to another.
In essence integration testing is no different from traditional system testing. First we do unit testing, then we do integration testing. The additional factor is that we have to do regression testing on the applications whose web services we are using while we do the integration testing.
The other two kinds of testing (stress testing/performance testing and acceptance testing) also need this extra element. We need to test everything that is linked together and test it so that it runs in the way that it will run in the ?live? environment.
Maybe you?ve spotted the problem here. Things can get very complicated. In the pleasant but inefficient world of application silos, testing was relatively easy. You tested the silo. You unit tested the components of the application. Next you integration tested the whole. Then, if you had any qualms about performance, you stress tested the whole thing. Ultimately you gave it over for acceptance testing. Creating the test bed was not a problem; you just imitated the silo run-time environment and built up a set of testing data, possibly borrowing some data from ?live? systems.
The problem with SOA is that it?s end-to-end. The silos have gone and with them went the ability to easily set up a test bed. As you make your way down the SOA road, you will need to have much more versatile test-bed creation capabilities. Indeed you will probably want tools that can create and dismantle virtual testing environments and draw data from live systems. Without such tools simply creating an adequate test bed will be difficult.
Stress testing/performance testing is also going to be a bit of a challenge as regards the test bed. Depending on how a new composite application is going to run, you will probably want the stress testing to be modeled around the whole set of end-to-end software that will quite probably span many different servers.
What currently makes the testing situation manageable with SOA is that very few, if any, organizations are building complex end-to-end composite applications at the moment. Their projects tend to involve the integration of adjacent application silos or the adding of browser interfaces to core systems, plus a little bit of BPM here and there. When businesses get more ambitious, they?ll discover how complex SOA testing can get.