Will it or won't it

John is hired as a Lead Software Engineer to look after a complex application which his employer relies on for it's business. John's Manager requests that John provide a briefing in two weeks for the company board on the reliability of the application and specifically he asks John to ensure to the board that the application will never crash or exit unexpectedly.

What information can John provide ?

  • A : With suitable analysis he can guarantee it will never crash under any circumstances
  • B : Not Enough Information
  • C : He can report on the defensive mechanisms within the application and the quality of the testing, but he cannot sensibly guarantee that the application will never crash (without access to significant time, resources and people).
B C A

This section requires Javascript.
You are seeing this because something didn't load right. We suggest you, (a) try refreshing the page, (b) enabling javascript if it is disabled on your browser and, finally, (c) loading the non-javascript version of this page . We're sorry about the hassle.

2 solutions

Suneet Bendre
Sep 21, 2015

Guarded code will help to protect application from unexpected behaviour for explored code flows, but you cannot finite complete code flow hence caanot guarantee 100% behaviour.

Suneet - I would sort of agree, although It is only human to tend to guard for those errors that we expect might happen - rather than the ones that are really unexpected (I have had a system library crash not due to anything I could guard against or prevent, but due to a race condition in the library which had gone undetected for 10 years - until we found it under heavy load conditions) Your main weapon is good and thorough testing - including mocking out interfaces to allow you to get good coverage on each of your modules/units in isolation. With a good mocking framework - you can make your libraries generate all the possible errors - even ones you can't reproduce normally.

Tony Flury - 5 years, 8 months ago
Tony Flury
Sep 20, 2015

Obviously with very simple programs you can look at the code and by inspection prove that the code shouldn't ever crash - but it is unlikely that business will be dependent on a such a simple program.

What the manager is asking is related to the work that Alan Turing and others did to find a solution to the Halting Problem

The Halting Problem that Alan Turing and others looked at is whether you could develop an algorithm - call it the Tester - (which could be another application) which determines if any program will complete or not for a given input. Turing proved that you could never construct the Tester algorithm to prove that any program halts - as you can then construct another program which will contradict itself - essentially by running the Tester on itself.

You can extend the logic to any application and any possible result, and state it is generally impossible to develop an algorithm to analyse a program to determine that it will always generate a specific result - or in our case will never produce a particular result. (Since you can always trivially change a program to halt when a given result is generated, the Halting Problem - and it's lack of solution means that you can never guarantee by analysis that a given result will be generated)

Although we only have one program to test (the business application), the Halting problem does not "strictly" apply, it is however a good start in thinking about what analysis will do for you. There is an implication that it might be possible to build a Tester for your application that proves that you get the right answer (i.e. no crashes) in ALL cases, when you consider how complex most applications are - and how many possible options there are for the input data - it should start to give you an idea of how complex your Tester might need to be to Guarantee that your application will Always behave.

The critical thing here is analysis - the work Turing did says nothing about testing, and you can test that your application generates the expected result with particular inputs.

Practically the best that John can do is the following :

  • Prove that all the tests provided results as expected.
  • Demonstrate that your tests covered a reasonable level of invalid inputs and error conditions
  • Perform soak or load tests and confirm that no unexpected results were encountered.
  • Provide an analysis of the error handling to show that every conceivable error has been caught

Even with all of this, John cannot possibly test with all possible inputs, or soak test for long enough to guarantee that the application doesn't have a slow fault (say a memory leak that happens once a month) or any other of the myriad condition which could cause problems.

0 pending reports

×

Problem Loading...

Note Loading...

Set Loading...