Monthly Archives: June 2016

Hypothesis testing and why we’re all actually research scientists

Testers are scientists.

You might not wear a lab coat to work (but if you don’t, you should try it), you might not think of yourself as a scientist, but what you do every day is craft a hypothesis and then design and execute experiments to try and falsify it. Many testers might not think of themselves as experimental scientists, it may seem like we have nary a test tube in many an office and no
Bunsen burners in the server room, so how can we be scientists?

Science isn’t defined by the equipment, but by the methodology. Karl Popper’s philosophy of science* defines science, in a horrible paraphrase, as the framing of a hypothesis and repeated attempts to disprove it – which is exactly what you do as a tester. You have a hypothesis, which I’ll designate as the hypothesis under test. The hypothesis under test for software testers is, generally, that the software behaves as expected.

So why does this matter? Well first you need to define what the expected behaviour is, what it should do when presented with different inputs/situations. Then you need to see if you can falsify that hypothesis under test by designing experiments which will provide a series of inputs and situations and observe the outcome. You tend to call these test cases. But are you really doing this? Often it’s too easy to fall into the trap of merely checking that the software behaves properly when given correct input, however that’s not falsification. You definitely should see if this happens, but there has to be an element of falsification; you have to assume that when given abnormal input or situation it also behaves correctly. Of course the label “situation” here covers a multitude of sins, from throttled CPU, to network time outs, to RAM. Your experiment will probably be best served by being broken down into many experiments to isolate each input and environmental variable.

Your experiment is starting to look a lot like a test plan and test cases/areas under test. Now you might be testing something like tens of thousands of connections to a database or packet transfer and the environment itself will introduce occasional failures, or you might simply have a tolerance for a certain level of failure. You accept that as long as X% or less of your experiments/tests fail, you’re willing to accept that the system as a whole works. Now this is a statistical method of hypothesis testing, you might have done something like this at school with confidence intervals and two tailed tests and the like. There’s a lot of maths around this area and if you’re so inclined, and I honestly think it’s worthwhile knowing, you can find lots about it on the internet**. However it’s probably enough to know an overview of the basic maths from the links below and you’ll see how you can account for things like systemic error.

I hope this has illustrated something of what I use to approach my testing, I was trained at university as a physicist, so I spent a lot of time designing ways to test hypotheses and even more time actually testing them. Then I spent even more time than that going through my results to extract meaning from the morass of data I’d generated.

I won’t be plunging this blog into a morass of mathematics and statistics, at least on my regular testing blog posts. Though I have an idea to explain randomness and its application to algorithms to generate efficient computations and good test coverage. This would require a whole series of posts to explain what randomness is, how it can be useful and why you shouldn’t fear things slipping through a procedurally generated test regime which employs stochastic elements in its tests. Whether anyone would want to read it though is another matter, if you feel strongly either way please let me know in the comments.


Getting started in Python

Recently I’ve had an enquiry or two about resources for learning Python and applying those to testing. Well luckily for people who want to know, I was not born knowing Python and I’ve amassed quite a collection of resources relating to python and a few ideas of my own that I hope can help.

First up, there’s the basic courses on Python.
Codecademy – £Free
This is a basic, and free, grounding in python and programming that takes you through several projects and quizzes to get you started in programming and python, definitely recommended for the more traditional starter who wants to learn about programming more than instant practical applications. This can easily be done in a couple of work days or a couple of weeks of evening, or a weekend. It’s a very accessible and aside from the final project you can do it all in the browser so it’s perfect for those who want to dabble.

Automate the boring things with Python – £31
Another course for absolute beginners, longer than Codecademy’s offering and with a lot more practical examples, but less structured towards general programming concepts and more towards getting things done. Yes it costs money, but you’ll be productive with it right off the bat.

Google Python lessons – +£Free
How I learnt Python. This assumes that you know at least one other programming language and the basics of programming, it’s teaching you Python and not how to write a program. As such it doesn’t go through a lot of things like basic program flow, algorithm design or the like. It teaches you how Python does things and gives a great basic understanding of the Python standard library. Doesn’t touch much on 3rd party libraries.

EDIT: Hat tip to Sorina for this Computer science 101 from Udemy. As she suggests it’s an excellent companion to the Google course as it fills in the gap in that course.

Next up there’s podcasts:
Talk Python to Me
Talk Python to me is a general python podcast, its level is definitely beyond beginner, so I’d recommend doing some python programming first to see if you like it. It’s very much focussed on Python development but it includes interviews with people doing quite fascinating things with Python. Expect to hear about massive Python suites being developed, python running at speeds you mightn’t have expected from an interpreted language and library development. I recommend the episodes on Requests and Fluent Python especially, once you’ve spent some time writing Python. Both are very revealing and Fluent Python is in my reading list later on.

Python Testing
This is probably the best match to what I’m writing about when I talk about testing with Python. It’s a little drier and more technical than Talk Python To Me, but it’s definitely worth the listen. Lots of interesting tit-bits about testing with Python, the frameworks, the practicalities. The presenter’s somewhat lubrugious delivery might require some adjustment to, but well worth the effort.

Code Newbie
Less Python focussed, this is just a series of interviews with people talking about their experiences learning to program. Personally I didn’t find this podcast to my taste, a little too general and rambling for me, but useful if you want some inspiration from other people’s experiences.

Finally, a reading list:
Learning Python
Great book with oodles of information about Python and probably life itself. Very comprehensive and a very good reference for anyone learning Python. Expect it to sprout post-its like mushrooms after the rain. If I could only have one Python book, it would be this book. It’s not really a book about learning by example, I recommend it in parallel with the courses above and as a reference book.

Fluent Python
Another brilliant O’Reilly book but definitely for the more advanced user. Lots of excellent information about the more advanced and deep aspects of Python. Very strongly recommended, but definitely not a first book

Black Hat Python
In the vein of Automate the Boring Things With Python, this lets you hit the ground running on security testing and some useful network testing scripts.

As previously stated however, the absolute best way to learn is to keep writing programs! Find something you are passionate about and write it!