Lesson 11: Testing

Homepage Content Slides Video


This lesson is under construction. Learn from it at your own risk. If you have any feedback, please fill out our General Feedback Survey.


Automated testing, commonly referred to as just ‘testing’, is writing code that tests your code.

This can be as simple as calling a function and expecting a specific output or as complicated as simulating button clicks on a webpage. The field of testing is broad and key to modern development.

def add_double(x, y):
    return 2*(x+y)

def test_add_double():
    expect(add_double(1, 2) == 6)

Why Testing Matters

mars testing example

The first program you wrote you probably tested manually. This means you ran the program, fed it inputs, and read the outputs by hand.

With automated testing you spend a slightly longer up-front in writing the tests up, but then you can run all of your tests with a single command (or with Continuous Integration, no commands at all!)

Testing is very important when there’s a lot of money or time (or both!) invested in a project.

Structure of a Test

Most tests consist of the same general structure:

Any pre-testing steps occur here. For instance, your application may use a database; this is where you would populate that database with test data.
Expected output
The expected results of your function calls are outlined.
Actual output
The functionality being tested is implemented and its results are recorded.
Now the two values (expected and actual) are compared, usually using some form of the syntax expect(expected == actual). If the contents of the expect call is false, then the test-runner (more on that later) raises an error, continues running the test, and produces a traceback at the end of all tests.
The setup and the tests are undone. If data was populated during the test that data is removed; if files were written they are deleted. This is to ensure that each test is completed in the same environment and each one is self-contained.

Types of Testing

There isn’t just testing. Testing happens at many stages of development and in many different ways. Here we will discuss three major types:

Unit Testing

Testing each component (function, struct, class, etc) individually, ignoring how the program works as a whole in favor of verifying each piece.

include app_library

def test_math_function():
    expected = 5.67
    actual = math_func(1, 2, 3, 4)
    expect(actual == expected)
Integration Testing
Testing how each function works together. Ensuring the components do what they are expected to do.
Systems Testing
Testing the program as a whole in an environment (computer, operating system) similar to its target platform(s).

Concept: Mocking

Simulating behavior external to a program so your tests can run independently of other platforms.

You’re testing your program, not somebody else’s. Mock other people’s stuff, not your own.

For example: If you are writing a library that wraps a web API you would mock that API so you can ensure the tests run even when the website it wraps is down.

Testing Frameworks

Testing frameworks range in the functionality they provide from simply detecting and running test functions, to helping programmers articulate tests closer to English, to forcing a very logical type of organization on your tests.

$ run tests
Finding tests...
Running tests in tests/foo.ext
Running tests in tests/bar.ext
Running tests in misc/test_baz.ext

Frameworks vs ‘The Hard Way’

While you can write tests the hard way:

var = some_function(x)
if var == expected_output:
    print("Test X failed!")
$ run test
Test 5 failed!

It’s usually easier to use a framework.

def simple_test():
    expect(some_function(x), expected_output)
$ run tests
Test 5 failed.
Debug information:

Teardown and Setup

One major advantage all testing frameworks offer is the concept of “setup” and “teardown”. This is the process of running a function, or set of instructions, before and after each test.

Useful for:
  • populating a test database
  • writing and deleting files
  • or anything else you want!

The advantage of setup/teardown is that each test is run in the same environment. This allows you to write the tests and not worry about “Wait, is there anything in the database when this is run? I specifically only need X in the database.”

def tests_setup():
    connect to database
    populate database with test data

def tests_teardown():
    delete all data from test database
    disconnect from database

def some_test()
    setup is called automatically
    use data in database
    assert something is true
    teardown is run automatically

TODO: Using Python’s unittest

Let’s suppose that we want to add a new view to the Flask app we created in the Frameworks lesson’s TODO. When the user enters the url /hello/<name>, where “name” is any string of the user’s choice, the view should return “Hello <name>!!” BEFORE you actually write this view, write a test that will test the desired functionality first– i.e., test that your hello.py returns “Hello bob!!” when “bob” is provided as the name variable. AFTERWARDS, implement the actual view to make your test(s) pass.

Unittesting in Flask
Check out the official Flask docs for help with syntax.

Answer excerpt:

def test_hello(self):
    rv = self.app.get('/hello/bob')
    assert 'Hello bob' in rv.data

Further Reading

CS 362
This OSU Course covers testing very in depth and even covers types of testing including Random testing and testing analysis.
Python Unittest Documentation
A good reference for using Python’s built-in unit-testing module.