# Python Testing Guide
## How to test
Here's some general guidelines for writing tests:
1. All functions (including helper functions and testing functions) require examples/test cases
2. Examples/test cases should reflect various input scenarios so that you exercise the possible main behaviors of your function. You do not need to test every input or every input combination, but rather enough to be confident that your function works with various inputs.
3. Test the “edge cases” of functions. For example, functions that take in numeric inputs and work differently depending on the range that number is in should test the boundaries of those ranges.
4. Even if your function does not work as expected, you can still write examples/test cases for it and receive full testing credit.
5. If a function has a side effect (like modifying a list), **test that side effect**.
6. If a function both has a side effect _and_ returns information, test **both the main return and the side effect**.
7. If a function _fails_ in certain cases, test that failure (see below example on how to test errors).
8. If a function modifies (or mutates) data, set up testing data rather than working on a global piece of `data/dataset`.
# Setting up pytest
**Initial setup**: To use Pytest, you should have your VSCode and Python environment configured per our [VSCode Setup Guide](https://docs.cs200.io/s/vscode-python-setup-guide). If you haven’t yet configured your VSCode and confirmed you can run tests on the HW00-setup project, you should do so before continuing.
**For every stencil you use:** Be sure to follow the instructions in the [**Python Stencil Setup guide**](https://docs.cs200.io/s/python-stencil-setup-guide) to open your stencil and make sure your interpreter is configured correctly. You will need to [set your interpreter](https://docs.cs200.io/s/python-stencil-setup-guide) each time you open a new stencil, and you should check your interpreter setting each time you run VSCode, as it can sometimes lose the setting!
# Working with tests in VSCode
This section describes a few common tasks you’ll need to do when testing with VSCode. Skim over these now and then return to the instructions as you need them.
## Running tests
Next, we’ll run some tests present in this project. To do this:
1. Open the testing panel by clicking on the beaker in the sidebar, which looks like this:
2. VSCode should automatically detect a set of tests in your project and your testing pane should look like the picture below. If not, see [this section](https://docs.cs200.io/s/python-stencil-setup-guide#If-you-need-it-Configuring-your-tests-manually) to configure your tests.
3. Once you are able to see your tests, click the **“Run tests”** button to run them!
If the tests all pass (all have green checkmarks) and your testing pane looks like the figure below, you are good to go! Yay!
## Adding New tests
Pytest and VSCode continually scan your code to look for new tests cases to run. It will find tests so long as they match the following conditions:
- All files with tests must start with test\_ and end with .py, ie. test\_thing.py
- All test functions must start with test\_ (ie, test\_for\_some\_thing)
When you write new tests, VSCode should detect them automatically and update the testing pane. However, this process is sometimes slow, so you can ask it to refresh the list like this:
1. In VSCode, open the testing pane (click on the beaker icon on the left side)
2. Click **“Refresh Tests”** on the top of the testing pane, which looks like this:
3. Your new tests should now show up in the testing pane. If they don’t, make sure they fit the conditions described above, then check [here](https://docs.google.com/document/d/1MQOK-lUaqbdOGZGp5X5uN92qM7VnnHLNd4RRrnHBulE/edit#bookmark=id.tyi49gb05291).
## Getting a REPL/Debugging with tests
The Python REPL is a great way to play around and explore things you can do with Python. In the REPL, you can type arbitrary lines of code and see the result immediately, without writing a full test! You can use VSCode’s debugger to get a REPL (which it calls the “Debug Console”) at _any point_ inside a test–this is a very useful feature for debugging or just figuring out how to approach writing code! To do this, you can do the following (also shown on the figure below):
0. Pick a test you want to debug. If you’re just getting started and aren’t ready to test any code yet, that’s okay! You can write a test that’s incomplete–just make a test that sets up some variables and then use the REPL to play around!
1. **Add a “pass” statement** at the point you want to stop and use the console. If you’re making a test just to play around, just add it on the last line
2. On the line where you want to stop, click in the region to the left of the line numbers to **set a breakpoint**–you can see this on the figure below
3. In the testing pane, hover over the name of the test and click the button **“Debug Test”** that pops up, like in the figure below
Once you start the debugger, VSCode will open the debugging window, which should look like this.
To use the REPL:
1. Find the debug console window, which should be at the bottom or on the right side of VSCode. If you don’t see it, open it in the VSCode menu by selecting **View > Debug Console**
2. The last line (with the >) is the REPL! To use it, type a line of code here and press enter. Python will evaluate it as if you typed it where the code is paused. You can even access variables and call functions! See the figure for examples. For more guides to use the debugger or other VSCode features, see [these examples from lecture](https://brown-csci0200.github.io/assets/lectures/23python1/Lec25_whiteboard_f22.pdf).
# Examples of common ways to test
The test\_hw4.py starter code has examples of common ways you might test different results. For reference, the contents are here:
```python
import pytest
import copy
import hw4 as h4
example_person_list = [
{"name": "Spongebob", "time in group": 2.3, "skills": ["jellyfishing", "frycooking"]},
{"name": "Sandy", "time in group": 0.2, "skills": ["karate", "science", "breathing"]},
{"name": "Patrick", "time in group": 2.3, "skills": ["eating", "jellyfishing"]}
]
'''
Similar to @Before in junit, pytest runs this before every test method
'''
def setup_function():
# set test_group as global so that other methods can find it
global test_group
# copy.deepcopy makes a brand-new copy of example_person_list
# and every piece of data contained in it on the heap
test_group = h4.Group(copy.deepcopy(example_person_list))
# now we can refer to test_group in any test function in this file
'''
pytest assertions simply check if boolean expressions evaluate to true
Make sure ALL of your testing functions start with test_ and are in a file that starts with test_!!
'''
def test_example():
assert 2 == 2
# use pytest.approx for decimal numbers
assert 2.5 == pytest.approx(5 / 2)
# since test_group was declared global and initialized in setup_function, we can call
# its methods here
assert test_group.measure_progress("Sandy") == pytest.approx(3 / 0.2)
# testing for exceptions:
with pytest.raises(LookupError):
test_group.measure_progress("Plankton")
# assertions on lists
assert "a" in ["a", "b", "c"]
assert "d" not in ["a", "b", "c"]
assert len(["a", "b", "c"]) == 3
false_boolean_expression = False
assert not false_boolean_expression
# value equality (==) vs. same object on the heap (is)
list1 = [1, 2, 3]
list2 = [1, 2, 3]
list3 = list1
assert list1 == list2
assert list2 == list3
assert list1 == list3
assert list1 is list3
assert not (list1 is list2) # list1 and list2 are different objects on the heap
list4 = list(list1) # makes a copy of list1 in a new location on the heap
assert list4 == list1
assert not(list4 is list1)
```
# Troubleshooting
We maintain a list of the most common Python setup issues here.
***
_Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CS200 document by filling out the_ [_anonymous feedback form_](https://docs.google.com/forms/d/e/1FAIpQLSdEK-ttEPd8atXbZBKY8BwUEU4_vXE-5etuVVJDcmIHKXMs7A/viewform)_!_
__