Writing Tests and Working with Wheat/Chaffs

# **Writing Tests and Working with Wheat/Chaffs** ## **Wheat and Chaff Tests** Wheat and chaff testing creates test suites to help separate “good implementations” from “bad, buggy implementations." For our purposes in CS200, the "wheat" represents the solution code that we run against the test-suite that you create. If the wheat passes, this means that your tests accurately reflect the intended functionality of the program (Congrats!) Meanwhile, chaffs represent buggy implementations of HW or projects that we purposely create to run against your test-suite. When running these implementations, your tests should be comprehensive enough to catch any inaccuracies or broken functionalities that these solutions might have. Overall, wheat-chaff errors ==**indicates something to fix with the test suite**==, NOT the implementation code! Take a look at this [CS17 lab](https://cs.brown.edu/courses/cs017/content/labs/02RacketIntro-racket.pdf) for more a more in-depth explanation about wheat & chaffs. ### **Common Questions & Debugging Tips** - **What do I do if my WHEAT does not pass (shows red on Gradescope)?** - When you don’t pass a wheat, it seems like the autograder is not giving you a lot of information, but this does NOT mean your implementation code contains a lot of bugs or even that there is something wrong with it. A wheat fail means your test suite is failing to test for the core intentions of the assignment. For example: - Our spec asks you to write a function that outputs even numbers and your test suite does not test and pass a case where running your function produces even numbers. - In conclusion, _a wheat fail can be fixed by fixing your test suite and **NOT** your implementation code_. - ==**ADVICE:** For debugging wheat failures, try **removing** tests from your code to identify which ones may be causing the failure. Additionally, make sure that your tests are adhering to the formating specified in the assignment (for example, check that you haven't put parameters in the wrong order for a particular method or data structure).== - **What do I do if one or more of my CHAFFS are not passing (shows red on Gradescope)?** - Every chaff is a broken implementation. Tests that catch chaffs should catch these fails. Ideally, your test suite should catch most broken aspects of your code (we never expect you to catch all the chaffs, just most of them because some chaffs reflect really nitty gritty fails that even we can’t all anticipate). - _When you don’t catch certain chaffs, we (the TAs) unfortunately cannot identify which exactly it is, but we can help you think through what makes a robust test suite and what edge cases you might be missing_. - ==**ADVICE:** For debugging chaff failures, try **adding** tests to your code for a more comprehensive test suite. Think about any edge cases that you haven't considered yet!== ## **What makes a good unit test?** **A unit test is a test that targets a specific and singular function or method (the unit).** A good unit test suite allows us as developers to be able to refactor our code or add new functionality without being worried about having added bugs to our code by altering existing code functionality. Good unit tests tend to have the following properties ([reference 1](https://medium.com/chris-nielsen/so-whats-a-good-unit-test-look-like-71f750333ac0), [reference 2](https://leanylabs.com/blog/good-unit-tests/)): - **Independence** – a unit test should have no side effects, i.e. you should be able to run your test suite in any order and all the tests should still pass. - **Descriptive** – the name of the test should describe what it is testing, or there should be documentation near the test describing it. This allows you as a developer to be able to immediately identify which section of your code may be broken without having to actually revisit the test. - **Well-structured** – tests should tend to follow the same three part structure: - _Setup_ objects and other necessary values - _Act_ on those objects and values using the method you want to test with - _Assert_ that the result of your method is exactly what you expect - **Deterministic** – the unit test should either pass 100% of the time, or should fail 100% of the time. Tests that sometimes fail and sometimes pass, can make it harder to track down bugs since they will seem to appear and disappear without you changing anything about the code. Notice that good unit test suites are **implementation agnostic**, i.e. they should **never test or assume** anything about the inner workings of the method it is testing. One way of thinking about this is that the method you are testing is a black-box or oracle that you are trying to determine always tells the truth, or sometimes lies to you! ## **Testing Practice** (Optional) ### Exercise #1 Consider the _Shopping Discounts_ problem from Homework 1a (CS15). An online clothing store applies discounts during checkout. A shopping cart is a list of the items being purchased. Each item has a name (a `String` like `“shoes”`) and a price (a real number like `12.50`). Each item is represented as follows: `public record Item(String name, double cost) { }` The `checkout` method in the `Cart` class adds up the costs of a list of Items and applies discounts as follows: - If the cart contains **at least $100 worth of shoes**, take 20% off the cost of all shoes (match only items whose exact name is “shoes”). - If the cart contains **at least two hats**, take $10 off the total of the cart (match only items whose exact name is “hat”). What would a good unit test suite look like for this? The `checkout` method acts on a list of items, therefore, we can test this method by supplying different input lists that should have different discounts applied. What are some potential `checkout` inputs we could test? :::success <details> <summary>Click here for the answer</summary> - An empty cart (empty list) - A cart without shoes or hats - A cart with one hat - A cart with two hats - A cart with three hats - A cart with shoes totaling less than $100 - A cart with a singular shoes item totaling more than $100 - A cart with multiple shoes each with price less than $100 that sum to a value less than $100 - A cart with multiple shoes each with price less than $100 that sum to a value greater than $100 - A cart that has both two hats and shoes whose total price is greater than $100 </details> ::: You may have more tests or fewer tests depending on how you set up your input carts, however, remember that it tends to be better to have numerous simple unit tests that each test a singular thing rather than few complicated unit tests that try to accomplish many things at once. #### ***Wheat/Chaff Practice*** Looking at the list above, think about potential chaffs (aka bad implementations) this unit-test suite might catch. *For example*, let's consider a chaff solution that applies the hat discount if there are **any hats present**. What possible input/test might catch this? :::success <details> <summary>Click here for the answer</summary> The input "a cart with one hat" would be able to catch this incorrect implementation. If we recall, a discount should only be applied when there are at least two hats in the cart. By checking that a discount is NOT applied when there is only one hat (via a test), we can catch this chaff/bad implementation where it IS applied. </details> ::: :::warning <details> <summary>Caution: Why would "a cart with two hats" not catch this specific chaff? </summary> If we recall the functionality of our shopping car from earlier, a discount SHOULD be applied to a cart that contains two or more hats. Since this chaff applies a discount if there are one or more hats, it would technically still pass a test that checks that a discount is applied for two hats. NOTE: This does NOT mean that you shouldn't include this test, but rather that you should include ADDITIONAL tests (like the one mentioned above) to ensure that chaffs/incorrect implementations are caught! </details> ::: ### Exercise #2 ***Here’s another example to try out for yourself!*** Suppose, you want to implement `quicksort` (from Homework 1b CS15 track), which takes in an unsorted list and sorts it in ascending order. Since you are an avid believer in test-driven development you want to write a quick unit test suite before implementing this method. What are some potential `quicksort` inputs we could create tests from? :::success <details> <summary>Click here for the answer</summary> - Empty lists - Already sorted lists - Lists with duplicate numbers - Lists with negative numbers </details> ::: The cool thing about this example is that if you then wanted to, say, try writing out other sorting algorithms like heapsort, mergesort, or [sleep sort](https://rosettacode.org/wiki/Sorting_algorithms/Sleep_sort), you can re-use this test suite! This is because you **haven’t** created a test suite that tests quicksort, but rather have created a test suite that tests **sorting algorithms in general.** This is because your test suite doesn’t (or at least should not) test internal features about the algorithm (like what element is the pivot in quicksort at a given iteration). ***Wheat/Chaff Practice*** As a part of your unit test-suite for `quicksort`, you incorrectly assume that `quicksort` removes all odd numbers from the output list. You submit your test-suite to Gradescope and recieve an error. Would this be a wheat or chaff error? :::success <details> <summary>Click here for the answer</summary> A wheat error! Here, we are creating a test that makes an incorrect assumption about the functionality of quicksort. In reality, the function does NOT remove odd-numbers. So, if we were to run the wheat (aka correct solution) against this test, it would fail and create an error! </details> :::   _Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CS200 document by filling out our_ [_anonymous feedback form_](https://docs.google.com/forms/d/e/1FAIpQLScCb0F4UsIOLZSr0xqPmdgDDWE4afNHXqnzSiyoK38Ce0C3vg/viewform)_!_