<style> h3 { border-bottom: 1px solid #ccc; } section { margin-bottom: 2rem; padding: 0em 1em; padding-bottom: 1em; border-radius: 4px; background-color: #f7f7f7; border: 1px solid #ccc; } summary { font-weight: bolder; } summary:hover { text-decoration: underline; } .todo { color: #ff00ff; border: 2px dashed #ff00ff; padding: 0em 1em; border-radius: 5px; //display: none; // UNCOMMENT TO HIDE TODOs } </style> # Lab 10: Working more with Python (Fall 2024) ## Learning Objectives 1. Learning what libraries and frameworks are in Python, and how to use them 2. Learning where/how to look for help when programming ## Setup Before beginning this assignment, clone the source code from GitHub to make your personal repository for this lab. Here is the **[GitHub Classroom link](https://classroom.github.com/a/9UiBkGpB)**.   :::warning :warning: **Warning**: **Be sure to read the [**Python Stencil Setup Guide**](https://docs.cs200.io/s/python-stencil-setup-guide)** for important instructions on how to set up VSCode to use the stencil.   In particular, **DO NOT FORGET TO [set up your environment](https://docs.cs200.io/s/python-stencil-setup-guide#Setting-up-your-environment)** per the instructions. This lab involves running several real-world Python libraries, which need to be configured before you can use them--**if you don't set up your environment, these won't work!** ::: **Environment setup**: Make sure your VSCode is set up as per our [setup guide](https://docs.cs200.io/s/vscode-python-setup-guide), and that you can [run the test files from HW0](https://docs.cs200.io/uEOO6M82SSuiX0717Xu5GA) to make sure your setup is working.  If you have issues during the setup process, see [here](https://docs.cs200.io/s/python-stencil-setup-guide#Troubleshooting). ## Intro Throughout the whole semester we have seen the use of different languages in the context of data structures and algorithms. Now, we want you to get some exposure to other ways Python is used within the field of Computer Science. One of the common uses for Python in the ‘real world’ is in Machine Learning. You've likely already got an idea of what "machine learning" might mean, but here's a few concrete definitions for it: - “Machine learning (ML) is a branch of artificial intelligence (AI) and computer science that focuses on the using data and algorithms to enable AI to imitate the way that humans learn, gradually improving its accuracy.” (IBM) - “The use and development of computer systems that are able to learn and adapt without following explicit instructions, by using algorithms and statistical models to analyze and draw inferences from patterns in data.” (Oxford Dictionary) We’ll start this lab with a machine learning example. ## Looking back at Decision Tree Let’s start by going back to your Decision Tree project. During this project you coded an algorithm that used a tree-like data structure to arrive at its predictions in new situations after having trained it on some attribute data. <section> ### Task 1 Turn to your partner and think of two limitations of the decision tree algorithm you implemented. </section> At the time of the project, a lot of you expressed concerns on the accuracy of your algorithm. What if we told you that from what you already know about Decision Tree and how it works, you can build a more accurate model and even scale it to use bigger datasets... We'll be exploring this idea for the first part of the lab. <section> ### Task 2 Read through the code in `mnist_example.py`. :::info **Disclaimer!** There’s code we’ve never seen before: don’t panic, you already have the tools to reason about it. Later on we'll show you where this code comes from. ::: </section> ## Diving deeper into the code <section> ### Task 3 Run the `data_unpack.py` file to load the MNIST datasets. </section> Before you run the code, let's start by breaking down exactly what's happening in the `mnist_example.py` file. #### Importing Dependencies In the beginning of the code, we import dependencies like `scikit-learn` (ML Library), `matplotlib` (Visualizer), `idx2numpy`(Converts IDX format to `numpy.ndarray`). These are external libraries or modules that provide additional functionality beyond what’s available in the core Python language. By giving us access to pre-built data structures and functions, dependencies simplifies our in-file coding and allows us to work with more specialized tasks, like the number classifier we are trying to implement. **More on this later**. #### Importing the Dataset <!-- Fill out --> - The [MNIST](https://en.wikipedia.org/wiki/MNIST_database) (Modified National Institute of Standards and Technology) dataset is a widely-used benchmark dataset in the field of machine learning. - Includes a list of 70,000 28x28 pixels gray-scsale **images** of handwritten digits (0 ~ 9) and a list of corresponding integer **labels** - Think of the image as input *x*, the model as the function *f*, and the label as the desired *f(x)* - 60,000 image-label pairs are in the **Training Set** for model training , and 10,000 image-label pairs are reserved for the **Testing Set** - The independent testing set is important to evaluate the model's performance after training is finished with the training set <section> ### Task 4 Run the `mnist_example.py` file, you will see the first 5 pictures in the training dataset appear. </section> #### Random Forest Classifier :::info **Tying it back to Decision Tree** Our original Decision Trees can be heavily impacted by a single point of data. What happens if we just had an outlier? Well one way to solve this is to create multiple decision trees, on different subsets of the data. Some will have the outlier, some won't. This lets us average each of their responses together, to get a more robust and stable prediction, reducing the influence of outliers. This is what the Random Forrest Classifier tackles. ::: - A random forest is a meta estimator that fits a number of decision tree (!!!!) classifiers on many sub-samples of the dataset. - It uses averaging to improve the accuracy and control over-fitting (aka. when the data becomes too specific to the training set and performs poorly on new, unseen data). **A random forest is a collection of decision trees that are trained on different subsets of the data.**  :::info **Note**: Don't overthink about the model's arithmetic details. You'll see later that understanding the bigger picture is more important than knowing the nitty-gritty! ::: #### Using the model - We create an instance of the `RandomForrestClassifier()` model: - `model = RandomForestClassifier(n_estimators=50, random_state=42)` - We feed the training data into the model and train it to identify and classify our images. - `model.fit(mnist_train_img.reshape(-1, 28*28), mnist_train_labels)` - `predictions = model.predict(mnist_test_img.reshape(-1, 28*28))` - We evaluate the accuracy of our model, meaning how well it classifies our images. We do so by running it on our `test_image `dataset, which will output the number (label) it predicts for each image, and compare the output to our `test_label` dataset, which contains the *actual* number each image represents. - In ML we refer to this as **supervised learning**. In this approach, we already know the expected output of the model and can compare it to the actual output. This comparison helps the model understand where it's making mistakes and how to improve its accuracy. - `accuracy = accuracy_score(mnist_test_labels, predictions)` <section> ### Task 5: - Uncomment line 35 onwards as indicated by the ***#TODO***. - Feel free to comment out the code we are using to visualize the images (lines 19-29) so they don't print each time you run the file. </section> :::success **Checkpoint!** Call over a TA, have them check your answers and code. ::: ### Dependencies: Packages, Libraries and Frameworks In Python, we rely on **packages, libraries, and frameworks** to do more with less effort. **Packages** are like organized folders of code bits that make it easier to manage and share our work. **Libraries** are pre-made sets of code that add special features to Python, either built-in or made by others in the Python community. **Frameworks** are big collections of these code bits that give us a roadmap for building different types of programs. Take a look at the `requirements.txt` file in the stencil. This `.venv` virtual environment we made you set up downloads and installs all of the libraries and packages mentioned in the file, which allows you to then import them at the top of `mnist_example.py` to actually be able to use them. It's not just the libraries we import, but the libraries that *those libraries* import (which we call "dependencies"), and so on all the way down. :::info **.venv** A virtual environment is simply a way to keep track of separate collections of libraries for each project we might write. It lets us use different libraries, or even different versions of the same library, and manages the relationships and dependencies automatically for us. ::: We'll explore this further in the context of the Random Forest Classifier algorithm we provided. We have this snippet of code: `model = RandomForestClassifier(n_estimators=50, random_state=42)` `RandomForestClassifier()` is a method that exists within the scikit-library, and that we imported earlier (look at lines 2 and 3). This class has a bunch of code under the hood, which we can't immediately see just by looking at the usage. In fact, as long as we know what inputs it takes and outputs it produces, **we don’t really *need* to know how the method works to be able to use it**. Later in this lab we'll be taking a look "under the hood" at the source code for this class, but for now let's practice using a class we can't inspect. ### Getting to work #### Hyperparameters In machine learning hyperparameters refer to parameters whose values are set before the learning process begins. You (the programmer) can choose their values before running the code. It controls the learning process itself and affects the behavior and performance of the model. In the case of `RandomForestClassifier()`, `n_estimators` is one such hyperparameter. You don't need to fully understand this to observe how changing the value changes the model. Here's the [`RandomForestClassifier()` documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier ). The descriptions that we presented in the [the explanation of the algorithm](https://docs.cs200.io/1ly3qPpzRJmRIA1riR4wLQ?view#Random-Forest-Classifier) were quoted (in part) from this same documentation. <section> ### Task 6: Explore the documentation above and experiment with 3 to 4 different values of `n_estimators` within the 1 to 500 range. Based on this information, in your own words what do you think `n_estimators` is doing? :::info You'll have to run the code to see how different values for `n_estimators` affect the model. **Be patient as the model might take some time to run!** ::: </section> :::info <details> <summary> Answer </summary> Think about the model's performance. How does this affect the time it takes to run? How does it affect the accuracy? </details> ::: #### A deeper change It's pretty easy to change inputs and get different results, even for methods we might not otherwise understand. But what if we want to change something deeper about how this code works? Scikit-learn is an open source machine learning library. This means anyone on the web has access to its source code, which consists of the actual snippets of code used to construct the library and its methods. This is [scikit-learn’s source code for the forest models]( https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/ensemble/\_forest.py). The `RandomForestClassifier()` class is defined between lines 1170 and 1552. Out of those 382 lines, 300 are dedicated to docstrings describing the class's structure, the purpose of each input parameter and how to run it. **This is why we've been stressing the importance of having clear documentation and well-organized code for your assignments. It's super important when dealing with large codebases like this!** The file is already so large, just imagine opening it up and trying to find anything without any documentation or proper formatting/code organization. It would be quite the struggle. <section> ### Task 7 Working with your partner, try to identify which place(s) in the code you would need to modify to implement the following changes. **1.** Change the default for n_estimators to 500. **2.** Change the partitioning strategy to allow data overlaps between trees in the random forest. Take some time to look through the source code. :::info <details> <summary> If you are confused after reading point 2 and looking at the source code... </summary> Move on to the "Main Takeaways" below. </details> ::: </section> :::info <details> <summary> Main Takeaways </summary> You don't need to understand what goes under the hood to be able to use scikit-learn! But… Scikit-learn’s core functionality is implemented in Python. To actually be able to complete the second point you would have to navigate out of `forest.py` and into `base.py`. Libraries build up from other files within the same library, and they also wrap other low-level libraries. In the case of scikit-learn, it uses like NumPy and SciPy. **See! Even library developers don't want to recreate functionality for themselves, they use other people's libraries too.** So, with the knowledge from this class, and some more practice looking at codebases like this source code (which can sometimes be overwhelming), you have all the tools you need to understand what is going behind the scenes of a library like scikit-learn. **Frameworks and libraries make things easier and faster, and can sometimes even make things possible with less necessary background knowledge.** </details> ::: :::success **Checkpoint!** Call over a TA to check your answers. ::: ### What now? #### So which library do I use? There’s new libraries and frameworks constantly coming out, while the already existing ones are always getting updated. It's impossible to know how every single one out there is built. In turn, one of the most important skills as a programmer is to ask the right questions in order to get the results you are looking for within a sea of information and misinformation, trustworthy and untrustworthy sources (both malicious and accidental). So it becomes not just a question of "what is available," but "what works correctly for my use case". So far the policies of this course have restricted your use of online resources to solve certain kinds of problems, to encourage you to learn certain concepts on your own. As you go further in your computing journey, whatever that may look like, you'll find that programmers often don't limit themselves in this way. Most programmers will often use online tools, from search engines (e.g., DuckDuckGo) to online forums (e.g., Stack Overflow). More recently, programmers have started integrating AI tools (e.g., ChatGPT, Copilot) into their workflow. Any of these tools can be helpful in navigating the vast amount of available libraries and frameworks available. While these tools can provide a helpful source for getting started or getting unstuck, none of them are perfect. The internet is a sea of information, not all of it accurate. There are trustworthy and untrustworthy sources, and any of these tools might (maliciously or accidentally) lead you down the wrong track. It is important as a programmer to learn how to both ask the right questions of these tools, and how to judge whether the answer from the tool is valid for your use case.. :::info **Scenario:** Imagine you're in your next CS course at Brown, and your final project involves building a machine learning model to classify images of animals into mammals, birds, fish, reptiles or amphibians. Your task is selecting the appropriate Python library or framework to implement your project efficiently and effectively. However, with so many options available, you're not sure which one to choose. You want to make an informed decision but are unsure where to start. ::: <section> ### Task 8 - Read [this article](https://codeeasy.io/blog/how-to-effectively-google-as-a-software-devel) on how to effectively search for answers as a developer. - Write 3 questions you would use to search for answers in the specific scenario we presented above. </section> <section> ### Task 9 Now, look up those 3 questions in a few different locations! What can you notice from the answers? </section> <!-- <section> ### Task 10 Let's assume ChatGPT indicated you should explore the framework called TensorFlow. What follow up questions would you make to find what model is the best? </section> --> :::success **Checkpoint!** Call over a TA to go over your answers. ::: ### Some Closing Thoughts In reality, Python relies heavily on its ecosystem of libraries and frameworks. Without them, Python's capabilities can sometimes feel quite limited. We discovered when diving into one of these libraries that it's essentially a collection of various code snippets put together, using knowledge of data structures. Looking forward, with the knowledge from class and some more time, you *could* build your own library! However, with libraries and frameworks already available, you can leverage the expertise of other programmers who have spent years refining and improving their codebases without the need to reinvent the wheel. Each library comes with its own unique design decisions made by their programmers, allowing you to avoid making decisions in areas where you lack expertise. You can use these tools where most of the heavy lifting is already done, and simply adjust the final details by tweaking input values, like we did with the hyperparameters in the random forest classifier (using the `n_estimators` field). You don't need to understand every little detail under the hood to use these tools effectively; you just need to know how to use them, and how to get answers when you have questions about them. If you are interested in researching other libraries for other computer science related subjects, you can start by these that ChatGPT suggested: 1. **Data Science and Analysis**: `pandas` 2. **Machine Learning and AI**: `scikit-learn` 3. **Web Development**: `Django` or `Flask` 4. **Natural Language Processing (NLP)**: `NLTK` or `spaCy` 5. **Web Scraping**: `Beautiful Soup` or `Scrapy` :::success Congratulations, you’ve completed the final lab! Good luck with the rest of the semester :) - *The CS0200 Staff* ::: *** _Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CSCI0200 document by filling out the_ [_anonymous feedback form_](https://docs.google.com/forms/d/1KqjKrDqaHAdjpmeXuQkOkeFKjuurzWHfc3HyJbXVxIs/edit)_._