1114 views
owned this note
<style>
h3 {
border-bottom: 1px solid #ccc;
}
section {
margin-bottom: 2rem;
padding: 0em 1em;
padding-bottom: 1em;
border-radius: 4px;
background-color: #f7f7f7;
border: 1px solid #ccc;
}
summary {
font-weight: bolder;
}
summary:hover {
text-decoration: underline;
}
.todo {
color: #ff00ff;
border: 2px dashed #ff00ff;
padding: 0em 1em;
border-radius: 5px;
//display: none; // UNCOMMENT TO HIDE TODOs
}
</style>
# Lab 8: Finding and Formatting Data: LLMs and String Diffs (Spring 2025)
Many scripting and programming tasks involve finding data that are formatted in specific ways, extracting pieces from formatted data, or converting data into specific formats as needed by other parts of a program. In this two-part lab, you will build a simple chat-based flight finder in which multiple formatting tasks arise. Along the way, you'll use three tools that can help work with formatted (or formatting) data: json, differencing libraries, and LLMs.
***Heads up:** We're going to use the two-part lab to help you get a realistic software experience. At the end of this week, you will zip up your work and turn it in. When you come back next week, you'll be given the code of another pair and be asked to build on that other code to finish creating the flight finder.*
<section>
## ‼️ Attendance ‼️
In order to get marked off for attendance for this lab, fill out [this extremely brief and anonymous feedback form](https://docs.google.com/forms/d/e/1FAIpQLSdPdSjfs_e1JGmhC32A_YmC7fhFwmsAN4HR5tRqE0PjK-8PSQ/viewform?usp=preview) (**as well as being present and completing other tasks**). The form is anonymous. Once you complete it, it will bring you to a page with another link that records your email so we can check off that you completed it.
</section>
# Learning Objectives
In this week's lab, you will learn to:
- Develop a project start to finish without starter code
- Write an application that calls out to an LLM to turn free-form user text into formatted data
- Use a string-difference library to find the closest match for a string among a set of choices
- Store data through JSON
# Setup
Use this **[GitHub Classroom link](https://classroom.github.com/a/j8dUhQS_)** to get the stencil code for this assignment. For instructions on how to clone the Stencil and set up VSCode to use it, see the **[Python Stencil Setup Guide](https://docs.cs200.io/s/python-stencil-setup-guide)**.
*This week's stencil has only a library file (for working with an LLM through the Brown CS network) and two sample CSV files. There is no starter code file. You are going to create your application from a blank file.*
:::info
Rainbow CSV is not a necessary extension for this assigment, although very helpful
:::
**Environment setup**: Make sure your VSCode is set up as per our [setup guide](https://docs.cs200.io/s/vscode-python-setup-guide), and that you can run the test files from HW0 to make sure your setup is working. If you have issues during the setup process, see [here](https://docs.cs200.io/s/python-stencil-setup-guide#Troubleshooting).
**Accessing the Brown CS VPN:** We'll be using an LLM that is installed on the CS department network. To access it, you'll need to use something called a VPN (virtual private network), a mechanism that lets you access the CS network securely from your own computer.
The [instructions on this page](https://cs.brown.edu/about/system/connecting/openvpn/) provide the instructions to (a) install an openVPN client (bottom of the page -- look for the link for your operating system), and (b) install the configuration specific to Brown CS (which will be customized for you based on your Brown login credentials).
If you did not install the VPN prior to lab, do that now.
:::danger
If you're in a lab of 15+ students, please only have ONE person per group connect to the VPN. Too many connections in a room will bottleneck that room's bandwidth!!!
:::
:::warning
Please connect to the `user_browncs`configuration, NOT the 2nd `user_browncs_gateall` (extra workload by routing all traffic throug the VPN)
:::
:::info
You're all set if you can see http://perfmon.cs.brown.edu/, even if public IP doesn't change.
:::
# First, the big picture
Here's a sample interaction of the flight chat system that we're going to build this week (just using basic input/output commands in terminal). The user indicates which city they want to leave from and when they want to go, and the system suggests a possible travel destination.
```
Where are you leaving from? bostn
When do you want to leave? mar 15
Searching for flights on 15/03/2025 from Boston
How about Zinnia Air flight 726 to New York? ok
Great! Let's go on to booking!
```
The system makes use of two CSV files: one containing information on flights, and one with information on the names of airlines and their two-letter abbreviations. For example:
```
# the flights
From,To,Date,Flight,Seats
Seattle,Houston,19/04/2025,PY348,18
Seattle,San Francisco,29/05/2025,ZU581,5
Boston,Seattle,29/05/2025,ZU111,15
# the airline abbreviations
abbrev,full
ZU,Zinnia Air
PY,Pyair
GS,Gamma Skies
```
**Task:** Based on the interactions, what sort of tasks will your program need to perform? Write down a list.
We'll now proceed to build the application in stages.
# Formatting Dates using LLMs
The sample interaction shows that the user entered a date of `mar 15`, and the program converted that to `15/03/2025` (dd/mm/yyyy format). Real users enter data in all sorts of ways (which is why many websites have you select dates on a calendar). For example, users here could write any of `march 15`, `15th of March`, `3/15/25`, and more. It would be quite tedious to have you write programs to handle all of these possible date formats.
This is exactly the sort of task that tools like ChatGPT are designed to handle well! AI-based chat tools are built on billions of samples of human text (including how we express dates). The tools depend on large summaries of patterns in our use of language (called "Large Language Models", aka LLM). Rather than us writing logical functions to turn user's dates into a common format, we get much more flexibility and power from asking an LLM to do the conversion based on its knowledge of language.
To get started, we'll prompt the user for a date and send it to an LLM for conversion into dd/mm/yyyy format. To keep things simpler, we'll assume that all years are meant to be 2025.
<section>
### Task 0
If you haven't already, create a new python file called `flight_chat.py` to hold this week's lab work. Place this file in the same directory as the stencil files that you downloaded.
Make sure you're setting up your [**virtual environment**](https://docs.cs200.io/s/python-stencil-setup-guide#Setting-up-your-environment) properly using `requirements.txt`
</section>
## I/O (input/output) in Python
You can prompt a user for input as shown in the following example:
```=python
user_name = input("What is your name? ")
print("Hello, " + user_name + "!")
```
Running this code will yield:
```
What is your name? Blueno
Hello, Blueno!
```
<section>
### Task 1
In the `flight_chat.py`, add code to ask the user to enter a date. For now, print that date back out to the user (to make sure you can read and print out user-entered data).
</section>
## Talking to an LLM
Many of you have likely used an LLM by now (like ATA or ChatGPT): you go to a webpage, enter a question, and get back an answer. To write our flight chat program, however, we can't be typing a query into a web page. We need a way to send a prompt to an LLM from *within a program*.
The file `llm_service.py` in the stencil provides a function called `ask_llm` that takes the prompt question as a string, sends the prompt to an LLM, then returns the response as a string. Here's an example of how to use it:
```=python
from llm_service import ask_llm
print(ask_llm("Do you prefer Python or Java?"))
```
<section>
### Task 2
Update your program so that it takes the date from the user, asks the LLM to convert it into mm/dd/yyyy format, and prints out the formatted date. Assume 2025 if the year is not provided.
The challenge for this task lies in figuring out how to construct the prompt so that you get the formatted answer that you are looking for. Experimenting with this is the point of this part of the assignment, so don't worry if it takes a few tries. You'll learn more from some early failures than if you get it right off the bat.
</section>
<section>
### Task 3
Figure out how to test the LLM portion so far, and write tests in `test_flight_chat.py` using `pytest`. It's fine if you have some tests that don't pass yet, though you should (a) write tests with a variety of possible user date formats and (b) adjust your prompt so that at least a couple of different formats are working.
</section>
:::success
**Checkpoint!** Call over a TA to check your work.
:::
# JSON: Communicating Structured Data
An LLM prompt returns a string. Dates, however, have structure (the month, day, and year). Before working with the date in the rest of the program, we should convert the string we received into a dictionary with separate keys for the month, day, and year.
LLMs can't return dictionaries. But the need to pass dictionaries between programs is so common that there is a standard string-based format for representing them. That format is called JSON.
:::info
**What is JSON?** JSON stands for [JavaScript Object Notation](https://json.org/example.html) and is a standardized format for representing data as lists, dictionaries, and values. Conveniently, the syntax looks very much like Python syntax. Here's an example:
:::
```=python
# Note that this is a STRING (see the ` ... ` marks)
'[
{
"name": "Sebastian",
"score": 2.3,
"skills": ["sleeping", "screeching"]
},
{
"name": "Claire",
"score": 0.2,
"skills": ["coding", "science", "breathing"]
},
{
"name": "Habram",
"score": 2.3,
"skills": ["eating", "sleeping"]
}
]'
```
<section>
### Task 4
Adjust your LLM prompt so that it returns the date as a JSON string for a dictionary with month, day, and year keys.
</section>
<section>
### Task 5
Run your program start to finish so far: it should prompt the user for a date string, then extract and print the month from the resulting JSON data.
</section>
What happened? JSON is a string-based *representation* of a dictionary, but it is still a string. To actually use the result as a dictionary, we have to build the dictionary object from the string. This is a common operation for JSON, so there is a built-in way to do this.
```=python
import json
result_dict = json.loads(response_string)
```
Now you should be able to extract the month from the LLM response.
# Recap
So far, we have seen two ideas about formatting:
- LLMs can be powerful tools for helping to format (or reformat) data, especially when there aren't clean rules that we could capture in code
- JSON is a useful tool for communicating data structures between separate programs (which can only communicate through strings). JSON defines rules to format certain data structures as strings; JSON libraries (which exist in most languages today) can read and write data structures from and to JSON-formatted strings.
Of course, there is more formatting work to do to finish building our mystery flight chat service!
# Identifying cities
Returning to the sample interaction at the start of the handout, notice that the user entered `bostn`, which the program correctly detected as `Boston`. This isn't a question of formatting. Rather, we have a set of inputs that the program could handle, and we have to figure out whether the user entered something that was close to one of those inputs.
<section>
### Task 6
Try using an LLM (a standalone one, not calling out to one in your program) to get to a known city from a possible user typo. The goal here is to explore what sort of prompt you would need to do this accurately.
</section>
<section>
### Task 7
Think again about the program you are writing. Is calling out to the LLM a good design decision to handle this task? Why or why not? What questions should you ask yourself to think this through? Write them down in python comments where you tried using LLM.
</section>
:::success
**Checkpoint!** Call over a TA to check your work.
:::
## Matching similar text: string differences
If we look at `bostn` vs `Boston`, we observe that the two strings are close: other than capitalization, there's just a single letter missing. There's a classic programming problem called *string differencing*, in which one computes how many character-level changes (add, remove, flip) would be needed to turn one string into another.
If we had a list of the city names that we wanted to support, we could use a string difference library to determine whether the user entered something close to a city that we know. Fortunately, string difference libraries are common in most programming languages. Here's an example of the python library for this:
```python
import difflib
user_dest = 'bostn'
city_names = ['New York', 'Boston', "Boise", "Ontario"]
closest_match = difflib.get_close_matches(user_dest, city_names, n=1, cutoff=.5)
if closest_match:
return closest_match[0]
else:
print("No Match Found")
```
<section>
### Task 8
Use a standalone LLM (ChatGPT/DeepSeek) to understand this code. Then, augment your program to prompt the user for a city and convert it to an expected city (the collection of expected cities is in the `flights200.csv` file in the stencil).
:::info
Hint: Ask an LLM how to use the `pandas` library to extract all cities
:::
Print a message of the following form to the user showing both of your converted inputs (the date and the city).
```
Searching for flights on 15/03/2025 from Boston
```
</section>
<section>
### Discuss
Real travel typically give you a drop-down list so you can choose a known city from the outset. Would that have been a better choice here? Why or why not?
</section>
# Clean Up Time!
Many of us prototype applications by getting down code that works. We don't put on tests, documentation, or even create good separation of code into multiple functions. It's good practice to stop periodically and clean up what you had as if you were going to show it to your boss (say for a code review).
</section>
<section>
### Task 9
Clean up your code. Consider questions like whether it is structured well for testing, follows model-view-controller, and is easy for someone else to read.
Eg: `closest_match` using `difflib` from the cities should all be in a helper method, rather than the main general body
Also make sure that it would continue to work with a different csv files of flights (you can assume that the different file would have the same column headers).
Leave a comment at the top of the file with instructions on how to run your code so far.
</section>
Whichever lab pair gets your assignment during next week's lab will be asked to comment on their experience working with the code that you left them. This is also really good practice for working in a software development team (as you would on an internship or a job), so don't just brush it off because we're in lab.
<section>
### Task 10
Now, you're going to upload your work from this week to Peerceptiv (which will help us shuffle the assignments among you all when we come back for lab next week). **Only one member should submit per group**.
Peerceptiv doesn't allow you to upload Python files, so you have two options:
- rename your file to `flight-chat.txt` and upload that
- zip up the folder with your `flight-chat.py` and upload the zip file
If you created any additional files (tests, additional csv files, etc), you'll want to create the zip file (since that lets you upload multiple files)
[Here's a link to the Peerceptiv assignment](https://app.peerceptiv.com/course/60daf3a6-a765-4c09-aa06-f344588f04c5/assignments) (or go through Canvas and look for a Peerceptiv assignment named Lab 8)
</section>
:::success
**Final Checkpoint:** Congratulations, you’ve completed this week’s lab! Check in with a TA, and then you can go home. Nice job!
:::
***
_Please let us know if you find any mistakes, inconsistencies, or confusing language in this or any other CSCI0200 document by filling out the_ [_anonymous feedback form_](https://docs.google.com/forms/d/1KqjKrDqaHAdjpmeXuQkOkeFKjuurzWHfc3HyJbXVxIs/edit)_._