Project 2: Predicting Credit Default

The second of the course's three projects. Python-based. Worth 10% of the course grade. Due Wednesday, July 15, 12:00 noon MT (also shown on the Canvas assignment).

1. Scenario: You Are the Analytics Lead on the AR Team

You work in the accounting group at a consumer lender. Your accounts-receivable team is overwhelmed: they cannot personally review all 29,881 customer accounts to work out who is about to default, so they cannot focus their collection effort where it matters. Extend credit to everyone and you absorb every default; deny everyone and you have no business. They need a third option: a model that flags the accounts most likely to go bad, so the team acts on the risky ones and leaves the rest alone.

Your manager asks you to build that model and bring a recommendation to Accounting leadership, who will decide whether to adopt it and then carry your case to the collections and financial planning & analysis (FP&A) teams. They do not want code. They want to know three things: does the model work, where is the risk outsized relative to the reward (that is, where do we draw the line between "flag" and "leave alone"), and what does it cost us to be wrong. Your job is to answer that and present it clearly enough that leadership can forward your deck without you in the room.

Concretely, your deck should answer:

1. Can we predict which accounts are likely to default, and how well?
2. Where should we set the cutoff: how aggressive should we be about flagging, given that a missed default and a wrongly flagged good customer do not cost the same?
3. Should we adopt this, and how should the team use it?

1.1. Learning Objectives

By completing this project, you will:

• Build and read a logistic regression model on realistic accounting data
• Turn model output into a business decision: where to set a risk cutoff, and what it costs to be wrong
• Use AI as an active editor, not a passive reader: challenge its work, verify it, and stay accountable for every number you present
• Present a data-driven recommendation to non-technical leadership

1.2. Deliverables

You submit three things on the single Project 2 assignment on Canvas: two files and one pasted prompt.

1. Your notebook (.ipynb). A complete Colab notebook that loads the data, fits the model, and produces the evaluation, with every output visible. Before you download it, run every cell top to bottom so the results are saved in the file (File → Download → Download .ipynb), same as Labs 7 and 8. The analysis is graded from this file, and its final cells hold your Challenge & Verify log (Section 4.3).
2. Your executive slide deck (.pdf only; export from PowerPoint or Google Slides via File → Export/Save as PDF). The submission form accepts only PDF for the deck. Your pitch to Accounting leadership, 7 to 9 slides plus an optional appendix (the title slide counts; the appendix does not). This is the main graded deliverable.
3. Your challenge prompt(s), the AI's response, and your follow-ups, pasted into the text box on the submission form: the exact prompt text you ran against your own work (Section 4.3), the full text of the AI's response, and any follow-up questions you generated. As text, not screenshots.

Tone for the deck: accounting and finance leadership, minimal jargon, clear and concise. Assume it will be forwarded to people who were not in the room.

1.3. Suggested Milestones

2. Grading

This project grades the judgment around the model, not your ability to type model code. You are encouraged to use AI to generate the regression, or you can code it yourself. What separates an analyst from an agentic workflow is everything around it: framing the decision, building confidence around a deliverable, choosing where to draw the cutoff and defending it in dollars, and reading the result back in business terms. If you cannot understand what the model is producing, you are not operating as an analyst; you are operating as an agent orchestrator. If a model gets something wrong, the person who wrote the prompt is still accountable.

The bar is the same as Project 1. Solid, by-the-book work scores well: a clear deck built on a reasonable analysis lands around 80%, even if a number or two is off. The top of each pool is reserved for the move that shows real judgment: a cutoff recommendation that genuinely reckons with the cost trade-off, and a pressure-test that caught something worth catching. You do not need to be a statistics expert or a lawyer to do well.

The full 100-point rubric is in Section 5 at the bottom of this page, sorted the way you will build: the notebook items first, then the deck.

3. Data

• File: ar_default_data.csv (click to download from the course GitHub repo). 29,881 customer account records.
• Prefer SQL? The same data is also on GitHub as a SQLite database, ar_default.sqlite, one table named accounts.
• Target: bad_ar: did the customer default next month (1 = defaulted, 0 = paid). About 22% of accounts default, so this is an imbalanced problem: a model that predicts "everyone pays" is already 78% accurate and completely useless. That is why you will judge the model on precision, recall, and cost, not on raw accuracy.

3.1. Data Dictionary

"Normalized" means credit_limit and outstanding_balance have been rescaled and are not dollar amounts, so do not expect them to look like money. Compare them relative to each other (a 2 is twice a 1); the model does not care about the units.

3.2. Getting and Loading the Data

In your notebook, load the file straight from the URL below; this is the authoritative address (the Canvas assignment links the same file):

import pandas as pd

url = "https://raw.githubusercontent.com/sean-mccaman/acctg5150-090/main/2026-summer/project-2/ar_default_data.csv"
df = pd.read_csv(url)
df.head()

Two shortcuts, both optional:

• Open the starter notebook in Colab and start in one click. It has the load cell already written, a section stub for each step of the workflow, and the Challenge & Verify log template at the end. Save a copy to your Drive (File → Save a copy in Drive) and work there.
• The in-browser data explorer: run SQL against the data right in your browser, with starter queries for the default rate and how it varies by feature and by demographic group. Nothing to install. First load takes a few seconds while the engine starts; after that it is cached.

4. Suggested Workflow

This is one path through the project. The slide order in 4.2 is flexible; the steps below are the analysis your deck has to be able to stand on. Use AI for the code if you like (most will); it is the same specify-fit-interpret rhythm as Lab 8. Your job is to direct the work, understand what it produced, and make the judgment calls a model cannot.

4.1. Step-by-Step

1. Load and inspect. Read the file, look at the columns, check the overall default rate and how it splits across a couple of features. Make sure you understand what one row represents.
2. Fit the model. Build a logistic regression model predicting bad_ar from the six Feature columns in the data dictionary (credit_limit, outstanding_balance, pay_this_month, no_activity_this_month, last_payment_portion, num_recent_payments). Leave the demographic columns out unless your Challenge & Verify log makes the case for them. If the Lab 8 feature-selection move showed you two predictors that carry the same information, dropping one is fine; say why. Have the AI write the fit, or write it yourself, then read the output: which features are significant (the model is confident they matter; the AI can point these out and explain each one in plain terms), and does the direction of each make business sense (being months behind should raise default risk). Your job is the sanity check; if a factor makes no business sense to you, ask before you put it in your deck.
3. Evaluate at a starting cutoff. The model gives each account a predicted probability of default; the cutoff (also called the threshold) is the line above which you flag the account. Start at 0.5 and build the confusion matrix: the four-way count of how the flags landed (defaults you caught, defaults you missed, good customers you wrongly flagged, and good customers you correctly left alone). From it, compute precision (of the accounts we flagged, how many actually defaulted) and recall (of the accounts that actually defaulted, how many we caught). Week 9 teaches all three. The mechanics are not the graded part, so here is the shape of it (the AI can produce the same):

   from sklearn.metrics import confusion_matrix, precision_score, recall_score
   
   pred = (p_default >= cutoff).astype(int)     # 1 = flag the account
   cm = confusion_matrix(y_actual, pred)        # rows: actual, columns: predicted
   precision = precision_score(y_actual, pred)
   recall = recall_score(y_actual, pred)

4. Sweep the cutoff. The 0.5 default is almost never the right line for an imbalanced problem. Try a range of cutoffs and watch precision and recall trade off: lower the cutoff and you catch more defaults but raise more false alarms. (An ROC curve, a standard chart of catch-rate against false-alarm-rate across all cutoffs, is a nice way to show the model's overall power. Week 9 introduces it; it is optional here.)
5. Make it a business decision (the heart of the project). A missed default and a wrongly flagged good customer do not cost the same. State your assumption about those two costs in dollars. Then, for each candidate cutoff, compute the total expected cost: (defaults missed × your cost of a miss) + (good customers flagged × your cost of a false alarm). Recommend the cutoff with the lowest total. A reasonable starting point, if you want one: cost a missed default at the average outstanding balance of the accounts that actually defaulted, and cost a false alarm as a collection call plus some strain on a good customer relationship, a number you estimate and defend. The AI can build this table for you. The judgment you are graded on is choosing defensible costs and standing behind the recommendation they produce.
6. Pressure-test your work. Run your challenge prompts (Section 4.3) against your own analysis, and log what came back and what you changed.
7. Build the deck. Assemble the pitch to Accounting leadership (Section 4.2).

4.2. Building the Deck (slide outline)

7 to 9 slides plus an optional appendix. The arc below lands at 7; splitting a dense slide into two is fine, just stay at 9 or fewer before the appendix. The order is flexible, but a complete deck covers all seven beats:

• Slide 1 (Title). Project title, your name, date.
• Slide 2 (The problem). The AR team cannot review 29,881 accounts by hand. Frame the decision: should we use a model to flag the risky ones, and where do we draw the line.
• Slide 3 (Data & approach). What the model looks at, in your own words. This is where you show you understand the analysis, not just that you ran it.
• Slide 4 (What predicts default, and how well). The factors that matter most, read in plain terms, and a plain-language sense of how well the model tells good accounts from bad.
• Slide 5 (The cutoff decision). The cost trade-off, your recommended cutoff, and what it means concretely: at this line we catch about X% of defaults and flag about Y good customers. Name the cost of being wrong.
• Slide 6 (Recommendation). Adopt it or not, and how the team should use it. One sentence a non-technical manager could repeat and forward.
• Slide 7 (Risks & what to watch). Where the model could be wrong, or what you would want before fully trusting it. If a challenge prompt surfaced something worth flagging (for example a fair-lending consideration), this is a natural home for it.
• [Optional] Appendix. Supporting charts and detail for anyone who wants to dig in.

4.3. The Challenge & Verify Log

This is where you show the skill the course is really about: directing an AI and then checking it, rather than trusting it. As you start to challenge the work of AI, you go from being a passive reader to an active editor. That is why the exchange itself is part of what you submit: the submission form has a text box where you paste the exact challenge prompt(s) you used, the full text of the AI's response, and any follow-up questions you generated.

Pick apart your own work. Run at least two prompts against your analysis (a third is welcome, not required; two well-developed entries can earn full marks). A prompt can challenge what you did ("what mistake might I be making?") or probe for what you are missing ("what regulatory risk could this create?"); either kind counts toward the two. Record each one in a short log in the final markdown cells of your notebook, and paste the prompt, the AI's full response, and your follow-ups into the submission form.

For each prompt, record four things in the log:

Then one closing line: the single most useful thing a prompt surfaced, and how you decided whether to trust it.

Smaller, targeted prompts also count. Two worked examples, one of each kind:

What earns the points: genuine engagement with what came back. A strong entry can end three ways: you changed something, you defended not changing it (disagreeing with the model on good grounds is exactly the skill), or your understanding of your own work got deeper, and you can say how. Any of those earns full marks. The only losing move is an entry with nothing behind it: a softball prompt that could not have hurt you, or "the AI said it looks good" with no follow-up. If a prompt surfaces nothing, ask a sharper one.

4.4. Using AI Honestly

This course is built around AI, and you are expected to use it here. The AI writes the modeling code, while you direct it, verify it, and interpret the results. Like the analyst in the private sector, your grade rides on the reasonableness of your cutoff decision, the accuracy of your business read, and the ability of your model to withstand a pressure test. While an AI can do it all, if Claude is putting its name on your work, you are not the person responsible for the creation of business value.

Your Challenge & Verify log doubles as your record of how you used AI, so keep it honest and specific. This project is completable at no cost with the University-provided AI access (linked on Canvas) and free Google Colab; if anything about access is a barrier, reach out on Canvas and it will be sorted out.

One more thing, so the point of all this is plain. I am not interested in whether the frontier models can do this work. I know that with the right prompts and tool calls, they can do this and so much more. I care entirely about your ability to self-generate educational material tailored to your learning style, so that you become an expert faster than you thought was possible.

5. Grading Rubric

Adapted from the shared course project rubric. Every criterion is graded for partial credit: a small, isolated issue costs a point or two, and the large drops are reserved for work that is missing or substantially wrong. Sorted the way you will build: notebook first, then the deck.

The notebook side carries 35 points: the analysis (criterion 1) plus the Challenge & Verify work (criterion 2, scored from the log in your notebook together with the prompt you paste into the submission form). The deck carries the other 65. The cutoff and interpretation work is computed in the notebook but earned by how the deck presents the decision; the notebook just has to back it up. A wrong number costs you in criterion 1; it does not sink a well-reasoned, well-presented deck.