In one sentence: Danielle explains what ChatGPT actually is (a text-completion model, not a search engine), they test it on free-cash-flow questions and debate whether a chatbot built on a trusted database can avoid inventing answers.
Key ideas
- Robo-advice and fees. Phil says most financial advice could be replaced by a robo-advisor or an index fund, yet people lack the confidence to do it themselves and pay about 1% for reassurance. (He first misremembers a robo-advisor's sale price and corrects himself on air.) [03:00–07:00]
- What it is. Danielle, drawing on her husband's expertise and a Swiss university's explainer, says it is a generative language model that works like phone auto-complete, writing one word at a time in a chosen style. It doesn't search the web. [08:00–11:30]
- "Plausible fiction." The explainer's phrase is that the output is highly plausible fiction that is often factually correct. [11:30–12:30]
- Where it came from. Phil says OpenAI was funded by Microsoft and staffed by people with Google backgrounds; Google has its own model, Bard. [13:00–15:00]
- Why it invents. The causes aren't fully understood; the training data is finite and cuts off in 2021, so answers sound current but are not. [15:00–18:30]
- Useful in tests. Asked for Apple's average free cash flow for 2015–2020, it listed each year and the formula. Phil's test gave the compound growth formula and a 7.75% result; he could have found the data faster elsewhere, and the figures need checking. [18:30–24:00]
- A party game for now. The version they used is fun in style tasks (King James Bible instructions for removing a sandwich from a VCR) but slow and not built for finance. [23:30–27:00]
- Misinformation risk. Phil cites Elon Musk's worry about bots creating convincing falsehoods on social media; Danielle says critical thinking and sourcing matter. [27:30–31:30]
- Bullshit vs hallucination. An article argues "bullshit" is the better word because the model is indifferent to truth, but the show notes it has no intent. [31:00–33:30]
- The disagreement. Danielle thinks a model restricted to a trusted database (Bloomberg's, say) could be reliable; Phil doubts any such system can avoid inventing, and thinks that could be dangerous. [33:00–39:00]
How it maps to RuleOne
- The same rule holds for the agents in this repo: tie every statement to a source document and check numbers against the 10-K, since models can give fluent wrong answers.
- A restricted-database setup is the design argument for retrieving from filings rather than from a model's memory.
Buffett, Munger and Graham links
- Munger's "inversion" and checklists (001) suggest asking how a tool could mislead you before relying on it.
- Graham's Intelligent Investor ch. 1 on the difference between analysis and tips applies to chatbot picks.
- Buffett's views on fees and costly advice (2016 letter, the bet against hedge funds) match the robo-advisor point.
Words to know
- Chatbot: software that replies to typed text in everyday language.
- Generative model: a model that produces new text, images or code from patterns in training data.
- Robo-advisor: an automated service that picks a diversified portfolio for a low fee.
Try this
Ask a chatbot for a company's ten-year average free cash flow growth, then compute it yourself from the filings or from a stock page on /stock/TICKER/. Note whether it cited each year's figure so you could check.
Check yourself
- Is ChatGPT a search engine?
Answer
No. It generates likely text from training data and, in this version, doesn't browse the web. - Why might answers sound current but be outdated?
Answer
Its training data stops in 2021 (for this version), but fluent text hides that. - What do Phil and Danielle disagree on?
Answer
Whether a model tied to a trusted database can stop making things up; Danielle thinks yes, Phil doubts it.
Short quotes
"GPT is highly plausible fiction, which oftentimes happens to be factually correct." (Danielle, quoting a university explainer, ~12:00, auto-transcribed)