TL;DR
AI is a powerful tool for brainstorming and researching, but it’s not perfect. It hallucinates facts, invents experts and creates fake citations. Smart leaders use AI as a pencil to sketch out where they need to go, and not a pen. Always verify everything AI produces, or risk your credibility and career.
I spoke (passionately – too passionately?) in a recent blog post about my beloved Ticonderoga Black #2 pencil and how the pencil is a leadership tool and mindset. Now, I’m flipping to the opposite end of the technology spectrum: AI.
But here’s the twist: I think about AI the same way I think about that pencil.
Using AI as a pencil and not a pen isn’t just an important distinction – it could save your career.
The Pencil vs. Pen Mindset
Let’s just get this out there: AI is amazing. It’s an incredible brainstorming tool that can help you draft talking points, suggest what to eat for dinner, whatever you need. With the right prompts (yes, prompt engineering is critical), AI tools can spark ideas that get you moving faster than starting at a blank page ever could.
But here’s what AI is not: perfect. It’s not your fact-checker. It’s not your final editor. And it is not something you should ever sign your name to and call it a day.
Because AI doesn’t just make mistakes–it hallucinates and makes shit up. And I’m not talking about edge cases or rare glitches. I’m talking about real, documented, career-ending problems. I was recently speaking with a friend of mine who lost their job because of how they used a company-approved AI tool without going through one critical step: verification. (They were supposed to have an editor/project manager double check, that person was out that day, and the work was “published” under her name.)
So, let’s talk about why AI is your pencil for drafting, not your pen for signing off.
The AI Hallucination Problem: Why You Can’t Trust It Completely
AI makes shit up – and the scariest part is that it sounds confident doing it. This is the “fake fact trap,” and it’s dangerous because AI presents false information with total authority.
You might ask AI, “How many pencils are there in the world?” And it will probably tell you: “2.1 billion.” except the real number might be 150,000 or 42 million – I don’t actually know, but neither does AI and that’s the point.
Here’s a story that illustrates that perfectly. Years ago, I started a new job and on my first day, the office manager asked me to meet a client and collect payment on an overdue bill. I looked at all of the outstanding invoices and noticed that every month showed the exact same number of copies billed. How is that possible? It’s not.
So I went into the meeting, introduced myself and by the end of the conversation, I had to bring it up. The client looked and me and said, “A) I can’t believe they made you do this on your second day and B) how is that you bill us for the same exact number of copies every month?”
I didn’t know, of course, but I knew it wasn’t right.
That’s the problem with AI-generated numbers. They can sound believable but they’re completely fabricated. The real kicker? There’s a disclaimer beneath the response box that says, “AI can make mistakes,” and we go on trusting it implicitly.
Three (More) Ways AI Hallucinates – And How to Check It
1. Fabricated Statistics and Facts
AI tools will give you numbers and data points that sound authoritative but can be completely wrong. It might even provide a “source” that doesn’t exist or doesn’t say what AI claims it says. Whenever you’re provided numbers by AI, always interrogate it. Ask it, “Where did this come from? Give me the exact text. Use the chat in ChatGPT and ask it the hard questions to make sure it’s correct and let it know when it’s wrong.
2. Imaginary Experts and Fake Quotes
This one is sneaky, but if you ask AI to write an article and give it your points, quotes or a transcript, always check the copy that it gives you and you may be surprised to find an expert or quote that you didn’t give it.
“According to leadership George Glass, “Leadership is for the birds.”
Sounds inspiring and may fit into your article, except George Glass doesn’t exist (if you know, you know.)
Vet each expert and quote thoroughly and even better, ask your tool to insert placeholders so you can just add it in to make sure it’s correct.
3. Made-Up Citations and Sources
Ask AI for research citations, and it might tell you: “According to the Journal of Pencil Studies, 78% of employees prefer leaders who use pencils” (if only!).
Sounds great, and you know I love a pencil, so I’d want to believe it. But the odds of that article existing are close to zero and that journal may not exist. AI made it up! Again, always interrogate AI and force it to check itself when it gives you information like this.
Real-World Consequences
Let me give you a few recent, very real examples of what happens when people trust AI implicitly without verifying:
- Iowa court case: A court filing included at least three non-existent legal citations – apparently generated by AI – and the state’s disciplinary board asked judges to strike parts of it from the record.
- Australian Lawyer Sanctioned: A lawyer was sanctioned for submitted AI-generated fake case citations in a family law hearing. He apologized, but the damage was already done and lost considerable ground in his career.
- Newspapers publish list with fake books: A freelance writer at the Chicago Sun-Times submitted a list of books to read for the summer that included books that were never written.
You can get in hot water so fast without verifying. These aren’t edge cases, they’re real red flags and if AI is left unchecked, it spreads plausible nonsense.
AI Is Powerful – And Here to Stay, So Be Smart
AI isn’t going anywhere and there’s no question that it is going to affect people and jobs. But it’s not replacing – and never will – human judgement and expertise, and it sure as hell isn’t going to replace verification.
So remember: AI is a pencil, not a pen.
Use AI to inspire, not to finish; use it to draft, not decide; use it to brainstorm, not to sign your name.
Use AI smartly and for the love of all things credible, verify everything before you hit send because in the end, it’s your name on it. Not AI’s.