Imagine creating a math problem that a human with basic economic knowledge can solve with ease, but a large language model (LLM) struggles to get right. That’s exactly what I’m trying to do. I need help crafting arithmetic-based questions within the economics domain that are solvable by humans but likely to fail for AI models.
The catch? These prompts must have a single correct numerical answer, no hidden tricks or assumptions, and use standard economic reasoning and arithmetic. Oh, and a human non-expert should be able to solve them with clear logic and attention to detail. The goal is to expose conceptual or reasoning flaws in current LLMs.
I’ve already come up with around 100 prompts, but most are too easy for AI agents. They solve them effortlessly. So, I’m turning to the community for ideas, examples, or suggestions on how to design such prompts. Maybe something that subtly trips up models due to overlooked constraints, misinterpretation of time frames, or improper handling of compound economic effects?
If you have any creative suggestions or ideas, I’d deeply appreciate them.