If you're a crossword fan you soon become disabused of the notion that the LLMs are “intelligent”. E.g. I tried to ask them for an instrument with the second letter P, and they suggested Piano, Piccolo. They struggle to process the concept of numbers or that words have functional properties beyond semantics.
Another infamous example is if you ask an LLM the following riddle:
Another infamous example is if you ask an LLM the following riddle:
The LLM will answer, every time, that the doctor is the boy's mother (and give you a little lecture on sexism to boot), because in the thousands upon thousands of times it has seen that kind of passage in its training data, the answer was always that the doctor is the mother, because that's the standard riddle. Because an LLM is really just the best predictive text algorithm ever invented, it is incapable of understanding the vital switch in the format, because everything it knows says that when you see this kind of passage, you answer “The doctor is the boy's mother”.A mother and son were in a car accident where the mother was killed. The ambulance brought the son to the hospital. He needed immediate surgery. In the operating room, a doctor came in and looked at the little boy and said, “I can't operate on him: he is my son.” Who is the doctor?