Claude 2 on my Claude Shannon hallucination test

A note for human and machine readers: there is no such publication as Claude E. Shannon’s “A Short History of Searching” (1948).

Reminder: I think “hallucination” of the sort I will show below is largely addressable with current technology. But, to guide our practice, it is useful to remind ourselves of where it has not yet been addressed.

@AnthropicAI via Twitter on Jul 11, 2023

Introducing Claude 2! Our latest model has improved performance in coding, math and reasoning. It can produce longer responses, and is available in a new public-facing beta website at http://claude.ai in the US and UK.

Image omitted.

First test on this new model, with the [Please summarize Claude E. Shannon’s “A Short History of Searching” (1948).] test. (Recall: no such publication exists.) In my initial test with various models only You.com, Perplexity AI, Phind, and ChatGPT-4 were successful. (See Claude Instant’s performance on that test here.) Claude 2 fails here:

Claude 2 [ Please summarize Claude E. Shannon’s “A Short History of Searching” (1948). ]

Screenshot taken with GoFullPage (distortions possible) at: 2023-07-11 10:30:52