Claude and LLaMa get basic human anatomy wrong

Not all LLMs are created equal

As we’re building chat.radiantai.health, I’m in the enviable position of testing different LLMs for medical answer generation. During my big tech days, I could kind of get a sense that not all LLMs are equal for software engineering purposes, the differences between the LLMs are stark when it comes to basic human anatomy.

A primer on human anatomy

The study of the structure of the human body is called anatomy. Gross human anatomy centers on large structures observable with the naked eye. One such structure is the diaphragm, which you may have heard of before. It is the muscle which drives breathing (inspiration and expiration). If someone were to slice you in half just below the lungs, the diaphragm would stretch across one of the halves, like a paper covering a cylinder.

Since the diaphragam stretches across the entire cross section of your body, it has several openings that allow large vessels to travel from the top of your body into the bottom part.

The caval opening

Ask any medical student about diaphragm openings and they will tell you “I ate 10 eggs at 12”. This mneomic encodes the following information:

As you can see, the inferior vena cava crosses through the diaphragm at T8, where T stands for throacic vertebrae. When we ask Claude-instant-100k about it, we get:

“Q: At what level is the caval hiatus?

A: The caval hiatus is located at the level of the twelfth thoracic vertebra (T12) in the human body.

Some key details:

• The caval hiatus is an opening in the diaphragm that allows the inferior vena cava to pass through the diaphragm from the abdomen into the thorax (chest cavity).

• It is situated posteriorly, close to the midline of the body.

• Specifically, the caval hiatus is found at the level of the twelfth thoracic vertebra (T12). This is a vertebra in the thoracic (mid-back) region of the spine.

• By being located at the T12 level, the caval hiatus provides an anatomical landmark to identify this region of the diaphragm and locate the inferior vena cava as it passes through the diaphragm.

• During surgeries or procedures involving the lower thorax, abdomen or pelvis, knowing the level of the caval hiatus is important for surgeons to avoid injury to the structures passing through this opening, like the inferior vena cava.

So in summary, the caval hiatus that allows the inferior vena cava to traverse the diaphragm is consistently found at the anatomical level of the twelfth thoracic vertebra.”

Here we can see that even though the LLM has correctly identified the meaning of caval opening, it gets the level completely wrong.

When we ask LLaMa-70B:

“Q: At what level is the caval hiatus?

A: The Cavalier King Charles Spaniel is a small breed of dog that is known for its gentle and affectionate nature. The breed is divided into four levels of size, which are determined by the height of the dog at the shoulder. The four levels are:

1. Toy: This is the smallest level of Cavalier King Charles Spaniel, with a height of 10-11 inches (25-28 cm) at the shoulder.

2. Miniature: This level is slightly larger than the toy variety, with a height of 11-12 inches (28-30 cm) at the shoulder.

3. Standard: This is the middle level of Cavalier King Charles Spaniel, with a height of 13-14 inches (33-36 cm) at the shoulder.

4. Maxi: This is the largest level of Cavalier King Charles Spaniel, with a height of 15 inches (38 cm) or more at the shoulder.

So, as of 2023-09-15, the Cavalier King Charles Spaniel is in the "Maxi" level.”

which doesn’t even recognize caval hiatus as a medical term.

Finally, when we ask ChatGPT:

“Q: At what level is the caval hiatus?

A: The term "caval hiatus" is typically used in reference to human anatomy, specifically the anatomical structure known as the "caval opening" or "caval foramen." The caval opening is an opening or hole in the diaphragm, the muscular partition between the chest cavity (thoracic cavity) and the abdominal cavity. It allows certain structures to pass through from the abdomen to the chest.

The caval opening, or caval hiatus, is located in the central tendon of the diaphragm, slightly to the right of the midline. It is positioned at the level of the 8th thoracic vertebra (T8) in most individuals. The inferior vena cava (IVC), the large vein that carries deoxygenated blood from the lower body to the heart, passes through this opening as it ascends from the abdomen into the thoracic cavity.

It is important to note that the location of anatomical structures can vary slightly among individuals, so there may be some natural variation in the precise position of the caval hiatus. ”

Though I am no fan of OpenAI for their closed source approach to AI, they do make a pretty good LLM.

If you’re interested in learning more about the adventures of a big tech employee turned medical student, please consider subscribing!