Google, DoD built an AI-powered microscope to help doctors spot cancer

On Sep 18, 2023

Dr. Niels Olson uses the Augmented Reality Microscope.

U.S. Department of Defense

In his office at the VA hospital in Seattle, Dr. Nadeem Zafar needed to settle a debate.

Zafar is a pathologist, the kind of doctor that carries out clinical lab tests on bodily fluids and tissues to diagnose conditions like cancer. It’s a specialty that often operates behind the scenes, but it’s a crucial backbone of medical care.

Late last year, Zafar’s colleague consulted with him about a prostate cancer case. It was clear that the patient had cancer, but the two doctors disagreed about how severe it was. Zafar believed the cancer was more aggressive than his colleague did.

Zafar turned to his microscope – a canonically beloved tool in pathology that the doctors rely on to help make their diagnoses. But the device is no ordinary microscope. It’s an artificial intelligence-powered microscope built by Google and the U.S. Department of Defense.

The pair ran the case through the special microscope, and Zafar was right. In seconds, the AI flagged the exact part of the tumor that Zafar believed was more aggressive. After the machine backed him up, Zafar said his colleague was convinced.

“He had a smile on his face, and he agreed with that,” Zafar told CNBC in an interview. “This is the beauty of this technology, it’s kind of an arbitrator of sorts.”

The AI-powered tool is called an Augmented Reality Microscope, or ARM, and Google and the Department of Defense have been quietly working on it for years. The technology is still in its early days and is not actively being used to help diagnose patients yet, but initial research is promising, and officials say it could prove to be a useful tool for pathologists without easy access to a second opinion.

A new tool for pathologists

Augmented Reality Microscope at Mitre

Ashley Capoot | CNBC

There are currently 13 ARMs in existence, and one is located at a Mitre facility just outside of Washington, D.C. Mitre is a nonprofit that works with government agencies to tackle big problems involving technology. Researchers there are working with the ARM to identify the vulnerabilities that could cause issues for pathologists in a clinical setting.

At first glance, the ARM looks a lot like a microscope that could be found in a high school biology classroom. The device is beige with a large eyepiece and a tray for examining traditional glass slides, but it’s also connected to a boxy computer tower that houses the AI models.

When a glass slide is prepared and fixed under the microscope, the AI is able to outline where cancer is located. The outline appears as a bright green line that pathologists can see through their eyepiece and on a separate monitor. The AI also indicates how bad the cancer is, and generates a black and white heat map on the monitor that shows the boundary of the cancer in a pixelated form.

CNBC demoed the ARM with researchers at the Mitre facility in August.

View of what pathologists see on their screen when they use the ARM.

Ashley Capoot | CNBC

Patrick Minot, a senior autonomous systems engineer at Mitre, said since the AI is overlaid directly onto the microscope’s field of view, it doesn’t interrupt the pathologists’ established workflow.

The easy utility is an intentional design choice. In recent years, pathologists have been contending with workforce shortages, just like many other corners of health care. But pathologists’ caseloads have also been mounting as the general population grows older.

It’s a dangerous combination for the specialty. If pathologists are stretched too thin and miss something, it can have serious consequences for patients.

Several organizations have been trying to digitize pathologists’ workflows as a way to increase efficiency, but digital pathology comes with its own host of challenges. Digitizing a single slide can require over a gigabyte of storage, so the infrastructure and costs associated with large-scale data collection can balloon quickly. For many smaller health systems, digitization is not yet worth the hassle.

The ARM is not meant to replace digital pathology systems, but Minot said it can help health organizations bypass the need for them. Pathologists have the option to take screen grabs of slides using ARM’s software, for instance, which are much less expensive to store.

The ARM will usually cost health systems between $90,000 to $100,000.

Minot added that the ARM ensures the physical microscope, not just a computer, remains an integral part of the pathologists’ process. Many have warned him not to mess with their microscopes, he joked.

‘Big data is what Silicon Valley does best’

Dr. Niels Olson uses the Augmented Reality Microscope.

Source: U.S. Department of Defense

Few understand the challenges facing pathologists quite like Dr. Niels Olson, the chief medical officer of the Defense Innovation Unit, or DIU, at the Department of Defense.

The DIU was created in 2015 as a way for the military to integrate cutting-edge technology developed by the commercial world. The organization negotiates contracts with companies so they can collaborate and circumvent long bureaucratic hang ups.

Olson is a pathologist, and before beginning his role at the DIU, he served in the U.S. Navy. In 2018, he was sent to Guam, a U.S. island territory in Micronesia, where he worked as the laboratory medical director and blood bank director in the Naval Hospital.

During his two years in Guam, Olson was one of two pathologists on the island, and the only pathologist in the Naval Hospital. This meant he was often making major decisions and…