Artificial intelligence models may be doing a lot more memorizing and a lot less reasoning when it comes to predicting biology, results from a new competition suggest.
Shortly after launching in April, a tiny startup in Utah called Leash Bio kicked off a challenge to test how accurately AI models predict molecules binding to specific protein targets. The results are in, Leash CEO Ian Quigley exclusively tells Endpoints News — and they aren’t good.
“No one did well,” said Quigley, summing up the results of about 2,000 teams that competed over three months.
For the competition, which ran on the data science competition platform Kaggle, Leash provided its training data, which consisted of lab results showing how millions of molecules bind — or don’t bind — to target proteins. Leash’s dataset is roughly 1,000 times larger than the largest publicly available database focused on protein-small molecule interactions, to Leash’s knowledge.
Despite that size, Quigley said the results showed there still aren’t enough data to solve the binding problem with AI. Models were best at predicting molecules that looked like the training data, but they got worse at guessing the binding of more unfamiliar drug candidates. The startup posted a blog post Thursday describing the results in detail.
“They are less good, and I mean pretty poor but still doable, at predicting molecules that look different from the others but share a very common central core,” Quigley said. “What the computers cannot do at this current stage is predict on molecules that don’t look anything like the ones they’ve been shown already.”
“They are pretty good at memorizing and pretty bad at extrapolating into novel chemical space,” he added. “When I say pretty bad, you could randomly choose molecules and ask if they are binders or not from a collection, and you would be doing as well as the winning models in this competition.”
That’s a sobering reality check for the buzzy field of AI bio, which has been characterized by megarounds and proclamations of R&D revolutions.
In a way, the contest’s struggles validate Leash’s belief that the field needs more data, not more models. The biotech, initially run out of Quigley’s basement-turned-lab, started out with a $7.9 million seed round and was co-founded by two former Recursion scientists: Quigley and Andrew Blevins.
Leash’s team plans to present detailed results from the BELKA contest in December at NeurIPS, a leading AI research conference.
Quigley asks AI field: ‘Show us how it’s done’
Still, there are big caveats to the results. Most notably, the field of 1,950 participating teams didn’t include any AI heavyweights, particularly computational-heavy biotechs working on small molecules like Nimbus Therapeutics, Relay Therapeutics, Iambic Therapeutics, Atomwise or Charm Therapeutics.
The winner was a data science graduate student named Victor Shlepov.
Kaggle requires winners to release their solutions and whether they used additional data, and doing so goes against the incentives of for-profit companies, Quigley said. Instead, Leash plans to publicly release its data in September, allowing others to participate outside the formal contest.
“If there are groups that feel they are superior in this task and did not want to reveal their solutions, we please invite them to show us how it’s done,” Quigley said. “We could be really wrong here. There could definitely be folks who have either the datasets or the fancy tricks that really have solved this task very effectively and we haven’t seen it yet.”
For its own part, Quigley said Leash will take its own stab at cracking the binding code. The company, now with 10 employees, continues to generate heaps of protein-small molecule binding data using DNA-encoded library screens. The biotech plans to study 200 proteins a month, Quigley said, while also advancing two drug programs that are in the earliest stages of development.
Leash’s bet is that far more data will yield better AI predictions. Quigley compared it to how prior breakthroughs in machine-learning problems, from playing chess to folding proteins, came only when there was a massive database full of examples to train on.
“We think the future is not that long off,” Quigley said. “But we’d also argue it’s not here today.”
Editor’s note: This story was updated to include a link to Leash Bio’s blog post on the contest results.