Every day, dozens of like-minded mathematicians gather on an online forum called Zulip to build what they believe is the future of their field. They’re all devotees of a software program called Lean.

Computers are extremely good with numbers, but they haven’t gotten many human mathematicians fired. Until recently, they could barely hold their own in high school-level math competitions. But now ...