CMU 教授创立 Safeworld，用仿真评估机器人安全性

The big trend in robots is handing the keys over to a generative AI model, but that brings with it a problem: that architecture isn’t predictable the way traditional algorithms are. How can you be sure your brand new humanoid will be safe?

Dr. Ding Zhao, who directs the Safe AI lab at Carnegie Melon University, has been working on this problem for almost his entire career. Now, along with veteran start-up executive Kyle Wong and machine learning engineer Simo Rachidi, he’s founded a company, Safeworld, intended to solve it.

“The safety challenge that we’re talking about is a combination of, one, really advanced generative AI probabilistic evals — how do you underwrite the risk of a probabilistic system?” Zhao says. “The second part that’s really hard is the trust part, and you need both to deploy a robot.”

Safeworld is emerging from stealth today with a seed round of more than $12 million, led by Shine Capital and a16z Speedrun, with additional investment from Box Group, Carnegie Mellon University Endowment, Innovation Endeavors and SV Angel.

“The time to build an industry safety standard is now while robots are being designed and deployed,” a16z Speedrun partner Jonathan Lai told TechCurnch. “By the time you have robots in households colliding with kids and causing safety incidents, that’s way too late.”

Safeworld’s speciality is evaluating a robotic control system in simulations that are populated with realistic human models. It’s akin to the challenge faced by companies like Tesla or Wayve, who must ensure that their vehicles respond appropriately to a variety of surprising incidents they may encounter on the road. But that will be more difficult for robots, Zhao argues, because they work in unstructured environments, and because each facility they are in will have different safety standards.

“One of the most common areas is if there is a blind corner in this particular factory,” Wong said. “What is the speed or what is the stopping distance that you need to make sure that this robot will not collide with a particular human? If a human is carrying boxes, for example, will the robot detect the human or not?”

为了回答这个问题，Safeworld 将在 Genesis 或 MuJoCo 等模型中构建该角落的数字版本，插入它所评估的机器人的仿真，并由其真实软件驱动，然后运行数千个场景，让人类模型与机器人相遇。据 Zhao 说，这比看起来要难，因为人是不可预测的。

“绊倒和摔倒也是我们通过仿真进行大量测试的一个很好的例子，”Wong 说。“否则，你就得亲自去为机器人绊倒和摔倒，而这就像是一件很难一直做的事情。”

Safeworld 正在构建的平台与机器人制造商内部使用的工具之间存在明显的相似之处。然而，创始人认为，除了他们特定的专业知识之外，机器人制造商也会希望由第三方来验证他们的工作，哪怕只是为了在竞争对手之间共享安全案例的信息。

“很多人低估了其中一些边缘案例会有多难解决，”Zhao 说。“我们担心的不是真空中的、演示中的机器人。而是大规模部署的机器人，面对的是可能从未操作过机器人的人。”

Vishal Dugar，首席技术官Gritt Robotics，正在为机器人开发 AI 大脑，这些机器人目前帮助工人在工业规模的太阳能农场安装光伏板，并希望承担更复杂的建筑任务。他的公司正在与 Safeworld 合作，共同开发他们的安全仿真。

“我们大多数系统的困难在于，很难通过做一些数学、写一些方程，然后说‘是的，这个系统已被验证是安全的’来正式证明它，”Dugar 说。“这必然要通过经验来完成。”

他的机器人与人类工人一起工作，确保其机械臂不会撞到他们显然是首要考虑的问题。要在实践中验证这一点，需要考虑各种潜在场景。

“人类有各种各样的外表，”Dugar 指出。“他们的身体可以有不同的姿态。他们可能跪着、站着。他们可能绊倒、摔倒。他们可能蹲着。他们可能奔跑。你必须对这些场所中人类可能表现出的所有这些行为做出响应，还要应对人类外表的各种变化，你知道，衣服、体型、身材、身高、肤色，以及其他一切。”

对于 Safeworld 和机器人领域的生成式 AI 来说，现在仍处于早期阶段，公司也仍在摸索最适合其产品的模式——是面向外部用户的平台，还是基于服务的方式？——但团队相信他们正在解决正确的问题。

“我们很可能会成为这个领域第一家盈利的公司，”Zaho 说。“因为如果有人想要部署，他们就需要付钱让我们来处理这种情况。”