Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory, working with Google and Northeastern University, have built software that generates a 3D model from a prompt and then lets users repair the parts that would fail in print.
Generative 3D tools usually know how an object should look, not how it should work, so a mug may not hold coffee and the mesh is hard to edit. The new program, InstructMesh, pairs Microsoft’s TRELLIS generator with GPT-4 so a user can highlight a region, describe the defect in language, and apply a local geometric fix, or use sliders for smaller tweaks such as thickness.
The CSAIL group printed household objects from those repaired meshes: a mug wrapped by a dragon whose tail is the handle, butterfly-wing glasses, a shiny blue shell-like whistle, and an octopus-style dispenser that pours from each tentacle. They also made a knee brace that looks like denim to match a patient’s jeans, and a shrimp-like bristle-bot enclosure with a motor inside that slides across a table.
“We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want,” said Faraz Faruqi, an MIT PhD graduate and lead author on the paper. He is now at Google.
To test whether newcomers could catch problems before fabrication, the team had TRELLIS recreate popular models from Thingiverse. Nearly 80 percent of those generated meshes were structurally flawed. Novices using InstructMesh identified and repaired the issues about 90 percent of the time, as reviewed by an expert. Users also made phone stands and vases, and found the sliders useful for enlarging or extruding a selected region.
“The users got what they prompted for and easily tweaked designs where needed,” said Faruqi. “What they saw is what they got, and the items worked as advertised, so to speak.”
Stefanie Mueller, an associate professor in electrical engineering and computer science and mechanical engineering at MIT, and a CSAIL member, is a senior author. Co-authors include researchers at Google, Northeastern University, and CSAIL. Google and the MIT-HPI Collaborative Research Program supported the work. The team will present it at the ACM Symposium on User Interface Software and Technology in November.
Source: csail.mit.edu
