MaterialVellum: Steering Generative AI by Leveraging Expressive Material Affordances to Form Design Constraints

Abstract

We present MaterialVellum, an interaction approach for steering generative AI through material-inspired manipulation of design constraints. In urban design, for instance, zoning bounds can be defined by the stretch of elastic bands, building density redistributed by molding clay, and functional relationships organized by the pull of magnets. MaterialVellum departs from current interfaces that rely on natural language prompts, and instead leverages users' existing knowledge of physical materials to represent and manipulate design constraints. We introduce a framework that maps material affordances---such as elasticity, malleability, or attraction---to spatial, relational, and proportional constraints that underpin design. These mappings are operationalized in user interfaces through familiar direct manipulation multi-touch gestures, including stretching, sculpting, and clustering, whose effects feed as control parameters into generative AI pipelines. We demonstrate this approach through technology probes, including an urban design, a flower arrangement design, and a graphical design scenario.A qualitative study (N=10) shows that material-based interactions improve accessibility of constraint expression, support exploratory workflows, and lower barriers for non-experts, while effectiveness depends on alignment with users' mental models.