Human-building interaction (HBI) is converging the fields of architecture and interaction design, leading to new and interesting tensions in perspectives and methodological approaches. One such tension is related to temporal constraints. Architecture and interaction design typically produce outcomes with very different lifetime expectancies and, predominantly, use methods with very different pace. As an example, fast, iterative approaches of contemporary interaction design, consisting of frequent updates and redesigns, contrasts with much slower, plan-driven and long-term vision driven approaches within architecture. One question emerging from this tension is how to meaningfully combine perspectives and approaches. One suggestion, among others, has been that interaction design methods such as participatory design can be used to heighten the involvement of inhabitants and other stakeholders in continuous adaptations of the buildings they inhabit. While an interesting proposal, we believe that methodological considerations only partly address the complexity of the tension at play from the different lifetime expectancies of buildings and interactive computer systems. Unfolding this complexity further, we therefore propose a framework of temporal constraints at three levels of abstraction: (1) rationale, (2) method, and (3) outcome. Inspired by previous work, we discuss temporal constraints in HBI at these levels. We argue that designing for HBI requires an understanding of temporally constrained design conventions that apply meaningfully to both the short term and long term.