Limitations in physical functioning can lead to dire health and financial consequences. However, the progression of physical function impairment interacts with many different factors, such as organ function, activities and the environment. It is suspected that the complexity of physical function evaluation methods contributes to the low prevalence of function assessments in routine care. Earlier studies have shown that a bidirectional and monotonic relationship exists between physical activity and function. Wearable activity monitors (WAM) -- devices equipped with one or more accelerometers, offer an opportunity to infer physical function from the observed physical activity objectively and at scale. However, efforts to infer function from such activity measurements have been limited. In this dissertation I present methods to infer physical function from daily activity data. Using free-living activity data from the physical activity studies in the Osteoarthritis Initiative, I show that activity measurements obtained by a WAM are sensitive to changes in physical function, if we use an appropriate representation for daily activity. Further, I show that a representation of physical activity that quantifies a subject's engagement in different classes of activity patterns, improves the performance in a physical function classification task. I also demonstrate how such an activity representation may be constructed in an unsupervised manner, obviating the need for time consuming annotation of activity data. Finally, I motivate the case for a composite score of physical function and present two methods for deriving such a score from free-living activity. I also present evaluations of the computed score on predicting key events in the natural history of knee osteoarthritis. Inferring physical function from wearable activity monitors will enable remote monitoring of function, which in turn will facilitate better delivery of care to patients, at a lower cost.