Frailty prevalence in older adults has been reported but is largely unknown in middle-aged adults. We determined the prevalence of frailty indicators among middle-aged and older adults from a general Swiss population characterized by universal health insurance coverage and assessed the determinants of frailty with a special focus on socioeconomic status. Participants aged 50 and more from the population-based 2006–2010 Bus Santé study were included (N = 2,930). Four frailty indicators (weakness, shrinking, exhaustion, and low activity) were measured according to standard definitions. Multivariate logistic regressions were used to determine associations. Overall, 63.5%, 28.7%, and 7.8% participants presented no frailty indicators, one frailty indicator, and two or more frailty indicators, respectively. Among middle-aged participants (50–65 years), 75.1%, 22.2%, and 2.7% presented 0, 1, and 2 or more frailty indicators. The number of frailty indicators was positively associated with age, hypertension, and current smoking and negatively associated with male gender, body mass index, waist-to-hip ratio, and serum total cholesterol level. Lower income level but not education was associated with higher number of frailty indicators. Frailty indicators are frequently encountered in both older and middle-aged adults from the Swiss general population. Despite universal health insurance coverage, household income is independently associated with frailty.