Get Updates
Association for Public Policy Analysis and Management
"Seeking Solutions to Complex Policy and Management Problems"
Washington Marriott Hotel & Westin Georgetown Hotel—Washington, DC—November 3-5, 2011
Children of Immigrants' SNAP Use and Food Insecurity
Heather Koball and Albert Yung-Hsu Liu
(Mathematica)
Food security, which is defined as a lack of consistent access to enough food for an active, healthy life, is a key indicator of well-being. It has been linked to positive health outcomes and improved development among children. Food security, however, is not universal. In 2008, 16 percent of United States households with children were food insecure in the previous year. These statistics mask substantial variation across demographic groups. Research indicates that immigrant households with children are twice as likely as native-born households with children to be food insecure. The primary public assistance program to reduce food insecurity is the Supplemental Nutrition Assistance Program (SNAP). Immigrant eligibility for this program has changed over time, and the rules differ for adults and children. Welfare reform in 1996 eliminated eligibility for legal, noncitizen adult immigrants, although some states maintained their own supplemental programs for immigrants. The 2002 Farm Bill restored eligibility to adult immigrants who lived in the United States for more than five years. SNAP eligibility rules for immigrant children are more inclusive. Although welfare reform restricted eligibility for legal immigrant children as it did for adults, it was restored in 1998 for legal immigrant children who moved to the United States prior to welfare reform. The 2002 Farm Bill expanded eligibility by making all legal immigrant children eligible for SNAP. Furthermore, the children of immigrants who were born in the United States, regardless of their parents' eligibility, are eligible for SNAP. Given the importance of food security for fostering child well-being, the high rate of food insecurity in immigrant households, and the ongoing changes in SNAP eligibility rules, policymakers would benefit from a better understanding of the relationship between SNAP participation and food insecurity among immigrant households with children. Our paper answers three research questions pertaining to this relationship. First, we examine whether SNAP eligibility rules affect SNAP participation among immigrant households with children. Second, we examine the demographic and labor market characteristics associated with SNAP participation in these households. And third, we document the causal effect of SNAP participation on the food insecurity among immigrant households with children. Our study uses data from the Current Population Survey-Food Security Supplement (FSS) from 1996 to 2009. The FSS contains information on food insecurity and SNAP benefits, as well as demographic information about immigrants. To address the first two research questions we estimate regression models that predict the likelihood of SNAP participation, based on policy indicators and household characteristics. Our methodological strategy for the third research question is to use state and federal policy changes as an instrument for SNAP participation as we examine the latter's effect on food insecurity.
Intergenerational Transmission of Wealth and the Black-White Wealth Gap
Alexandra Killewald
(Mathematica)
A persistent question concerning black-white inequality in the United States is how much of the racial inequality in adult outcomes can be explained by different starting points, rather than by further disadvantages experienced by black Americans during the life course. Parental wealth is one indicator of an individual's "starting point." This paper will estimate the fraction of the black-white wealth gap that is explained by racial differences in parental wealth at different points in the life course of younger adults. The results will be informative for public policies concerning the accumulation of assets. If racial wealth inequality in adulthood is primarily due to parental wealth differences early in life, then policies such as "baby bonds" or subsidized children's accounts can reduce the racial wealth gap. However, if race differences in wealth remain after adjusting for parental wealth, greater attention should be focused on processes, including racial discrimination in asset markets, that prevent black Americans from accumulating wealth at the same rate as similarly endowed whites. This paper will use data from the Panel Study of Income Dynamics (PSID) to analyze the mediating role of parental wealth in explaining black-white differences in wealth among younger adults. The analysis will first document the extent to which differences in parental wealth explain race differences in child wealth between the ages of 20 and 45, including whether the mediating role of parental wealth changes across this age range. Supplementary analyses will explore the contribution of parental wealth to racial differences in owning a home, the primary asset in most household portfolios. This analysis takes as its point of departure Conley's (1999) provocative finding of no significant wealth disadvantage for young black Americans after controlling for race differences in parental wealth. Conley's sample was limited because it excluded married women, thereby ignoring the possibility that race differences in marriage patterns may affect race differences in women's wealth accumulation. Furthermore, Conley's analysis examined outcomes only for adults ages 20-30. Parental wealth may be a weaker mediator of the black-white wealth gap at older ages: disadvantage in wealth accumulation for black Americans beyond the effects of differential parental wealth starting points may appear only later in life, after the second generation has begun to accumulate their own assets. The present analysis extends the age range considered and presents a more complete picture of the evolving role of parental wealth in influencing the black-white wealth gap.
Bending the Cost Curve and Improving Quality: Payment Reform to Reduce Hospital Readmissions
Deborah Chollet, Allison Barrett, and Timothy Lake (Mathematica)
New public payment incentives to reward more efficient and high-quality care are a major component of health reform in the United States. This study investigates whether either of two proposed payment reforms—pay-for-performance or episode-based payments—would reduce rates of hospital readmissions, a major and preventable source of health care costs. It simulates hospital behavior in one large state—New York—offering a major case study of the incentives and impacts of hospital payment reform. The study investigates whether either of two alternative hospital payment reforms would provide a sufficient business case for hospitals to invest in reducing readmissions, and how readmission rates and costs would be affected. We consider whether hospitals would adopt either of two evidence-based interventions—the Care Transitions Intervention and the Project RED model—that have been shown to reduce readmissions through improved discharge planning and post-discharge follow-up. The simulations consider hospitals' cost of the intervention, the likely impact on readmissions, and the resulting impact on both hospital revenue and the size of the incentive under each payment reform. The study uses New York's all-payer hospital discharge data, which include more than 6 million stays at New York hospitals from 2006 to 2008. All-cause 30-day hospital readmissions are considered, with clinical exclusion criteria applied to omit planned readmissions. APR-DRGs are used to adjust readmission rates for severity of illness. In 2008, 14.6 percent of admissions to hospitals in New York State resulted in a readmission, costing public and private payers $3.7 billion. Readmission rates varied widely among hospitals, but variation by geographic location was relatively low. About 40 percent of hospitals performed worse than expected, given their case mix. In nearly all hospitals, payers could reduce readmissions by intervening directly—that is, directly paying for staff to conduct evidence-based discharge management. Payment reforms aimed at encouraging hospitals to implement these programs would have less effect. With pay-for-performance incentives that would reduce future payments to high-readmission hospitals (such as Medicare will soon implement nationwide), only a small proportion of hospitals would adopt either evidence-based intervention to reduce readmissions. In contrast, with episode-based payments, most hospitals would adopt these interventions, substantially reducing readmissions. Because hospitals would be paid for risk-adjusted expected readmissions, aggregate payments would change relatively little in the short term, but rebasing per-episode payments over time to reflect reduced hospital cost for readmissions could gradually reduce payer costs as well. Implications for policy. If effective, payment incentives to reduce hospital readmissions could improve the quality of care and patient quality of life, while also achieving substantial savings for both private and public payers. Both Medicare and the New York Medicaid program are concerned about the quality and cost implications of high readmission rates, and both have taken steps to pilot or implement payment incentives intended to reduce readmissions. This study helps policymakers better understand the potential for these efforts to succeed.
Summative Findings from the Evaluation of the Individual Training Account Experiment
Irma Perez-Johnson, Quinn Moore, and Robert Santillano
(Mathematica)
The Individual Training Account (ITA) experiment was designed to provide federal, state, and local policymakers in the United States information about how to manage customer choice under ITAs. First introduced in 1998, ITAs represented an important shift for the workforce investment system, moving away from contract-based training in favor of individually managed accounts that afford customers greater flexibility and control over their training decisions. Workforce agencies were given flexibility in how to implement ITAs but had limited information on which to base such decisions. To help fill this gap, the ITA experiment tested three approaches that varied in the ITA award structure, counseling requirements, and the ability of program staff to veto customers' final training choices. Eight local sites participated in the ITA experiment and implemented the three approaches tested side-by-side, beginning in 2002 until early 2004. An important limitation of the original evaluation of the ITA experiment was its 15-month follow-up period. About 15 percent of all study participants were still in training at the end of this period, and their training and employment outcomes could not be observed. It was also possible that the full effects of the three approaches had not been observed. To help address these limitations, DOL funded an extended evaluation of the ITA experiment. The extended evaluation follows study participants for six to eight years after random assignment. The proposed paper will share summative findings from the evaluation of the ITA Experiment. It will bring together the most important findings from the original evaluation—on the implementation and operation of the three ITA approaches, customer participation in counseling services, receipt of ITA-funded versus other training, and occupation and training program choices—with new experimental estimates of the long-term impacts of the approaches on several outcomes, including customers' participation in training, employment and earnings, characteristics of jobs held, household income, and receipt of public assistance. The report also examines the relative returns on investment of the three approaches. By methodically examining the implementation of the ITA approaches tested, their relative impacts on a wide range of outcomes, and the net benefits of switching between approaches, the extended evaluation provides the best available evidence on the tradeoffs inherent in managing customer choice under different ITA approaches for customers, the government, and society as a whole.
Customer Responses to Personal Reemployment Accounts (PRAs): Findings from Seven Demonstration States
Gretchen Kirby
(Mathematica)
In an effort to foster good employment outcomes for recipients of Unemployment Insurance (UI), the U.S. Department of Labor launched a demonstration of Personal Reemployment Accounts (PRAs) in seven demonstration states in 2004. PRAs were intended to help unemployed workers build job skills and find work through a flexible customer-managed account. Targeted to a subset of UI recipients, PRAs of $3,000 allowed recipients to choose how and when to spend funds from their account to purchase reemployment services, including training. Workers could also elect to receive the funds as cash bonuses for reentering the workforce and keeping a job. This presentation will summarize key findings from the PRA demonstration project that examined this strategy through an implementation study and an analysis of individual-level PRA and UI data. The lessons from the PRA experience are of value to policymakers and program administrators as the concept of self-managed accounts that support employment and training continues to evolve. The PRA experience sheds light on considerations in structuring individually managed accounts and in aligning their use with goals for particular customers. Findings from the implementation and focus group studies were used to develop a framework for examining the patterns in PRA use by different types of users and in different time periods. PRA recipients were divided into 5 distinct groups based on PRA use in the bonus qualification period (the first 13 weeks after the UI claim) in order to explore the specific strategies they pursued in using their accounts for reemployment, as well as examining their employment, earnings, and UI outcomes. The findings suggest some considerations about PRAs specifically, and customer-managed accounts in general. The broad purposes of the PRA gave recipients a great deal of flexibility in supporting their reemployment efforts, but the bonus and service purchase components might have sent mixed messages and served different purposes. The right amount for a reemployment account may be a function of its purpose to the recipient. The $3,000, while generous for some, was not enough of an incentive for many to speed their reemployment in the context of long-term career interests and goals. The $3,000 was similarly "low" in comparison with the amount of potential support offered through an ITA for which individuals could qualify to assist with training costs. PRAs are fully customer-managed, and few recipients choose to use the funds to purchase intensive career counseling and job search assistance. There was no way to assess, through this evaluation, how the outcomes of PRA recipients may have been different given some upfront development of career plans and associated strategies for using the accounts. PRA recipients had the $3,000 at their disposal for up to one year. The analysis shows that among all PRA users, the last disbursement from the account is made, on average, within the first four and a half months. This suggests that the one-year length may not be necessary, particularly if an account is focused on quick reemployment.
Place-Based Strategies and Evaluation (Roundtable)
Keri-Nicole Dillman (Mathematica), Moderator
While comprehensive community initiatives (CCI) and place-based strategies have been advanced for the past two decades, we are currently seeing major and unprecedented investment in place-based policies by the Federal government through new initiatives (such as the Choice Neighborhoods and Promise Neighborhoods Initiatives and the umbrella Neighborhood Revitalization Initiative) and calls for a place-conscious approach across all federal agencies. Place-based strategies have also attracted growing interest from locally rooted community and family foundations. While much of the debate in the evaluation field regarding the first generation of these place-based efforts centered on the obstacles to using experimental designs to evaluate them, the evaluation field has, in some areas, shifted its focus regarding these efforts. Funders (particularly in the philanthropic community) are increasingly interested in ‘contribution rather than attribution'—that is real-time information from formative and developmental evaluations as well as (if not more than) summative evaluation. Moreover, performance measurement and community indicators are now central to CCI targeting, implementation, and evaluation as small-area data sources and methods (e.g. GIS, neighborhood indicator partnerships) and web-based performance management systems (e.g. "efforts to outcomes" measurement systems) have advanced. Accompanying the new attention to place-based efforts is a reinvigorated critique and dialogue at conferences and in the literature regarding the role of evaluators and evaluation in marshaling information and evidence, providing ongoing feedback and assistance, and assessing outcomes (see, for instance, a review of the field by the Aspen Institute's Roundtable on Social Change, a 2010 Federal Reserve Bank Conference on improving the outcomes of CCIs, and the 2009 inaugural issue of Foundation Review dedicated to CCIs). This roundtable will examine the richness of this new debate by convening practitioners, funders, and evaluators for a critical discussion of the field. We envision a lively and frank discussion of the kinds and quality of information that can be gathered and used, and the investment of time, money, and technical capacity required to use it effectively to make policy and improve programs.
The House Next Door: A Comparison of Residences by Disability Status Using New Measures in the American Housing Survey
Gina Livermore (Mathematica)
Researchers have thoroughly documented the struggles of working-age people (18 to 64 years old) with disabilities in terms of their employment, health insurance coverage, access to health care, and poverty status. However, the state of housing for this group has yet to be researched extensively, perhaps due in part to data limitations. Now, because of the inclusion of disability-related questions in the 2009 American Housing Survey (AHS), this issue can be analyzed in detail for the first time. Understanding the housing needs of working-age people with disabilities is crucial to developing housing policies for this population, such as the Section 811 program. By analyzing the differences in housing between adults with and without disabilities, we can identify areas in which housing for people with disabilities is lacking and assess the effect of housing policies on the likelihood that people with disabilities will have poor or unstable housing. For this study, we conducted multivariate analyses of the likelihood of particular housing and neighborhood features while controlling for sociodemographic characteristics. In particular, we assessed the extent to which aspects of housing and neighborhood quality differ between working-age people with and without disabilities, holding income and other household characteristics constant. We also estimated additional models to explore differences based on the level of disability severity. Our results suggest that people with disabilities reside in housing units with significantly lower levels of quality and in less desirable neighborhoods, as measured by 14 housing and neighborhood attributes. Specifically, people with disabilities live in homes with significantly fewer amenities (such as a dishwasher or central air conditioning) and more deficiencies (such as indoor or outdoor leaks and plumbing problems) that are smaller in size and are more likely to live in a manufactured or mobile home compared to people without disabilities. We also find that people with disabilities are significantly more likely to live in neighborhoods with lower median incomes, fewer neighborhood benefits (such as satisfactory police protection and proximity to public transportation), and more neighborhood problems (such as roads in need of repair and proximity to abandoned buildings) compared to their nondisabled counterparts. To assist people with disabilities, policymakers have implemented several federal and local housing policies that aim to help people with disabilities find suitable, affordable housing. We examine the role that this type of assistance plays in the housing attributes of people with disabilities. We find that housing assistance reduces the negative impact of disability on the housing and neighborhood characteristics considered. Further, results of multivariate regressions reveal that the benefit of housing assistance differs for persons with and without disabilities. For people with disabilities, housing vouchers are the most beneficial form of housing assistance, associated with significant increases in satisfaction rating of housing unit, number of amenities, and average area median income. Our results are particularly important as the federal government is in the process of reducing funding for the Housing for People with Disabilities Program and shifting the balance of available funds across several housing assistance programs.
Paying More for Primary Care Cognitive Services May Help Bend the Medicare Cost Curve
James Reschovsky (Center for Studying Health System Change), Arkadipta Ghosh, Kate Stewart, and Deborah Chollet (Mathematica)
The Affordable Care Act temporarily increased Medicare fees 10 percent for evaluation and management (E&M) visits, when conducted by primary care providers. The five-year temporary nature of the policy is unlikely to significantly reorient our health care system to be more primary care oriented, as urged by many. Permanently changing relative prices paid for primary care and specialist services could, however, have a more significant effect in the long run. This study develops a micro simulation model to predict the long-term effect of permanently increasing fees for primary care evaluation and management visits, relative to other physician services. We investigate the impact on the provision of primary care cognitive services as well as on total Medicare fee-for-service costs. We obtained Parts A&B claims for a national sample of 2.7 million elderly, non-ESRD Medicare beneficiaries enrolled in both Parts A and B, who received services from respondents to the 2004-05 Community Tracking Study Physician Survey (N=6627) at any time between 2004-06. We created measures of service use by using total allowed provider reimbursements, standardized to remove geographic- and policy-based variations in provider payments. We then constructed a complex micro simulation model that involved estimation of models to predict future provision per beneficiary for 19 categories of services (e.g., primary care E&M visits, major procedures, inpatient care, SNF). Primary care E&M growth predictions services entered into prediction equations for each of the other 18 service categories. Physician responses to fee changes were based on supply equations and a fee variable developed by Hadley, et. al, 2010. We assumed input costs and Medicare fee increases would follow average trends over the past decade in our base case. We then assumed a permanent 10 percent fee increase for E&M visits by primary care physicians, effective Jan. 1, 2011, while leaving other physician fee increases unchanged from the base case. Projections extended 10 years to 2020. A 10 percent increase in Medicare fees for primary care E&M visits is predicted to increase supply over the long term by 8.6 percent, accounting for an increase in costs that equals less than 0.2 percent of total medical spending for Medicare beneficiaries. However, over the long term, total medical spending per beneficiary is predicted to decline by 1.9 percent, resulting in a substantial benefit cost ratio. The increase in primary care E&M visits achieves these savings primarily by reducing inpatient services, as well as post-acute and hospital outpatient services. Results suggest that promoting primary care can help bend the Medicare cost curve.
Innovative Technology- and Outreach-Based Public Benefit Access Initiatives
Emily Sama-Miller and Jacqueline Kauff (Mathematica)
Low-income households may qualify for a range of public benefits, but the process to apply for each program may be cumbersome, confusing, or in conflict with other programs. New technology-driven tools that allow for online screening and benefits application are of particular interest now as states facing budget crises and seeking to create health insurance exchanges seek viable ways to streamline benefits enrollment. Under contract to ASPE, with additional funding support from ACF and the Center for Faith-Based and Community Partnerships, Mathematica undertook a project to catalogue state and local initiatives to improve access to public, means-tested benefits and to study a small number of these in greater depth. The first phase of the study, a national scan of relevant efforts, focuses on initiatives that leverage technology (especially web-based technology) and outreach to clients in order to improve the access that those clients have to benefits for which they may be eligible, but are not receiving. Using a systematic literature review and a series of interviews with key contacts involved in these efforts, Mathematica compiled a broad resource listing these initiatives and their characteristics in terms of public programs involved, populations served, funding and technological support, and some information on outcomes. The presentation will review the key findings of this scan with respect to the reach, maturity, and components involved in these initiatives. It will also describe early findings from the second phase of the study: selecting and visiting a set of eight case study sites to learn about some promising initiatives in greater detail. These case study sites were selected, based on information collected during the scan, to illustrate the range of technologies, programs, and partnerships that are represented in benefits access efforts. In-depth site visits to observe operations and interview staff in each of these eight sites will provide information on the context in which these efforts operate; their development, implementation, and operations; and the outcomes and costs associated with each effort. Within each effort and across sites, the study will identify factors that contribute to the sustainability, expansion, and replication of these efforts. This study builds on some previous studies that have examined program-specific benefits access initiatives (for example, CHIP and SNAP), but will contribute to the field by bringing together information on a broader spectrum of benefits programs that cross the boundaries of funders and federal agencies to deliver benefits to vulnerable populations. The presentation will be an opportunity to disseminate freshly collected information (work to complete the scan and select case study sites will be completed in March 2011, with site visits occurring in late spring 2011) to an audience outside the group funding and advising the study for the first time.
Assessing the "Rothstein Critique": Do We Really Know When Teacher Value-Added Models are Biased?
Dan Goldhaber (University of Washington) and Duncan Chaplin (Mathematica)
There is a growing body of literature that examines the implications of using value-added models (VAMs) in an attempt to identify causal impacts of schooling inputs, and the contribution that individual teachers make toward student learning (e.g. Ballou et al., 2004; McCaffrey et al., 2004, 2009; Rothstein, 2009; Rothstein, 2010; Todd and Wolpin, 2003). In a provocative and influential paper, "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," Jessie Rothstein (2010) reports that value-added models (VAMs) used to estimate the contribution individual teachers make toward student achievement fail falsification tests, which suggests that VAM teacher effect estimates are themselves biased. In particular, he shows that teachers assigned to students in the future have statistically significant predictive power in predicting current student achievement, a finding that obviously cannot be viewed as causal. Instead, it appears to signal that student-teacher sorting patterns in schools are not fully accounted for by the set of variables typically included in these statistical models, implying a correlation between omitted variables affecting student achievement and teacher assignments. Rothstein uses this finding to form the basis of a falsification test designed to indicate bias in VAM teacher effect estimates of teacher contributions to student learning. Rothstein's finding is significant since there is considerable interest in using VAM teacher effect estimates for policy purposes such as pay for performance (Podgursky and Springer, 2007) or determining which teachers maintain their eligibility to teach after some specified period of time, such as when tenure is granted (Goldhaber and Hansen, 2010a; Gordon et al., 2006; Hanushek, 2009). If VAMs are shown to produce biased teacher effect estimates, it casts considerable doubt upon the notion that they can be used for high-stakes policy purposes. As Rothstein (2010) puts it, his "results indicate that policies based on these VAMs will reward or punish teachers who do not deserve it and fail to reward or punish teachers who do (31)." In this paper we describe conditions under which the Rothstein falsification test suggests bias when none exists and provide results from Monte Carlo simulations that illustrate this point. Specifically, we show that the estimated effects of current teacher effects can be unbiased in a value-added model where the Rothstein test falsifies VAM models, i.e. the estimated impacts of future teachers are statistically significant. This finding is due to endogeneity of future teachers. On the whole, our findings show that the "Rothstein falsification test" is not definitive in showing bias, which suggests a much more encouraging picture for those wishing to use VAM teacher effect estimates for policy purposes. Read the working paper.
Making Use of Unreliable Measures: The Importance of Value-Added for Teachers
Steven Glazerman (Mathematica), Dan Goldhaber (University of Washington), Susanna Loeb (Stanford University), Stephen Raudenbush (University of Chicago), and Douglas Staiger (Dartmouth University)
Value-added measures of teacher performance are controversial. Recent analysis has shown the estimates to have low reliability for making distinctions among teachers (Schochet and Chiang 2010). Others have raised similar concerns because the year-to-year correlation can be in the range of 0.20 to 0.40. We acknowledge that such data confirms that value-added estimates, especially when there is scarce data available for any given teacher, can be noisy. But we go on to argue that even noisy measures can be a vast improvement over current teacher evaluation systems that often fail to make any meaningful distinctions among teachers, who are known to vary widely in their contribution to student learning. We provide a decision-theoretic framework within which even noisily measured value added estimates can be very useful and offer several arguments for incorporating teacher value-added estimates into a larger system of teacher evaluation. We then discuss a method by which policymakers at the federal or state level can assess the adequacy of a value-added system for identifying exceptional teachers. Finally, we summarize a policy proposal for America's Teacher Corps, which capitalizes on the ideas generated in the first part of the presentation. The work presented will summarize three papers by the Brookings Institution Task Force on Teacher Effectiveness and relate them to the panel theme of using imperfect statistical data to make decisions about worker productivity in complex professions.
Designing Physician Incentives When Performance Measurement Is Unreliable: What Can Medicare Learn from Education Reform?
Gregory Peterson and Eric Schone
(Mathematica)
In 2017, Medicare will start paying all physicians differently depending on the value of the care they deliver using a "value-based modifier." For example, a physician who scores as ‘high value' could be paid $105 for stitches, rather than the usual $100. The objective of the value-based modifier is to promote more efficient use of resources, but ensure quality of care by tying reimbursement to objective measures of health care quality. Similar payment methods are increasingly in use by commercial health plans and state Medicaid plans. In this paper, we compare value-based payment strategies used for physicians to performance-pay strategies adopted by some innovative school districts trying to promote more effective teaching. We explore how incentive designs in education have addressed the unreliability of performance measures and what Medicare could learn from these efforts as it designs its program. Our analysis indicates that methodologies used to determine performance pay for teachers in some districts respond to uncertainty in performance measures through both measure and incentive design: (1) Measure precision is increased by setting minimum sample sizes, requiring several years of data for some incentives, using econometric models to decrease noise, and using composite measures. (2) Incentive design differs among districts. Some districts reduce the likelihood of incorrect performance designations by focusing incentives on those whose performance is at the bottom or the top of the distribution. Bonuses and sanctions are then quite large—bonuses can be 10-30 percent of a teacher's annual salary and at least one school district subjects the lowest performers to termination Other districts offer smaller awards that a larger fraction of teachers are likely to attain. Existing physician programs also mitigate imprecision by making use of many different measures. However, most are based on short time frames and implicit or explicit reliability calculations that tolerate low reliability. Incentives are generally small in relation to total payments and offered in the form of a continuous increment or in multiple tiers. We compare incentive designs in education with those in health care from both theoretical and empirical perspectives to identify design features that can provide the strongest incentives for improved performance despite the unreliability of the performance measures. From this analysis, we identify which design features in innovative school districts would be most appropriate for pay for performance by Medicare.
True Alternative Licensure in Teaching: Research on American Board Certification
Steven Glazerman, Christina Clark Tuttle, and Duncan Chaplin (Mathematica)
Alternative certification of teachers involves lowering the barriers to entry in the profession by eliminating initial licensure requirements or replacing them with new requirements considered to be less burdensome. The goal of such policies is to allow career changers and other non-traditional candidates who have unique talents and might otherwise be deterred from doing so to become a teacher. We provide evidence from a four-year study of the American Board for Certification of Teacher Excellence (ABCTE), one of the few true alternative certification programs in the country. ABCTE certifies teacher in several subject areas on the basis of passing written exams and completing a criminal background check. Coursework and mentoring are offered, but not required. The ABCTE credential is a portable credential accepted in nine states. This paper provides survey evidence on the types of people who obtain ABCTE credentials, the reasons for obtaining a credential, and the subsequent career paths of those who obtain the credential. We report on the relative difficulty of the exams that ABCTE uses to certify teachers compared to traditional teacher tests, known as the Praxis II. We also provide evidence on principal ratings of ABCTE teachers' effectiveness and estimate the impact that ABCTE teachers have on student achievement.
Cohort Trends in Employment and Use of Work Incentives for Participants in the Supplemental Security Income Program
Yonatan Ben-Shalom and David Stapleton (Mathematica)
As the number of individuals receiving Supplemental Security Income (SSI) increased over the last decade, the characteristics as well as dynamics of program participation and service use among new beneficiaries changed too. During the same period, several administrative and policy changes occurred that might have affected these dynamics and eventually pathways that adult SSI beneficiaries can take on the way to exit for work. These changes included the introduction of Medicaid Buy-In program starting in the late 1990s, the 1999 increase in the Substantial Gainful Activity earnings amount, the rollout of Ticket to Work from 2002 through 2004, the release of grants for Benefit Planning and Outreach, later replaced by Work Incentive Planning and Assistance, the concerted administrative efforts to reduce post-entitlement backlogs, and others. This paper compares longitudinal statistics on characteristics, program participation, service use, employment and earnings between earlier and more recent adult SSI awardees, and explores potential effects of the above policy changes as well as the economic cycle on any cohort differences we may see. We construct annual files from 1996 through 2006 for each annual award cohort, defined as those who first received an award for SSI as an adult during each calendar year from 1996 to 2005. Most of our data are drawn from SSA's Ticket Research File (TRF08), then separately linked to the Master Earnings File and the Rehabilitation Services Administration's 911 file. Key longitudinal statistics include demographics, benefit amounts, achievement of earnings above SGA, benefit suspension or termination for work, employment service enrollment, and earnings. Results are tracked annually for each cohort following the corresponding award year through 2006, and will be presented by age, gender and state of residence, and compared across cohorts.
Youth Perspectives on Transition to Adulthood
Bonnie O'Day (Mathematica)
The Youth Transition Demonstration is a Social Security Administration-funded project to assist youth with disabilities to make a successful transition to employment, a transition made more difficult by issues unique to this population, such as health, social isolation, service needs, and lack of access to supports. To obtain a perspective that cannot be gleaned from more quantitative data sources and to hear the youths' point of view, we conducted two rounds of focus groups in six demonstration sites. The purpose of these focus groups was to gather information on participants' experiences while participating in the project, including their awareness and utilization of services. We also explored expectations for employment and higher education, experiences finding and keeping work, barriers to meeting goals, perceived gaps in services, and suggestions for ways to improve transition services. To obtain participants, we designated criteria for each group (employed versus unemployed, in-school versus out-of-school, younger versus older) and selected youth who met these criteria at random from program participants. This paper analyzes the results of these focus groups to explore the following questions: What goals and expectations for higher education, employment, and careers do youth with disabilities have upon leaving high school? What do youth perceive are the primary barriers and facilitators that affect goal achievement? What gaps in services exist, and what services are most helpful? What are their expectations about whether they will continue to receive Supplemental Security Income or Social Security Disability Insurance benefits in five to ten years? Focus group participants stated that they wished and expected to work. They viewed work as an integral part of a successful future—because it gave meaning to their lives and provided the income necessary to participate in social activities, make purchases important to them, and in the future to maintain a home and start a family. Some youth were well on their way to achieving these goals, but others faltered because they had no clear employment goal or direction, or did not know what steps they should take to achieve their goal if they had one. Primary barriers were lack of available training, employer flexibility in accommodating their needs, lack of assistance for people with disabilities to succeed in college, lack of transportation, and overprotective parents. Primary facilitators included supportive parents, service agency personnel who helped them find the financial and other resources necessary to reach their goals, and agency staff with employer connections to "get them in the door." Youth stated that benefits counseling was very important to them and their parents; most said they wished to decrease their dependence on benefits or leave the benefit rolls entirely in 5 or 10 years.
Federal Expenditures for Working-Age People with Disabilities in Fiscal Year 2008
Gina Livermore and David Stapleton (Mathematica) and Meghan O'Toole (AmeriCorps/City Year)
A surprisingly large fraction of federal expenditures in the United States go toward providing assistance to working-age people with disabilities. We estimate that in fiscal year (FY) 2008, the federal government spent $357 billion on this population; these expenditures represented 12 percent of all federal outlays and 2.5 percent of the gross domestic product (GDP). Federal expenditures for this population have increased by 56 percent since FY 2002 (the last year for which comparable estimates were produced)—much faster than inflation, GDP growth, or growth in all federal outlays. Most of these expenditures paid for health care and income maintenance. In this paper, we describe the methodology used to calculate these federal expenditures, present findings on total expenditures and 63 specific types of federal expenditures, and conclude with a discussion of the policy implications. The paper updates previous estimates developed by Stapleton and Goodman (2007) in order to examine changes in total expenditures and the composition of expenditures, and to consider the policy implications of the trends observed. Our findings indicate that federal expenditures on income maintenance and health care grew at high rates, and continue to represent the majority of expenditure (95 percent). Expenditures on education, training, and employment services remain a very small percentage of total expenditures and declined slightly, in real terms, from 2002 to 2008. The findings have important policy implications. Most notably, the sheer size of federal expenditures for working-age people with disabilities and the large number of programs that serve them reflect a strong social commitment to providing support for this population. Yet these expenditures represent such a large and growing share of all federal outlays that any serious effort to rein in federal spending will have to consider limiting the growth in expenditures for this population. Federal health care expenditures, which are growing rapidly for all covered groups, are particularly likely to be targeted. Several factors will likely continue to fuel the growth of federal expenditures for working-age people with disabilities in the near future. First, the severe recession appears to have spurred a large increase in SSDI entry, a trend that continued beyond FY 2008; SSDI awards increased by 10.2 percent in FY 2009 and by another 7.9 percent in FY 2010. Second, expenditures for Veterans' programs are expected to continue to grow rapidly because of the rising number of disabled veterans from the wars in Iraq and Afghanistan, combined with aggressive government efforts to meet their needs. Finally, the Affordable Care Act of 2010 is expected to increase federal expenditures for health care in the short term, and a disproportionate but small share of that increase will likely pay for services delivered to working-age people with disabilities The rapid growth in costs for this population, combined with strong political pressure to reduce the federal deficit, has greatly increased the premium on finding ways to make the disability support system more efficient. Failure to do so could lead to funding cuts that cause significant harm to working-age people with disabilities.
Statistical Power for Regression Discontinuity Designs in Education: Empirical Estimates of Design Effects Relative to Randomized Controlled Trials
Lisa Dragoset and John Deke (Mathematica)
The regression discontinuity design (RDD) has the potential to yield findings with causal validity approaching that of the randomized control trial (RCT), as reflected in the decision by the What Works Clearinghouse (WWC) to allow RDD studies the potential to be classified in the same category as RCTs (U.S. Department of Education, 2010). This, combined with nationwide efforts to improve state educational data systems, is likely to lead to an expansion in the use of RDD for identifying the impacts of interventions in education. Researchers planning to conduct prospective studies that rely on RDD will need to choose a sample size that allows them to detect meaningful effects. The "RDD design effect" is defined as the ratio of the variance of the RDD impact to the variance of an RCT impact for a study of the same size. Estimating the RDD design effect enables researchers to know how many additional schools/students an RDD study would need to include in order to detect effects of the same size as one could detect in an RCT with the same outcome variables. Schochet (2008) estimated RDD design effects for clustered RDDs and found that, on average, an RDD study would need to include 3-4 times as many schools/students as an RCT in order to produce impacts with the same level of statistical precision. However, Schochet (2008) did not take into account the use of an optimal bandwidth or functional form, nor the Lee & Card (2008) adjustment for random misspecification error, both of which increase the RDD design effect and are required by the WWC RDD standards. The current paper draws on data from past education RCTs conducted for the Institute of Education Sciences covering a total of 30,000 students in kindergarten to grade 9 in 27 states and 500 schools to calculate empirical estimates of the RDD design effect. We estimate design effects under the null hypothesis of no treatment effect using (1) 23 different test score outcomes, (2) 22 different pre-test scores used as the RD assignment variable, (3) 3 different cutoff values of the assignment variable, (4) the Lee & Card (2008) adjustment for random misspecification error, and (5) both an optimal bandwidth or optimal functional form estimation approach. We find that taking into account the Lee & Card adjustment and the use of an optimal bandwidth or functional form noticeably increases the RDD design effect beyond that reported in Schochet (2008), which was primarily focused on the case of an RDD analysis using a linear functional form and all available data without any adjustment of standard errors for random misspecification error.
Statistical Power for Education RCTs with Binary Outcomes
Peter Schochet (Mathematica)
For randomized control trials (RCTs) of education interventions, there has been a growing literature on methods for assessing appropriate sample sizes for obtaining precise impact estimates for continuous outcomes, and in particular, for student achievement test scores. The literature in the education area, however, is much smaller on methods for calculating statistical power for impacts on binary (0/1) outcomes, such as student dropout or graduation rates, postsecondary school enrollment rates, student proficiency rates in math and reading, and so on. The calculation of statistical power for impacts on binary outcomes is considerably more complex than for continuous outcomes for several reasons. First, nonlinear binary choice models—such as logistic (logit) regression models—are typically used to estimate impacts for binary outcomes rather than linear models. This makes it more difficult to find tractable formulas for the variances of the impact estimates. Furthermore, unlike the linear model, the treatment effect parameter in the logit model (in log odds terms) differs depending on whether or not baseline covariates are included in the model. It is surprising also that the inclusion of baseline covariates in the logit model tends to increase the variance of the estimated treatment effects. Finally, the variances of the impact estimates for binary outcomes depend on the population means. This heteroscedasticity complicates the definition of the intraclass correlation (ICC) for clustered designs. This article develops a tractable approach for calculating appropriate sample sizes for education RCTs with binary outcomes using logit models with and without baseline covariates. The theoretical analysis develops sample size formulas for nonclustered designs, and for clustered designs at the school or teacher level using GEE methods. The article focuses on the impact parameter pertaining to rates and proportions rather than to the log odds of response, which has been the focus of the previous literature (mostly in the medical and public health fields). This focus is warranted because most large-scale RCTs in the education area report intervention effects on binary outcomes as percentage point differences in proportions. Furthermore, a key result proved in the article is that the impact parameter for proportions is the same for logit models with and without baseline covariates. Thus, this article provides a unified framework for examining statistical power for binary outcomes. The article also compiles ICCs for the clustered design for 27 diverse binary outcomes using data from seven education RCTs. Finally, the article conducts a simulation analysis using the power formulas with empirically based parameter values to assess appropriate school sample sizes for RCTs with binary outcomes. The key finding from the simulation analysis is that sample sizes of 40 to 60 schools that are typically included in clustered RCTs for student test score outcomes will often be insufficient for binary outcomes. The key reason is that the potential for precision gains from regression adjustment is likely to be smaller for binary outcomes.
Implementing Expanded Medicaid Eligibility Under ACA: Perspectives from the States
Cheryl Camillo (Mathematica)
The Patient Protection and Affordable Care Act (ACA), as amended by the Health Care and Education Reconciliation Act of 2010, specifies that income eligibility for medical assistance for certain categories of individuals under state plans and waivers be determined using "modified adjusted gross income" (MAGI) for an individual or, if that individual is a member of a family, the income of the household. The federal tax code defines MAGI as adjusted gross income plus tax-exempt interest income and the foreign earned income exclusion. The household includes not only the individual or couple applying for assistance but all of the dependents that they claim for tax purposes. MAGI will be used to determine eligibility for an expansion of medical assistance (to 133 percent of poverty for adults) and for the new exchanges from which persons or families with incomes up to 400 percent of poverty will be able to purchase subsidized health insurance. MAGI will also be used to determine the point at which individuals and families transition between Medicaid as it existed prior to ACA and the new eligibility category. State expenditures for benefits paid under Medicaid are reimbursed by the federal government at a rate defined by the federal medical assistance percentage (FMAP), which varies among the states as a function of state per capita income. When expanded Medicaid eligibility goes into effect in 2014, state expenditures for persons made eligible by ACA will be reimbursed by the federal government at a rate of 100 percent for all states, initially. This new FMAP will decline in stages to 90 percent while the FMAP for expenditures for persons who would have been eligible previously (but based on MAGI in most cases) rises to 90 percent. States will have to submit one set of expenditures for reimbursement at the new FMAP and another set for reimbursement at the "old" rate—much as they do for the Children's Health Insurance Program (CHIP), which reimburses state expenditures using an enhanced FMAP. This paper examines the issues that are likely to confront the states as they implement expanded Medicaid eligibility under ACA. First, the paper will outline the changes to Medicaid eligibility and to the eligibility determination process mandated in the law and discuss their possible implications. Second, the paper will review any relevant regulations that have been issued to date. Third, drawing from discussions with participants in a workshop held in the spring, the paper will pose and answer the following questions: What challenges do states foresee in implementing these elements of ACA? How are states preparing to address these challenges? What systems enhancements are needed to make MAGI operational? What policy choices at the state level could reduce the challenges in implementing and operating a new eligibility system? What lessons can be drawn from the implementation of CHIP? Based on these answers we offer recommendations to the broader community of states.
Impacts of Charter Management Organizations on Student Achievement: Promising Management Structures and Practices
Joshua Haimson, Brian Gill, Josh Furgeson, and Bing-ru Teh
(Mathematica)
Charter management organizations (CMOs), non-profit organizations that operate multiple charter schools, have been expanding over the past decade. These schools are now serving substantial numbers of students, a large fraction of which come from low-income and minority families. CMOs have attracted significant media attention and substantial funding from foundations and federal grant programs, in part because some CMO schools have impressive student outcomes. There is a great deal of variation in CMO management strategies and educational practices. For example, CMOs vary in their size and growth rates; the amount of time they allocate for instruction; how they coach teachers and approach teacher evaluation; and whether they ask teachers to use specific instructional models, formative student assessments, or behavior strategies. The National Study of CMO Effectiveness is the first rigorous national evaluation of CMOs impacts. This presentation will summarize the findings from the study including those related to (1) the range of management strategies and educational practices employed by CMOs, (2) the average impacts and range of impacts of CMO schools on student achievement, and (3) the CMO management strategies and practices that are positively associated with impacts. Our analysis of CMO impacts on student achievement combines experimental methods making use of CMO admission lotteries and quasi-experimental methods using matched comparison groups. To gauge the reliability of the quasi-experimental impact estimates, they are benchmarked against the experimental estimates in the same CMO schools. The quasi-experimental methods are then applied to a larger number of CMO schools including those where it is not possible to implement a randomized experimental design. The presentation will coincide with the release of a report from this comprehensive four-year study conducted by Mathematica and the Center for Re-inventing Public Education at the University of Washington. The study is sponsored by the New Schools Venture Fund and funded by the Bill & Melinda Gates Foundation and the Walton Family Foundation, each of which has approved this presentation.