Exclude

Spring 2024 Newsletter

Anonymous — Mon, 08 Apr 2024 16:24:07 +0000

Spring 2024 Newsletter Anonymous (not verified) Mon, 04/08/2024 - 10:24

What’s New at the Center for Assessment, Design, Research and Evaluation

As of Fall 2023, CADRE now has a physical location within the new Miramontes Baca Education building (4th floor) along the southern boundary of the University of ��ֱ�� campus in Boulder. We have two new Research Associates, Adam York and Kaitlin Nath. One of our longstanding staff members, Jessica Alzen, has been promoted to Senior Research Associate. We are also excited to have a new faculty partner in Oded Gurantz, who joined the School of Education as an assistant professor in the Research and Evaluation Methodology program in the fall of 2022.

As always, CADRE staff have been keeping busy on a variety of projects.

CADRE Director Derek Briggs recently finished a chapter (with co-authors Andy Maul and Josh McGrane) for the 5th edition of the edited volume Educational Measurement entitled “On the Nature of Measurement.” Unfortunately, the book isn’t set to be published until the end of 2024, but a pre-print version of the chapter is available. Derek also enjoyed co-authoring a recently published journal article with CADRE faculty partner Ben Shear entitled “Measurement Issues in Causal Inference.” Another fruitful collaboration has been with CADRE faculty partner Lorrie Shepard and CADRE graduate student researcher Nicolás Buchbinder for a study conducted for the NAEP Validity Studies Panel. The result of this study, a white paper entitled “What Can NAEP Mathematics Subscales and Subscale Weights Tell Us ��ֱ�� Common Core Effects?” will be released later this spring. They can’t wait to share the results. Derek continues his ongoing research on content-referenced growth reporting, described in more detail below.
Associate Director Elena Diaz-Bilello recently co-authored a chapter with Jared Anthony for the upcoming volume by the National Academy of Education (NAEd), The Implementation and Use of Balanced Assessment Systems. This book reconceptualizes balanced assessment systems to support and inform equitable and ambitious teaching and learning. She is also writing a commentary for a set of essays addressing the implications of culturally responsive assessment for classroom assessment practices. Her commentary will be published in a new volume, Culturally Responsive Assessment in Classrooms and Large-Scale Contexts: Theory, Research, and Practice, to be released by the National Council on Measurement in Education (NCME). Elena currently collaborates with the Hawaiʻi Department of Education, the National Center for the Improvement of Educational Assessment, and the Learning Policy Institute on the statewide Performance Assessment Development Initiative (PADI) in Hawaiʻi, while continuing her dedicated partnership work with the ��ֱ�� Department of Education and several school districts in ��ֱ��.
Senior Research Associate Jessica Alzen finishes up two multi-year projects this year. The first is a collaboration with CADRE faculty partner, William Penuel, as well as colleagues from Northwestern University and University of California Davis. The project focuses on teacher learning regarding the Next Generation Science Standards and the pedagogical shifts necessary to fully enact curriculum aligned to those standards. The most recent paper from that collaboration discusses different ways teachers ask students to work together and how those choices can influence students’ opportunities to build science knowledge. The second project that ends this year is a collaboration with Eric Vance from the applied math department at University of ��ֱ�� Boulder. A recent paper from that project considers the extent to which the Laboratory for Interdisciplinary Statistical Analysis at the university trains students to be effective interdisciplinary statistical collaborators.
Postdoctoral Research Associate Kaitlin Nath continues three projects in partnership with the ��ֱ�� Department of Education (CDE) and local school districts. The first is a collaboration with the CDE and two districts to understand the use of performance-based assessments (PBA) and capstone projects to evaluate graduation readiness. The project focuses on the extent to which information from PBAs aligns with information from other graduation menu options and whether PBAs support equity goals. The second is a continued evaluation of a pilot grant that gives money to schools to hire additional mental health professionals and provide mental health services to students. The third is ongoing work with ��ֱ��’s School Transformation Network, which provides targeted supports to low-performing schools across the state. So far, Kaitlin’s work with this department has included evaluating the effect of ��ֱ�� turnaround programs on student outcomes and conducting case studies to understand how schools engage in turnaround work. Within the University of ��ֱ�� Boulder, Kaitlin continues work on the “CODE:SWITCH” project, which aims to train undergraduate students in the intersection of statistics and humanities. Kaitlin looks forward to continued collaboration with state partners to evaluate education initiatives.
Research Associate Adam York continues work on partnerships with the ��ֱ�� Department of Education (CDE). One is a continuation of the evaluation of the state’s pilot K-5 Social Emotional Health grants, which funds schools to add mental health professionals that deliver social and emotional supports for students. Another is ongoing work with CDE’s Adult Education Initiatives, exploring programmatic features that support adult basic education and family literacy in ��ֱ��. Adam recently completed a policy memo for the National Education Policy Center, titled How the Practices of Schools of Opportunity Illustrate Recent Research on Learning. The paper describes one school that detracked their curriculum, and two schools that developed culturally sustaining curriculum, noting the intersections with research on learning and implications for school leaders and policymakers.
Faculty Partner Oded Gurantz is primarily partnering with the California Student Aid Commission (CSAC) on a number of studies examining how postsecondary policies impact family’s financial health. He is also partnering with a number of federal agencies as part of his position as a Fellow with the federal Office of Evaluation Sciences. One project is building off a prior relationship with the U.S. Department of Education, where they are examining the causal effects of the federal Return of Title IV Funds policy, which can require students who withdraw early in an academic term to return some of the grant aid to which they were entitled. This policy was the previous focus of a descriptive paper for the Institute of Education Sciences. Oded is also engaged in a randomized control trial project currently in the field, which is studying whether small encouragements from the US Citizenship and Immigration Services office induce permanent residents to file for naturalization as U.S. citizens. The description of this project is pre-registered at https://oes.gsa.gov/projects/increasing-naturalization/.
Faculty Partner Ben Shear continues his work aimed at improving the uses and evaluation of standardized tests with various partners. In 2023 his journal article "Causal inference and COVID: Contrasting methods for evaluating pandemic impacts using state assessments" examining the use of state assessment data to study pandemic impacts was published and was also the basis for a Provost’s Faculty Achievement Award at the University of ��ֱ��. This project developed from a long-standing partnership with the ��ֱ�� Department of Education Accountability Unit and complements his recent co-authored article with Derek Briggs mentioned above. As a member of the Stanford Education Data Archive (SEDA) Development Team he contributed methodological expertise to support the creation of two new publicly available SEDA data products published in 2024 - SEDA2023 and SEDA 5.0. In June 2024 Ben will present new work examining the potential to incorporate matrix sampling techniques into state assessment programs for accountability and system monitoring purposes at NCSA with colleagues from the National Center for the Improvement of Educational Assessment and multiple state departments of education. In other news, Ben celebrated the birth of his first child in January 2024 and looks forward to the adventures ahead and analyzing a new type of growth percentile data.
Faculty Partner Lorrie Shepardpublished a chapter, “Authentic Assessment Embedded in Project-Based Learning,” with Jeff Palladino, principal of Fannie Lou Hamer Freedom High School. The chapter appears in NEPC’s new book, Schools of Opportunity: 10 Research-Based Models of Equity in Action, edited by Adam York, Kevin Welner, and Linda Molner Kelley. Lorrie is awaiting the publication of her chapter on “Standards-Based Reform and Test-Based School Accountability,” coauthored with Scott Marion and Christopher Saldaña submitted over a year ago for publication in AERA’s Handbook of Education Policy Research (Vol. 2). As a member of the steering committee for an on-time volume, Lorrie will join editor Scott Marion in presenting an overview to NCME of the National Academy of Education’s new volume, Toward Balanced Assessment Systems: Considerations for Implementation and Use. Lorrie is also concluding her term (plus extra Covid years) as chair of the Professional Development Committee for the National Academy of Education. A particularly rewarding aspect of this role has been overseeing mentoring and professional development retreats for NAEd/Spencer dissertation and postdoctoral fellows.

Featured studies

Content-Referenced Reporting of Student Growth

By Derek Briggs

��ֱ�� Multi-Tiered System of Supports Partnership

By Jessica Alzen, Elena Diaz-Bilello

K-5 SEL Pilot Grant Evaluation

By Adam York, Kaitlin Nath, Elena Diaz-Bilello

Studying the role of postsecondary policies in shaping families’ financial health

By Oded Gurantz

Off

Traditional

White

Content-Referenced Reporting of Student Growth

Anonymous — Thu, 28 Mar 2024 18:39:56 +0000

Content-Referenced Reporting of Student Growth Anonymous (not verified) Thu, 03/28/2024 - 12:39 Derek Briggs

Over the past four years, CADRE has been involved in a Research-Practice Partnership with assessment specialists from the organization Curriculum Associates on a project that is exploring innovations in the presentation and interpretation of student growth from large-scale educational assessments. CADRE’s central innovation in this space is what we have come to describe as taking a “content-referenced growth reporting” approach, in contradistinction from the two more predominant approach to thinking about student growth (norm-referenced and criterion-referenced growth). The idea, in a nutshell, is to shift attention from growth interpretations that emphasize “points earned” to growth interpretations that focus on conceptual changes in student thinking, changes that can be characterized in concrete terms by the content of the assessment in the form of exemplar items.

The embodiment of this idea is found in a prototype we have developed for an interactive digital score reporting interface. CRG reporting is intended to help teachers understand the changes in thinking that occur as students advance in their learning of different skills and standards across content areas. The goal is for teachers to identify common ways that students think about problems, and to connect changes in this thinking to the growth a student has made in assessment scores.

The reporting interface is designed to accomplish this by giving users the opportunity to visualize and interpret student growth across multiple assessment occasions, using a score scale whose interpretation is facilitated by qualitatively distinct reference locations. Importantly, we also want the reporting interface to be informed by cognitive principles for the effective representation of quantitative data, and by the intended answers—as well as new questions—that the interface should provide and elicit about student growth.

To get the gist of what we are trying to accomplish, we invite the reader to interact with version 1.0 of CADRE’s digital reporting prototype which can be accessed at https://remcu.github.io/crg-prototype/.

Two of the major inspirations for the CRG approach come from item mapping, as instantiated in the National Assessment of Educational Progress, and the research literature on learning progressions (e.g., Clements & Sarama, 2004; Alonzo & Gotwals, 2012;). Other related explications of the ideas behind CRG can be found in Briggs & Peck (2015), Briggs (2019) and Wilson (2023).
Figure 1. Theory of Action Behind CRG Reporting

A high-level theory of action for CRG reporting is outlined in Figure 1. The top of the diagram starts with the end goal which is to facilitate improved learning outcomes for students. This is accomplished through teachers adjusting their instruction based upon the inferences about growth and learning they can make from using the CRG reporting interface. There are two secondary goals of the prototype; 1) to contribute to teachers’ professional learning, and 2) to improve attitudes about the usefulness of assessment. We believe that as teachers attempt to connect student growth to changes along an underlying learning progression, they will also be engaging in a process of professional learning that deepens their content area expertise. Similarly, as teachers learn—with support—to interact with the reporting interface, we hope they develop an improved relationship with assessment and data use. In the Figure 1 diagram, the foundation for the CRG reporting interface depicted by four building blocks: a domain-specific learning progression, targeted test design, a scale established using item response theory (i.e., the Rasch model), and items mapped from the scale to levels of the learning progression.

To date we have developed two different prototypes for a CRG reporting interface specific to learning progressions for the core concepts of fractions and phonics knowledge within the content domains of mathematics and reading. Reports on learning progression development and validation in these two areas can be found in Wellberg, Briggs & Student (2023) and Cox & Briggs (2023). Additional prototypes are in development in mathematics for algebraic functions and geometric measurement.

We have engaged in two rounds of interviews with teachers in 2022 and 2023 as they have interacted with the CRG protoypes for fractions and phonics. The results from these interviews have helped us to refine our theory of action (e.g., the details of what we expect in the arrows from reporting interface, educator interaction and inferences about learning and growth shown in Figure 1) and have motivated ongoing revisions that will lead to version 2.0 of the CRG reporting protype (currently in progress). For reports on these interviews and lessons learned, see Briggs et al. (2023) and Buchbinder et al. (2023).

A full manuscript describing the CRG approach is currently in progress with plans to submit for peer-review by July 2024. In the meantime, we will be presenting this work as part of a symposium at the annual meeting of the National Council on Measurement in Education in Philadelphia on Friday April 12 (“Challenges and Innovations in Creating Interactive Reports of Student Progress and Growth”) and also at the annual National Conference on Student Assessment in Seattle in June.

The CADRE PI for the CRG research project is Derek Briggs. Major contributions have come from CADRE graduate student researchers (in alphabetical order) Nicolas Buchbinder, Olivia Cox, Kyla McClure, Sarah Wellberg and Erik Whitfield.

References

Briggs, D. C. & Peck, F. A. (2015). Using learning progressions to design vertical scales that support coherent inferences about student growth. Measurement: Interdisciplinary Research & Perspectives, 13, 75-99.

Briggs, D. C. (2019). Interpreting and visualizing the unit of measurement in the Rasch Model. Measurement, 46 (2019) 961–971. https://doi.org/10.1016/j.measurement.2019.07.035

Clements, D. H., & Sarama, J. (2004). Learning Trajectories in Mathematics Education. Mathematical Thinking and Learning, 6(2), 81–89. https://doi.org/10.1207/s15327833mtl0602_1

Alonzo, A. & Gotwals, A. (2012). Learning progressions in science: current challenges and future directions (2012). SensePublishers.

Wilson, M. (2023). Constructing measures : an item response modeling approach. Routledge.

Off

Traditional

White

Studying the role of postsecondary policies in shaping families’ financial health

Anonymous — Thu, 28 Mar 2024 18:28:06 +0000

Studying the role of postsecondary policies in shaping families’ financial health Anonymous (not verified) Thu, 03/28/2024 - 12:28 Oded Gurantz

Background

A college degree provides substantial benefits, including lower unemployment, higher wages, and increased social mobility. However, as the price of a degree has risen, families increasingly use debt to finance a college education, with parents shouldering a larger share of the burden. For example, originations of federal “Parent PLUS” loans rose from $2.2 billion in 1991-1992 to $15.2 billion in 2017-2018 (Ma and Pender 2023). Parent PLUS loans are just one of many ways in which a child’s college decisions affect parents, leading to articles that declare parents as the “Hidden Causalities of the Student Debt Crisis” (Granville 2022).

Motivated by these trends, Dr. Gurantz has begun to engage in a number of projects that look at how attending college affects family finances. Surprisingly there is very little research on this topic, in large part as there is rarely the ability to observe parent-student linked data at any reasonable scale. As a result, prior studies typically focus on how attending college impacts just the finances of the individual student (e.g., Scott-Clayton & Zafar, 2019), which paints an incomplete picture of how the costs of higher education are affecting society.

With funding from the Spencer Foundation and Arnold Ventures, we have begun to answer these questions, with a research team that includes Palaash Bhargava and Dr. Sandra Black from Columbia University, Dr. Jeffrey Denning from Notre Dame, and Dr. Robert Fairlie from UCLA. These results rely on a newly-available, confidential, restricted-access administrative dataset that captures the universe of Free Application for Federal Student Aid (FAFSA) filers in California, which is then linked at the individual-level to detailed credit data records from a large credit bureau. These FAFSA data have an innovative feature: for students who are still dependents – which is most 18 and 19 year olds but can continue as long as students are not yet 26 – we can link applicants to their parents and consider how the child’s college experience affects their parent’s financial health. Although credit data do not capture many important outcomes, we can observe debt balances and default rates on a variety of loan types, such as educational loans, credit cards, and other forms of credit.
Impacts of state grant aid on financial health

The first project using these data focuses on how families react to changes in the price of college as a result of receiving state aid; we anticipate releasing this working paper in 2024. In our data we can observe families who just met the criteria to become eligible for state aid receipt and compare to them to essentially identical families who are just ineligible for state aid. As a result, these luckier families receive about $17,500 in additional aid over the following six years, which induces their children to be slightly more likely to enroll in a broad access, four-year California State University instead of a two-year community college. Even though state grant aid induces a small increase in more expensive, four-year college enrollment, we find that parents reduce the educational loan balances they take out on behalf of their students by about 10%. We also find reductions in HELOC balances (Home Equity Lines of Credit). These reductions vary by household wealth, with families that have mortgages showing larger reductions in HELOCs, whereas families who do not own a home having larger reductions in educational loan balances. Grant aid also reduces the chance that parents are delinquent on their debt but only for parents with a prior history of delinquency, with no effect on parents without prior delinquencies.

Interestingly, we do not find any evidence that receiving grant aid reduces student borrowing. This project demonstrates that, at least in this case, focusing just on student outcomes would have missed all of the positive financial effects of aid receipt on families.

Future Work

The paper described above is just the first in a sequence of topics that we hope to investigate over the coming years. A second paper, just underway, will examine the complex role of federal Parent PLUS loans. These loans are only available to parents if their child has exhausted their own available federal credit and have become an increasingly important source of financial support. There has been considerable controversy over the benefits of Parent PLUS loans and who should have access to them. As with other educational loans, the primary benefit is that they could enable students to attend college who otherwise could not. However, parents may become overextended by these sizeable loans, which could create adverse financial consequences if it causes them to miss mortgage, credit card, or other necessary payments.

For most of recent history, PLUS loans could be originated through the Department of Education’s Direct Loan (DL) program or the Federal Family Education Loan (FFEL) program, which were bank-issued loans backed by the federal government. However, after the federal government became the sole provider of federal loans via the Direct Loan Program, they chose to synthesize policies and tighten loan standards, which had the effect of denying PLUS access to some families with adverse credit histories. This change led to significant public outcry, particularly from HB��ֱ��s and other colleges that enrolled more Black students who were negatively impacted by these adverse credit history standards (Stratford 2014). As a result, the federal government changed the eligibility criteria yet again in 2014, thus increasing access for a subset of parents.

Using the same data listed above – FAFSA applicants linked to their credit histories – we will examine whether changes in access to Parent PLUS loans impacted whether students attended college. More importantly, we can examine whether access to Parent PLUS loans changed how parents finance their child’s education, potentially forcing them into riskier types of credit with higher interest payments, such as credit cards. Thus we will determine whether losing or gaining access to this program had financial implications for these families with worse credit histories.

Conclusion

We hope the two papers described above are just the beginning of an avenue for producing policy relevant research that touches on some of the most important factors facing higher education today.

Off

Traditional

White

��ֱ�� Multi-Tiered System of Supports Partnership

Anonymous — Wed, 27 Mar 2024 16:27:49 +0000

��ֱ�� Multi-Tiered System of Supports Partnership Anonymous (not verified) Wed, 03/27/2024 - 10:27 Jessica Alzen , Elena Diaz-Bilello

The ��ֱ�� Department of Education’s (CDE) Office of Learning Supports contracted with CADRE to support CDE’s efforts to improve school and district systems at the organizational level with the ��ֱ�� Multi-Tiered System of Supports (COMTSS). COMTSS is a framework for considering prevention-based approaches to improving student outcomes. The framework has five key components: (1) team-driven shared leadership; (2) data-based problem solving and decision-making; (3) family, school, and community partnerships; (4) comprehensive screening and assessment systems; and (5) a layered continuum of supports.

Districts sign up to receive professional learning from CDE using the COMTSS framework. Staff members from the Office of Learning Supports work with districts to identify priority improvement areas around academic outcomes (e.g., results from standardized test scores or locally developed assessments) and/or behavioral outcomes (e.g., out-of-school suspensions, and chronic absenteeism). The CDE staff then provide professional learning to districts so that districts then have capacity and resources to provide professional learning to schools in designing interventions aimed at improving those outcomes.

CADRE’s role as the external evaluator is to serve as a thought partner to the Office of Learning Supports as their team makes decisions about professional learning implementation and how to evaluate effectiveness. Key indicators of organizational growth toward addressing priority improvement areas include measures of organizational capacity to conduct this work, a school fidelity tool aligned to the five components of the COMTSS framework, and course completion data from the Online Academy, a platform for professional development courses aligned to the COMTSS framework.

Partnerships like this one with CDE’s Office of Learning Supports is one way CADRE fulfills its mission to collaborate with state organizations and engage in research designed to make a positive difference in schools and communities.

Off

Traditional

White

K-5 SEL Pilot Grant Evaluation

Anonymous — Wed, 27 Mar 2024 16:25:36 +0000

K-5 SEL Pilot Grant Evaluation Anonymous (not verified) Wed, 03/27/2024 - 10:25 Adam York , Kaitlin Nath , Elena Diaz-Bilello

In 2022-2024 CADRE researchers partnered with the ��ֱ�� Department of Education to evaluate a state grant pilot program. This program emerged from ��ֱ�� HB 19-1017, which provided pilot K-5 Social-Emotional Learning (SEL) grants to 14 schools. One central purpose of these grants was to ensure that school health professionals (SHPs) at each elementary school could provide SEL support to all students, while also identifying and matching those with more severe behavioral and mental health needs with community partners to assist them and their families outside of school. Another purpose was to reduce demands on teachers who are often the first ones called upon to provide social-emotional, behavioral, and mental supports to students. The hope is that support provided to students by SHPs hired through this grant will reduce the number of students requiring more intensive interventions.

The schools that participated in this study are located in seven distinct geographical regions across the state of ��ֱ��. Twelve of the 14 schools have a Title 1 designation because over 40% of their student body is eligible for Free and Reduced Lunch. While this grant was intended to be implemented in schools beginning in the 2020-21 school year, the launch of these grants was delayed until the 2021-22 school year due to the COVID-19 pandemic.

To evaluate the effectiveness of the K-5 SEL pilot and to chart the ongoing SEL work taking place at all 14 sites, we collected and analyzed data for this report using a combination of qualitative and quantitative methods. We used data collected from the 14 schools to track the extent to which the pilot grants met their intended objectives. These data included surveys of teachers and SHPs, mental health systems assessments, and performance measures that tracked services provided for students. Within the initial 14 schools in the pilot, we conducted an exploratory case study in 2022-24 at four schools. Our goal was to gain an understanding of the context for the implementation work underway. We consulted with CDE staff to select four schools with distinct characteristics and levels of SEL program implementation so that we could learn about the common and unique ways in which SHPs provided students with SEL, as well as other mental and behavioral supports. At these four schools we conducted interviews with SHPs and principals, focus groups with teachers, and observations of both whole class and small group SEL activities.

Key findings from this report include:

SHP hires supported by this grant have lowered the service ratio of mental health professionals to students, and this appears to have increased each pilot site’s capacity to respond to students in crisis and to provide more personalized interventions. Compared to the first year of the grant (2021-22) in which several schools indicated a need to broaden the levels or tiers of behavioral support provided to students, all pilot schools reported in 2022-23 that mental health practices were implemented.
Although the COVID-19 pandemic is no longer creating massive disruptions (i.e., closures), a common theme running through selected case study sites is that although teachers indicated spending less time on behavioral management in year 2 compared to year 1 of the grant, behavioral disruptions appear to be persistent.

The study results point to positive developments achieved by grantees, but also indicate challenges for sustaining this work over time. The grants increased the capacity of schools to address student emotional needs by hiring new SHP or expanding existing SHP roles; this is encouraging. However, personnel at case study sites and the secondary data analyzed both indicate that further resources and support are needed to help mitigate the persisting and growing behavioral and mental health needs at all 14 elementary schools.

This pilot was initiated prior to the COVID-19 pandemic, in response to a reported trend of increased mental-health and behavioral needs in elementary schools. The massive disruptions and closures occasioned by the COVID-19 exacerbated this trend in the schools involved in this study, and likely many others across the state. Negative ramifications from pandemic-related disruptions linger at all 14 schools in this study, indicating the need to continue attending to the behavioral and mental needs of elementary students.

Our work continues in 2024 as we complete the final report on the pilot program. In addition to ongoing review of secondary data across all of the pilot schools, our case study analysis will explore themes related to improvement to SEL practices and school-wide coordination, the expanded use of SEL screeners in some schools, and communication with families by SEL specialists.

Off

Traditional

White

Spring 2022 Newsletter

Anonymous — Tue, 19 Apr 2022 00:21:07 +0000

Spring 2022 Newsletter Anonymous (not verified) Mon, 04/18/2022 - 18:21

What’s New at the Center for Assessment, Design, Research and Evaluation

Although the CADRE Newsletter went on something of a hiatus during the COVID-19 pandemic, our team of researchers has remained active throughout.

CADRE Director Derek Briggs recently completed a book he had been researching and writing over a five year period entitled Historical and Conceptual Foundations of Measurement in the Human Sciences (Routledge, 2021). Since 2020, he has been leading a project in collaboration with the organization Curriculum Associates with the goal of enacting improvements and refinements to the methods used to model and report information about student growth. The collaboration includes two ��ֱ�� graduate students (Sarah Wellberg and Sandy Student) and a team of Curriculum Associates leadership and staff (Laurie Davis, Kristen Huff, Amanda Brice, Logan Rome and Dan Mix). They will be presenting the progress they have made on an approach they call “content-referenced growth” at both the NCME conference and CCSSO’s National Conference on Student Assessment. In this newsletter, Derek shares some insights motivated by his team’s work with Curriculum Associates on the topic of summer learning loss. Specifically, Derek illustrates how the statistical artifact of regression to the mean can lead to problematic policy recommendations about summer learning interventions.

Associate Director Elena Diaz-Bilello has kept the CADRE ship sailing along while also leading a number of projects with ��ֱ�� school districts and the ��ֱ�� Department of Education. In this newsletter Elena shares some insights from an ongoing project with Denver Public Schools that focuses on the evaluation of learning experiences for students in the Arts throughout the district.

Research Associate Jessica Alzen has been working with CADRE Faculty Affiliate Bill Penuel along with collaborators at Northwestern University and University of California Davis on a “Sensemakers Project” to identify variations in teachers' knowledge, beliefs, and practices as they enact science curriculum designed to support the Next Generation Science Standards. In this newsletter, Jessica shares a reflection on the role that student identity and belonging play in helping students to think and act like scientists.

Faculty Partner Ben Shear has been involved with ongoing projects for the ��ֱ�� Department of Education’s Accountability and School & District Transformation Units. Ben has been actively involved in using ��ֱ�� test data to evaluate the effect of the pandemic on student learning. He has a recently completed manuscript that provides guidance and caveats to researchers using test data with the intent of drawing causal inferences that we look forward to sharing soon. In this newsletter, Ben calls attention to an important methodological issue that needs to be taken into consideration when tracking annual changes in student enrollment in K-12 public schools.

Finally, we want to highlight one in a long list of accomplishments for Faculty Partner, Lorrie Shepard. Lorrie and co-authors Chris Saldaña and Scott Marion have a forthcoming chapter in the second volume of the AERA Handbook of Education Policy Research entitled, “Standards-Based Reform and School Accountability.” After spending many years leading and advising the Research and Evaluation Methodology (REM) program at the School of Education, Lorrie will be retiring from teaching at the end of this school year. She will, however, stay connected to us and we look forward to the continued guidance and wisdom she provides to CADRE, REM, and the larger School of Education and ��ֱ�� Boulder community.

Featured projects

Gain Scores and the Regression Fallacy

By Derek Briggs

In an article for Phi Delta Kappan entitled “Rethinking summer slide: the more you gain, the more you lose” Kuhfeld (2019) uses NWEA MAP test data to show that students with the largest test score gain from fall to spring of an academic school year are those likely to have the largest score declines from spring to the fall of a subsequent school year...

Examining the Question of Access to Arts Programming

By Elena Diaz-Bilello

Much has been written about the importance of the Arts, particularly with respect to how exposure to arts programming can meet social emotional learning needs of students and enhance their well-being. For this reason, education researchers highlight the importance of providing access to these disciplinary areas and decry how the Arts tend to be the first areas to be cut at schools and districts facing budget constraints...

Considering Student Identity and Belonging

By Jessica Alzen

Measuring Public School Enrollment Changes During the COVID-19 Pandemic

By Benjamin Shear

Conferences and NCME Presidential Address

All of us at CADRE have been busy this Spring preparing to share our work at NARST, NCME, AERA and NCSA. We look forward to reconnecting with you at these annual conferences!

This year at NCME is especially important to us since CADRE’s Derek Briggs closes out his one year term as 2021-22 NCME President. We look forward to seeing many of our partners and colleagues at Derek’s NCME Presidential Address in San Diego on Saturday, April 23 at 4:40 p.m. (PT). For long-time NCME members please note the change in schedule, as the NCME Presidential Address will take place on Saturday evening, rather than at breakfast as in past years. The Presidential Address will take place in the California Ballroom at the Westin San Diego Gaslamp Quarter and will also be held as a hybrid event, so anyone who registers for the NCME conference can live-stream the event even if they aren’t in San Diego. For those attending the event in-person, please join us for the Presidential Reception that will follow the address at 6:30 p.m. (PT) at the Garden Terrace.

Off

Traditional

White

Measuring Public School Enrollment Changes During the COVID-19 Pandemic

Anonymous — Fri, 15 Apr 2022 19:05:07 +0000

Measuring Public School Enrollment Changes During the COVID-19 Pandemic Anonymous (not verified) Fri, 04/15/2022 - 13:05

Benjamin Shear

Introduction

The COVID-19 pandemic led to historic changes in public school enrollment in the US. There has been considerable interest in quantifying the magnitude of these changes, and in understanding their causes and consequences (e.g., Dee & Murphy, 2021). An analysis of enrollment data across all 50 states and the District of Columbia by EdWeek documented a nearly 3% decline in public school enrollment in the 2020-21 academic year (AY), relative to 2019-20 AY enrollments (Pendharkar, 2022). Although enrollments declined in all states, the declines tended to be larger in lower grades; kindergarten enrollment declined by more than 10% in 20 states, for example, which was often larger than the declines in other grades. In ��ֱ��, for example, while total preschool through 12th grade enrollment declined by about 3.3%, kindergarten enrollment declined by about 9.1% (��ֱ�� Department of Education, 2021).

Tracking changes to public school enrollments is critical to understanding the impacts of the pandemic and planning to support schools and students moving forward. Studying the within-grade changes in enrollment, however, can potentially be misleading when comparing the changes across different grade levels, because these statistics confound changes due to pandemic-related factors with pre-existing trends in changing cohort sizes over time.

This brief describes two different ways that grade level enrollment changes can be calculated. The first is the within-grade change in enrollments, which has been most commonly reported. The second is the cohort-adjusted change in enrollment that calculates changes to grade level enrollment adjusting for prior grade enrollment. State level enrollment data from two states are used to illustrate the differences between these two methods. The brief also highlights the importance of comparing changes to pre-pandemic trends and shows how comparing enrollment change patterns across states, particularly with the cohort-adjusted metric, can document variation in cross-state enrollment dynamics that may provide opportunity for future studies.

Data

The analyses in this brief are based on publicly reported state level enrollment data (the fall census counts of enrolled students) in ��ֱ�� and Massachusetts, as reported by the ��ֱ�� Department of Education (CDE) and the Massachusetts Department of Elementary and Secondary Education (MDESE) for the 2015-16 through 2021-22 AY¹.

Two Methods for Calculating Grade Level Enrollment Change

Although ��ֱ�� public school enrollment declined drastically in 2020-21, these changes were not consistently distributed throughout the state or across grade levels. Declines in enrollment were much larger in the lower elementary grades. These changes have primarily been tracked by comparing the grade level enrollment in the 2020-21 AY to the grade level enrollments in the prior 2019-20 AY. These changes in enrollment are what I will refer to as “within-grade” changes, and are calculated as:

where is the enrollment in grade g in year y, and is the enrollment in grade g in the prior year². Figure 1 plots the within-grade percent changes in enrollment from kindergarten through 12^th grade in the 2020-21 AY in ��ֱ��. As reported, the largest decline was observed in kindergarten. The figure also illustrates within-grade declines were larger in elementary and middle school grades, with either very small declines or in some cases positive changes in enrollment in high school. But the pattern is not entirely easy to explain. Aside from kindergarten, the largest decline was observed in 5^th grade, with slightly smaller declines in other elementary grades. The smallest decline in elementary school was in first grade, which seems counterintuitive given the much larger decline in kindergarten enrollment.

Figure 1. Within-Grade Percent Change in Grade Level Enrollments in CO (2020-21).

These counterintuitive results may occur at least in part because the within-grade changes in enrollment confound year specific changes in enrollment with changes to the size of ��ֱ�� public school student cohorts. To illustrate this, Table 1 presents 4^th and 5^th grade enrollment counts from 2019-20 and 2020-21³. The gray cells highlight an intact “cohort” – students who were enrolled in 4^th grade in 2019-20 and 5^th grade in 2020-21 as an example. Since the 2015-16 AY, the size of the incoming first grade cohorts in ��ֱ�� have been steadily declining, by as much as 3% in some years. The relatively large decline in 5^th and 6^th grade enrollment seen in Figure 1 were due in part to the fact that these cohorts of students were already smaller than the cohorts ahead of them. The within-grade change in enrollment for 5^th grade in 2020-21 can be calculated as:

Table 1 also shows that in 2019-20, prior to the pandemic, the cohort of students who would move on to 5^th grade in 2021 was 66,172, substantially smaller than the cohort of 68,592 students ahead of them. Even if all 66,172 of the 2019-20 4^th graders had progressed to 5^th grade in 2020-21, the within-grade decline in enrollment would have been about -3.5%, similar in magnitude to the average changes across elementary grades in Figure 1. This occurs because the within-grade change in enrollment is sensitive both to students disenrolling from school and to differences in the size of successive cohorts.

Table 1. Grades 4 and 5 Enrollment Counts in ��ֱ�� in 2019-20 and 2020-21.

Calculating the within-grade change to enrollment is important and is certainly in no way “incorrect.” For funding and planning purposes, it is often necessary to know, for example, how many kindergarten or first grade students are enrolled in the current year, and how this compares to enrollments the prior year. However, to understand the dynamics that cause changes to enrollments or to track trends over time, this within-grade change in enrollment can potentially be misleading.

A better metric for measuring changes to grade level enrollment for some purposes is the “cohort-adjusted” change in enrollment. The cohort-adjusted change in enrollment compares the grade level enrollment in a given grade and year relative to the number of students enrolled in the prior grade in the prior year. That is, it compares the relative change in the size of a particular cohort as they progress from one grade to the next. This metric can be calculated as:

In this example, we would compare the number of 5^th grade students in 2020-21 to the number of 4^th grade students in 2019-20, to calculate a cohort-adjusted change in 5^th grade enrollment of:

Enrollment in 5^th grade did decline relative to what we would have expected if all 4^th graders from 2019-20 had progressed one grade and returned to public school, but the decline is significantly smaller than the within-grade change suggests.

Figure 2 shows the cohort-adjusted changes to grade level enrollments in ��ֱ�� in 2020-21. Note that because preschool is not universal in ��ֱ��, public preschool enrollments tend to be inconsistent and substantially smaller than kindergarten enrollments. The cohort-adjusted enrollment change is calculated starting in first grade, which uses kindergarten enrollments in the prior year in the calculation.

Figure 2. Cohort-Adjusted Percent Change in Grade Level Enrollments in CO (2020-21).

Although the cohort-adjusted changes tell a similar overall story (larger declines to enrollment in lower grades) there are three important substantive differences that the cohort-adjusted metric highlights. First, there is now a clear trend across the elementary and middle school grades that is straightforward to explain – it appears that parents were most often holding their youngest children out of school, and the proportion of families doing so became steadily smaller from 1st through 8^th grade. This pattern was not obvious in the within-grade changes that confounded enrollment changes due to the pandemic with changes in overall cohort sizes over time. Second, there were noticeable increases in enrollments in 9^th and 12^th grade, relative to the number of 8^th and 11^th graders enrolled in school in the 2019-20 AY. The increase in cohort-adjusted 12^th grade enrollment was nearly 6.5%, or about 4,350 students. Finally, while the within-grade changes in enrollment suggested either very small declines or increases in 10^th and 11^th grade enrollments relative to 2019-20, the cohort-adjusted rates suggest fairly sizable declines in 10^th and 11^th grade enrollments relative to the number of students in 9^th and 10^th grade in 2019-20.

Because both metrics are based on aggregate enrollment counts rather than longitudinal student-level data, neither metric can directly quantify the number of students leaving and entering the public school system. Both metrics are affected by students who repeat a grade and by students who enter the public school system (for example if they move into the state), in addition to students who leave the school system. As an example, the count of 64,791 5^th graders in 2020-21 will include students who are new to the state (and were not enrolled in 4^th grade in 2019), students who skipped 4^th grade, and students who remained in 5^th grade in both 2019-20 and 2020-21. Each metric adjusts for (and is sensitive to) different regularities and trends over time. The within-grade enrollment change adjusts for changes in grade level enrollment caused by structural features of the school system that are constant over time. If many students enter the public school system in 9^th grade each year after enrolling in private school in K-8, for example, the within-grade change metric will “adjust” for this phenomenon by comparing enrollments to the same grade in the prior year. The cohort-adjusted metric, however, will be affected by this because it will lead to an increase in the cohort enrollment relative to 8^th grade. The opposite is true for changes in the size of cohorts over time, which the cohort-adjusted metric adjusts for, and the within-grade metric does not.

Analyzing the changes in grade level enrollments raises a natural question: to what extent are these changes unique to the pandemic in 2020-21, and to what extent do these reflect stable patterns of school enrollment dynamics? Comparing the changes in Figure 1 and Figure 2 to historical trends can provide a more complete understanding of the changes that occurred in 2020-21 and any potential re-enrollments in 2021-22.

Putting Values in Historical Context

Figure 3 shows the within-grade and cohort-adjusted changes in enrollment in ��ֱ�� for grades 1-12 from 2015-16 through 2021-22. These figures highlight that the declines in enrollment in the elementary and middle school grades in 2020-21 clearly stand out regardless of which metric we use. The changes were more obvious when looking at the cohort-adjusted rates, where 2020-21 was the first year that there were noticeable negative changes in elementary grades. While the negative changes to within-grade enrollment were much larger in 2020-21 relative to prior years, we also see that some of these negative changes were likely due to pre-existing trends. We see, for example, that the 1^st grade cohort in 2015-16 was smaller than the prior cohort, and this difference continued to show up in successive grades and years.

Figure 3. Cohort-Adjusted and Within-Grade Changes in Enrollment in CO, 2015-16 to 2021-22.

Other features of the cohort-adjusted rates are also noteworthy. The increase in 9^th and 12^th grade cohort-adjusted enrollments is not unique to 2020-21. This may occur because students who were enrolled in home schooling or private school in elementary and middle school enter the public school system in 9^th grade for high school. The large and consistent changes in 12^th grade are more difficult to explain. It could be caused by students who remain enrolled in 12^th grade for a second year to complete additional credits needed to graduate, but there could be other explanations. The consistent declines in 10^th and 11^th grade may be due to students who leave school before graduating and are another regularly occurring phenomenon that is not apparent based on within-grade changes.

Understanding Changes in 2021-2022

The final column of Figure 3 shows changes to enrollment for the most recent year, 2021-22. These enrollment counts have been of great interest to evaluate whether students appear to be returning to public schools. Both sets of metrics are skewed by the atypical decline seen in 2020-21, but they also point to different stories about students’ return to public school. Statewide, ��ֱ�� saw an overall increase in PreK-12 enrollment of about 0.4% in 2022 relative to the prior year, with the largest within-grade changes to enrollment in preschool and kindergarten (��ֱ�� Department of Education, 2022). Based on the within-grade changes, half of all grades saw additional declines relative to 2020-21. However, the cohort-adjusted values tell a different story. The large positive change in 1st grade indicates that there were about 4% more 1st grade students enrolled in school this year relative to the number of kindergarten students in 2020-21. This suggests that a non-trivial number of families may have decided to enroll their children in first grade, despite not having attended public kindergarten in the prior year. The consistent small positive changes in other elementary grades suggest that there was potentially a small return to school in these grades, although the increases in grades 2-6 were similar to those seen prior to the pandemic and could reflect more typical annual patterns of students moving into the ��ֱ�� system or potentially being retained within grade.

Variation Across States

A final question to consider is: To what extent do these findings generalize to other states? Variation in enrollment changes suggest decisions about sending children to school varied across states, and not all states will have observed such large changes to cohorts of students over the past 10 years. The consistent increases in 9th and 12th grade enrollment seem to happen annually in ��ֱ�� but may not occur elsewhere.

Figure 4 shows both cohort-adjusted and within-grade enrollment changes in Massachusetts for the same time span as a contrasting example. The within-grade changes in MA were similar in 2020-21 to those in ��ֱ�� and also show signs of being impacted by changing cohort sizes over time (i.e., inconsistent trends across grades).

The cohort-adjusted trends tell a different story, however, both relative to the within-grade changes and relative to ��ֱ��. There is a consistently positive change in cohort-adjusted enrollment in first grade, suggesting that many students do not enroll in public kindergarten before enrolling in first grade. Thus, while the decline in 2020-21 cohort-adjusted enrollment appeared smaller in first grade relative to other grades, in historical context this negative change was noteworthy relative to the positive changes in cohort-adjusted 1st grade enrollment observed in every prior year. There is also a spike in 9th grade enrollments as in ��ֱ��, but rather than an additional spike in 12th grade enrollments, there are consistent declines in 10th-12th grade cohort-adjusted changes, and these appear to have been about the same in 2021 as they were in the prior five years. This is similar to ��ֱ�� in that the pattern of cohort-adjusted high school enrollment appears to have been less impacted by the pandemic, but differs from ��ֱ�� in the overall pattern of high school enrollments.

Figure 4. Cohort-Adjusted and Within-Grade Changes in Enrollment in MA, 2015-16 to 2021-22.

Finally, the most recent data from 2021-22 are similar to ��ֱ��. The cohort-adjusted changes suggest enrollments are rebounding in first grade and were similar to historic norms in other grades, while the within-grade enrollment changes tell a more mixed story suggesting additional but inconsistent declines in enrollment. Taken together, these comparisons suggest there might be meaningful variation in the state-specific patterns of student enrollment across grades that would be worth studying further both in general and to better understand how the pandemic may have impacted students’ trajectories through school.

Conclusion

Tracking the historic changes in public school enrollments will continue to be important in the coming years. Analysts should consider using both within-grade and cohort-adjusted change metrics to do so, which can provide a more complete understanding of enrollment changes. This is particularly true when investigating variation in changes across grade levels, as the analyses in this brief illustrate that each metric is sensitive to different changes and thus helps us to understand different aspects of changing enrollments. When calculating total public school enrollment across all grades, the two metrics will likely not yield such different results. The two metrics can also lead to different inferences about students returning to public schools in 2021-22, and the same will likely be true in future years. Finally, the differences observed between ��ֱ�� and Massachusetts, particularly in the regular patterns of within-cohort grade level changes, suggest there is interesting heterogeneity of state-specific enrollment dynamics that could be investigated further.

¹Data from CDE are available at: https://www.cde.state.co.us/cdereval/pupilcurrent in the file “2021-22 PK-12 Pupil Membership by Grade with Historical Data.” Data from the MDESE are available here: https://profiles.doe.mass.edu/statereport/enrollmentbygrade.aspx. For these analyses, MA students enrolled in special education beyond grade 12 (reported as grade “SP”) are not included in grade level enrollment counts.

²Although enrollments are based on fall census counts, when using a single year to represent an AY I refer to the spring of the AY. For example, referencing year = 2020 would refer to enrollment counts for the 2019-20 AY.

³A complete table of all grade levels is included in the Appendix.

References

��ֱ�� Department of Education. (2021, January 22). News release—Official count of 2020-21 ��ֱ�� students confirms 3.3% decrease in state enrollment with biggest drop in early grades. https://www.cde.state.co.us/communications/newsrelease-enrollment

��ֱ�� Department of Education. (2022, January 19). News release—��ֱ�� school enrollment increases slightly after 2020’s decline. https://www.cde.state.co.us/communications/newsrelease2021enrollment

Dee, T. S., & Murphy, M. (2021). Patterns in the pandemic decline of public school enrollment. Educational Researcher, 50(8), 566–569. https://doi.org/10.3102/0013189X211034481

Pendharkar, E. (2022, February 1). More than 1 million students didn’t enroll during the pandemic. Will they come back? https://www.edweek.org/leadership/more-than-1-million-students-didnt-enroll-during-the-pandemic-will-they-come-back/2021/06

Appendix: Complete Grade Level Enrollment Changes in 2020-21 and 2021-22

��ֱ�� K-12 public school enrollment counts and changes for the past three years.

Massachusetts K-12 public school enrollment counts and changes for the past three years.

Off

Traditional

White

Considering Student Identity and Belonging

Anonymous — Fri, 15 Apr 2022 18:46:06 +0000

Considering Student Identity and Belonging Anonymous (not verified) Fri, 04/15/2022 - 12:46 Jessica Alzen

My current projects have me spending a lot of time in various classroom spaces. I recently used Zoom to observe in middle and high school science classrooms in Illinois and New Jersey. I spent one day in Cañon City, CO where I visited every classroom in an elementary school with a team of researchers, educators, and community members. Another day, I attended a multi-tier system of support (MTSS) team meeting at a Boulder Valley School District middle school, and on yet another day, I observed language arts classes in an elementary and middle school in Greely, CO. I also recently sat in on a statistical collaboration class meeting at University of ��ֱ�� Boulder.

Each visit had its own purpose. For one project, we are thinking about how teachers take up new science curriculum to support students in thinking and acting like scientists. In another, we are serving as thought partners alongside district leadership in building and improving local accountability systems. In yet another project, we are studying effective ways of training interdisciplinary statistical collaborators. Although these projects are relatively disparate in nature, a theme that has come up repeatedly in one way or another are the ideas of student identity and belonging. As educators, we care that students see themselves as belonging in the school, classroom, and curriculum. We want students to see themselves as scientists, and we want students from all backgrounds and experiences to envision themselves in the same potential career spaces. This manifests itself when we characterize what it means to think and act like a scientist, the kinds of books we look for in classroom libraries and images on school walls, the ways students talk about themselves and their futures, and the verbiage we use for context in an assessment or survey question.

I’ve particularly enjoyed thinking about these ideas in my project involving secondary science teachers. On that project, our team provides science curricula and related professional development to help teachers navigate shifting their classroom practices to better encompass the Next Generation Science Standards. We’ve been studying the teachers’ moves that can provide more opportunities for students to have more authority and agency in the classroom and to think and act as scientists. Teachers can encourage students to share their ideas and tell students that their ideas are important for figuring out scientific ideas. However, simply inviting students to take up more authority in the classroom or telling them that their ideas matter doesn’t always convince students to participate, nor is it necessarily an authentic invitation. Sometimes when students share their ideas, the class moves in another direction. This can make students feel as though their ideas don’t actually matter, and they are simply playing school by searching for the “right” ideas to share when their teacher asks a question. This sort of inauthentic experience can happen for plenty of reasons. Perhaps the teacher is unsure about having students direct the class conversation. Maybe she doesn’t quite know how to create an environment where students follow their own noticing about scientific ideas to increased understanding about the world. It’s also possible that scheduling constraints don’t allow for that level of student engagement on a given day.

Photo by Allison Shelley/The Verbatim Agency for EDUimages.

One thing we’ve found in our work is that the more teachers create spaces and opportunities for students to engage in thinking together as a class, the more reasons students have for engaging in knowledge-building about science ideas. When teachers ask students to make connections between their ideas or ask the class to make sense of two competing ideas that come from different students, they ask all students in the room to engage at a deeper level than simply looking for the “right” science answer. When students don’t agree about an idea and teachers invite students to share why they don’t agree or what causes them to be unconvinced, students think and act like scientists. The more teachers use differing student ideas as a catalyst for class conversation, the more students see their ideas and experiences as mattering. They can see themselves in the curriculum because the class centers around students sharing and making sense of their different ideas and lived experiences.

We’ve found that this idea of creating a space where students engage in a collective enterprise of understanding science ideas can lead to more buy-in from students and gives them more of a reason to participate and think and act like scientists. We’ve seen teachers using this approach to engage more students in whole class discussion and challenge students to consider a wider array of ideas. The pursuit of collective knowledge-building opens up space in the class for more students to see themselves and their ideas as integral to making sense of science ideas and the world around them.

At CADRE, our mission is to produce generalizable knowledge that improves the ability to assess student learning and to evaluate programs and methods that may have an effect on this learning. A large body of research suggests that students fare better when they have a sense of belonging or they can identify with their classrooms and schools. Research projects, such as the one discussed here, help stakeholders explore and understand the ways that curriculum and teaching practices address the ideas of identity and belonging and are one way that we at CADRE work to accomplish our mission.

Off

Traditional

White

Examining the Question of Access to Arts Programming

Anonymous — Mon, 11 Apr 2022 20:25:34 +0000

Examining the Question of Access to Arts Programming Anonymous (not verified) Mon, 04/11/2022 - 14:25 Elena Diaz-Bilello

Much has been written about the importance of the Arts, particularly with respect to how exposure to arts programming can meet social emotional learning needs of students and enhance their well-being (Bowen & Kisida, 2019; Greene et al., 2018; Kisida et al., 2020). For this reason, education researchers highlight the importance of providing access to these disciplinary areas and decry how the Arts tend to be the first areas to be cut at schools and districts facing budget constraints. Further, schools that cut arts programming are often the same sites serving large populations of historically underserved students (Bassock et al., 2016; Kisida et al., 2020; West, 2007).

In Bowen and Kisida’s (2019) landmark randomized control study of an arts education program in Houston, the authors contend that the results of their study “provide critical evidence that increasing students’ arts educational opportunities has positive impacts on meaningful outcomes [and that] the narrowing of educational offerings and objectives to align with accountability assessments has had adverse effects on the arts in K-12 education” (p. 17). Bowen and Kisida were only able to study these opportunities with respect to the access students have to arts programming. But access to arts programming is one thing, access to high quality programming is another.

Access vs. Quality?

To illustrate this point, in 2017-18, I began a collaboration with an instructional specialist in the visual arts to collect qualitative observations on teaching practices. The observations I collected over the course of two years were intended to help map out the features of high quality instruction in this domain. As part of this process, we visited one of the few alternative high schools in the Denver Public Schools (DPS) that offered visual arts courses to students. Similar to other urban school districts, the vast majority of students served by this—and all of the district’s alternative high schools for that matter—are students of color.

During these visits we observed a wide variety of classroom interactions, spoke with teachers about their expectations for students, and documented how the teachers designed their courses to meet those expectations. In year one at the alternative high school, we observed a class where individual students sat at separate tables constructing sculptures using selected shapes. The stated larger objective of this class was, “students will be able to engage in the creative process by applying mathematical measurements to create sculptures.” Although students were “on task,” and the school principal had praised this teacher for providing students with interdisciplinary opportunities – in this case, melding the Arts with Math – the activities and lessons we observed during the school year were only aligned to state standards for an elementary level visual arts class. And when we interviewed the teacher, we learned that she chose to have students work silently and independently on art projects. Not because independent work would foster learning, but because it served to better manage classroom behaviors. In the class sessions we observed, students were given few opportunities to make meaningful personal connections to the products they created.

Fast forward one year later, a new teacher to the school, but a veteran teacher of the visual arts took over the visual arts program. The first thing this teacher did was to dismantle the approach taken by the previous teacher. She enacted collaborative group projects, displayed the artwork produced by students throughout the halls, set up tasks and lessons aligned to high school visual arts standards, and encouraged active dialogue by engaging students in discussions about art as they negotiated what needed to be done to complete each group project. She provided ample opportunities for students to deepen disciplinary understandings and skills in the visual arts while simultaneously allowing creativity and thinking to expand. For example, in one class we observed, she had a group of students working together on a large oil painting project reflecting the group’s personal interests. She checked in with the group to discuss the progress made on their artwork and had them consider solutions for accentuating perspective and depth in the painting. The students debated about the colors and how they wanted to play with the space, and ultimately settled on a solution that implemented the ideas forwarded by two students in the group. Within this context, students acquired and applied understandings of art elements while cultivating important life skills through group collaborations.

Figure 1. A collaborative group project: oil painting in progress

Figure 2. Exhibiting student artwork at an alternative high school

Compared to the approach taken by the teacher in year one, it wasn’t difficult to conclude that the year two teacher at this alternative high school was offering a higher quality visual arts learning experience to students. Strong formative assessment practices, coupled with a firm belief that every student in her class had valuable contributions to make to the learning process for other students, made this teacher’s class stand out - not just within the context of this school but also within the context of other visual arts classrooms we observed at five other schools in the district.

Focusing on Quality Arts Programming

While I wholeheartedly agree with recommendations made by education researchers to invest and restore arts education programs across schools in this country, I’m not convinced that simply expanding access or providing after school arts programs is an adequate policy solution. The National Core Arts Standards highlight that like other core disciplinary areas, Arts courses require careful planning and execution to address important learning targets linked to big ideas. This also means understanding and appreciating that high quality learning can take on different expressions and forms in the Arts when designing classroom activities and assessments. For example, forms of extended discourse in a music class can look quite different but accomplish similar aims to extended discourse using verbal communication in core subject areas. In a strings class, students engaging in developing new patterns and sounds can actively communicate with one another through their instruments as they individually and jointly contribute to that group dynamic through evolving tempo, rhythm and the collective sound produced. The careful listening and contributions that lead to each phrase, followed by the continuous adjustments made by students in subsequent phrases, reflect similar dialogue-based goals accomplished through extended discourse. This can be facilitated when the structures set by teachers in music and composition explorations encourage students to contribute and enrich the learning of one another. Knowing whether this type of dynamic and rich learning experience driven by student interactions occurs in music classrooms would require stepping into these spaces. Observing these rich interactions also reflect a student-centered approach that moves away from more typical “authoritative” teacher-centered approaches that dominate even advanced level studio courses in music (Colwell, 2011).

Our Continued Work in the Arts

In our next phase of work with the DPS Arts and Physical Education department (APED), we look forward to examining the intersection of both access and high-quality arts programming. CADRE’s long-term partnership with APED continues because we share a strong mutual interest in promoting inclusive classrooms designed to advance and deepen student learning. An aspirational goal for both APED and CADRE is that students not only have access to the Arts, but that the disciplinary understandings and complexities of Arts instruction are recognized and enacted to enrich the learning experiences for students across all schools in the district.

References

Bassok, D., Latham, S., Rorem, A. (2016). Is kindergarten the new first grade? AERA Open, 1(4), 1–31. https://doi.org/10.1177/2332858415616358

Bowen, D.H. & Kisida, B. (2019). Investigating causal effects of arts education experiences: experimental evidence from Houston’s Arts Access Initiative. Houston, TX: Houston Education Research Consortium, Rice University.

Colwell, R. (2011). Roles of direct instruction, critical thinking, and transfer in the design of curriculum for music learning. In Colwell, R., Webster, P. R. (Eds.), MENC Handbook of Research on Music Learning, 1 (pp. 84–139). Oxford, UK: Oxford University Press.

Greene, J. P., Erickson, H. H., Watson, A. R., Beck, M. I. (2018). The play’s the thing: Experimentally examining the social and cognitive effects of school field trips to live theater performances. Educational Researcher, 47(4), 246–254. https://doi.org/10.3102/0013189X18761034

Kisida, B., Goodwin, L., & Bowen, D. H. (2020). Teaching History Through Theater: The Effects of Arts Integration on Students’ Knowledge and Attitudes. AERA Open. https://doi.org/10.1177/2332858420902712

West, M. (2007). Testing, learning, and teaching: The effects of test-based accountability on student achievement and instructional time in core academic subjects. In Finn, C. E., Ravitch, D. (Eds.), Beyond the basics: Achieving a liberal education for all children (pp. 45–61). Fordham Institute.

Off

Traditional

White

Exclude

Spring 2024 Newsletter

What’s New at the Center for Assessment, Design, Research and Evaluation

Featured studies

Off

Content-Referenced Reporting of Student Growth

References

Off

Studying the role of postsecondary policies in shaping families’ financial health

Background

Future Work

Conclusion

Off

����ֱ�� Multi-Tiered System of Supports Partnership

Off

K-5 SEL Pilot Grant Evaluation

Off

Spring 2022 Newsletter

What’s New at the Center for Assessment, Design, Research and Evaluation

Featured projects

Conferences and NCME Presidential Address

Off

Measuring Public School Enrollment Changes During the COVID-19 Pandemic

Introduction

Data

Two Methods for Calculating Grade Level Enrollment Change

Putting Values in Historical Context

Understanding Changes in 2021-2022

Variation Across States

Conclusion

References

Appendix: Complete Grade Level Enrollment Changes in 2020-21 and 2021-22

Off

Considering Student Identity and Belonging

Off

Examining the Question of Access to Arts Programming

Access vs. Quality?

Focusing on Quality Arts Programming

Our Continued Work in the Arts

References

Off

��ֱ�� Multi-Tiered System of Supports Partnership