Family & Community Adult Education Adult & Community Learning Services
Adult and Community Learning Services

Policy Update Assessment Requirements for FY 03

This policy update is to inform you of new requirements for student assessment that most ABE programs funded by the Department of Elementary and Secondary Education must implement beginning July 2002. No decision has been made with respect to how pre/post testing will be accomplished in Transitions and Distance Learning programs. Please note that these new requirements apply only to the primary assessments that programs use to report educational gain to the Department and not to other assessment instruments that programs use in support of the teaching and learning process. Programs are encouraged to continue using other assessments they find useful to the instructional program, particularly since many of the instruments you are currently using at the local level do meet the need of informing instruction.

Let me give you some background that informed the above decision. As you know, the Performance Accountability Work Group (PAWG), was made up from representatives from the field. One of their challenges was to make recommendations for a Massachusetts ABE assessment system that would meet the following three overriding goals:

• The assessments must meet standards of validity (testing what is meant to be tested), reliability (producing consistent results) and comparability ( the scores are able to be correlated to other assessments).
• The assessments must be aligned with the learning standards in the ABE curriculum frameworks; this is essential if the assessments are to be "valid."
• The assessments must provide useful information to teachers and students.

The array of fifty or more assessments now used by ABE programs to report educational gain to the Department do a far better job of meeting the second and third goals than the first. The "crosswalks" between the assessments in use and grade level equivalents (GLE) and student performance levels (SPL) that programs provided to the Department constituted a temporary solution during the first three years of the Workforce Investment Act (WIA). These crosswalks cannot achieve the level of statistical comparability that assessments in a high stakes environment are required to meet. This temporary solution cannot continue since: (a) we are working in a high stakes environment, i.e., both federal incentive funding and local program funding are at stake; and, (b) assessments must be valid, reliable and comparable by law nationally and by state policy in Massachusetts.

Further, a review of the literature and consultations with prominent experts in the field of assessment also led us to conclude that the results of multiple assessments cannot be reliably compared to one another. Our evaluations of various assessment options were heavily influenced by the preceding three goals.

The PAWG spent several months reviewing various assessment tools and evaluating the extent to which the tests are aligned with the standards in the ABE curriculum frameworks for English/Language Arts (ELA), Numeracy (Math) and English for speakers of other languages (ESOL). The group concluded that a combination of standardized tests and performance assessments needs to be selected and/or developed that clearly align with the standards in our frameworks. Within these frameworks, all standards in Math will be assessed statewide; for ELA, the Reading, Writing and Critical Thinking standards will be assessed statewide, and for ESOL, selected standards in Oral Communication, Written Communication and Language Structure and Mechanics will be assessed statewide. While we have many programs with experience in performance based assessment, there will be a need for gradual introduction and mastering the complexity of establishing validity and reliability for this type of assessment.

The Department has accepted the following recommendations that will become our official assessment policy starting July 2002.

ELA & MATH: LONG-TERM RECOMMENDATION

For ELA and Math, the PAWG found NO commercially available standardized assessments that are substantially well-aligned with the reading, writing and critical thinking standards of these frameworks. Hence, the long-term recommendation is that the Department leads the development of a combination of tests and performance assessments that are fully aligned with these frameworks. We estimate that it will take a minimum of two to three years to develop, field test and standardize these assessments, however, funding constraints are likely to stretch this out for an additional year or two.

ELA & MATH: SHORT- TERM RECOMMENDATION

The two commercially available assessments that come closest to alignment with our ELA and Math frameworks are the TABE and the AMES. Unfortunately, at their best this alignment among test items and framework learning standards is just slightly more than 50% which is one of the reasons why they are not recommended as the long term solution for ELA and Math assessment. Since there is some uncertainty in the publication of the AMES, the TABE, forms 7 and 8 have been selected; no other forms of the TABE may be used. However, the PAWG found very little alignment at the beginning literacy levels (GLE 0-2), hence, the TABE is not currently recommended for assessments at this level. We are continuing to work on a solution to this problem.

We have negotiated a 20% discount with the publisher, CTB/ McGraw Hill for purchases of TABE testing materials. All programs in Massachusetts will receive an 18% discount on all products ordered no matter the quantity and a 2% shipping discount. The attached order form should be used when purchasing materials as it reflects this special pricing. The following materials must be purchased and optional ones are also listed.

Required

• Locator Test and Practice Exercise
• TABE Forms 7 and 8 TABE Complete Battery* (includes examiner's manual)
• Level E (GLE 1.6 - 3.9)
• Level M (GLE 3.6 - 6.9)
• Level D (GLE 6.6 - 8.9)
• Level A (GLE 8.6 - 12.9)
• Norms Book
  *Please note that the Department analyzed the shorter "survey form" of the TABE in the hope that we could cut in half the time needed to administer the test. Unfortunately, our analysis of the standard error of measurements (the amount the test-taker's score is expected to fluctuate around the true or "ideal" score) for the survey forms indicated that they are too great to obtain reliable measures of progress for our ABE students over the course of an average year of instruction.

Optional

• SCOREZE Answer Sheets (recommended)
• TABE-PC (computer administered version)

ESOL: SHORT-TERM RECOMMENDATION

The two assessments that best align with the adult ESOL framework learning standards are two performance-based assessments: the BEST - for speaking and listening, and the REEP (Refugee Education and Employment Program, developed by the Arlington, VA public schools) - for writing. We are concerned that the BEST begins to "plateau", or not test higher level skills, before students complete the high ESOL level (which Massachusetts will define as ending at SPL 8 beginning FY 2003). We are also aware that the one-on-one format will be time consuming to administer. We are in the process of negotiating with the publisher, the Center for Applied Linguistics, to obtain a better price than programs could obtain on their own. We recommend that you wait for further notice from us before ordering the test. We'll get back to you shortly. The REEP appears to be suitable at all ESOL levels, however, we are still evaluating whether there are enough standardized prompts (topics to generate student responses) to support its use for statewide assessment of ESOL student writing. We are also still awaiting licensing and pricing information from the test's developers. We will also get back to you on the REEP a.s.a.p.

So beginning in FY03, these tests will be the only tests that can be administered to assess educational gain every four months according to our statewide assessment policy. Our office will be working closely with SABES to ensure that adequate training is made available to your staff over the coming months. In addition, we strongly encourage programs to use other assessments, particularly performance assessments that may be more useful in the teaching and learning process. Since performance tests require extensive standardization, it will take time before we can use any Massachusetts' specific assessments statewide to obtain reliable measures of educational gain. For the longer term, we will evaluate the new forthcoming BEST test and local assessments for their potential to be standardized for use on the state level.

We understand that this means a trying period of adjustment for practitioners and students alike. We further understand that some important areas of our work are not yet addressed by these new assessment requirements: beginning ABE literacy; native language literacy; ESOL reading; and I'm sure there are more. We want to recognize and express our appreciation for your patience and your contributions to this effort as we move forward. We will continue to work with you to ensure that our short and long term assessment strategies are the best we can do on behalf of our students - they need and deserve nothing less.