|
|
 |
Frequently Asked Questions for Performance Standards
Table of Contents:
- General
- Attendance
- Average Attended Hours
- Pre/Post Testing
- Learner Gains
- Student Goals
General
Why is there a range for the standards? Shouldn't a standard met be one number or one percentage?
Answer: Performance standards were set across a range rather than a single number to allow programs wider latitude in meeting performance standards for the four different benchmarks. Performance standards that span a range also more accurately reflect the reality of statewide adult basic education system performance. How can a program that has had its funding reduced be expected to meet all performance standards?
Answer: ACLS and PAWG reviewed performance data and designed the performance standards so that a wide variety of program designs and student populations will be able to obtain proficiency. Analysis of data reveals that a majority of ABE funded programs currently meet and exceed performance standards. Thus there is no single predictor of a program's ability to meet the standards, including size of grant, program intensity, or the program's duration of service. What is a negotiated program outcome?
Answer: Negotiated program outcomes were established prior to the current FY2006-2010 funding cycle to allow ACLS to capture outcomes on a programmatic level relevant to health education, civic participation, family literacy, and workplace education. Negotiated program outcomes are no longer a component of the Countable Outcomes policy. What happens if a program doesn't meet the performance standards?
Answer: ACLS provides support for continuous improvement for all programs, including technical assistance for programs that do not meet performance standards. The resources include dissemination of promising practices, technical assistance from the program specialist, assistance developing an action plan for improvement where needed, and professional development / support from SABES. Programs that do not earn the total number of available performance points each year will have a reduced total number of past performance points for the next funding cycle. Past performance points will be one of several criteria used to score the next competitive RFP. What is the status of programs serving the homeless in terms of accountability? What sort of performance data do we collect on these programs?
Answer: Performance standards for programs serving the homeless will be developed from data collected on homeless programs. ABE for the homeless will have its own performance accountability standards and data, and this data will not be included in the data reported for community adult learning centers (fund codes 340, 345, 359).
Attendance
When calculating attendance, will ESE filter out students enrolled in ADP classes?
Answer: The enrollment of students in ADP classes will not impact any of the program's performance measures including student attendance.
Average Attended Hours
How can low-intensity programs be expected to meet the average attended hours standard?
Answer: After analysis of statewide ABE performance data, PAWG recommended (and ACLS accepted) performance standards that are based on the state averages in four areas of performance (the fifth, student goals, is in pilot). Further analysis of data revealed that a majority of ABE funded programs (representative of a diversity of program types and student populations) currently meet and exceed the adopted performance standards. How can programs be expected to meet the average attended hours standard if, due to weather-related issues, we can't get to our learning center?
Answer: ACLS is aware of weather-related and other issues outside of a program's control that affect attendance and average attended hours. For this reason ACLS set the attendance and average attended hours standards near the average figures for the state, both of which incorporate multiple years of data. My program has a July summer session, then a break in August, and starts again in September. Our end-of-year ceremonies are in July. A number of students attend class all year, finish in July, and then don't come back in September. Will these students lower our attended hours average, or will students who attend July, after having attended the previous year, and then leave, be excluded before the average is calculated?
Answer: Average attended hours are calculated on the state fiscal year calendar (July 1-June 30). Currently all of ACLS's reports follow the state fiscal year. Students who leave a program during the fiscal year cycle will cause the average attended hours figure to be lower for that fiscal year. If a student is in a ESE-funded slot and then moves to a non-ESE funded slot, will their average attended hours include their time in the non-ESE slot?
Answer: Yes. Regardless of the class focus or funding type, all student hours are included in the program's average attended hours. This includes rates and non rate based classes as well as classes funded through ESE Direct, ESE Indirect, and Non ESE (match). When calculating the average attended hours, will ESE filter out students enrolled in EDP/ADP classes?
Answer: The enrollment of students in EDP/ADP classes will not impact any of the program's performance measures including average attended hours. Are students who are enrolled in non-rates based classes included in the average attended hours?
Answer: Yes, average attended hours includes all students enrolled in rates and non-rate based classes. If a student is only enrolled in a non-rates based citizenship class, then his/her attended hours will be low. Will this bring the program's average attended hours down?
Answer: Yes. If the non-rates based class runs for a short period of time, then the average attended hours will be fractionally lower. While ACLS realizes that not all students will be able to attend all planned instructional hours, the performance standard for average attended hours is not set so high as to require all students to attend 100% of planned hours in order to meet the standard. A program meets the standard if students attend an average of 117 planned hours. Students who are enrolled in a ESE program that doesn't have summer courses want to enroll in a six-week summer class at a neighboring program. Will these students who attend for only six weeks negatively impact the neighboring program's retention rate?
Answer: ACLS Guidelines for Effective Adult Basic Education state that programs must immediately enroll any student in good standing who seeks to transfer from another ESE funded ABE program or place this student at the top of the appropriate waiting list. The number of performance points to be gained or lost in the summer enrollment scenario is negligible because the performance accountability system is balanced over five separate and individually weighted components that will be averaged across multiple years of the funding cycle. The Performance Standards are not set so high as to require programs to get the maximum points in all five categories to remain funded or even to avoid action planning. Our programs have an open-entry/open-exit policy. In 2005 approximately one-third of the students enrolled [exited] after fewer than 117 class hours remained in the year. Even if they'd maintained 100% attendance these students would still negatively affect the Average Attended Hours performance measure. If we remove these students (who can't meet the standard due to late in the year enrollment) from the calculation, our Average Attended Hours increase by nearly 10 hours. Would ESE consider exempting late enrollees from the calculation of Average Attended Hours, (similar to the April 1st exemption for the Pre/Post Testing performance measure)?
Answer: Analysis of data prior to setting performance standards revealed that a majority of ABE funded programs currently meet and exceed performance standards. While ACLS realizes that not all students will be able to attend all planned instructional hours, the performance standard for average attended hours is not set so high as to require all students to attend 100% of planned hours in order to meet the standard. A program meets the standard if students attend an average of 117 planned hours. Our program's average attended hours are going to be negatively impacted by the ½ year class that we added in February 2006 with the additional funding we were awarded. When looking at average attended hours, perhaps programs specialists will have to filter out those half/partial year classes for the data to be valid.
Answer: Because increases were equally distributed within the same region, and because the increases were mandated to provide instructional services, any impact will be minimized within the same region.
Pre/Post Testing
My program has a July summer session, then a break in August, and starts again in September. Our end-of-year ceremonies are in July. A number of students attend class all year, finish in July, and then don't come back in September. If students attend the previous year, their assessment is rolled over to July, and they attend class in July and then leave, will they be excluded before the average is calculated?
Answer: In the above case, if the students attend at least 12 hours into the new fiscal year (beginning July 1), those students will be included in the pre/post test calculation. Will programs be penalized when students enroll for only a short time, get their GED, and don't take a post test (fast track students)?
Answer: Students with an initial TABE scale score of 585 and above who earn their GED are not included in the pre/post test calculation, regardless of the length of time the student is enrolled in the program. Do we need to pre-test students who are not enrolled in rates based classes?
Answer: Yes, all students must be pre and post tested. Will students who are enrolled in non-rates based classes be included in our pre/post percentage?
Answer: Yes. According to ESE guidelines, 25% of students in a citizenship class can be from outside the ESE program. Do these students need a primary assessment to enroll in the non-rates citizenship class?
Answer: Yes, a student needs a primary assessment to be enrolled in a ESE-funded rates or non-rates based class. Will students who attend more than 12 hours but not enough hours for a posttest be included in the pre- post-test performance standard?
Answer: Yes. Students may only be excluded from the pre/post test requirement in the following circumstances:
- Students with < 12 hours attendance
- Students with an initial scale score in TABE Reading, Writing, or Math > 584
- Students with an intake date after 4/1
Will a program's past performance points for pre-post testing be affected because their citizenship class is a short-term class and the students who are only in citizenship won't be around long enough for a post test?
Answer: Yes, past performance points for pre- and post-testing will be influenced by the number of students who are pre- and post-tested. While ACLS realizes that programs will not be able to pre- and post-test all students, the performance standard for pre- and post-test is not set so high as to require 100 percent of the students to be pre- and post-tested in order to meet the standard. A program meets the standard if at least 66% of students are pre- and post-tested. Students are excluded from the pre/post test requirement in the following circumstances:
- Students with < 12 hours attendance
- Students with an initial scale score in TABE Reading, Writing, or Math > 584
- Students with an intake date after 4/1
Learner Gains
Is the learning gains standard based on all learners in the program, or all learners who are pre/post tested? The standard says "Total percent of learners," but I believe in the RFP process the percentage was based only on the learners who had a pre and post test.
Answer: Learning gains are calculated on the scores earned by all students who are pre and post tested. Students who are not pre and post tested cannot demonstrate learning gains. (However, if a program post tests less than 50% of their students, then the points awarded for learner gains will be reduced by 50%.) What is a learning gain?
Answer: A learning gain (significant or meaningful gain) can be defined in a number of ways. For the purposes of ACLS reporting, a learning gain represents a statistically significant increase in skills / knowledge as measured by one of the standardized assessments (Best Plus, REEP, TABE) or one of the non-standardized tests (e.g., Beginning Literacy or Pre Literacy ESOL) which are measured by one GLE. The Center for Educational Assessment at the U Mass Amherst recommended the number of scaled score points for the standardized assessments, and ACLS uses those amounts in the performance standard for learning gains.
Student Goals
What is the difference between a goal and a countable outcome?
Answer: Countable outcomes are the stated goals of students who enroll in our programs. Through the Countable Outcomes policy, programs attempt to capture the major reasons that motivate students to enroll in ABE programs, in addition to increasing their basic skills in reading, writing, math, speaking, and listening. Countable outcomes include items such as attaining citizenship, getting a driver's license, getting a job, helping children with homework, increasing income, or learning about health issues. How can one suggest a new goal be added to the student goal sheet and list of countable outcomes?
Answer: Programs can request additional goals for inclusion in SMARTT by sending the request to their program specialist. ACLS will determine whether or not this goal can be added to SMARTT and the student goal sheet. The program specialist will notify the program of the result of the request. Which goals can undocumented students set/achieve?
Answer: With the help of the staff in the program, there are many goals that undocumented students could achieve. While undocumented students might not be able to open a checking account, these students could, for example, acquire the skills needed to improve their lives as family members (read, write, do mathematical problem-solving with child and/or help child with homework) and community members (participate in community activities; learn about and/or use community organizations; get a library card). When does the achievement of a goal that is tracked through data matching "count," i.e., which year?
Answer: Each of the follow up goals has a different reporting timeframe. In the table below, the following three goals are data matched (obtain a GED is data matched for ALL students while the job related goals are data matched only for those who provide a social security number).
| Goal | Timeframe |
|---|
| Obtain GED | Date met anytime prior to 8/5 since all scores for tests taken prior to 6/30 have been received. (Ex. Student tested on 6/16/05 and the scores were reported on 7/25/05 so that is counted for FY05.) |
| Entered Employment | Date met in next quarter after goal is set |
| Retained Employment | Date met is two quarters after date met for entered employment. |
* Note: All job related goals are reported using prior fiscal year data since the follow-up time frame takes 3 - 9 months and employers take at least 6 months to provide the information to Department of Revenue. This is timeframe is now required by USDE.
What goals can low-literacy learners achieve?
Answer: With the help of the staff in the program, there are many goals that low-literacy learners could achieve. While most low-literacy students will not be able to achieve a GED in a program year, these students could, for example, acquire the skills needed to improve and enrich their lives as family members and can achieve the column A goal "read, write, do mathematical problem-solving with child and/or help child with homework." These students can also learn skills to help them in their jobs and achieve the column A goal "increase earnings" or "retain a new job for 6-9 months." Low-literacy learners can also give back to their community by becoming active community members and achieve the column B goal "participate in community activities." I've heard that it is illegal to photocopy the required documentation for Citizenship (goal in column A, No. 7 A-D). How should programs document the achievement of this goal?
Answer: It is not illegal to photocopy the citizenship certificate if the copy is used to secure a particular social service or economic or other benefit for the certificate holder (i.e., food stamps). ACLS no longer requires programs to photocopy the citizenship certificate for countable outcome 7D "Receive Citizenship Certificate." Programs may use teacher verification to verify attainment of citizenship goal. However, programs should continue to collect required documentation for the achievement of the countable outcomes (7A - 7C) leading up to 7D "Receive Citizenship Certificate." This change in policy has been incorporated into the Countable Outcomes Definitions and Required Documentation chart. How should programs differentiate between short and long-term goals?
Answer: ACLS will eliminate the distinction between short and long term goals in FY2007, and programs will no longer need to make this distinction. However, the three goals below are linked to an assessment score in SMARTT and will continue to be automatically set as short or long term goals in SMARTT according to the following conditions:
"Obtain GED" - Will be set as Short Term Goal if the assessment is greater than or equal to TABE scale score of 550
"Enter transitional/developmental education" - Will be set as Short Term Goal if the assessment is greater than or equal to TABE scale score of 550, REEP scale score of 4.2, or BEST PLUS scale score of 507
"Enter post-secondary education leading to Associate degree or higher" - Will be set as Short Term Goal if the assessment is greater than or equal to TABE scale of 550, REEP scale score of 4.2. or BEST PLUS scale score of 507.
Is "Obtain a GED" always a long term goal, even if the scale score is high?
Answer: If the student's TABE scale score is greater than or equal to 550, then "Obtain a GED" is a short term goal. Some students are uncomfortable having their choices of reading documented for the "Get and Use Library Card" goal. Are there any suggestions on alternatives to document this goal?
Answer: The student may show the library card and some library material that has been checked out to the teacher. After verifying the checked-out library material and card, the teacher should place a note in the student's file. Many of the student goals require interim steps in order to reach the final goal (although they are not long term goals). How can we collect and record documentation on these interim steps that indicate that the student is making progress toward a goal?
Answer: The accomplishment of interim goals (or "mini goals") are meaningful in that they are necessary for student goal achievement; however, for the purposes of state and federal accountability, the interim goals should be developed and monitored in the classroom and/or in consultation with the student's counselor rather than in SMARTT or on the student goal sheet. Is the Goals Performance Standard affected whether the goal is short or long term?
Answer: There is no distinction between short and long term student goals in the FY2006 Student Goals Pilot. Where and how do we document unanticipated goals in order to gain 'credit' for them?
Answer: Unanticipated goals can be entered in any column (A, B, C). Programs will receive the same credit for unanticipated goals as they will for all other goals. Do goals that require follow up, which are carried over from a prior fiscal year, affect the NEW Goal Performance Standard of goals set? Must such a student STILL set 2 NEW goals?
Answer: All goals that have not been met or removed are carried over from one fiscal year to the next fiscal and are considered "current" goals. What documentation is required for attempts to contact a student? Programs that deal with high-risk students have difficulty contacting students for follow up survey who have been incarcerated. Will the PS understand and make reasonable accommodations for a low response rate, given the particular population served?
Answer: An attempt to contact a student for follow up is not part of the Countable Outcomes Definitions and Required Documentation and therefore cannot be documented. Follow up that is successful is documented through either the survey method or data matching. To insure that Mass ESE maintains reliable follow up goal reporting to USDE, programs should insure a contact rate of at least 50% for these six goals: enter employment; retain employment; obtain GED; obtain secondary school diploma; enter postsecondary education; enter training program. If a student meets a data-matching goal, like enter employment, after exiting from a program, does a student need to have already set that goal in order for data matching to occur?
Answer: Yes. All goals that require follow up (enter employment; retain employment; obtain GED; obtain secondary school diploma; enter postsecondary education; enter training program) must be set before they can be met. What documentation is required for each goal that is achieved?
Answer: Please see the Countable Outcomes Definitions and Required Documentation chart. What documentation is required for the follow up survey method?
Answer: Programs should maintain a copy of the completed survey instrument on file at the program. Is there one follow-up survey that programs should use?
Answer: ACLS does not require programs to use a specific survey for follow up on student goal achievement. Programs can request a sample survey from ACLS or use one of the sample surveys provided on the NRS Web site link: NRS Implementation Guidelines Appendix A Sample Surveys If a student selects the goal "obtain employment" but does not provide a social security number, can we still follow up with that student by survey to show a student met that goal?
Answer: Yes. In the above case, the program should follow up via the survey method. When a student's phone number has been changed or disconnected and the program is unable to reach the person, why does the student continue to be listed in the follow-up due page in SMARTT? At what point is this student dropped from the list, since the program has no way of contacting him or her?
Answer: The student's name continues to appear in the follow-up due page until the program contacts the student or for 18 months, whichever occurs first. Should programs enter long-term goals such as obtain a GED, obtain a diploma, or enter postsecondary education or training if those goals are long-term goals (e.g. a low-level ESOL student wanting a H.S. diploma)? Will entering these long-term goals hurt the program's and the state's performance in federal reports?
Answer: If a student sets a goal to obtain the GED, the SMARTT system automatically determines whether the goal is long-term or short-term based on the student's initial assessment. In general, any goal that is not achieved within a single program year will not negatively impact reporting on the state or the federal level, regardless of whether it is a short or long-term goal. What is the definition of a primary goal and a secondary goal and how are they counted by ACLS and USDE?
Answer: ACLS uses the categories primary goal and secondary goal for NRS reporting. A primary goal represents a student's main reason for enrolling in a program; secondary goals represent all other reasons students enroll. ACLS counts all primary and secondary goals that are countable outcomes in the student goals pilot. Primary and secondary goals that are assessment-based learning goals (improve reading skills, improve writing skills, improve math/science skills, communicate more effectively in English) are captured through the assessment of learning gains. The only goals that are reported to USDE are the six NRS follow up goals: obtain GED, obtain ADP/EDP, enter training program, enter postsecondary education, enter employment, retain employment. Are countable outcome goals reported the same as assessment-based learning goals to ACLS? To USDE?
Answer: No. On the state level, countable outcomes are reported according to the Countable Outcomes Documentation and Required Definitions chart; the assessment-based learning goals (improve reading skills, improve writing skills, improve math/science skills; communicate more effectively in English) are captured through the assessments of learner gains. The only countable outcome goals that are reported to USDE are the six NRS follow-up goals found on table 5: obtain GED, obtain ADP/EDP, enter training program, enter postsecondary education, enter employment, retain employment. The assessment based learning goals are reported to USDE on table 4 and are not linked to countable outcome goals. If a learner achieves the goal "Improve Reading or Writing" or "Improve Speaking and Listening" by making significant educational gain, does this also count as a countable outcome goal met?
Answer: No. In the above case, the achievement of an assessment-based learning goal contributes to the "Learner Gain" performance measure and not the achievement of a countable outcome goal. If a student sets obtain GED as a goal, signs a release of information (ROI), passes the GED, and was in a ESE-funded program this year, does that program get credit for a goal met? What if any of these variables change?
Answer: Yes, in the case cited above, the program gets credit for the student's achievement of the goal. The goal must be set in order for the data match to indicate that the goal has been met. The release of information (ROI) form is not required for the program to obtain credit for a student's achievement since the ROI merely allows the program to view the achievement of the goal. See question #6 in this section above and #36 in this section below for variables governing the data matching timeframe. If the student sets a goal to pass the GED but gets a diploma instead, how does that affect the GED report? Does the diploma get counted?
Answer: Obtaining the GED and obtaining the diploma (obtain ADP/EDP) are two separate outcomes, each of which requires an independent goal setting process. Therefore achieving the ADP/EDP (diploma) goal cannot be counted as achievement of the obtain GED goal. Could we get clarification on this question on the FAQ website: Student Goals Pilot, Question #1: "What is the difference between a goal and a countable outcome?" Our question is, are countable outcomes goals?
Answer: Countable outcomes are column A, B, and C goals. Assessment-based learning goals (improve reading skills; improve writing skills; improve math/science skills; communicate more effectively in English) are not countable outcomes because their achievement is measured by the student's assessment score. How should we record the date set for unanticipated outcomes?
Answer: For all goals except the six NRS follow up goals, it is not necessary to record the date set for an unanticipated outcome. Because of NRS reporting requirements, there are no unanticipated outcomes for the six NRS follow up goals: obtain GED, obtain ADP/EDP, enter training program, enter postsecondary education, enter employment, retain employment. When does follow up occur on goals students achieve after exit?
Answer: Goals that require follow up through data matching and survey have different reporting timeframes, as listed in the table below. Follow up for these goals can be reported up to 18 months after a student exits a program. If a program chooses to follow up after that timeframe, they can do so, but the data cannot be entered into SMARTT.
| Goal | Timeframe |
|---|
| Entered Employment * | Next quarter after goal is set |
| Retained Employment * | Two quarters after "Entered Employment" is met |
| Increase Earnings* | To be determined |
| Obtained a GED and Obtained Secondary School Diploma (EDP/ADP) | Until August 5th after fiscal year ends (all GED scores will have been submitted by that time). Dates met up to August 5th will be counted in the fiscal year ending June 30 to be consistent with the GED timeframe. |
| Enter Postsecondary Education; Enter Training Program | Until November 1st after fiscal years ends (all students will have entered postsecondary education by that time) Dates met up to 11/1 will be counted in the fiscal year ending 6/30 |
*Note: All job related goals are reported using prior fiscal year data to coincide with Title I reporting timeframes. This is timeframe is now required by USDE.
Can programs use the follow-up survey method instead of using data matching?
Answer: The Obtain GED goal can only be followed up via matching of student data. The Enter Employment and Retain Employment goals require follow up via either data matching or the survey method, depending on whether the student provides a SSN. The Obtain EDP/ADP goal, the Enter Occupational Training goal, and the Enter Post-Secondary Education goal can only be followed up via the survey method. The table below summarizes the above conditions.
| Goal | Follow up via Data Matching | Follow up via Survey |
|---|
| Obtained a GED | Yes | No |
| Obtained Secondary School Diploma (EDP/ADP) | No | Yes |
| Entered Employment | Yes (if SSN provided) | Yes (if SSN not provided) |
| Retained Employment | Yes (if SSN provided) | Yes (if SSN not provided) |
| Increase Earnings | To be Determined | To be Determined |
| Enter Occupational Training | No | Yes |
| Enter Post-Secondary Education | No | Yes |
Are students who are enrolled in non-rates based classes expected to set and meet countable outcomes?
Answer: Yes. Can an undocumented student who gets a job be counted for the goal of getting a job? Can programs use both data matching and surveying of students for job information? If a program has undocumented students who get jobs, can the program get a pay stub as the documentation for that student?
Answer: See question # 37 and the table in this section above. Regarding documentation for the "enter employment" goal, please see the Countable Outcomes Definitions and Required Documentation chart. If an employed student comes into the program and becomes unemployed, should that student enter "seeking employment" as a goal?
Answer: Yes, but only if it is the student's goal. Where can we find additional information on goal setting?
Answer: Programs can find goal-setting information at NRS, ALRI, SABES, or by asking
their Program Specialist. Following is a link from the NRS website on goal setting
http://www.nrsonline.org/reference/index.html?chapter=2§ion=5&topic=1&subtopic=9 Does the Performance Measure for Goals affect only Column A items?
Answer: The performance measure for goals affects Column A, B, and C goals. If all goals are kept in the SMARTT system, even goals that have been removed because they are no longer relevant to the student, are they included in the performance measure as well?
Answer: It would be completely removed and not considered as an active goal and not included as a goal set or met. At the CO workshop, participants had the opportunity to suggest other goals, perhaps intermediary, that are not currently listed on the SMARTT goals sheet. Another discussion that people have concerns the verification that is listed for some of the goals. In particular, the goal "helping child with homework" requires fairly tedious documentation. Is there a process in place or being considered to revisit or reconsider what verification is sufficient and realistic to ask students to present?
Answer: Yes, ACLS regularly reviews the Countable Outcomes policy including the required documentation for goals. Programs can suggest changes to the required documentation for goals by sending the request to their program specialist. ACLS will determine whether or not this request can be granted. The program specialist will notify the program of the result of the request.
|
|
