Selected Good Practices in Implementing the Assessment of Student Learning Outcomes
The University has worked consistently and successfully to implement the assessment of student learning outcomes across the curriculum as stipulated by the NEASC reaccreditation report of March 2004. Beginning in 2004-05, each academic dean was charged with the responsibility of:
- reviewing the assessment status of all programs in the school or division;
- providing copies of the NEASC statements on evidence to all department chairs;
- asking departments which were already successful in assessing learning outcomes to agree on a common format for reporting assessment results;
- conferring with the chairs of departments judged to be in need of assessment and explaining the procedures; and
- ensuring annual reporting of assessment findings, including changes in instruction and curricula.
The latest annual assessment reports were submitted to Academic Affairs in May and June 2006. Faculty have set goals, collected and analyzed assessment data for at least one year, discussed the findings in department meetings, and proposed curriculum modifications. The Assessment Committee will continue to review plans and reports and issue recommendations for assessment practices throughout the University. For purposes of this report, a minimum requirement for good practice is that a degree program meets one of the following criteria:
- analyses of student assessment data were conducted and no substantive changes were considered necessary;
- changes in instruction were implemented if indicated by assessment findings; and/or
- changes in curriculum were implemented if indicated by assessment findings.
The following departments and degree programs are considered to have good practices in one or more aspects of assessment. The order of presentation is arbitrary and no rankings should be inferred from it.
Good Practice in the Assessment of Undergraduate Programs
B.A. and B.S. in Biological and Environmental Sciences. As a result of analysis of assessment data from the capstone course, the department is revising the senior research project structure to better address several objectives, including: (1) design and execution of experiments; (2) understanding of the scientific process and methodology; (3) application of quantitative methods; and (4) demonstration of independence, creativity, and initiative in scientific design. The course in Scientific Communication will be reorganized, expanded, and offered earlier in the course sequence for majors.
B.A. and B.S. in Communication. Fifty undergraduate student theses were rated by a faculty committee using a department-developed rubric. After reflecting on the results, faculty plan to implement three changes in 2006-07: (1) assessment will be integrated into the teaching of Senior Thesis rather than occurring at the end of the semester; (2) students who do not complete theses in the prescribed time frame will be included in the analysis in an effort to discover reasons for non-completion; and (3) in order to generate better recommendations for each option within the major, the department will undertake separate analyses of student theses representing each option. WCSU Good Assessment Practices Page 2 11/17/06
B.A. in Computer Science. Faculty focused on three learning objectives: (1) students will understand the facts, concepts, principles and theories relating to computer science; (2) develop in-depth understanding of software applications; and (3) develop abilities to design and implement computer programs. Four faculty members rated student programming assignments sampled from three courses. Measures of completeness and testing revealed lower mean scores for students in Software Engineering than in Web Applications Development. After reflecting on data collected in 2004-05, the faculty implemented several changes to instruction and curriculum. For example, a special session on testing and documentation was added to Data Structures, and two alternatives to the Software Engineering course were developed.
B.A. and B.S. in English and Professional Writing. Attainment of four key learning objectives was measured: (1) ability to organize a lengthy piece of writing; (2) appropriate academic documentation; (3) maintain own voice while incorporating research; and (4) mastery of standard English. Analysis focused on the application of a rubric to student work sampled from baseline courses and compared to work sampled from advanced courses. Faculty concluded that the rubric needs to be customized for each program/option in order to make specific recommendations.
B.A. in History. The program was evaluated by sampling final papers from 2002 through 2005 from the capstone course, Senior Seminar. Analysis of the papers revealed proficiency in the students’ choice of appropriate topics and in the quality of scholarship. However, the building of arguments was somewhat weaker. In response to these findings, the department is developing more rigorous writing requirements in two required courses, Introduction to Historical Research and Historiography.
B.A. and B.S. in Physics, Astronomy and Meteorology. Meteorology students presented weekly television weathercasts and faculty judged their proficiency on three main objectives: (1) processing weather graphics; (2) forecasting and assessing weather situations; and (3) oral communication. An end of semester discussion and presentation was also used to assess forecasting proficiency and written communication skills. In addition, students prepared daily forecasts for a national contest. All but one student on one objective was judged to be “moderately proficient” or better.
B.A. in Theatre Arts. The senior portfolio, comprising a written proposal, oral presentation, and written final paper, was used to assess three objectives: (1) perform competently on creative projects, technical projects, or theatre management; (2) communicate clearly orally and in writing; and (3) apply relevant theories and develop research tools. All full-time faculty evaluated 13 portfolios using a department-developed rubric. Faculty concern focused on areas where substantial numbers of portfolios were rated less than proficient. After discussion of the findings, faculty resolved to implement four recommendations: (1) tell students to begin assembling the portfolio earlier in the semester; (2) integrate a greater diversity of plays across the curriculum; (3) develop a handbook to aid students develop a style of presentation; and (4) begin assembly of production books for all main-stage shows as a resource for students.
B. A. and B.S. in World Languages and Literature. Student writing samples from Spanish conversation and literature courses were evaluated using a department-developed rubric based on national guidelines. Three key learning objectives were addressed: (1) variety in sentence components; (2) support of opinions; and (3) transitions between subtopics and distinctions between principal and secondary ideas. Student performance improved from 100% below proficiency at the beginning level to 70-90% “proficient-to-superior” evaluations at the upper level, depending on objective. After reflecting on the results of the assessment, faculty recommended more complex writing assignments for beginning level students. SPA 203, currently a conversation course, is being modified to include composition. Moreover, a research paper will be required in all 300-level courses, and there will be ongoing assessment of writing proficiency at all levels.
B.S. in Justice and Law Administration. The final research paper in the Research Seminar in Justice and Law was used to measure student attainment of four key learning objectives, including: (1) communicates effectively in written composition; (2) understands concepts of Justice and Law Administration; (3) exhibits critical thinking within the discipline; and (4) demonstrates research capabilities. A sample of ten papers was selected randomly and student identification was concealed. Each of the papers was rated by three faculty WCSU Good Assessment Practices Page 3 11/17/06 members using the same rubric. Nine of the ten papers revealed an adequate understanding of concepts and eight demonstrated competent research skills. However, four papers were judged less than competent in written communication, and five were judged less than competent in critical thinking. To address the two areas of weakness, faculty will be encouraged to add writing components, put more emphasis on APA formatting, and stress critical thinking in each course.
B.B.A. in Accounting. Faculty improved the assessment grid for the program by mapping learning objectives to the required courses in which attainment of the objectives is to be demonstrated. Analysis of the results of a comprehensive exam revealed a clear pattern of lesser competence in three areas: cost behavior; job order; and process costing. These topics will receive greater emphasis. Supervisors rated the oral and written communication skills of 22 interns on a four point scale. The same set of internship evaluations revealed that all interns were rated competent or highly competent in computer technology skills. In 2006-07 the department will meet to discuss implications of the comprehensive exams for curricular change, as well as possible improvements to the internship program.
B.B.A. in Finance. Assessment efforts focused on two types of portfolio management: an Internet investment simulation in Investment Analysis and Portfolio Management, and case studies in the capstone course, Cases in Managerial Finance. The rubric for evaluating the presentation of case studies has six items, including knowledge of the financial topic, analytical competency, and ability to solve a financial problem. In addition, student projects in Financial Decision Models and Quantitative Methods in Finance are reviewed by faculty teams. Changes to instruction include increased emphasis on cash flow analysis and problem solving, in response to demonstrated weaknesses in student projects.
B.B.A. in Management. First, faculty developed and revised a class presentation instrument and administered it to students in the Current Issues in Management course. The primary purpose was to assess written and verbal communication skills for professional business settings. Second, faculty developed a Team Effectiveness Critique for application to group projects. Each team member was asked to rate 11 dimensions of effectiveness for its team. Endpoints of the scale were specific statements, such as “One person dominates…” and “There is full participation in leadership….” Third, faculty utilized curriculum mapping to gather information on the methods for assessing all six objectives for student learning.
B.B.A. in Marketing. In the capstone course, Marketing Management, students are expected to develop a comprehensive marketing plan. One learning outcome is to transform the strategies into marketing programs that can be implemented. The plans are examined annually, both by faculty in the department and by external evaluators. Faculty review is documented in minutes of department meetings. External evaluators, most of whom are marketing plan clients, rate the plans on several criteria including thoroughness, usability, situational analysis, and plan integration. Several changes to both instruction and curriculum have been implemented based on the findings. Instructional changes include an increased emphasis on secondary marketing research sources, data acquisition, analysis and statistics. Curricular changes include a two course sequence in Marketing Communications, and specific prerequisites for the capstone course.
B.S. in Education. Three instruments were used for program evaluation of elementary education. Two of the instruments, PRAXIS I and II, were designed and administered by the Educational Testing Service (ETS). PRAXIS I measures the level of candidate proficiency in reading, writing and mathematics. A passing score is required for formal admission to the professional Elementary Education program. PRAXIS II measures knowledge and understanding of curriculum, instruction and assessment in language arts, mathematics, social studies and science as it applies to elementary education. The third instrument is the state Student Teacher Evaluation Instrument (STEI), designed to assess how well students accomplish their initial teaching experiences in actual classrooms. It is based on Connecticut’s Common Core of Teaching established by Connecticut’s State Department of Education. For example, the question, “How did the teacher communicate with students about assessment and evaluate student performance?” is addressed by four performance indicators, each of which is rated on a continuum indicating levels of proficiency. PRAXIS II results for 2004-05 revealed that 96.4% of the undergraduate elementary education program’s certification candidates WCSU Good Assessment Practices Page 4 11/17/06 were proficient or better in language arts, mathematics, and science. Similar results were obtained using the STEI: 20.4% proficient and 79.6% superior in language arts, mathematics, and science. Similar results were obtained for undergraduate secondary education program candidates. Based on the evidence of student proficiency, no changes in either curriculum or instruction are warranted.
B.S. in Health Promotion and Exercise Science (HPX). The department offers degree programs in Health Education, Community Health, and, beginning in fall 2006, Health Promotion Studies. Students completing the degree in Health Education should be able to integrate the National Health Education Standards for students with relevant health content into a competency-based curriculum. To measure changes in student abilities to assess individual and community needs for health education, faculty devised a pretest-posttest approach. Content drawn from eight required courses was used to construct a test item bank. A 50-item pretest is given to all incoming students in HPX 100, and a posttest of similar format will be given to all completers of these degree programs in their final semester. First administered in January 2006, the pretest revealed a need for making two courses as required instead of elective courses: Mental Health and Human Sexuality. After completion of the student teaching assignment, each student has an exit interview with the faculty coordinator. The cooperating teacher assessment of the student’s performance is compared to the student teacher’s self-evaluation. The student’s strengths, weaknesses, and career goals are discussed during the exit interview. All eight student teachers were judged competent on all aspects of the 2005 performance evaluation.
B.A., B.M., and B.S. in Music and Music Education. Key objectives for all music degree programs include: (1) performance ability, assessed at the end of each semester of applied study during jury exams; (2) knowledge of music history, theory, and analysis, measured during four semesters of music theory and two semesters of music history; and (3) skill-based areas, measured daily during four semesters of ear training and sight singing. At a jury exam, each student performs repertoire specific to his or her instrument or voice type. In addition, all students sight-read and perform rudiments (scales, arpeggios, percussion). There is a rubric for each area of competency. In addition, students enrolled in the B.M.-Jazz and B.M.-Classical programs perform recitals at the end of their junior and senior years. These recitals are evaluated by a faculty panel, and include extensive program notes which give the student an opportunity to display theoretical, analytical and musicological knowledge of the works being performed. Students in the B.S. and the B.M.- Classical programs take a capstone course, Musical Form and Analysis, for which they are required to analyze 15 musical compositions and provide stylistic, theoretical and historical views of each work. Finally, B.S. students must demonstrate competence in planning, instructing and assessing the learning of K-12 students. They must also pass the PRAXIS I and II exams before receiving provisional certification. One result of faculty reflection on assessment results is the adoption of stricter standards for admission to the B.M.-Jazz program as well as the sophomore barrier assessment prior to enrollment in upper division music courses. Another finding is that one-fifth of incoming majors were required to take Music Essentials. Because faculty consider this an unacceptably large proportion, plans are underway to implement an online theory assessment which will include tutoring in basic fundamentals. Some work remains to be done before comprehensive assessment results are available. Jury sheets, currently hand-scored, are being adapted to scanner technology to facilitate analysis. Similarly, there is a proposal to administer the Advanced Measures of Audiation, a tool to measure music aptitude in adults, predict program success, and facilitate advising and placement.
B.S. in Nursing. The department continues to meet the standards of its national accrediting body, the Commission on Collegiate Nursing Education. Graduates of the baccalaureate programs are expected to be able to think critically, communicate effectively, and perform nursing interventions appropriate to their practice roles. Three types of direct measure of student learning are utilized: Preceptor Evaluation; Computerized Adaptive Testing (CAT); and Cumulative Professional Portfolio (CPP, for RN-B.S. only). Employers are also asked to rate the same skills and competencies used in the CPP. Over several years, one can compare the perceptions of faculty, employers and graduates to see the extent of agreement. The preceptor evaluation was designed to be administered at the end of the capstone course. The CAT test is a WCSU Good Assessment Practices Page 5 11/17/06 standardized mock exam designed to prepare students to perform well on the National Council Licensure Examination for Registered Nurses (NCLEX-RN). CAT measures several dimensions of nursing competency, including pharmacological therapies, reduction of risk potential, physiological adaptation, safety and infection control, and basic care and comfort. One measure of the success of CAT and coursework is the 100% NCLEX-RN pass rate for students graduating in 2004, 2005 and 2006. In addition, all degree candidates scored 90% or better on CAT. The rubric for the CPP operationalizes the broad program goals. For example, critical thinking comprises use of the process of scientific inquiry and research findings to improve nursing care, in addition to implementing and evaluating the care provided to individuals, families and communities. Application of the nursing process is also evaluated using reflective writing samples, including the capstone experience.
B.A. in Social Work. The department continues to meet the standards of its national accrediting body, the Council on Social Work Education. In 2005-06 the department focused on three student learning outcomes mandated by CSWE: (1) understand personal, professional, organizational and client system values and practice in a manner consistent with the NASW Code of Ethics; (2) use communication skills differentially across client populations, colleagues and communities; and (3) use supervision and consultation appropriate to social work practice. Each of the outcomes is measured by three instruments: (1) the Final Senior Field Practicum Evaluation (FSFPE); (2) assessments by field agency personnel; and (3) the Baccalaureate Social Work Education Assessment Package Exit Survey (BEAP). FSFPE, completed by instructors of the field practicum, evaluates student performance across nine areas in the two-semester practicum. This evaluation is conducted at the end of spring semester of senior year. Self-study data from 2003 compared to 2005 data indicate that field instructors continue to give high ratings to students’ understanding of values and ethics, as well as communication and supervision/consultation skills. Assessments by field agencies are similar to internship evaluations used to evaluate of the degree programs. In alternate years the department mails surveys to field agency personnel, including 15 items measuring students’ functioning in the context of program objectives. Nearly all agency staff (94%) responded that students “usually” or “consistently” applied professional values and standards to practice; and similar results were obtained for effective use of supervision and consultation (98%). However, the results for communication skills were somewhat less positive: 86% of agency staff who supervised seniors reporting that those students “usually” or “consistently” applied communication skills appropriately; and only 75% of agency staff whose observations were based on juniors reported the same levels of communication skills. After discussing these findings, faculty recommended keeping course content as designed but putting more emphasis on student perspectives of the scope and nature of all types of communication.
Good Practice in the Assessment of Graduate Programs
M.F.A. in Visual Arts. The core of the program is the student’s individual portfolio developed over four semesters of increasing intensity designed to develop individual originality and enhance expertise of technique and expression. Student portfolios are evaluated weekly by resident faculty and biweekly by visiting artists. Day-long midterm and final group critiques are conducted with resident faculty. The culmination of the program is a public thesis exhibition, which is open to peer review by visiting artists, advisory board members, university officials, faculty, students, media representatives, and the public. A curriculum modification initiated in 2005-06 was an exhibition of graduate work in New York. This external review has been formally added to the program. In 2006-07 faculty will work on tying evidence to specific measures and refining a standardized rubric for critiques.
M.A. in History. Candidates have the option of a comprehensive essay examination or a thesis. After reviewing recent essays and theses, faculty identified student deficiencies in research and writing. As a result of discussions within the department, three curricular changes are underway: (1) revising standards and procedures for the comprehensive exam; (2) adding a required seminar in Research and Historiography; and (3) strengthening the area of United States history with four new courses. WCSU Good Assessment Practices Page 6 11/17/06
M.H.A. in Health Administration. Candidates are required to complete a final project in conjunction with the capstone course, Strategic Management in Health Care Organizations. Examinations and case analyses are also used to assess student attainment of three primary goals: (1) students will demonstrate a thorough understanding of the concepts, tools, and techniques of health services finance: (2) in-depth knowledge of the history and development of the American health care system; and (3) an understanding of health services management practices. In the spring of 2006, outside evaluators were chosen from the university and a nearby hospital. An assessment rubric was used both for peer review and external review, and the scores were examined for consistency. No program changes are considered necessary.
M.B.A. in Business Administration. Two key learning objectives for MBA students were assessed in 2005- 06: (1) students will demonstrate knowledge of the fundamental business and organizational functions and activities, and (2) students will demonstrate ability to think strategically about business and organizational problems. The capstone course, Strategic Management, provided opportunities for faculty and external reviewers to evaluate student projects. Five faculty members served as evaluators of capstone projects in spring 2005, and gave each project a rating on a 100-point scale for each of 14 dimensions, including “understanding company needs” and “implementation detail.” Faculty will continue to refine the capstone rubrics as needed. After discussion of the project ratings, faculty modified the course sequence to strengthen students’ quantitative skills. Specifically, two prerequisites were mandated: (1) Statistics will be a prerequisite for Economic Analysis for Managers; and (2) either Statistics or Economic Analysis for Managers will be a prerequisite for Managerial Finance.
M.S. in Counselor Education. The Council for Accreditation of Counseling and Related Educational Programs (CACREP) provides standards for the assessment of student learning. Site supervisor evaluations of practica and internship experiences are used to measure the development of experiential skills important for the functioning of professional counselors. Student video sessions and clinical folders are reviewed for evidence of knowledge of the concepts, theories and practices of counseling. Results of the National Counselor Exam (NCE) are used to assess the students’ preparation for certification in counseling. Finally, faculty evaluation of the practicum research paper provides additional evidence of all of the competencies. All evidence gathered to date supports the efficacy of the program as designed and currently implemented.
M.S. in Education. Program evaluation focused on two goals: (1) students in all options will demonstrate the ability to be both consumers of information and producers of educational research through the use of inquiry, critical analysis and synthesis; and (2) students will become proficient in the use of assessment techniques and analysis of student learning. Direct measures were applied to research papers produced in Introduction to Educational Research and to three assignments in Measurement and Evaluation. Two curricular changes were made as a result of reflection on the findings. First, Introduction to Educational Research and the Educational Research Seminar have been redesigned to relate more directly to classroom practices. In an effort to avoid conducting research with human subjects, the content of projects has tended to move away from action research to projects based on aggregate data which can be less meaningful for the student teacher. To counter this trend, an action research approach will be re-infused into the course sequence. Second, the combination of activities offered in Measurement and Evaluation satisfies the state and national standards for measurement and evaluation. However, all sections of the course do not emphasize the same concepts and outcomes. The course focus will be realigned across sections to ensure that students have a successful course sequence.