The Reading, Writing, Speaking, and Listening Tasks Important for - ETS

Loading...

Monograph Series

MS - 21 NOVEMBER 2001

The Reading, Writing, Speaking, and Listening Tasks Important for Academic Success at the Undergraduate and Graduate Levels Michael Rosenfeld Susan Leung Philip K. Oltman

The Reading, Writing, Speaking, and Listening Tasks Important for Academic Success at the Undergraduate and Graduate Levels

Michael Rosenfeld Susan Leung Philip K. Oltman

Educational Testing Service Princeton, New Jersey RM-01-03

®

®

Educational Testing Service is an Equal Opportunity/Affirmative Action Employer. Copyright © 2001 by Educational Testing Service. All rights reserved. No part of this report may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Violators will be prosecuted in accordance with both U.S. and international copyright laws. EDUCATIONAL TESTING SERVICE, ETS, the ETS logos, TOEFL, the TOEFL logo, TSE, and TWE are registered trademarks of Educational Testing Service. The Test of English as a Foreign Language, Test of Spoken English, and Test of Written English are trademarks of Educational Testing Service.

To obtain more information about TOEFL products and services, use one of the following:

Email: [email protected] Web site: http://www.toefl.org

Foreword The TOEFL Monograph Series features commissioned papers and reports for TOEFL 2000 and other Test of English as a Foreign Language (TOEFL®) test development efforts. As part of the foundation for the TOEFL 2000 project, a number of papers and reports were commissioned from experts within the fields of measurement and language teaching and testing. The resulting critical reviews and expert opinions have helped to inform TOEFL program development efforts with respect to test construct, test user needs, and test delivery. Opinions expressed in these papers are those of the authors and do not necessarily reflect the views or intentions of the TOEFL program. These monographs are also of general scholarly interest, and the TOEFL program is pleased to make them available to colleagues in the fields of language teaching and testing and international student admissions in higher education. The TOEFL 2000 project is a broad effort under which language testing at Educational Testing Service® (ETS®) will evolve into the 21st century. As a first step, the TOEFL program recently revised the Test of Spoken English (TSE®) and introduced a computer-based version of the TOEFL test. The revised TSE test, introduced in July 1995, is based on an underlying construct of communicative language ability and represents a process approach to test validation. The computer-based TOEFL test, introduced in 1998, takes advantage of new forms of assessment and improved services made possible by computer-based testing, while also moving the program toward its longer-range goals, which include: • • • •

the development of a conceptual framework that takes into account models of communicative competence a research agenda that informs and supports this emerging framework a better understanding of the kinds of information test users need and want from the TOEFL test a better understanding of the technological capabilities for delivery of TOEFL tests into the next century

Monographs 16 through 20 were the working papers that laid out the TOEFL 2000 conceptual frameworks with their accompanying research agendas. The initial framework document, Monograph 16, described the process by which the project was to move from identifying the test domain to building an empirically based interpretation of test scores. The subsequent framework documents, Monographs 17-20, extended the conceptual frameworks to the domains of reading, writing, listening, and speaking (both as independent and interdependent domains). These conceptual frameworks guided the research and prototyping studies described in subsequent monographs that resulted in the final test model. As TOEFL 2000 projects are completed, monographs and research reports will continue to be released and public review of project work invited. TOEFL Program Office Educational Testing Service

i

Abstract The primary purposes of this project were: 1) to aid in translating theoretical frameworks developed in reading, writing, speaking, and listening by the TOEFL Framework Teams into task statements that undergraduate and graduate students need to perform in order to complete their academic programs; 2) to have undergraduate and graduate faculty experienced in teaching nonnative speakers of English, as well as undergraduate and graduate students who are nonnative speakers of English, review and evaluate the statements through a survey; 3) to provide analyses of these results that aid in the design of test specifications and assessment measures for TOEFL 2000; and 4) to document these results to help support the validity of TOEFL 2000. Toward this end, 155 undergraduate faculty, 215 graduate faculty, 103 undergraduate students, and 242 graduate students from 21 universities across the United States and Canada rated 42 task statements developed from the frameworks. Both faculty and students rated the importance of each task statement to the successful completion of coursework; in addition, faculty indicated the degree to which tasks are more often characteristic of more academically successful, nonnative speakers than their less successful counterparts. Responses were analyzed by respondent groups as well as specific subgroups. The results obtained confirm the judgments of the linguistic specialists who formulated the tasks. Faculty and students agree that the tasks are relevant and important for completing coursework at both the undergraduate and graduate levels. These findings support the use of this pool of tasks in the design of both test specifications and assessment measures for undergraduate and graduate students across a wide range of subject areas. Key Words:

Language testing, task analysis, validity, international students, academic communication skills

ii

Acknowledgments This report is the product of the efforts of a great many people to whom the authors are indebted. In particular, we would like to recognize the members of our Advisory Committee and the TOEFL 2000 Framework Teams for their assistance throughout all phases of this project. Our appreciation also goes out to coordinators at participating universities for their aid in data collection, and to all faculty and students who took time out of their busy schedules to complete our survey instrument. The authors would also like to recognize Regina Mercadante for her administrative assistance throughout this project; Gerry Kokolis, who was responsible for data analysis; and Paul Angelis, who was our project monitor.

iii

Table of Contents Page 1. Introduction .................................................................................................................................................... 1 .Purpose ..................................................................................................................................................... 1 2. Methods ......................................................................................................................................................... 3 Overview of the Methodology................................................................................................................... 3 The Steering Committee ........................................................................................................................... 3 Defining the Draft Domain of Task Statements........................................................................................ 3 Developing the Survey Instruments .......................................................................................................... 5 Selection of Schools, Faculty, and Students ............................................................................................. 7 Pilot Testing of Draft Survey Instruments ............................................................................................... 10 Production and Administration of Survey Instruments ........................................................................... 10 Data Analyses ........................................................................................................................................... 11 3. Results .......................................................................................................................................................... 14 Response Rate .......................................................................................................................................... 14 Respondent Demographics ....................................................................................................................... 14 4. Faculty .......................................................................................................................................................... 16 Undergraduate Faculty ............................................................................................................................. 16 Graduate Faculty ...................................................................................................................................... 23 Comparing Ratings of Undergraduate and Graduate Faculty ................................................................. 29 5. Students ........................................................................................................................................................ 31 Undergraduate Students ........................................................................................................................... 31 Graduate Students .................................................................................................................................... 34 Comparing Ratings of Undergraduate and Graduate Students................................................................ 42 Comparing Ratings of Faculty and Students............................................................................................ 44 6. Summary and Conclusions .......................................................................................................................... 46 Summary.................................................................................................................................................... 46 Conclusions and Implications .................................................................................................................. 47 References ......................................................................................................................................................... 51 Appendices ........................................................................................................................................................ 53

iv

List of Tables

Table 1

Page Task Statements Rated Below 3.50 by Undergraduate Faculty Respondents ............................ 17

Table 2

Task Statements Rated Most Important by Undergraduate Faculty Respondents ..................... 18

Table 3

Task Statements Rated 3.50 or Higher by Undergraduate Faculty Across Subject Areas......... 19

Table 4

Intercorrelations of Mean Importance Ratings by Subject for Undergraduate Faculty ............ 20

Table 5

Task Statements Rated Highest by Undergraduate Faculty Respondents .................................. 21

Table 6

Task Statements Rated Below 2.0 by Undergraduate Faculty Across Subjects......................... 22

Table 7

Intercorrelations of Mean Relationship Ratings by Subject for Undergraduate Faculty .......... 23

Table 8

Task Statements Rated Below 3.50 by Graduate Faculty Respondents ..................................... 24

Table 9

Task Statements Rated Most Important by Graduate Faculty Respondents............................... 25

Table 10

Intercorrelations of Mean Importance Ratings by Subject for Graduate Faculty ...................... 26

Table 11

Task Statements Rated Highest by Graduate Faculty Respondents............................................ 27

Table 12

Task Statements Rated Below 2.0 by Graduate Faculty Across Subjects .................................. 28

Table 13

Intercorrelations of Mean Relationship Ratings by Subject for Graduate Faculty..................... 28

Table 14

Task Statements Rated Below 3.50 by Undergraduate Student Respondents............................ 32

Table 15

Task Statements Rated Most Important by Undergraduate Student Respondents ..................... 32

Table 16

Task Statements Rated Below 3.50 by Undergraduate Students by Educational Status............ 33

Table 17 Table 18

Intercorrelations of Mean Importance Ratings by Educational Status for Undergraduate Students ............................................................................................................... 34 Task Statements Rated Below 3.50 by Graduate Student Respondents ..................................... 36

Table 19

Task Statements Rated Most Important by Graduate Student Respondents............................... 36

Table 20

Task Statements Rated Below 3.50 by Graduate Students by Subject ....................................... 38

Table 21

Intercorrelations of Mean Importance Ratings by Subject for Graduate Students .................... 40

Table 22

Task Statements Rated Below 3.50 by Graduate Students by Geographic Region of Origin ........................................................................................................................... 41

Table 23

Intercorrelations of Mean Importance Ratings by Geographic Region of Origin for Graduate Students ........................................................................................................................ 42

Table 24

Intercorrelations of Importance Ratings Across All Respondents .............................................. 44

Table 25

Percent Agreement of Mean Importance Ratings Across All Respondents ............................... 45

Table 26

Example Tasks that Meet Suggested Criteria for Inclusion in TOEFL 2000 ............................ 49

v

1. Introduction The TOEFL 2000 project has developed theoretical frameworks for conceptualizing the reading, writing, listening, and speaking domains that will be used to guide the development of a new language proficiency examination to replace the current Test of English as a Foreign Language (TOEFL) examination. Currently, these frameworks do not include a detailed description of the reading, writing, listening, and speaking tasks that are important for the competent academic performance of students at the undergraduate and graduate levels. This study was conducted at the request of the TOEFL 2000 Framework Teams in order to define sets of tasks consistent with the frameworks that are important for competent academic performance at these levels. This study will provide a link between the frameworks and the specific English language proficiency tasks required for competent academic performance at the undergraduate and graduate levels. This study can be conceived of as a “job analysis” of the English language proficiency tasks that need to be carried out competently if students are to be successful in their course work. Job analysis methodology has often been used in language assessment. McNamara (1996) cites the statement by Jones (1979) that the first step in language performance assessment is the job analysis (the “job” in this case being that of the student). McNamara also quotes Davies (1984), who states that when designing a language proficiency test, “The best safeguard against an unsatisfactory test is a professional job analysis at the outset” (p. 52). McNamara describes some of his own work in developing the Occupational English Test, in which he made extensive use of job analysis methodology. He developed detailed listings of the reading, writing, speaking, and listening tasks required in the practice of several health professions, and used judgments of these tasks made by experts in the fields to design a battery of English proficiency tests. According to Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999), job analysis is one of the primary methodologies used to demonstrate the content validity of assessment measures. The data obtained from this study should be very useful in the design and validation of TOEFL 2000. Purpose This study was designed to provide a number of primary outcomes relevant to the TOEFL 2000 project. First, it was intended to provide a link between the frameworks and the specific English language proficiency tasks (reading, writing, speaking, and listening) judged by undergraduate and graduate faculty and students to be important for competent academic performance across a range of subject areas. This information can also be used to modify aspects of the frameworks where necessary. Second, the data generated from this study should provide information useful for building new TOEFL 2000 assessment measures by identifying specific tasks that can be simulated in the assessment or used to identify the language proficiencies necessary to perform them competently. Third, test specifications that can be linked to tasks that are judged to be important across a range of subject areas at both the undergraduate and graduate levels provide an important aspect of content validity evidence to support the use of TOEFL 2000. Fourth, the task statements judged to be important for competent academic performance

1

can be used to build criterion measures to evaluate the criterion-related validity of the TOEFL 2000 at some later date. A secondary benefit of the study might include the discovery of a more task-related way of describing to both faculty and students what TOEFL 2000 measures. In addition, the information gathered about tasks can be used to inform the development or modification of English as a Second Language (ESL) curricula throughout North America. The study was designed to answer the following 10 major research questions: 1. What task statements are judged to be important by faculty for satisfactorily completing coursework in each of the six fields of study1 at the undergraduate level? 2. What is the overlap in task statements judged to be important by faculty for satisfactorily completing coursework across the six fields of study at the undergraduate level? 3. What task statements are judged to be important by faculty for satisfactorily completing coursework in each of the six fields of study at the graduate level? 4. What is the overlap in task statements judged to be important by faculty for satisfactorily completing coursework across the six fields of study at the graduate level? 5. What is the overlap in task statements judged to be important by faculty for satisfactorily completing coursework across the undergraduate and graduate levels? 6. Of the task statements judged to be important by faculty for satisfactorily completing coursework, which distinguish more academically successful, nonnative speakers of English from their less successful counterparts? 7. What task statements are judged by graduate students within each of the six fields of study to be important for satisfactorily completing the courses they have taken thus far in their programs? 8. What is the overlap of task statements judged to be important by graduate students across the six fields of study? 9. What task statements are judged by undergraduate students to be important for satisfactorily completing the courses they have taken thus far? 10. What is the overlap of task statements judged to be important for satisfactorily completing coursework between faculty and students? The answers to these questions are important to the design of TOEFL 2000 assessment measures as well as the documentation of their validity.

1

The six subject areas selected for inclusion in this study were chemistry, computer and information sciences, electrical engineering, business/management, psychology, and history. The rationale for this selection is provided in the next section.

2

2. Methods Overview of Methodology The process described here was designed to involve several groups with differing perspectives in ways that reflect their expertise and experience. A Steering Committee consisting of one member from each of the four TOEFL 2000 Framework Teams as well as one representative from the TOEFL program (see Appendix A) provided overall guidance and advice on each of the major steps in the study. This ensured that the procedures employed in this study provided data that would be useful in the development of assessment measures for TOEFL 2000. Members of the Framework Teams (see Appendix B) assisted in developing the initial list of reading, writing, speaking, and listening task statements. The intent was to write these statements in language that was clear and understandable to nonlinguists. Graduate and undergraduate faculty members who teach a range of subjects and courses, as well as undergraduate and graduate students, also participated in describing the communication tasks important for competent academic performance. The final set of task statements were placed in survey format, along with appropriate rating scales, and were administered to faculty and students across a range of subject areas at the undergraduate and graduate levels. Analyses were conducted to identify tasks that were judged to be important by faculty and students at the undergraduate and graduate levels as well as those that may be unique to one of the levels. These important tasks should play a primary role in the development and validation of the TOEFL 2000 assessments. The Steering Committee A Steering Committee comprised of one member from each of the four TOEFL 2000 Framework Teams as well as one representative from the TOEFL program staff was established. The Committee's charge was to provide guidance and advice on all major project decisions. Their role ensured that the research and development needs of TOEFL 2000 were reflected in each step of this project and that the procedures used were acceptable to the linguistic community. A member from each of the Framework Teams volunteered to assist in the development of the proposal for this study and to be considered for membership on the Steering Committee. However, the final selection of Steering Committee members (including the TOEFL program representative) was made by the TOEFL 2000 Research and Development Oversight Committee (see Appendix C). A good deal of the communication with the Steering Committee was conducted by email and telephone. Defining the Draft Domain of Task Statements Several steps were used to define the reading, writing, speaking, and listening tasks thought to be important for academic success at the undergraduate and graduate levels. Each of these is described below. First draft of task statements. The first draft of task statements was developed by each of the four Framework Teams. Each Team was asked to generate from 10 to 20 task statements that they believe are consistent with their frameworks and important for competent academic

3

performance across a range of subjects. These tasks were viewed as important to academic success for both native and nonnative speakers of English. Educational Testing Service (ETS) research staff worked with an on-site member of each Framework Team to aid in the generation of the initial list of task statements. Task statements from previous needs analysis studies (Bridgeman & Carlson, 1983; Hale et al., 1996; Powers, 1986) were reviewed and considered for possible inclusion in the initial draft. Research staff then reviewed the task statements related to each of the four framework areas with all team members in a given area by email or telephone. Comments from each team were reviewed by the Steering Committee, and a second draft of the statements was developed. Since these statements were to be used with faculty members and students across a range of subject areas, they were written using language that was as clear and straightforward as possible. The intent was to develop a single set of statements to which faculty members as well as students could respond. This revised draft of task statements served as the starting point for review and comment by faculty and students, who were later given the opportunity to add, delete, or modify task statements. Review of revised draft by faculty and students. Twenty-six faculty members and students at both the graduate and undergraduate levels, representing a range of schools (see Appendix D) and subject areas, reviewed the revised draft of task statements. Six subject areas were selected for inclusion in this study. Chemistry, computer and information sciences, electrical engineering, and business/management were selected because they are areas of high enrollment by foreign students (Davis, 1995). Psychology was selected to represent the social sciences because, of all the social sciences, it has the highest enrollment of foreign students. History was also selected because it is a basic requirement in almost all undergraduate programs. Participants in this review process were selected from a range of colleges and universities that enroll large numbers of nonnative speakers of English. The top 100 schools, in terms of the foreign student enrollment, were identified (Davis, 1995) and served as the sampling frame from which to select colleges and universities for participation in the study. Geographic diversity, size, and whether the school is public or private were considered in the selection of institutions for participation in this phase of the study. Research staff consulted with TOEFL program staff, as well as members of the Steering Committee, for guidance in the final selection of schools to participate in this review. Faculty members from each of the six subject areas, who are also experienced in teaching nonnative speakers of English, were identified at each of the selected schools. ETS research staff emailed these individuals an explanation of the study and the review process, along with the draft list of task statements. Faculty members were asked to first review the draft list of tasks. They were asked to then indicate if the task statements were clear and understandable, and covered the reading, writing, speaking, and listening tasks they believe are important for competent academic performance in their subject areas. They were also asked to identify any important tasks they felt were missing. Each faculty member was asked to recommend and encourage one of their students to participate in the review as well. Both faculty and students were given the option of providing their review comments to ETS research staff by return email or telephone — whichever was more convenient for them. The majority of reviews were

4

conducted by email. Participants' involvement in the review process took approximately one hour in total. Faculty responses were received from 13 different institutions. Student responses were received from seven different institutions. Several of the institutions provided responses from multiple faculty members in different subject areas. One of the institutions provided two student responses from two different subject areas. A total of 18 faculty responses (eight at the undergraduate level and 10 at the graduate level) and eight student responses (four undergraduate students and four graduate students) were received. The majority of graduate and undergraduate students involved in the review were nonnative speakers of English. Respondents from all six fields of study were included in the review process. The comments obtained from the faculty and student reviews were used in the development of the final list of task statements. Developing the final list of task statements. Faculty and student comments related to each of the four content areas were next reviewed with an ETS member of each Framework Team using email and teleconferencing. Changes were made and circulated by email to each member of each team for comment and review. After comments were received and collated, they were reviewed with the Steering Committee. The Steering Committee, in conjunction with Research staff, considered the comments obtained from each of the Framework Teams, as well as the progress made on other TOEFL projects that were currently underway (based on input from the Oversight Committee), to arrive at the final set of task statements. Developing the Survey Instruments Separate survey instruments were developed for faculty, graduate students, and undergraduate students. Each of the three survey instruments contained the same set of task statements. However, the instruments contained different rating scales and asked for different background information. Each instrument is described below. Faculty survey instrument. The faculty survey instrument contained the reading, writing, speaking, and listening tasks thought to be important for competent academic performance, rating scales for use in evaluating each of the task statements, and background questions that would be used to describe the faculty completing the survey instrument. All aspects of the survey instrument — including the background questions developed by Research staff — were reviewed and approved by the Steering Committee. The two rating scales provided below were used with the faculty survey instrument. The first, an “Importance Rating Scale,” was designed to assess the importance of each task statement for competent academic performance. This scale was intended to identify the tasks important for competent academic performance for all students — both native and nonnative speakers of English. (For the purposes of the Importance Rating Scale, “competent academic performance” was operationalized as “the satisfactory completion of course work.”) The second scale — the

5

“Relationship Rating Scale” — was intended to focus on tasks that distinguish more academically successful, nonnative speakers of English from their less successful counterparts. While each scale provided information about different groups of students, both types of information are useful to the development of test specifications. Importance Rating Scale How important is it for students to perform this task competently in order to satisfactorily complete the courses you teach? (0) (1) (2) (3) (4) (5)

A student in my courses would not need to perform this task. Slightly important Moderately important Important Very important Extremely important

Relationship Rating Scale Do the more academically successful nonnative speakers of English in your classes perform this task better than the less academically successful nonnative speakers of English? (1) All nonnative speakers of English perform this task equally well. (2) The more academically successful nonnative speakers of English GENERALLY perform this task better. (3) The more academically successful nonnative speakers of English ALMOST ALWAYS perform this task better. Graduate student survey instrument. The graduate student survey instrument contained the same task statements as the faculty instrument. As with the faculty survey instrument, the Steering Committee also reviewed and approved the background questions developed by Research staff that would be used to describe the graduate students who completed the survey instrument. However, this survey instrument was designed to be completed by graduate students who are nonnative speakers of English. Thus, in order to make it easier for students to complete and thereby increase the likelihood of a higher return rate, the Steering Committee recommended that it contain only one rating scale. The Importance Rating Scale on the following page was designed to assess the importance of each task statement for competent academic performance.

6

Importance Rating Scale How important has it been for you to perform this task well in order to complete the courses you have had in your graduate program? (0) (1) (2) (3) (4) (5)

I did not need to perform this task. Slightly important Moderately important Important Very important Extremely important

Undergraduate student survey instrument. The undergraduate student survey instrument contained the same task statements as the faculty and graduate student instruments. As with those instruments, the Steering Committee reviewed and approved the background questions developed by Research staff that would be used to describe the undergraduate students who completed the survey instrument. Again, because this survey instrument was designed to be completed by undergraduate students who are nonnative speakers of English, it contained one rating scale, provided below, that was very similar to the one included in the graduate student survey. Importance Rating Scale How important has it been for you to perform this task well in order to complete the courses you have had in your undergraduate program? (0) (1) (2) (3) (4) (5)

I did not need to perform this task. Slightly important Moderately important Important Very important Extremely important

Selection of Schools, Faculty, and Students ETS Research staff worked with the Steering Committee and TOEFL program staff in the initial selection of colleges and universities for possible participation in this study. The primary factor considered in the selection process was that the schools have a large enrollment of foreign students and, to the extent possible, be on the list of the top 100 schools with 1,000 or more

7

foreign students enrolled (Davis, 1995). A second factor necessary for selection was that the schools have undergraduate and graduate programs in all, or most, of the six subject areas identified for the study (chemistry, computer and information science, electrical engineering, business/management, psychology, and history). Diversity with respect to geographic region and type of school (public vs. private) was also considered. Following the initial selection of more than 20 schools, a variety of techniques were employed by ETS Research staff to identify a contact person at each of the colleges and universities who could either serve as Study Coordinator or refer us to a colleague or graduate student who would be interested in this position. The main resource used in identifying contacts was the “TOEFL Committee Lists” notebook, which contains names of people who have served on a variety of TOEFL committees over approximately the past 10 years. These committee lists included members of the TOEFL Board, as well as members of the Examiners, Executive, Outreach and Services, Research, Test of Spoken English, and Test of Written English (TWE®) Committees, among others. When no contact could be found for a school among these lists, a second technique was employed: an Internet search was conducted of the Web sites of individual schools to locate a contact person affiliated with an office or department related to international student admissions or ESL studies. Most frequently, the Director of the Office of International Admissions was the initial contact at these schools. A recruitment letter (see Appendix E) was emailed to the initial contact person at each school explaining the purpose of the study, the involvement required of participating schools, and the responsibilities of a study coordinator at each school. Initial contacts were asked to decide, after reviewing the information in the letter, if they would like to take on the position of TOEFL 2000 Study Coordinator or could refer us to a colleague or graduate student who might be interested in this role. Once a contact person had committed to be a study coordinator, additional information was sent with more details about the study and suggestions for recruitment of study participants on campus. The majority of the communication between Research staff and study coordinators was conducted through email, but study coordinators were always given the option of communicating by telephone or fax when they needed clarification about any aspect of their involvement in this study. The coordinator at each participating institution was asked to identify a total of 24 faculty members (12 undergraduate and 12 graduate faculty) who currently teach or have previously taught students who are nonnative speakers of English, and 18 students (6 undergraduate and 12 graduate students) who are nonnative speakers of English, to participate in the study. The 24 faculty members were to be comprised of two undergraduate and two graduate faculty in each of the six subject areas. The 18 students were to be comprised of one undergraduate and two graduate students from each of six subject areas. Each coordinator was responsible for identifying the 24 faculty members to participate in the study and for overseeing the recruitment of student participants. ETS Research staff advised coordinators to ask each of the 12 participating graduate faculty members to recommend one

8

graduate student from his/her class/department to complete the survey (for a total of 12 graduate students, two in each of the six subject areas). Similarly, undergraduate faculty were asked to recommend six undergraduate students to complete the survey (one in each of the six subject areas). Since applicants to undergraduate school are not usually admitted to a major, an analysis by major was not planned, and therefore the sample size for the undergraduate survey was smaller. Each coordinator was also responsible for the distribution of surveys to faculty members and students and for collection of completed surveys. In addition, coordinators were responsible for encouraging faculty and students to complete the surveys. If originally selected faculty and students failed to complete the surveys, coordinators were asked to find replacement faculty or students to complete them. The coordinator was then expected to return the completed surveys to ETS Research staff. Coordinators were each paid a stipend of $500 upon completion of their work. The procedures used here are similar to those used in other TOEFL studies that yielded return rates ranging from 50% to 82% (Bridgeman & Carlson, 1983; Hale et al., 1996; Powers, 1986). Twenty-two schools, and therefore 22 coordinators at those schools, agreed to participate in the survey phase of this study. These 22 schools, listed below by geographic region, included: Northeast Boston University Columbia University University of Maryland - Baltimore County University of Pennsylvania South American University Georgia State University University of Missouri - Columbia North Carolina State University of Texas - Austin Midwest Indiana University Iowa State University University of Michigan - Ann Arbor University of Minnesota - Twin Cities Ohio State University University of Wisconsin - Madison

9

West Arizona State University Brigham Young University University of Hawaii - Manoa University of Oregon Stanford University University of Washington Canada University of Toronto Pilot Testing of Draft Survey Instruments During the first half of January 1999, a pilot test was conducted before the actual full-scale administration of the survey was undertaken. The purpose of the pilot test was to determine if the survey instruments and their directions were clear and easy to use. The pilot test was also used to determine if the procedures for contacting faculty and students and distributing survey materials were workable. Three of the 22 colleges and universities selected for participation in the study were included in the pilot test. The TOEFL 2000 Study Coordinator at each of these three schools was asked to distribute the surveys, along with a cover letter and questions, to two faculty members (one graduate and one undergraduate) and two students (one graduate and one undergraduate). Consequently, the pilot test included 15 participants: three coordinators, six faculty members, and six students. Faculty and students were asked to complete the surveys and then answer written questions concerning the clarity of the survey directions, rating scales, and statements. The results of the pilot test indicated that only a few minor editorial changes needed to be made to the survey instrument. Since these changes were minimal, the survey data obtained for two of the three schools (eight participants) could be used in the final analyses of survey results. (See Appendix F for a list of the pilot schools as well as the cover letter and questions pilot participants were asked to answer.) Production and Administration of Survey Instruments The final survey instruments were formatted as scannable booklets and were printed by National Computer Systems. Survey packets containing a survey instrument (see Appendix G), a cover letter from ETS (see Appendix G), and postage-paid return envelopes were mailed to coordinators at each participating university at the end of January 1999. Coordinators were asked to distribute the survey packets to each faculty member and student participating in the study. The completed packets were returned to the coordinator at each school, who was responsible for tracking the returns, following up with nonrespondents, and sending all completed surveys back to ETS for processing.

10

Data Analyses Several types of analyses were conducted at multiple levels of aggregation. Analyses were conducted for each group of survey respondents (i.e., undergraduate faculty, graduate faculty, undergraduate students, and graduate students). These group-level analyses were then followed by subgroup analyses. To ensure the stability and accuracy of the outcomes, a minimum of 25 respondents was needed for a subgroup to be included in any formal analyses. Description of separate data analyses are provided below. Faculty. Faculty analyses were designed to identify reading, writing, speaking, and listening tasks that faculty deemed to be important for competent academic performance within and across subject areas at both the graduate and undergraduate levels. Analyses also focused on identifying those tasks that faculty felt were performed better by more academically successful, nonnative speakers of English than by their less successful counterparts. Specific analyses of faculty input are described below. • Percent zero analyses. The percentage of responses allocated to the zero (0) category on the Importance Rating Scale was computed for each task statement. The zero category provided faculty members an opportunity to indicate that they believe students in their courses do not need to perform a given task. If a majority of respondents (more than half) recorded a zero rating for a particular task statement, it was flagged as not being part of the students' job (flagged task statements are shaded gray in tables). Clearly, if the job relatedness of a reading, writing, speaking, or listening task statement is to be supported, a majority of respondents should indicate that the statement is a legitimate part of the students' performance domain. Percent zero analyses were conducted separately for all undergraduate and all graduate faculty members, as well as separately within each of the six areas of study at both the undergraduate and graduate levels. • Means, standard deviations, and standard errors. Means, standard deviations, and standard errors were computed for each task statement for all nonzero faculty responses to the Importance Rating Scale. The mean provides an absolute index of importance and is used to differentiate more important task statements from less important ones. The standard deviation provides an index of the variability of the ratings for each statement, and the standard error estimates the variability of the mean rating for each statement (as the standard error decreases, the sample mean becomes a more accurate estimate of the population mean). Means, standard deviations, and standard errors were computed separately for all graduate and undergraduate faculty, as well as separately within each of the six subject areas at the graduate and undergraduate levels. With respect to the Importance Rating Scale, the mean rating obtained from faculty provided an indication of the importance of each task as it relates to competent performance for all students (nonnative speakers of English and native speakers of English). A mean rating of 3.50 (“very important”) or higher was selected to distinguish the most important tasks

11

from those of less importance. It should be noted, however, that mean ratings that rounded to 3.00 were classified as “important” and could be used in the design of TOEFL 2000 test specifications and assessments. The cutpoint of 3.50 was selected so that test developers could more easily identify those tasks that were rated to be “very important.” (Since, as it turned out, most items were rated either “important” or “very important,” task statements rated below 3.50 were flagged.) While all judgmental standards may be subject to debate, a rating of “very important” provides a solid foundation for claims of job relatedness. With respect to the Relationship Rating Scale, the mean was used to identify the degree to which task statements were judged to be more related or less related to the successful academic performance of nonnative speakers of English. Means, standard deviations, and standard errors were computed for each task statement. These analyses were computed across all undergraduate and graduate faculty and separately within each of the six subject areas for each of the two levels. A mean rating of 2.00 (“The more academically successful, nonnative speakers of English GENERALLY perform this task better.”) was selected to identify tasks that distinguish more academically successful, nonnative speakers of English from their less successful counterparts. Task statements with mean ratings of less than 2.00 were flagged (i.e., shaded gray in tables) so that test developers could more easily identify those task statements judged to be most related to academic success. • Intercorrelations. Intercorrelations were computed across undergraduate and graduate faculty ratings of importance and relationship, within subject areas at the undergraduate level for importance and relationship ratings, and within subject areas at the graduate level for importance and relationship ratings. These analyses were conducted to determine the level of agreement between undergraduate and graduate faculty ratings of importance and relationship, as well as the level of agreement among faculty within subject areas at each school level. Undergraduate students. Undergraduate analyses focused on identifying the reading, writing, speaking, and listening tasks judged by undergraduate students to be important for competent academic performance. Since applicants to undergraduate school are usually not admitted to a major, the steering committee recommended that educational status be used for undergraduate students in place of area of study. Thus, analyses were conducted across all undergraduate students as well as for each of three levels of educational status (first- and second-year students combined2, third-year students, and fourth-year students). As with other subgroups, a minimum of 25 respondents was needed for a cell to be included in any formal analyses. Percent zero, mean rating, and intercorrelational analyses similar to those described for faculty respondents were also conducted for undergraduate students; however, the reader will recall that in the case of undergraduate students only one rating scale was involved.

2

Responses from first- and second-year students were combined because there was not a sufficient number of respondents in each category (fewer than 25) to conduct a separate analysis.

12

Graduate students. Graduate student analyses were designed to identify those tasks that are judged by graduate students to be important for competent performance within and across the six subject areas. In addition to analysis by area of study, analysis by geographic region of origin was also conducted. As most students participating in the study indicated they came from Europe, Latin America, and South Asia/East Asia, analyses were conducted for each of these three regions. As with other subgroups, a minimum of 25 respondents was needed for a cell to be included in any formal analyses. Percent zero, mean rating, and intercorrelational analyses similar to those described for undergraduate students were also conducted for graduate students; again, only one rating scale was involved.

13

3. Results Response Rate As noted earlier, surveys were sent to coordinators at 22 universities, and those coordinators distributed surveys to undergraduate and graduate faculty and to undergraduate and graduate students at their colleges and universities. Return rates for schools and for respondent groups are discussed here. Schools. After the project began, one school indicated that it would be unable to participate. Therefore, 21 of the 22 coordinators who agreed to participate in the study actually completed and returned surveys, for a response rate of 95%. A request was made to the remaining 21 schools to collect additional data if possible in order to compensate for the “missing” data from the school that could not participate. Undergraduate Faculty. Of the 264 surveys that were distributed to undergraduate faculty, 155 were completed and returned. The response rate for undergraduate faculty was 59%. Graduate Faculty. Of the 264 surveys that were distributed to graduate faculty, 215 were completed and returned. The response rate for graduate faculty was 81%. Undergraduate Students. Of the 132 surveys that were distributed to undergraduate students, 103 were completed and returned. The response rate for undergraduate students was 78%. Graduate Students. Of the 264 surveys that were distributed to graduate students, 242 were completed and returned. The response rate for graduate students was 92%. Respondent Demographics Demographic distributions of the four groups who responded to the survey are presented in Appendix H. A summary of the demographic characteristics of each group of respondents is presented here (percentages may not sum to 100 due to rounding). Schools. The vast majority (86%) of the 21 participating schools were on Davis' (1995) list of the top 100 schools with 1,000 or more foreign students enrolled. Approximately three-fourths (76%) were public schools, and the remaining 24% were private schools. The geographic distribution of schools was as follows: Northeast (four schools, 19%), Midwest (six schools, 29%), South (four schools, 19%), West (six schools, 29%), and Canada (one school, 5%). Undergraduate Faculty. A total of 155 undergraduate faculty returned questionnaires. The majority of respondents were female (77%) and White (87%). About two-thirds (62%) reported that they had more than 10 years experience teaching nonnative speakers of English. Undergraduate faculty were distributed by subject area as follows: history (23%), psychology (17%), business/management (16%), chemistry (16%), computer and information science (16%), and electrical engineering (14%). The geographic distribution of the schools of undergraduate

14

faculty was as follows: Northeast (12%), Midwest (31%), South (21%), West (28%), and Canada (8%). Graduate Faculty. A total of 215 graduate faculty members returned questionnaires. Like undergraduate faculty, the majority of respondents were female (78%) and White (84%), and approximately two-thirds (65%) reported that they had more than 10 years experience teaching nonnative speakers of English. The distribution of graduate faculty by subject area was: business/management (18%), chemistry (17%), computer and information science (17%), psychology (15%), electrical engineering (14%), and history (12%). The geographic distribution of the schools of graduate faculty was: Northeast (14%), Midwest (33%), South (20%), West (28%), and Canada (5%). Undergraduate Students. A total of 103 undergraduate students returned questionnaires. Slightly more than half of the respondents (55%) were female. Over one-third of respondents (36%) indicated that they were fourth-year students, while another third (34%) reported that they were third-year students. The geographic distribution of undergraduate students' schools was as follows: Northeast (15%), Midwest (32%), South (18%), West (28%), and Canada (7%). Graduate Students. A total of 242 graduate students returned questionnaires. Two-thirds (66%) of the respondents were female, and the majority (62%) of graduate students indicated that they came from South Asia/East Asia. Areas of study reported by respondents were: business/management (24%), computer and information science (17%), electrical engineering (17%), chemistry (13%), history (12%), and psychology (10%). The geographic distribution of graduate students' schools was as follows: Northeast (16%), Midwest (33%), South (19%), West (26%), and Canada (5%).

15

4. Faculty This section of the report describes survey results obtained from undergraduate and graduate faculty who instruct nonnative speakers of English in their classes. Undergraduate Faculty Percent zero analyses. The percentage of zero responses provided by undergraduate faculty for each of the reading, writing, speaking, and listening task statements contained in the survey instrument are provided in Appendix I. As noted earlier, a zero response on the Importance Rating Scale was used to indicate that an undergraduate faculty member believes students in their courses do not need to perform that task, and when a majority of faculty responded to a task statement this way, the task was flagged as irrelevant to competent academic performance. Analysis of zero responses was conducted across all undergraduate faculty respondents and within each of the six subject areas. All of the 42 ratable tasks3on the survey were judged by undergraduate faculty to be part of the job of an undergraduate student; that is, no task statement was deemed irrelevant by a majority of faculty. Task statement 25(b) in the speaking domain — “giving directions or instructions” — was judged by the highest percentage of respondents to be outside the job of an undergraduate; 13% of undergraduate faculty indicated that it is not part of the job. At least 87% of undergraduate faculty respondents judged all of the tasks surveyed to be part of the job. Likewise, within each of the six subject areas, at least 71% of respondents judged all of the tasks surveyed to be part of the job. Task statement 13 in the writing domain — “Show awareness of audience needs and write to a particular audience or reader” — was judged by highest percentage of undergraduate faculty (29%) as outside the role of an undergraduate chemistry student. These results demonstrate that, across all subject areas and within each subject area, a large majority of undergraduate faculty members believe that the reading, writing, speaking, and listening tasks included in the survey instrument are relevant for satisfactorily completing the courses they teach. Mean importance ratings. Mean importance ratings were computed for each task statement across all undergraduate faculty respondents and separately for each subject area (see Appendix I). Zero ratings were not included in the computation of the mean. In an attempt to separate the more important task statements from those of less importance, any statement with a mean rating of less than 3.50 (midpoint between a rating of “important” and “very important”) was flagged (i.e., shaded gray in tables). Across all undergraduate faculty members, mean importance ratings ranged from 4.45 (“very important”) for task statement 26 in the listening domain — “Understands factual information and details” — to 3.03 (“important”) for task statement 25(c) in the speaking domain — “describing objects.” About a fourth of the 42 ratable statements (24%) received mean ratings of

3

While there were 36 numbered task statements, one statement included several parts to be rated, bringing the total ratable statements to 42.

16

less than 3.50. These statements — the lowest rated task statements — are provided in Table 1. The reader should note, however, that these statements were still rated as important for satisfactorily completing courses taught by undergraduate faculty. Table 1 Task Statements Rated Below 3.50 by Undergraduate Faculty Respondents Task statement

Mean rating

WRITING [content] 13. Show awareness of audience needs and write to a particular audience or reader

3.33

WRITING [development] 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

SPEAKING 24. Speak clearly and accurately enough to make presentations in class

3.41

3.38

25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information

3.44

b. giving directions or instructions

3.12

c. describing objects

3.03

d. giving and supporting opinions

3.42

LISTENING [communicative functions] 34. Understand the difference among communicative functions, such as suggestions, advice, directives, and warnings

3.41

35. Recognize the use of examples, anecdotes, jokes, and digressions

3.11

36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm )

3.05

The 10 statements with the highest mean importance ratings are presented in descending order in Table 2. Three of these task statements were in the reading domain, one in the writing domain, and six in the listening domain.

17

Table 2 Task Statements Rated Most Important by Undergraduate Faculty Respondents Rank

Task statement

Category

Mean rating

1

LISTENING [facts and details]

26. Understand factual information and details

4.45

2

READING [learning]

7. Read text material with sufficient care and comprehension to remember major ideas

4.43

3

READING [basic comprehension]

5. Read and understand written instructions/ directions concerning classroom assignments and/or examinations

4.39

4

LISTENING [main ideas]

30. Understand the main ideas and their supporting information

4.34

5

LISTENING [facts and details]

27. Understand the instructor’s spoken directions regarding assignments and their due dates

4.24

6

READING [learning]

6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present

4.23

7

LISTENING [vocabulary]

28. Understand important terminology related to the subject matter

4.23

8

LISTENING [main ideas]

31. Distinguish between important information and minor details

4.20

9

WRITING [organization]

16. Organize writing in order to convey major and supporting ideas

4.19

10

LISTENING [inferences]

32. Make appropriate inferences based on information in a lecture, discussion, or conversation

4.10

Analysis by subject area provided mean ratings that ranged from 4.68 (“extremely important”) for task statement 26 in the listening domain — “Understand factual information and details” — for chemistry, to 2.49 (“moderately important”) for task statement 2 in the reading domain — “Locate and understand information provided in nonprose documents (e.g., charts, graphs, and tables)” — for history. Thirty of the 42 ratable task statements (71%) received ratings of less than 3.50 in one or more of the subject areas. However, all statements but one — task statement 2, rated as being moderately important in history, as noted above — were rated as

18

being at least “important.” The 12 task statements that received mean importance ratings of 3.50 or higher from undergraduate faculty in all six subject areas are shown below in Table 3. Seven of these task statements were located in the listening domain, four in the reading domain, and one in the writing domain. Table 3 Task Statements Rated 3.50 or Higher by Undergraduate Faculty Across Subject Areas Task statement

Overall mean rating

READING [locating information] 1. Locate and understand information that is clearly stated in the text by skimming and scanning

3.74

READING [basic comprehension] 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

4.39

READING [learning] 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present

4.23

7. Read text material with sufficient care and comprehension to remember major ideas

4.43

WRITING [development] 18. Use relevant reasons and examples to support a position or idea

4.09

LISTENING [facts and details] 26. Understand factual information and details

4.45

27. Understand the instructor’s spoken directions regarding assignments and their due dates

4.24

LISTENING [vocabulary] 28. Understand important terminology related to the subject matter

4.23

29. Use background knowledge and context to understand unfamiliar terminology

3.68

LISTENING [main ideas] 30. Understand the main ideas and their supporting information

4.34

31. Distinguish between important information and minor details

4.20

LISTENING [inferences] 32. Make appropriate inferences based on information in a lecture, discussion, or conversation

19

4.10

Table 4 provides intercorrelations of mean importance ratings by subject area for undergraduate faculty. These results indicate moderate to high correlations across subject areas. It should be noted that these moderate correlations may be due to some degree to unreliability in the ratings. It is possible that the true relationships are actually somewhat higher than those shown in this table. The reader should keep this comment in mind when reviewing the results of similar analyses later in this report. Table 4 Intercorrelations of Mean Importance Ratings by Subject for Undergraduate Faculty Subject area taught

1

2

3

4

5

1.

Chemistry

1.00

2.

Computer and information science

.86

1.00

3.

Electrical engineering

.88

.78

1.00

4.

Business/management

.61

.65

.60

1.00

5.

Psychology

.67

.69

.59

.67

1.00

6.

History

.37

.53

.40

.64

.68

6

1.00

Overall, the analysis of mean importance ratings indicate that the total group of undergraduate faculty considers all 42 tasks surveyed to be either “important” or “very important” for satisfactorily completing the courses they teach. This same finding occurred within each subject area with only one exception (task statement 2 in the reading domain —“Locate and understand information provided in nonprose documents (e.g., charts, graphs, and tables)” — which was rated “moderately important” in history). Mean relationship ratings. Undergraduate faculty were also asked to respond to a three-point Relationship Rating Scale described earlier in this report. This scale asked respondents to indicate whether they believe that more academically successful nonnative speakers of English in their classes perform this task better than less academically successful nonnative speakers of English. This scale was designed to supplement the information gathered from the Importance Rating Scale. Mean relationship ratings were computed for each task statement across all undergraduate faculty respondents and separately for each subject area; these ratings are presented in Appendix J. In an attempt to separate the statements that respondents believe are more related to the successful academic performance of nonnative speakers of English from those that are less related to successful academic performance, any statement with a mean rating below 2.0 (“The more academically successful nonnative speakers of English GENERALLY perform this task better.”) was flagged (shaded in gray in tables).

20

Across all undergraduate faculty members, mean relationship ratings ranged from 2.41 (“The more academically successful nonnative speakers of English GENERALLY perform this task better.”) for task statement 26 in the listening domain — “Understands factual information and details” — to 1.85 (rounds to “... GENERALLY perform this task better.”) for statement 36, also in the listening domain — “Recognize the speaker's attitudinal signals (e.g., tone of voice, humor, sarcasm).” The 10 statements with the highest mean relationship ratings are provided in descending order in Table 5 below. Four of the 10 highest rated task statements were in the reading domain, two in the writing domain, and four in the listening domain. Table 5 Task Statements Rated Highest by Undergraduate Faculty Respondents Rank

Category

Task statement

Mean rating

1

LISTENING [facts and details]

26. Understand factual information and details

2.41

2

READING [learning]

6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present

2.41

3

WRITING [organization]

16. Organize writing in order to convey major and supporting ideas

2.40

4

READING [learning]

7. Read text material with sufficient care and comprehension to remember major ideas

2.38

5

LISTENING [main ideas]

31. Distinguish between important information and minor details

2.38

6

LISTENING [inferences]

32. Make appropriate inferences based on information in a lecture, discussion, or conversation

2.36

7

READING [basic comprehension]

4. Determine the basic theme (main idea) of a passage

2.35

8

WRITING [language]

20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation

2.35

9

READING [integration]

11. Synthesize ideas in a single text and/or across texts

2.34

10

LISTENING [main ideas]

30. Understand the main ideas and their supporting information

2.34

21

Five of the 42 ratable task statements (12%) received mean relationship ratings below 2.0 — the lowest mean relationship ratings by undergraduate faculty — and are provided in Table 6. The reader should note, however, that ratings obtained on these statements would round to a rating of “… GENERALLY perform this task better.” Table 6 Task Statements Rated Below 2.0 by Undergraduate Faculty Across Subjects Task statement SPEAKING 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: b. giving directions or instructions c. describing objects

LISTENING [communicative functions] 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm )

Mean rating

1.94 1.96

1.96 1.89 1.85

Analysis by subject area provided mean relationship ratings ranging from 2.67 (“The more academically successful nonnative speakers of English ALMOST ALWAYS perform this task better.”) for statement 26 in the listening domain — “Understands factual information and details” — for electrical engineering, to a mean of 1.71 (“… GENERALLY performs this task better”) for task 35 in the listening domain — “Recognize the use of examples, anecdotes, jokes, and digressions” — for business/management. Fourteen of the 42 ratable task statements (33%) received mean relationship ratings below 2.0 (these statements are shaded in gray in Appendix J). However, the mean rating for all of these statements round to a rating of 2.0 (“…GENERALLY performs this task better”). Table 7 provides intercorrelations of mean relationship ratings among subject areas for undergraduate faculty; these results indicate moderate correlations across subject areas.

22

Table 7 Intercorrelations of Mean Relationship Ratings by Subject for Undergraduate Faculty Subject area taught

1

2

3

4

5

1.

Chemistry

1.00

2.

Computer and information science

.49

1.00

3.

Electrical engineering

.35

.43

1.00

4.

Business/management

.42

.63

.64

1.00

5.

Psychology

.47

.57

.49

.49

1.00

6.

History

.57

.70

.64

.73

.60

6

1.00

In summary, analyses of mean relationship ratings show that undergraduate faculty believe more academically successful, nonnative speakers of English perform all reading, writing, speaking, and listening tasks better than less academically successful, nonnative speakers of English. This information, along with the analyses of importance ratings described above, should be considered when designing test specifications and in selecting tasks to be assessed in the TOEFL 2000 Examination. Graduate Faculty Percent zero analyses. Percentages of zero responses provided by graduate faculty for each of the reading, writing, speaking, and listening task statements contained in the survey are provided in Appendix K overall and by subject. All of the 42 ratable task statements were judged by the total group of respondents to be part of the job of a graduate student. The highest percentage of respondents indicating that a statement is not part of the job was 3%, and that occurred on four task statements — 9, 15, 19, and 25(b). These results indicate that at least 97% of all graduate faculty respondents believe all of the tasks surveyed are part of a graduate student's job. Analysis by subject area indicated that the highest percentage of graduate faculty members who stated that a task is not part of the job of a graduate student was 10%. This occurred for statement 19 in the writing domain — “Produce sufficient quantity of written text appropriate to the assignment and the time constraints” — for electrical engineering. These results indicate that at least 90% of the graduate faculty within each of the six subject areas believe all of the tasks surveyed are part of a graduate student's job. Further, these results demonstrate that the vast majority of graduate faculty — across all subject areas as well as within each of the six subject areas — believe that the tasks are relevant for satisfactorily completing the courses they teach.

23

Mean importance ratings. Mean importance ratings were computed for each task statement across all graduate faculty respondents and separately for each subject area; they are presented in Appendix K. Zero ratings were not included in the computation of the mean. In an attempt to separate more important task statements from those of less importance, any statement with a mean rating of less than 3.50 (midpoint between a rating of “important” and “very important”) was flagged (shaded in gray). Across all graduate faculty members, mean importance ratings ranged from 4.46 (“very important”) for task statement 16 in the writing domain — “Organize writing in order to convey major and supporting ideas” — to 3.16 (“important”) for task statement 36 in the listening domain — “Recognize the speaker's attitudinal signals (e.g. tone of voice, humor, sarcasm).” Three of the 42 ratable statements (7%) received mean ratings of less than 3.50; these statements — the lowest rated task statements — are presented in Table 8. It should be noted, however, that these statements were still rated as being important to the satisfactory completion of courses taught by graduate faculty. Table 8 Task Statements Rated Below 3.50 by Graduate Faculty Respondents Task statement

Mean rating

SPEAKING 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: c. describing objects

3.44

LISTENING [communicative functions] 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm )

3.27 3.16

The 10 statements with the highest mean importance ratings are presented in Table 9 in descending order. Three of these statements were in the reading domain, two in the writing domain, and five in the listening domain.

24

Table 9 Task Statements Rated Most Important by Graduate Faculty Respondents Rank

Category

Task statement

Mean rating

1

WRITING [organization]

16. Organize writing in order to convey major and supporting ideas

4.46

2

LISTENING [main ideas]

30. Understand the main ideas and their supporting information

4.45

3

LISTENING [facts and details]

26. Understand factual information and details

4.41

4

READING [learning]

7. Read text material with sufficient care and comprehension to remember major ideas

4.40

5

WRITING [development]

18. Use relevant reasons and examples to support a position or idea

4.34

6

LISTENING [vocabulary]

28. Understand important terminology related to the subject matter

4.32

7

LISTENING [main Ideas]

31. Distinguish between important information and minor details

4.29

8

READING [basic comprehension]

5. Read and understand written instructions/ directions concerning classroom assignments and/or examinations

4.29

9

LISTENING [inferences]

32. Make appropriate inferences based on information in a lecture, discussion, or conversation

4.28

10

READING [learning]

6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present

4.25

Analysis by subject area yielded mean ratings that ranged from 4.81 (“extremely important”) in history for two task statements in the writing domain — task statement 16, “Organize writing in order to convey major and supporting ideas,” and task statement 18, “Use relevant reasons and examples to support a position or idea” — to 2.55 (rounds to “important”) in chemistry for task statement 36 in the listening domain — “Recognize the speaker's attitudinal signals.” Thirteen of the 42 ratable task statements (31%) received ratings of less than 3.50 in one or more of the subject areas and so were flagged (shaded in gray in tables). Five of the statements were in the reading domain, three in the writing domain, two in the speaking domain and three in the listening domain. However, all of the tasks surveyed were rated as “important” or rounded to a rating of “important.” Overall, these analyses show that the total group of graduate faculty respondents believe all 42 ratable tasks are either “important” or “very important” for satisfactorily completing the courses they teach. Mean importance ratings were intercorrelated among each of the six subject areas and are provided in Table 10. These results show moderate to high correlations across all six subject areas. Ratings for chemistry, computer and information science, and electrical engineering

25

correlated higher with each other than with ratings for business/management, psychology, and history. Similarly, ratings for business/management, psychology, and history correlated higher with each other than with ratings for chemistry, computer and information science, and electrical engineering. Table 10 Intercorrelations of Mean Importance Ratings by Subject for Graduate Faculty Subject area taught

1

2

3

4

5

1.

Chemistry

1.00

2.

Computer and information science

.73

1.00

3.

Electrical engineering

.91

.78

1.00

4.

Business/management

.68

.57

.64

1.00

5.

Psychology

.69

.58

.63

.83

1.00

6.

History

.57

.45

.49

.86

.72

6

1.00

Mean relationship ratings. Mean relationship ratings, based on responses to the Relationship Rating Scale, were computed for each task statement for all graduate faculty respondents and separately for each subject area; they are presented in Appendix L. In an attempt to separate statements that respondents believe are more related to the successful academic performance of nonnative speakers of English from those that are less related to successful academic performance, any statement with a mean rating below 2.0 (“The more academically successful nonnative speakers of English GENERALLY perform this task better.”) was flagged (shaded in gray in tables). Across all graduate faculty members, mean relationship ratings ranged from 2.49 (“The more academically successful nonnative speakers of English GENERALLY perform this task better.”) for task statement 16 in the writing domain — “Organizes writing in order to convey major and supporting ideas” — to 1.88 (which rounds to the same descriptor as above) for statement 36 in the listening domain —“Recognize the speakers' attitudinal signals (e.g., tone of voice, humor, sarcasm).” The 10 statements with the highest mean relationship ratings are provided in Table 11. Three of the highest rated tasks were in the reading domain, one in the writing domain, two in the speaking domain, and four in the listening domain.

26

Table 11 Task Statements Rated Highest by Graduate Faculty Respondents Rank

Category

Task statement

Mean rating

1

WRITING [organization]

16. Organize writing in order to convey major and supporting ideas

2.49

2

LISTENING [main ideas]

30. Understand the main ideas and their supporting information

2.46

3

READING [learning]

7. Read text material with sufficient care and comprehension to remember major ideas

2.45

4

READING [integration]

11. Synthesize ideas in a single text and/or across texts

2.45

5

READING [learning]

6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present

2.45

6

SPEAKING

25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: f. developing or structuring hypotheses

2.44

7

LISTENING [inferences]

32. Make appropriate inferences based on information in a lecture, discussion, or conversation

2.43

8

LISTENING [main ideas]

31. Distinguish between important information and minor details

2.42

9

SPEAKING

25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: g. explaining or informing

2.40

10

LISTENING [facts and details]

26. Understand factual information and details

2.40

Two of the 42 ratable task statements (5%) received mean relationship ratings below 2.0. These statements, which received the lowest mean relationship ratings by graduate faculty, are presented in Table 12.

27

Table 12 Task Statements Rated Below 2.0 by Graduate Faculty Across Subjects Task statement

Mean rating

LISTENING [communicative functions] 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

1.95 1.88

Analysis by subject area provided mean relationship ratings ranging from 2.69 (“The more academically successful nonnative speakers of English ALMOST ALWAYS perform this task better.”) for statement 30 in the listening domain —“Understand the main ideas and their supporting information” — for computer and information science, to 1.76 (rounds to “... GENERALLY perform the task better”) for statement 36 in the listening domain — “Recognize the speakers' attitudinal signals (e.g., tone of voice, humor, sarcasm)” — for business/management and history. Eight of the 42 ratable task statements (19%) received mean relationship ratings below 2.0 (these statements are shaded in gray in Appendix L). However, the mean rating for all of these task statements round to a rating of 2.0 (“ ... GENERALLY perform the task better”). Table 13 provides intercorrelations of mean relationship ratings among subject areas for graduate faculty. These results show moderate correlations across subject areas. Table 13 Intercorrelations of Mean Relationship Ratings by Subject for Graduate Faculty Subject area taught

1

2

3

4

5

1.

Chemistry

1.00

2.

Computer and information science

.64

1.00

3.

Electrical engineering

.53

.66

1.00

4.

Business/management

.74

.79

.72

1.00

5.

Psychology

.49

.80

.60

.71

1.00

6.

History

.59

.56

.53

.69

.53

6

1.00

In summary, analyses of mean relationship ratings indicate that graduate faculty believe more academically successful, nonnative speakers of English perform all reading, writing, speaking, and listening tasks surveyed better than less academically successful, nonnative speakers of English. This information, along with the analyses of importance ratings described above, should

28

be considered when designing test specifications and selecting tasks to be assessed in the TOEFL 2000 Examination. Comparing Ratings of Undergraduate and Graduate Faculty Here, we compare some of the results that are relevant for consideration in designing test specifications and selecting tasks for inclusion in the TOEFL 2000 assessment measures. Importance ratings. Percent zero analyses and mean importance ratings were computed for all undergraduate and graduate faculty members, as well as separately within each of the six subject areas, with the following results: •

At least 87% of undergraduate faculty respondents judged all 42 ratable tasks to be part of the undergraduate student's job.



At least 97% of the graduate faculty respondents judged all 42 ratable tasks to be part of the graduate student's job.



Within each of the six subject areas, at least 71% of undergraduate faculty respondents judged all 42 ratable tasks to be part of the undergraduate student's job.



Within each of the six subject areas, at least 90% of graduate faculty respondents judged all 42 ratable tasks to be part of the graduate student's job.



Overall, undergraduate faculty judged all tasks surveyed to be either “important” or “very important” to satisfactorily complete courses they teach.



Overall, graduate faculty judged all tasks surveyed to be either “important” or “very important” to satisfactorily complete courses they teach.



Analyses of undergraduate faculty responses by subject area yielded one task that is “moderately important” to the completion of undergraduate work, while all others are either “important” or “very important.”



Analyses of graduate faculty responses by subject area indicated that all surveyed tasks are either “important” or “very important” to the completion of graduate work.

All 42 ratable tasks were judged to be part of the jobs of both undergraduate and graduate students by large majorities of undergraduate and graduate faculty respondents across all six subject areas as well as within each of the subject areas. In addition, all tasks were rated as being important enough to be eligible for consideration in designing test specifications appropriate for

29

both undergraduate and graduate students. In order to simplify the process of designing test specifications and selecting tasks for inclusion in the TOEFL 2000 assessments, it is recommended that primary emphasis on faculty ratings be placed on findings for the total groups of undergraduate and graduate faculty. Even though all task statements were rated high enough to be considered in the design of test specifications, some tasks were clearly rated as being more important than others. Appendix M presents the rank ordering of tasks by importance for both undergraduate and graduate faculty. A Spearman Rank-Order correlation (rho) was computed between the ranks for undergraduate and graduate faculty ratings. The correlation was .86, indicating a high level of agreement by undergraduate and graduate faculty on the tasks judged to be most important. Relationship ratings. Mean relationship ratings were computed for all undergraduate and graduate faculty members as well as separately within each of the six subject areas. Across all undergraduate and faculty respondents and within all six subject areas, mean ratings indicated a perception that the more academically successful, nonnative speakers of English either generally or almost always perform these tasks better than less academically successful, nonnative speakers of English. Again, in order to facilitate the design of test specifications and selection of tasks for TOEFL 2000 assessments, it is recommended that primary emphasis on faculty ratings be placed on findings for the total groups of undergraduate and graduate faculty. Even though all task ratings were judged to have some value for distinguishing between more academically successful and less academically successful, nonnative speakers of English, some tasks were judged to be more effective than others. Appendix M presents the rank-ordering of tasks for the relationship scale for both undergraduate and graduate faculty. Rho was computed between the ranks for undergraduate and graduate faculty ratings. The correlation was .89, indicating a high level of agreement by undergraduate and graduate faculty on those tasks most related to academic success by nonnative speakers of English. Importance and relationship ratings. Rho was also computed between undergraduate faculty rankings of importance and relationship and yielded a correlation of .76. A similar correlation was computed between importance and relationship rankings for graduate faculty and yielded a correlation of .73. These findings indicate a moderately high relationship between the two scales. However, the relationship is not so high that only one of the scales should be used in designing test specifications or selecting tasks for use in TOEFL 2000 assessments. Both scales provide unique information, and it is recommended that both be considered.

30

5. Students This section of the report describes survey results obtained from undergraduate and graduate students who are nonnative speakers of English. Undergraduate Students Percent zero analyses. Percentages of zero responses provided by undergraduate students for each of the reading, writing, speaking, and listening tasks in the survey are provided in Appendix N. Analyses of these responses were conducted across all undergraduate student respondents as well as by educational status: first- and second-year students combined, third-year students, and fourth-year students. The total group of undergraduate students judged all of the 42 ratable tasks to be part of their job. The highest percentage of respondents indicating that a task was not part of the job of an undergraduate was 3%; this occurred for task statement 25(b) in the speaking domain — “giving directions or instructions.” These results indicate that at least 97% of all undergraduate student respondents believe all of the tasks surveyed are part of their job. Analyses by educational status indicated that the highest percentage of undergraduate students who believe a given task is not part of the job of an undergraduate was 9%; third-year students indicated this for task statement 25(b) in the speaking domain — “giving directions or instructions.” These results indicate that at least 91% of the respondents within each of the three levels of educational status believe all of the tasks surveyed are part of the job. Further, these results demonstrate that the vast majority of undergraduate students — across all educational status levels as well as within each of the three levels of educational status — believe the tasks included in the survey are relevant for satisfactorily completing their courses. Mean importance ratings. Mean importance ratings were computed for each task statement across all undergraduate student respondents and separately for each of the three levels of educational status; these ratings are presented in Appendix N. Zero ratings were not included in the computation of the mean. In an attempt to separate the task statements that respondents believe are more important to their success from those they believe are less important, any statement with a mean rating of less than 3.50 (midpoint between a rating of “important” and “very important”) was flagged (shaded in gray in tables). Across all undergraduate students, mean importance ratings ranged from 4.37 (“very important”) for task statement 27 in the listening domain — “Understand the instructor's spoken directions regarding assignments and their due dates” — to 3.44 (“important”) for task statement 35 in the listening domain —“Recognize the use of examples, anecdotes, jokes, and digressions.” Three of the 42 ratable task statements (7%) received mean ratings of less than 3.50; these statements — the lowest rated task statements — are shown in Table 14. The reader should note, however, that undergraduate students still rated these tasks as important to satisfactorily completing their courses.

31

Table 14 Task Statements Rated Below 3.50 by Undergraduate Student Respondents Task statement

Mean rating

WRITING [content] 13. Show awareness of audience needs and write to a particular audience or reader

3.45

SPEAKING 25. Demonstrate facility with standard spoken English, including grammar, word choice, fluency, and sentence structure, while performing the following linguistic tasks: c. describing objects

3.48

LISTENING [communicative functions] 35. Recognize the use of examples, anecdotes, jokes, and digressions

3.44

The 10 task statements with the highest mean importance ratings are presented in Table 15 in descending order. Four of these task statements were in the listening domain, three in the reading domain, and three in the writing domain. Table 15 Task Statements Rated Most Important by Undergraduate Student Respondents Rank

Category

Task statement

Mean rating

1

LISTENING [facts and details]

27. Understand the instructor’s spoken directions regarding assignments and their due dates

4.37

2

READING [basic comprehension]

4.

Determine the basic theme (main idea) of a passage

4.30

3

READING [basic comprehension]

5.

Read and understand written instructions/directions concerning classroom assignments and/or examinations

4.24

4

LISTENING [main ideas]

30. Understand the main ideas and their supporting information

4.24

5

LISTENING [facts and details]

26. Understand factual information and details

4.20

6

LISTENING [vocabulary]

28. Understand important terminology related to the subject matter

4.19

7

WRITING [organization]

16. Organize writing in order to convey major and supporting ideas

4.18

8

WRITING [development]

18. Use relevant reasons and examples to support a position or idea

4.17

9

WRITING [language]

20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation

4.15

10

READING [learning]

7.

4.03

Read text material with sufficient care and comprehension to remember major ideas

32

Analysis by educational status provided mean ratings that ranged from 4.46 (“very important”) for two statements — statement 26 and 27 in the listening domain, “Understand factual information and details” and “Understand the instructor's spoken directions regarding assignments and their due dates” — for third-year students, to 3.17 (“important”) for statement 25(b) in the speaking domain, “giving directions or instructions” — for first- and second-year students combined. Thirty-two of the 42 ratable task statements (76%) received mean importance ratings of 3.50 or higher from undergraduate students in all three levels of educational status. The 10 task statements that received mean importance ratings of less than 3.50 in one or more of the educational status levels are shown below in Table 16. Four of these task statements were located in the speaking domain, three in the listening domain, two in the reading domain, and one in the writing domain. It is interesting to note that mean ratings of all 10 of these statements by first- and second-year students combined were flagged. Mean ratings of only two of the 10 statements by third-years students were flagged, and no ratings by fourth-year students were flagged. Table 16 Task Statements Rated Below 3.50 by Undergraduate Students by Educational Status Mean rating Task statement READING [learning] 9. Distinguish factual information from opinions READING [integration] 10. Compare and contrast ideas in a single text and/or across texts WRITING [content] 13. Show awareness of audience needs and write to a particular audience or reader SPEAKING 25. Demonstrate facility with standard spoken English, including grammar, word choice, fluency, and sentence structure, while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects f. developing or structuring hypotheses LISTENING [vocabulary] 29. Use background knowledge and context to understand unfamiliar terminology LISTENING [communicative functions] 35. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 36. Recognize the use of examples, anecdotes, jokes, and digressions Note. Ratings below 3.5 are shaded gray.

33

First- & second-year students

Third-year students

Fourth-year students

3.41

3.57

3.67

3.41

3.74

3.50

3.44

3.20

3.67

3.45 3.17 3.18 3.46

3.59 3.75 3.53 3.53

3.79 3.85 3.64 3.74

3.48

3.86

4.00

3.21

3.40

3.66

3.48

3.66

3.79

Table 17 provides intercorrelations of mean importance ratings by educational status for undergraduate students. These results indicate moderate to high correlations across subgroups. Table 17 Intercorrelations of Mean Importance Ratings by Educational Status for Undergraduate Students Educational status

1

2

1.

First- and second-year students

1.00

2.

Third-year students

.73

1.00

3.

Fourth-year students

.69

.82

3

1.00

Overall, analyses of mean importance ratings indicate that the total group of undergraduate students, and each subgroup, believe that all 42 ratable tasks are either “important” or “very important” for satisfactorily completing their courses. Graduate Students Percent zero analyses. Percentages of zero responses provided by graduate students for each of the reading, writing, speaking, and listening tasks in the survey are provided in Appendix O and Appendix P. Analyses of these responses were conducted across all graduate student respondents, within each of the six areas of study, and within each of three geographic regions of origin — Europe, Latin America, and South Asia/East Asia. Percentages of zero responses provided by graduate students, overall and by area of study, are provided in Appendix O; Appendix P presents percentages of zero responses provided by graduate students overall and by geographic region of origin. The total group of graduate students judged all of the 42 ratable tasks to be part of their job. The highest percentage of respondents indicating that a task was not part of the job was 2%; this occurred for eight task statements — 2, 3, 13, 19, 25(b), 25(c), 25(f), and 36 — across the four domains. These results indicate that at least 98% of graduate student respondents believe all of the tasks surveyed are part of their job. Analyses by area of study (Appendix O) indicated that the highest percentage of graduate students who believe a given task to be outside of the job of a graduate student was 8%; this occurred for electrical engineering on task statement 36 in the listening domain — “Recognize the speaker's attitudinal signals (e.g., tone of voice, humor, sarcasm).” These results indicate that, within each of the six areas of study, at least 92% of respondents believe all of the tasks surveyed are part of their job.

34

Analyses by geographic region (Appendix P) indicated that the highest percentage of graduate students who believe a task to be outside of the job of a graduate student was 5%; this occurred for students from Europe on task statement 2 in the reading domain — “Locate and understand information provided in nonprose documents (e.g., charts, graphs, and tables).” These results indicate that at least 95% of the respondents within each of the three geographic regions of origin believe all of the tasks surveyed are part of their job. Overall, analyses of zero responses demonstrate that the vast majority of graduate students believe that the reading, writing, speaking, and listening tasks included in the survey are relevant for satisfactorily completing their courses. This finding occurred across all areas of study and all geographic regions of origin, as well as within each of the six areas of study and each of the three regions of the world. Mean importance ratings. Mean importance ratings were computed for each task statement across all graduate student respondents, as well as separately for each of the six areas of study and each of the three geographic regions of origin; these ratings are presented in Appendix O and Appendix P. Zero ratings were not included in the computation of the means. In an attempt to separate tasks respondents believe are more important from those they believe are less important, any statement with a mean rating of less than 3.50 (midpoint between a rating of “important” and “very important”) was flagged (shaded in gray in tables). Mean importance ratings by graduate students, overall and by area of study, are provided in Appendix O; Appendix P presents mean importance ratings by graduate students overall and by geographic region of origin. Across all graduate students (see Appendix O or P), mean importance ratings ranged from 4.39 (“very important”) for task statement 30 in the listening domain — “Understand the main ideas and their supporting information” — to 3.23 (“important”) for two task statements — task statement 19 in the writing domain, “Produce sufficient quantity of written text appropriate to the assignment and the time constraints,” and task statement 35 in the listening domain, “Recognize the use of examples, anecdotes, jokes, and digressions.” Four of the 42 ratable task statements (10%) received mean ratings of less than 3.50; these statements — the lowest rated task statements — are provided in Table 18. The reader should note, however, that graduate students still rated these statements as important to satisfactorily completing their courses.

35

Table 18 Task Statements Rated Below 3.50 by Graduate Student Respondents Task statement

Mean rating

WRITING [development] 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints SPEAKING 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: c. describing objects LISTENING [communicative functions] 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm )

3.23

3.43

3.23 3.38

The 10 task statements with the highest mean importance ratings are presented in Table 19 in descending order. Four of these task statements were in the listening domain, three in the reading domain, two in the speaking domain, and one in the writing domain. Table 19 Task Statements Rated Most Important by Graduate Student Respondents Rank

Category

Task statement

Mean rating

1

LISTENING [main ideas]

30. Understand the main ideas and their supporting information

4.39

2

WRITING [organization]

16. Organize writing in order to convey major and supporting ideas

4.32

3

READING [basic comprehension]

4. Determine the basic theme (main idea) of a passage

4.24

4

SPEAKING

22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions

4.18

5

LISTENING [facts and details]

26. Understand factual information and details

4.13

6

SPEAKING

24. Speak clearly and accurately enough to make presentations in class

4.12

7

LISTENING [facts and details]

27. Understand the instructor’s spoken directions regarding assignments and their due dates

4.08

8

LISTENING [vocabulary]

28. Understand important terminology related to the subject matter

4.05

9

READING [learning]

7. Read text material with sufficient care and comprehension to remember major ideas

4.05

10

READING [learning]

8. Read text material and outline important ideas and concepts

4.05

36

Analysis by area of study (see Appendix O) provided mean ratings that ranged from 4.71 (“extremely important”) for statement 30 in the listening domain — “Understand the main ideas and their supporting information” — for history, to 2.79 (“important”) for statement 19 in the writing domain — “Produce sufficient quantity of written text appropriate to the assignment and the time constraints” — for psychology. Twenty-four of the 42 ratable tasks (57%) received mean importance ratings of 3.50 or higher from graduate students in all six areas of study. The 18 task statements that received mean importance ratings of less than 3.50 in one or more of the six areas of study are shown in Table 20. Eight of these task statements were located in the writing domain, four in the speaking domain, three in the reading domain, and three in the listening domain.

37

Table 20 Task Statements Rated Below 3.50 by Graduate Students by Subject Mean rating Task statement

Chemistry

Computer & information science

Electrical engineering

Business/ management

Psychology

History

READING [learning] 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 9. Distinguish factual information from opinions

3.61

3.65

3.63

3.90

3.48

3.91

3.78

3.43

3.18

3.64

3.16

3.91

READING [integration] 10. Compare and contrast ideas in a single text and/or across texts

3.78

3.71

3.49

3.95

3.35

4.09

3.97

4.18

3.90

3.90

3.28

4.06

3.50

3.98

3.29

3.51

3.04

3.55

3.91

3.80

3.54

4.02

3.40

4.09

3.94

3.62

3.95

3.81

3.44

4.06

3.91

3.75

3.48

3.92

3.88

3.56

3.47

3.20

2.97

3.36

2.79

3.62

3.56

4.08

3.72

3.93

3.36

3.94

3.72

3.58

3.49

3.58

3.16

3.74

WRITING [content] 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others WRITING [organization] 17. Use appropriate transitions to connect ideas and information WRITING [development] 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints WRITING [language] 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic Note. Ratings below 3.5 are shaded gray.

38

Table 20, continued Task Statements Rated Below 3.50 by Graduate Students by Subject Mean rating Task statement

SPEAKING 23. Speak clearly and accurately enough to participate in class discussions 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects LISTENING [main ideas] 31. Distinguish between important information and minor details LISTENING [communicative functions] 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm) Note. Ratings below 3.5 are shaded gray.

39

Chemistry

Computer & information science

Electrical engineering

Business/ management

Psychology

History

4.13

4.03

3.85

4.19

3.40

4.32

3.84 3.38 3.59

3.78 3.45 3.29

3.48 3.56 3.28

3.79 3.47 3.43

3.68 3.20 3.28

3.76 3.71 3.62

4.03

4.13

3.77

3.85

3.48

4.09

3.47 3.50

3.33 3.56

3.18 3.30

3.24 3.52

2.80 3.00

3.29 3.15

Table 21 provides intercorrelations of mean importance ratings by area of study for graduate students. These results indicate moderate to high correlations across subgroups. Table 21 Intercorrelations of Mean Importance Ratings by Subject for Graduate Students Area of study

1

2

3

4

5

1.

Chemistry

1.00

2.

Computer and information science

.69

1.00

3.

Electrical engineering

.74

.72

1.00

4.

Business/management

.78

.81

.80

1.00

5.

Psychology

.68

.66

.70

.77

1.00

6.

History

.74

.76

.72

.82

.73

6

1.00

Analyses by geographic region (see Appendix P) provided mean ratings that ranged from 4.49 (“very important”) for statement 30 in the listening domain — “Understand the main ideas and their supporting information” — for Europe, to 3.13 (“important”) for statement 35 in the listening domain — “Recognize the use of examples, anecdotes, jokes, and digressions” — for Latin America. Thirty of the 42 ratable tasks (71%) received mean importance ratings of 3.50 or higher from graduate students in all three regions. The 12 task statements that received mean importance ratings of less than 3.50 in one or more geographic regions are shown in Table 22. Five of these tasks were located in the listening domain, three in the writing domain, two in the reading domain, and two in the speaking domain.

40

Table 22 Task Statements Rated Below 3.50 by Graduate Students by Geographic Region of Origin Task statement Europe READING [locating information] 2. Locate and understand information provided in nonprose documents (e.g., charts, graphs, and tables) READING [learning] 9. Distinguish factual information from opinions WRITING [content] 13. Show awareness of audience needs and write to a particular audience or reader WRITING [development] 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints WRITING [language] 21. Demonstrate facility with a range of vocabulary appropriate to the topic SPEAKING 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: b. giving directions or instructions c. describing objects LISTENING [vocabulary] 29. Use background knowledge and context to understand unfamiliar terminology LISTENING [inferences] 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion LISTENING [communicative functions] 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 36. Recognize the use of examples, anecdotes, jokes, and digressions Note. Ratings below 3.5 are shaded gray.

41

Mean rating Latin South Asia/ America East Asia

3.46

3.73

3.79

3.73

3.22

3.43

3.57

4.17

3.34

3.51

3.30

3.14

3.68

3.61

3.49

3.33 3.46

3.30 3.23

3.45 3.39

3.81

3.48

3.87

3.57

3.43

3.74

3.70

3.43

3.60

3.51

3.13

3.16

3.57

3.68

3.33

Table 23 provides intercorrelations of mean importance ratings by geographic region of origin for graduate students. These results indicate high correlations across subgroups. Table 23 Intercorrelations of Mean Importance Ratings by Geographic Region of Origin for Graduate Students Geographic region

1

2

1.

Europe

1.00

2.

Latin America

.70

1.00

3.

South Asia/East Asia

.82

.73

3

1.00

Overall, these analyses of mean importance ratings indicate that the total group of graduate students, as well students within each geographic region, believe all 42 ratable tasks are either “important” or “very important” for satisfactorily completing their courses. This same finding is seen within each of the six areas of study also, with the exception that four task statements were rated as being “extremely important” (4.50 or higher) in history; and one of these four task statements was also rated “extremely important” in chemistry. The four task statements rated “extremely important,” by subject, were: •

history — task statement 16 in the writing domain, “Organize writing in order to convey major and supporting ideas”



history — task statement 22 in the speaking domain, “Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions”



history — task statement 26 in the listening domain, “Understand factual information and details”



history and chemistry — task statement 30 in the listening domain, “Understand the main ideas and their supporting information”

Comparing Ratings of Undergraduate and Graduate Students This section compares some of the results for graduate and undergraduate students that are relevant for consideration in designing test specifications and selecting tasks for the TOEFL 2000 assessment. Importance ratings. Percent zero analyses and mean importance ratings were computed for all undergraduate and graduate students. Mean importance ratings were also computed for

42

undergraduate students by educational status. In addition, mean importance ratings were computed for graduate students separately by area of study and by geographic region of origin. The following results were obtained: •

At least 98% of all undergraduate student respondents, judged all 42 ratable tasks to be part of the undergraduate student's job.



At least 98% of all graduate student respondents judged all 42 ratable task statements to be part of the graduate student's job.



Within each of the three levels of educational status, at least 81% of undergraduate student respondents judged all 42 ratable tasks to be part of the undergraduate student's job.



Within each of the six areas of study, at least 92% of the graduate student respondents judged all 42 ratable tasks to be part of the graduate student's job.



Within each of the three geographic regions of origin, at least 95% of the graduate student respondents judged all 42 ratable tasks to be part of the graduate student's job.



Overall, undergraduate students judged all surveyed tasks as either “important” or “very important” to satisfactorily complete their courses.



Overall, graduate students judged all surveyed tasks as either “important” or “very important” to satisfactorily complete their courses.



Analyses of undergraduate student responses by educational status indicated that all tasks surveyed are either “important” or “very important” to the completion of undergraduate work.



Analyses of graduate student responses by area of study yielded four tasks that are “extremely important” to the completion of graduate work, while all others are either “important” or “very important.”



Analyses of graduate student responses by geographic region of origin indicated that all tasks surveyed are either “important” or “very important” to the completion of graduate work.

All 42 of the ratable tasks were judged to be part of the jobs of both undergraduate and graduate students by large majorities of undergraduate and graduate student respondents across all subgroups, as well as within each of those subgroups. In addition, all tasks surveyed were rated as being important enough to be eligible for consideration in designing test specifications that are appropriate for both undergraduate and graduate students. In order to facilitate the

43

process of designing test specifications and selecting tasks for TOEFL 2000 assessments, it is recommended that primary emphasis on student ratings be placed on the total groups of undergraduate and graduate findings. Even though all tasks surveyed were rated high enough to be considered in the design of test specifications, some tasks were clearly rated as more important than others. Appendix M presents the rank ordering of task statements by importance for both undergraduate and graduate students. As with faculty ratings, rho was computed between the ranks for undergraduate and graduate student ratings. The correlation was .84, indicating a high level of agreement by undergraduate and graduate students on the tasks that are most important. Comparing Ratings of Faculty and Students Undergraduate and graduate faculty and students all responded to an Importance Rating Scale. The results obtained from their ratings can be summarized as follows: •

Large majorities of undergraduate and graduate faculty respondents across and within all six subject areas judged all 42 of ratable tasks to be part of the jobs of both undergraduate and graduate students.



Large majorities of undergraduate and graduate students across and within all subgroups — educational status for undergraduate students and six subject areas and three geographic regions for graduate students — judged all 42 ratable tasks to be part of the jobs of both undergraduate and graduate students.

Rho was computed for each of the total groups of undergraduate and graduate faculty and undergraduate and graduate students. The resulting intercorrelation matrix is presented in Table 24. These results indicate moderate to high correlations across faculty and student ratings. These results also indicate that there is reasonably good agreement among undergraduate and graduate faculty and undergraduate and graduate students regarding the relative importance of the 42 ratable task statements. Table 24 Intercorrelations of Importance Ratings Across All Respondents Student

Students/faculty

Undergraduate

Faculty Graduate

Undergraduate

1.

Undergraduate students

1.00

2.

Graduate students

0.84

1.00

3.

Undergraduate faculty

0.78

0.66

1.00

4.

Graduate faculty

0.71

0.73

0.86

44

Graduate

1.00

In addition to the correlations described on the previous page, percent agreement indices were computed between undergraduate faculty, graduate faculty, undergraduate students, and graduate students based on their mean importance ratings. Contingency tables were generated using the 3.50 standard, and the percent of classification agreement between groups was determined. For example, in the case of undergraduate faculty and undergraduate students, the percent agreement between their mean importance ratings, relative to the 3.50 standard, was computed. These results, presented in Table 25, indicate a high level of agreement by all four respondent groups concerning those tasks that were judged to be “very important” and those judged to be of somewhat less importance. Table 25 Percent Agreement of Mean Importance Ratings Across All Respondents Student

Students/faculty

Undergraduate

Faculty Graduate

Undergraduate

1.

Undergraduate students

100%

2.

Graduate students

93%

100%

3.

Undergraduate faculty

83%

86%

100%

4.

Graduate faculty

95%

98%

83%

Graduate

100%

Overall, this comparison of student and faculty ratings indicates that both undergraduate and graduate faculty and students believe all 42 ratable tasks in reading, writing, listening, and speaking are relevant aspects of undergraduate and graduate student work. In addition, all respondents judged all tasks to be either “important” or “very important” for completing coursework at both the graduate and undergraduate levels. There was also good agreement among the four respondent groups regarding those tasks that are “very important” (a rating of 3.50 or higher) to the completion of coursework and those that are of somewhat less importance. These results indicate that the 42 tasks surveyed constitute an appropriate pool of tasks to be considered in designing test specifications and assessments for TOEFL 2000. It is recommended that primary consideration be given to those task statements judged to be most important by all four respondent groups.

45

6. Summary and Conclusions Summary The primary purposes of this project were: 1. to aid in translating theoretical frameworks developed in reading, writing, speaking, and listening by the TOEFL Framework Teams into task statements that undergraduate and graduate students need to perform in order to complete their academic coursework 2. to have undergraduate and graduate faculty experienced in teaching nonnative speakers of English, as well as undergraduate and graduate students who are nonnative speakers of English, review and evaluate these statements through the use of a survey 3. to provide analyses of these results that aid in the design of test specifications and assessment measures for TOEFL 2000 4. to document these results to support the validity of TOEFL 2000 The frameworks for reading, writing, speaking, and listening describe the literature and linguistic theory used by the framework teams as they conceptualized TOEFL 2000. The task statements used in the present study operationalized this theory into tasks that the framework teams believe are necessary for satisfactorily completing coursework at both the undergraduate and graduate levels. Survey instruments were designed and administered to undergraduate and graduate faculty experienced in teaching nonnative speakers of English, as well as to undergraduate and graduate students who are nonnative speakers of English, at 21 universities across the United States and Canada. The purpose of the faculty surveys was to determine how important undergraduate and graduate faculty felt it was for their students to perform these tasks competently in order to satisfactorily complete the courses they teach, and whether the more academically successful, nonnative speakers of English in their classes perform these tasks better than the less academically successful, nonnative speakers of English. The purpose of the student surveys was to determine how important undergraduate and graduate students who are nonnative speakers of English feel these tasks are for completing their courses. Judgments of faculty and students were analyzed to determine whether they supported or refuted judgments of linguistic specialists on the framework teams. As noted in the introduction to this report, the study was designed to answer 10 major research questions. The answers to those questions can be summarized as follows: •

Large majorities of both undergraduate and graduate faculty and students judged all 42 surveyed reading, writing, speaking, and listening tasks to be relevant aspects of the job of undergraduate and graduate students. Faculty subgroups within each of the six subject areas of study at both the undergraduate and graduate levels, subgroups of undergraduate students by each of the three levels of educational status, and

46

subgroups of graduate students by each of six subject areas and three geographic regions, all produced the same judgment. •

All 42 ratable task statements were judged to be either “important” or “very important” for completing coursework at the undergraduate and graduate levels. This finding occurred for faculty within each of the six subject areas of study at both undergraduate and graduate levels, with one exception for undergraduate faculty: One task statement for one subject area was rated as being “moderately important.” Subgroups of undergraduate students (educational status) and subgroups of graduate students (area of study and geographic region) rated all task statements as “important” or “very important” for satisfactorily completing their coursework. In addition, four task statements were judged to be “extremely important” to the completion of graduate work in history, and one of these task statements was judged to be “extremely important” to the completion of graduate work in chemistry.



Across the total groups of undergraduate and graduate faculty and within all six subject areas, mean ratings on the relationship scale indicated that undergraduate and graduate faculty perceive that the more academically successful, nonnative speakers of English either generally or almost always perform all 42 tasks surveyed better than less academically successful, nonnative speakers.

Conclusions and Implications The results obtained in this study confirm the judgments of the linguistic specialists who formulated the reading, writing, speaking, and listening tasks. Faculty and students agree that the tasks are relevant for completing coursework at both the undergraduate and graduate levels. These findings support the use of this pool of tasks in the design of both test specifications and assessment measures for use with undergraduate and graduate students across a wide range of subject areas. Even though undergraduate and graduate faculty and students rated all 42 tasks as “important” or “very important,” some tasks were judged to be more important than others. It is recommended that TOEFL 2000 test committees give consideration to those tasks judged to be most important by all four respondent groups. This information, along with faculty judgments on the Relationship Rating Scale, provide a sound basis for designing test specifications and assessment measures. Appendix M provides a useful summary and rank ordering of the ratings provided by each of the four respondent groups. These job analysis results can provide important documentation of the job relatedness and validity of TOEFL 2000 assessment specifications and questions. The reader should note that the high level of agreement among the four respondent groups was based on broad definitions of the reading, writing, speaking, and listening task domains. The operationalization of these domains into actual test items may not necessarily generate the same

47

high level of agreement, especially if the difficulty levels and specific academic contexts for the items are not chosen appropriately. Since it is unlikely that all 42 surveyed tasks will be assessed in TOEFL 2000, it will be necessary to identify the subset of tasks that will be assessed. Selecting tasks for inclusion in test specifications involves the delineation of selection criteria and the exercise of sound professional judgment. By way of example, the authors have specified a set of criteria that could be used to select tasks for TOEFL 2000. These criteria were designed to identify tasks that faculty and students judged to be “very important,” as well as those that faculty judged to be most related to successful academic performance. Example selection criteria. It is proposed that to be selected statements must meet all three of the following criteria: 1. The task statement must have been given an importance rating of 4.0 or higher by either undergraduate or graduate faculty respondents. 2. The statement must have been given an importance rating of 3.5 or higher by both undergraduate and graduate student respondents. 3. The statement must be in the top half of the relationship ratings provided by either undergraduate or graduate faculty respondents, or have a mean importance rating that ranks it in the top 10 by either undergraduate or graduate student respondents. Table 26 presents example task statements that meet these criteria (the mean importance and relationship ratings for these task statements can be found in Appendix Q). These criteria identify a pool of 20 statements (48% of the domain), with representation in all four content domains, and could be considered to be essential elements of the performance domain. Clearly, other criteria could be specified and other task statements could be identified, as all 42 surveyed tasks were judged to be important enough for inclusion. These criteria are offered as a starting point for discussion. If test development staff would like to consider tasks or criteria other than those shown on the following page, they should offer a rationale that explains why those tasks or criteria should be employed. For example, another criterion might involve the inclusion of tasks that cover a range of ability levels, so that more diagnostic information may be provided to candidates.

48

Table 26 Example Tasks that Meet Suggested Criteria for Inclusion in TOEFL 2000 Content domain

Example task statement

READING Basic comprehension

(4) Determine the basic theme (main idea) of a passage. (5) Read and understand written instructions/directions concerning classroom assignments and/or examinations.

Learning

(6) Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present. (7) Read text material with sufficient care and comprehension to remember major ideas.

Integration

(10) Compare and contrast ideas in a single text and/or across texts. (11) Synthesize ideas in a single text and/or across texts.

WRITING Content

(12) Write in response to an assignment and stay on topic without digressions or redundancies. (14) Use background knowledge, reference or nontext materials, personal viewpoints, and other sources appropriately to support Ideas, analyze, and refine arguments.

Organization

(16) Organize writing in order to convey major and supporting ideas.

Development

(18) Use relevant reasons and examples to support a position or idea.

Language

(20) Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation.

SPEAKING

(22) Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions. (25) Demonstrate facility with standard spoken English, including grammar, word choice, fluency, and sentence structure, while performing the following linguistic tasks: (f) developing or structuring hypotheses explaining or informing (g) explaining or informing

LISTENING Facts and details

(26) Understand factual information and details. (27) Understand the instructor’s spoken directions regarding assignments and their due dates.

Vocabulary

(28) Understand important terminology related to the subject matter.

Main ideas

(30) Understand the main ideas and their supporting information. (31) Distinguish between important information and minor details.

Inferences

(32) Make appropriate inferences based on information in a lecture, discussion, or conversation.

49

Other criteria that could eventually affect the final test specifications might include the time and cost required to develop and author the software required for certain item types or performance tasks, as well as the time and cost associated with administration and scoring of such items. Faculty and student reactions to prototypes should also be considered. Also, as data are collected on certain item types or performance tasks, it might become apparent that the results obtained from one type of task are highly related to performance on another task. Again, the criteria described on the previous page are offered as a starting point for discussion and development; other factors and criteria will be involved in the final selection of content and item types to be included in TOEFL 2000. The task statements that were rated in this study to be most important and most related to successful academic performance could also be used to design criterion measures for use in gathering additional validity information about TOEFL 2000 after the examination has been developed and used with undergraduate and graduate students. In addition to providing results that are useful in the design, development, and validation of TOEFL, the study results could also be used for two other related purposes. First, since the task statements contained in this study describe reading, writing, speaking, and listening tasks that both undergraduate and graduate faculty and students judged to be important for competent academic performance across a wide range of subjects, it would seem reasonable to consider ways to use them in score reporting. If feasible, TOEFL should explore ways of using these task statements to report scores or to provide feedback to schools, faculty, and to students. These tasks could provide useful descriptors of the meaning of TOEFL test scores and might be more helpful to admission's officers, faculty, and students than current score reporting procedures. Second, these tasks could also be useful for developing or modifying curricula designed to assist nonnative speakers of English in improving their English language skills; these speakers may benefit from focused instruction in skills identified as important to academic success. Such enhancements may help them succeed in educational programs conducted in English.

50

References American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. Bejar, I., Douglas, D., Jamieson, J., Nissan, S., & Turner, J. (2000). TOEFL 2000 listening framework: A working paper (TOEFL Monograph Series Report No. 19). Princeton, NJ: Educational Testing Service. Butler, F. A., Eignor, D., Jones, S., McNamara, T., & Suomi, B. (2000). TOEFL 2000 speaking framework: A working paper (TOEFL Monograph Series Report No. 20). Princeton, NJ: Educational Testing Service. Bridgeman, B., & Carlson, S. (1983). Survey of academic writing tasks required of graduate and undergraduate foreign students (TOEFL Research Report No. 15). Princeton, NJ: Educational Testing Service. Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework: A working paper (TOEFL Monograph Series Report No. 18). Princeton, NJ: Educational Testing Service. Davies, A. (1984). Validating three tests of English language proficiency. Language testing 1(1), 50-69. Davis, T. M. (Ed.). (1995). Open doors 1994/1995: Report on international educational exchange. New York: Institute of International Education. Enright, M., Grabe, W., Koda, K., Mosenthal, P., Mulcahy-Ernt, P., & Schedle, M. (2000). TOEFL 2000 reading framework: A working paper (TOEFL Monograph Series Report No. 17). Princeton, NJ: Educational Testing Service. Hale, G., Taylor, C., Bridgeman, B., Carson, J., Kroll, B., & Kantor, R. (1996). A study of writing tasks assigned in academic degree programs (TOEFL Research Report No. 54). Princeton, NJ: Educational Testing Service. Jamieson, J., Jones, S., Kirsch, I., Mosenthal, P., & Taylor, C. (2000). TOEFL 2000 framework: A working paper (TOEFL Monograph Series Report No. 16). Princeton, NJ: Educational Testing Service. Jones, R. L. (1979). Performance testing of second language proficiency. In E. J. Briiere and F. B. Hinofotis (Eds.), Concepts in language testing: Some recent studies (pp. 50-57). Washington, DC: TESOL. McNamara, T. F. (1996). Measuring second language performance. London and New York: Longman. Powers, D. E. (1986). Academic demands related to listening skills. Language Testing, 3(1), 1-38.

51

Appendix A — TOEFL 2000 Steering Committee

Alister Cumming University of Toronto, Canada Barbara Suomi Educational Testing Service Dan Douglas Iowa State University Pat Mulcahy University of Bridgeport

53

Appendix B — TOEFL 2000 Framework Teams

Listening

Reading

Claudia Gentile

Mary Enright

ETS

ETS

Dan Douglas

Bill Brabe

Iowa State University

Northern Arizona University

Jean Turner

Keiko Koda

Monterey Institute of International Studies

Carnegie Mellon University

Joan Jamieson

Mary Schedl

Northern Arizona University

ETS

Susan Nissan

Pal Mulcahy

ETS

University of Bridgeport

Speaking

Writing

Frances Butler

Alister Cumming

University of California — Los Angeles

University of Toronto (Canada)

Stan Jones

Robbie Kantor

Statistics Canada

ETS

Dan Eignor

Terry Santos

ETS

Humboldt State University

Tim McNamara

Carol Taylor

The University of Melbourne (Australia)

ETS

Barbara Suomi

Don Powers

ETS

ETS

Irwin Kirsch ETS

54

Appendix C — TOEFL 2000 Research and Development Oversight Committee

Paul Angelis

Carol Chapelle

Southern Illinois University

Iowa State University

William Grabe

Joan Jamieson

Northern Arizona University

Northern Arizona University

Stan Jones

Irwin Kirsch, Ex Officio

Carleton University (Canada)

ETS

Antony J. Kunnan

Phil Oltman, Ex Officio

California State University — Los Angeles

ETS

Carol Taylor

Kentaro Yamamoto

ETS

ETS

55

Appendix D — Schools that Reviewed the Revised Draft Survey

Colorado State University Drexel University Georgia Tech University Hawaii Pacific University Kansas State University University of Nebraska Northwestern University University of Central Oklahoma St. Louis University State University of New York Temple University Tufts University University of Wisconsin

56

Appendix E — TOEFL 2000 Study Coordinator Recruitment Letter Dear ________ , ETS Research staff are currently in the process of conducting a study on behalf of the TOEFL 2000 Program. Your school has been selected as one of 25 sites from which we would like to obtain information for this study. Because of your involvement with TOEFL or other language related activities, we thought that you, or someone you can recommend, might be interested in the position of Study Coordinator at your school. Each coordinator will receive a stipend of $500 for his or her involvement. Some background information about the study and a description of the coordinator’s responsibilities follow. Background: TOEFL is in the process of developing a new examination to measure the reading, writing, listening, and speaking skills important for competent academic performance at the undergraduate and graduate levels. We want the new test to be as relevant as possible to you by reflecting tasks that are judged to be important for competent academic performance at the undergraduate and graduate levels. We want to make sure that professors and students think these tasks are important. We have developed a survey instrument to gather this information. It consists of approximately 40 statements that study participants will be asked to rate. (It should only take about 15 minutes to complete the survey.) We need help in identifying faculty and students to participate in the study. Purpose: The data generated from this study will provide information useful for building the new TOEFL 2000 assessment measure by identifying specific tasks that could be simulated in the assessment or used to identify the language proficiencies necessary to perform them competently. Job of the Coordinator: A)

Each coordinator will be asked to identify a total of 24 faculty members: 12 undergraduate and 12 graduate faculty to participate in the study. (These 24 faculty should be comprised of 2 undergraduate and 2 graduate faculty members selected from each of the 6 subject areas listed below.) Faculty should have experience teaching nonnative speakers of English. The 6 subject areas are: 1. 2. 3. 4. 5. 6.

chemistry computer and information science electrical engineering business/management psychology history

B)

Each coordinator will also need to oversee the recruitment of a total of 18 students for participation: 6 undergraduate and 12 graduate students who are nonnative speakers of English. Coordinators will need to ask participating faculty members to identify these students from each of the 6 subject areas. All 12 graduate faculty will each be asked to recommend 1 student from his or her department for participation in the study (i.e., 12 students total: 2 in each of the 6 subject areas). We only need 6 undergraduate students to participate, so undergraduate faculty will be asked to recommend 1 student from each of the 6 subject areas.

C)

Each coordinator will be responsible for both the distribution to, and collection of completed surveys from, faculty and students. Coordinators will also be responsible for encouraging faculty and students to complete their surveys. If surveys are not completed, they will be asked to find replacement faculty or students to complete them.

57

D)

Surveys will be mailed from ETS to each coordinator in January 1999. Each coordinator will be expected to return the completed surveys to ETS Research staff by the specified deadline in February 1999.

If you are interested in being a Study Coordinator, or if you can refer us to someone else at your school, we would be most appreciative. Please contact me as soon as possible with your decision or referral(s) as we are trying to get commitments from all of our coordinators by the end of November. If possible, please include phone numbers as well as e-mail addresses for yourself and/or any people you refer to us. And please feel free to call me at (609) 734-5221 or Dr. Michael Rosenfeld (if I am out of the office at the time of your call) at (609) 734-1193 if you prefer to discuss the details of the study participation over the phone. Thank you very much for your time in considering this matter. Susan Wilson Leung Senior Research Associate

58

Appendix F — Pilot Study Materials Schools That Participated in Pilot Study

North Carolina State University University of Oregon Ohio State University

59

Participant Letter

January 6, 1999 Dear __________ , Thank you for agreeing to participate in the pilot phase of the TOEFL 2000 Study that is being conducted by Educational Testing Service. More information about the study and directions for completing the survey are included on the top page of the attached survey (inventory). The major purpose of the pilot study is to make sure the directions, rating scales, and statements in the survey are clear and understandable. The information we obtain from you will help us to improve the clarity of the survey before it is finalized and sent to the rest of the study participants later this month. We need your assistance with the pilot study in two ways. First, please read the survey directions and statements carefully and actually use the rating scales to complete the survey (except please skip question number 37 in Part II — Background Information at this time). Your responses will be included in the data analysis for the full study. Second, let us know of any confusion or difficulty you encounter while completing the survey by answering the following questions in the spaces provided below. (If you prefer to give your responses to an ETS Research Staff member over the phone rather than in this written format, please email Susan Leung at [email protected] with your phone number and the time when you can be reached for a phone interview next week between Monday, January 11, through Wednesday, January 13.) Please circle “Yes” or “No” for each of the questions listed below. 1)

Are the directions clear and understandable? YES

NO

(If no, please indicate below what is confusing.)

60

2)

Are the rating scales clear and understandable? YES

NO

(If no, please indicate below what is confusing.)

3)

Are there any statements that were confusing? YES

NO

(If yes, please indicate which statements were confusing and what confused you.)

4) If there are any other comments about the survey that you would like to make, please write them below.

Thank you very much for completing the survey and answering our questions. The information you provided will be a valuable part of the pilot study; we greatly appreciate your help. Sincerely, Michael Rosenfeld, Ph.D. TOEFL 2000 Study Project Director

61

Appendix G — Survey Materials Participant Letter January, 1999 Dear TOEFL 2000 Study Participant: ETS is in the process of redesigning The Test of English as a Foreign Language (TOEFL) examination. We want to make sure that the new examination covers the content most important for academic success. ETS Research staff are conducting a study on behalf of the TOEFL 2000 program to define sets of reading, writing, listening, and speaking tasks that are important for competent academic performance at both the undergraduate and graduate levels across a range of subject areas. The results obtained from this study will be used in the design of the new TOEFL 2000 examination. In order to ensure that this new exam is as relevant as possible to you, we have developed a survey instrument to gather information about the tasks that faculty and students believe are important for a student’s academic success. Since the survey currently contains the reading, writing, listening, and speaking tasks THOUGHT to be important for academic success, it is critical for us to get your opinion as to how important these tasks really are in the academic world. Therefore, we are asking you to rate each of the task statements in the survey. Only you — students who are nonnative speakers of English and the faculty members providing instruction to them — are the best judges of what these important academic tasks are. The input we gather from you through the survey data will be forwarded for review by the team who will actually develop the new TOEFL 2000 examination. (As a point of clarification: These statements are not the items that will be used on the TOEFL exam, but rather they will serve as a guide to linguists who will develop test items to measure a test taker’s ability to perform these tasks.) Enclosed is the survey which was sent to our TOEFL 2000 Study Coordinators at 25 universities across the United States and Canada to distribute to undergraduate and graduate faculty and students. It is relatively short; we anticipate that the survey will only take about 15 minutes to complete. You will notice that we have also asked you to provide some information about yourself and your school in the Background Information section of the survey. This information will be used to describe the respondent group, and where appropriate, to conduct analyses of the data. All information provided in this section and the survey itself will be kept completely confidential and no individual data will be reported. If you would like an electronic version of the Executive Summary to the Final Report summarizing the results of this study, please send your name and e-mail address to Susan Leung at [email protected] and we will send you the summary after the study is completed. Your participation in this study is very important to us and will aid in the development of a new TOEFL 2000 examination which we believe will better meet the needs of both faculty and students at schools that serve an international population. Thank you in advance for your cooperation in this important endeavor. Sincerely,

Michael Rosenfeld, Ph.D. TOEFL 2000 Study Director

62

Appendix H — Demographic Distributions: Undergraduate and Graduate Students and Faculty Undergraduate students College/university American University Arizona State University Boston University Columbia University Georgia State University Iowa State University North Carolina State University Ohio State University Stanford University University of Hawaii at Manoa University of Maryland — Baltimore County University of Michigan — Ann Arbor University of Minnesota University of Missouri University of Oregon University of Pennsylvania University of Texas — Austin University of Toronto — Canada University of Washington University of Wisconsin — Madison Brigham - Young University University of Indiana No response Total

Number Percent Undergraduate students Region 0 1 5 3 2 4 7 6 1 7 5 8 3 6 6 2 3 7 5 6 9 6 1 103

0.0 1.0 4.9 2.9 1.9 3.9 6.8 5.8 1.0 6.8 4.9 7.8 2.9 5.8 5.8 1.9 2.9 6.8 4.9 5.8 8.7 5.8 1.0 100.0

46 57 0 103

44.7 55.3 0.0 100.0

Northeast Midwest South West Canada No response Total

15 33 18 29 7 1 103

14.6 32.0 17.5 28.2 6.8 1.0 100.0

12 17 35 37 2 103

11.7 16.5 34.0 35.9 1.9 100.0

4 0 15 9 6 64 4 1 103

3.9 0.0 14.6 8.7 5.8 62.1 3.9 1.0 100.0

Current status at college/university First-year student Second-year student Third-year student Fourth-year student No response Total

Geographic Region of Origin Africa Canada Europe Latin America Middle East South Asia/East Asia Other No response Total

Gender Male Female No response Total

Number Percent

75

Graduate students College/university American University Arizona State University Boston University Columbia University Georgia State University Iowa State University North Carolina State University Ohio State University Stanford University University of Hawaii at Manoa University of Maryland — Baltimore County University of Michigan — Ann Arbor University of Minnesota University of Missouri University of Oregon University of Pennsylvania University of Texas — Austin University of Toronto — Canada University of Washington University of Wisconsin — Madison Brigham - Young University University of Indiana No response Total

Number Percent Graduate students Region 0 5 10 9 11 8 12 12 11 13 5 15 19 14 9 15 9 12 11 12 14 14 2 242

0.0 2.1 4.1 3.7 4.5 3.3 5.0 5.0 4.5 5.4 2.1 6.2 7.9 5.8 3.7 6.2 3.7 5.0 4.5 5.0 5.8 5.8 0.8 100.0

82 160 0 242

33.9 66.1 0.0 100.0

Northeast Midwest South West Canada No response Total

39 80 46 63 12 2 242

16.1 33.1 19.0 26.0 5.0 0.8 100.0

32 40 40 59 25 28 10 8 242

13.2 16.5 16.5 24.4 10.3 11.6 4.1 3.3 100.0

2 1 37 23 16 151 6 6 242

0.8 0.4 15.3 9.5 6.6 62.4 2.5 2.5 100.0

Area of study Chemistry Computer and information sciences Electrical engineering Business management Psychology History Other No response Total

Geographic region of origin Africa Canada Europe Latin America Middle East South Asia/East Asia Other No response Total

Gender Male Female No response Total

Number Percent

76

Faculty

All Undergraduate Graduate No Response Number Percent Number Percent Number Percent Number Percent

College/university American University Arizona State University Boston University Columbia University Georgia State University Iowa State University North Carolina State University Ohio State University Stanford University University of Hawaii at Manoa University of Maryland — Baltimore County University of Michigan — Ann Arbor University of Minnesota University of Missouri University of Oregon University of Pennsylvania University of Texas — Austin University of Toronto — Canada University of Washington University of Wisconsin - Madison Brigham -Young University University of Indiana No response Total

0 8 19 13 18 17 19 17 14 23 10 19 22 24 20 8 17 23 13 24 28 21 4 381

0.0 2.1 5.0 3.4 4.7 4.5 5.0 4.5 3.7 6.0 2.6 5.0 5.8 6.3 5.2 2.1 4.5 6.0 3.4 6.3 7.3 5.5 1.0 100.0

0 4 8 5 9 2 7 4 5 11 4 7 10 10 7 1 7 12 8 12 8 13 1 155

0.0 2.6 5.2 3.2 5.8 1.3 4.5 2.6 3.2 7.1 2.6 4.5 6.5 6.5 4.5 0.6 4.5 7.7 5.2 7.7 5.2 8.4 0.6 100.0

0 4 11 8 9 15 12 12 9 12 6 12 11 13 11 6 8 10 5 12 20 8 1 215

0.0 1.9 5.1 3.7 4.2 7.0 5.6 5.6 4.2 5.6 2.8 5.6 5.1 6.0 5.1 2.8 3.7 4.7 2.3 5.6 9.3 3.7 0.5 100.0

0 0 0 0 0 0 0 1 0 0 0 0 1 1 2 1 2 1 0 0 0 0 2 11

0.0 0.0 0.0 0.0 0.0 0.0 0.0 9.1 0.0 0.0 0.0 0.0 9.1 9.1 18.2 9.1 18.2 9.1 0.0 0.0 0.0 0.0 18.2 100.0

50 120 78 106 23 4 381

13.1 31.5 20.5 27.8 6.0 1.0 100.0

18 48 33 43 12 1 155

11.6 31.0 21.3 27.7 7.7 0.6 100.0

31 70 42 61 10 1 215

14.4 32.6 19.5 28.4 4.7 0.5 100.0

1 2 3 2 1 2 11

9.1 18.2 27.3 18.2 9.1 18.2 100.0

Region Northeast Midwest South West Canada No response Total

77

Faculty

All Undergraduate Graduate No Response Number Percent Number Percent Number Percent Number Percent

Subject area Chemistry Computer and information sciences Electrical engineering Business management Psychology History Other No response Total

62 62 48 62 60 65 16 6 381

16.3 16.3 12.6 16.3 15.7 17.1 4.2 1.6 100.0

25 24 15 24 26 36 4 1 155

16.1 15.5 9.7 15.5 16.8 23.2 2.6 0.6 100.0

36 36 31 38 33 26 12 3 215

16.7 16.7 14.4 17.7 15.3 12.1 5.6 1.4 100.0

1 2 2 0 1 3 0 2 11

9.1 18.2 18.2 0.0 9.1 27.3 0.0 18.2 100.0

155 215 11 381

40.7 56.4 2.9 100.0

155 0 0 155

100.0 0.0 0.0 100.0

0 215 0 215

0.0 100.0 0.0 100.0

0 0 11 11

0.0 0.0 100.0 100.0

9.4 16.0 9.7 63.8 1.0 100.0

21 24 14 96 0 155

13.5 15.5 9.0 61.9 0.0 100.0

13 36 23 140 3 215

6.0 16.7 10.7 65.1 1.4 100.0

2 1 0 7 1 11

18.2 9.1 0.0 63.6 9.1 100.0

Perspective of responses Instruction of undergraduate students Instruction of graduate students No response Total

Experience teaching nonnative English speakers 1 to 3 years 4 to 7 years 8 to 10 years More than 10 years No response Total

36 61 37 243 4 381

78

Faculty

All Undergraduate Graduate No Response Number Percent Number Percent Number Percent Number Percent

Gender Male Female No response Total

80 295 6 381

21.0 77.4 1.6 100.0

35 120 0 155

22.6 77.4 0.0 100.0

42 168 5 215

19.5 78.1 2.3 100.0

3 7 1 11

27.3 63.6 9.1 100.0

6 0 31 11 322 6 5 381

1.6 0.0 8.1 2.9 84.5 1.6 1.3 100.0

2 0 9 5 135 4 0 155

1.3 0.0 5.8 3.2 87.1 2.6 0.0 100.0

4 0 20 6 180 2 3 215

1.9 0.0 9.3 2.8 83.7 0.9 1.4 100.0

0 0 2 0 7 0 2 11

0.0 0.0 18.2 0.0 63.6 0.0 18.2 100.0

0 0 6 5 3 41 0 326 381

0.0 0.0 1.6 1.3 0.8 10.8 0.0 85.6 100.0

0 0 3 2 2 11 0 137 155

0.0 0.0 1.9 1.3 1.3 7.1 0.0 88.4 100.0

0 0 3 3 1 27 0 181 215

0.0 0.0 1.4 1.4 0.5 12.6 0.0 84.2 100.0

0 0 0 0 0 3 0 8 11

0.0 0.0 0.0 0.0 0.0 27.3 0.0 72.7 100.0

Race/ethnicity African American (non-Hispanic) American Indian/Alaskan Native Asian or Pacific Islander Hispanic White (non-Hispanic) Other No response Total

Students’ geographic regions of origin Africa Canada Europe Latin America Middle East South Asia/East Asia Other No response Total

79

Appendix I — Undergraduate Faculty: Percent Zero and Mean Importance Ratings (Overall and by Subject) Task statements

Overall

Chemistry

Computer science

Electrical engineering

Business/management

Psychology

History

(N = 150)

(N = 25)

(N = 24)

(N = 15)

(N = 24)

(N = 26)

(N = 36)

Mean

SD

SE

% 0 Mean

SD

SE

3.74

1.04

0.09

1

3.71

1.16

0.09

% 0 Mean

SD

SE

3.60

1.08

0.22

0

2

4.44

0.58

0.12

3.57 1.05 0.09 4.06 0.98 0.08 4.39 0.86 0.07

1 3 0

3.12 3.52 4.16

4.23

0.81

0.07

3

4.43

0.68

0.06

% 0 Mean

SD

SE

3.96

0.93

0.19

4

0

4.04

0.88

0.18

1.09 0.22 0.92 0.18 0.99 0.20

0 0 0

3.65 4.09 4.54

4.12

0.73

0.15

0

1

4.25

0.61

0.12

3.74 1.10 0.09 3.66 1.14 0.09

3 8

3.54 3.48

3.51 3.90

1.18 1.09

0.10 0.09

3 3

3.84

1.05

0.09

3.33 3.51

1.18 1.26

3.50

1.18

% 0 Mean

SD

SE

3.67

0.90

0.23

0

4

3.87

1.25

0.32

0.98 0.20 0.75 0.16 0.66 0.13

4 8 0

2.93 1.22 3.36 1.39 4.13 0.92

4.23

0.92

0.19

8

3.80

0

4.26

0.81

0.17

4

4.20

1.14 0.23 1.08 0.22

4 16

3.29 3.15

1.19 0.99

0.24 0.21

2.83 3.48

1.09 1.12

0.22 0.22

0 4

3.48 3.87

1

3.25

1.03

0.21

4

0.10 0.10

10 5

2.82 3.41

0.95 0.19 1.30 0.26

0.10

8

3.14

1.06

4.19 1.05 0.09 3.56 1.11 0.09

3 4

3.82 3.36

4.09 0.91 0.07 3.41 1.18 0.10

3.70

1.11

3.62

1.00

% 0 Mean

SD

SE

3.50

1.14

0.23

0

0

3.70

0.93

0.19

0.32 0.36 0.24

0 7 0

0.94

0.24

0.77

13 13

1.20 0.25 1.14 0.23

3.39

0.94

29 12

0.22

% 0 Mean

SD

SE

%0

3.73

1.19

0.23

0

3.91

0.98

0.16

3

4

4.31

0.79

0.15

0

2.49

1.04

0.17

3

3.58 1.14 0.23 4.13 1.01 0.21 4.50 0.78 0.16

0 0 0

3.77 0.95 0.19 4.15 0.88 0.18 4.40 1.04 0.21

0 0 0

3.97 0.80 0.13 4.59 0.66 0.11 4.47 0.77 0.13

3 3 0

0

4.17

0.89

0.18

4

4.50

0.71

0.14

0

4.34

0.68

0.11

3

0.20

0

4.50

0.66

0.13

0

4.62

0.70

0.14

0

4.58

0.55

0.09

0

4.21 0.89 3.15 1.21

0.23 0.32

7 7

4.00 1.06 0.22 3.74 1.05 0.22

0 0

3.65 1.06 0.21 3.92 1.25 0.24

0 4

3.86 1.09 0.18 4.06 1.06 0.18

0 6

4 4

2.85 3.43

1.14 1.34

0.30 0.35

13 7

3.58 4.13

1.21 0.92

0.25 0.19

0 4

3.69 3.88

1.09 1.07

0.22 0.21

0 0

4.06 4.23

1.01 0.94

0.17 0.16

3 3

0.19

4

3.33

1.29

0.33

0

4.08

1.02

0.21

0

4.00

0.98

0.19

0

4.44

0.65

0.11

0

3.35 3.43

1.18 0.25 0.93 0.19

17 13

2.67 2.93

1.40 1.49

0.36 0.39

0 7

3.88 1.12 0.23 3.83 1.13 0.23

0 0

3.42 1.14 0.22 3.73 1.31 0.26

0 0

3.41 3.47

1.10 0.18 1.34 0.22

11 0

16

3.11

1.24

0.25

17

2.69

1.18

0.31

13

3.17

1.01

0.21

0

3.88

1.21

0.24

0

4.21

0.88

0.15

6

1.18 0.24 1.18 0.24

12 12

4.09 3.19

0.68 0.14 1.03 0.21

8 13

3.21 3.00

1.53 1.36

0.39 0.35

0 0

4.58 3.63

0.58 0.88

0.12 0.18

0 0

4.08 3.65

1.23 1.26

0.25 0.25

0 0

4.67 4.03

0.63 0.88

0.11 0.15

0 0

2 5

3.80 1.04 0.21 3.29 1.23 0.25

0 13

4.05 3.52

0.67 0.14 0.98 0.20

13 13

3.53 1.25 2.71 1.44

0.32 0.37

0 7

4.29 0.62 0.13 3.17 1.05 0.21

0 0

4.00 1.02 0.20 3.38 1.30 0.25

0 0

4.47 0.65 0.11 3.86 1.02 0.17

0 0

0.09

1

3.48

1.29

0.26

0

3.52

1.04

0.21

4

3.40

1.24

0.32

0

3.96

1.08

0.22

0

3.65

1.23

0.24

0

3.97

0.84

0.14

0

0.08

2

3.52

1.08

0.22

0

3.74

0.81

0.17

4

3.29

1.27

0.33

7

3.63

0.82

0.17

0

3.58

1.10

0.22

0

3.77

1.00

0.17

3

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

80

Task statements, continued

Overall

Chemistry

Computer science

Electrical engineering

Business/management

Psychology

History

(N = 150)

(N = 25)

(N = 24)

(N = 15)

(N = 24)

(N = 26)

(N = 36)

Mean

SD

SE

% 0 Mean

SD

SE % 0 Mean

SD

SE

%0

Mean

SD

SE

%0

Mean

SD

SE % 0 Mean

SD

SE

3.81

1.06

0.09

0

3.51 3.38

1.17 1.18

0.10 0.10

3.44 3.12 3.03 3.42 3.54 3.53 3.74

1.04 1.16 1.08 1.13 1.04 1.14 1.08

4.45 4.24

% 0 Mean

SD

SE

%0

3.96

0.93

0.19

0

3.83

0.96

0.20

0

4.07

1.16

0.30

0

3.92

1.10

0.22

0

3.42

1.14

0.23

0

3.77

1.06

0.18

0

2 12

3.36 3.33

1.22 1.33

0.24 0.27

0 28

3.57 3.30

1.24 0.25 1.13 0.24

4 13

3.07 3.23

1.21 1.24

0.31 0.32

7 13

4.13 3.96

0.74 0.93

0.15 0.19

0 0

3.08 3.14

1.29 1.32

0.26 0.27

4 12

3.66 3.28

1.08 0.18 1.11 0.19

0 6

0.09 0.10 0.09 0.09 0.09 0.09 0.09

6 13 8 8 7 5 3

3.38 3.32 3.10 3.00 3.67 3.30 3.83

1.12 0.22 16 1.45 0.29 24 1.18 0.24 16 1.03 0.21 20 1.02 0.20 13 1.49 0.30 8 1.23 0.25 8

3.50 3.41 3.43 3.35 3.57 3.64 3.78

0.91 1.01 0.99 0.99 1.12 1.00 0.95

0.19 0.21 0.20 0.20 0.23 0.20 0.19

8 8 4 17 13 8 4

3.00 3.08 3.20 3.07 3.14 3.27 3.60

1.20 0.31 1.19 0.31 1.15 0.30 1.33 0.34 1.10 0.28 1.22 0.32 1.35 0.35

0 13 0 0 7 0 0

3.71 3.13 2.83 3.79 3.57 3.57 4.13

0.95 1.08 0.98 1.25 0.95 0.95 0.85

0.19 0.22 0.20 0.26 0.20 0.19 0.17

0 0 4 0 4 4 0

3.08 2.59 2.71 3.08 3.28 3.56 3.40

1.14 0.23 0.96 0.19 1.00 0.20 1.18 0.24 1.10 0.22 1.12 0.22 1.19 0.24

8 15 8 8 4 4 4

3.71 3.20 3.00 3.82 3.79 3.70 3.68

0.91 1.21 1.13 0.87 0.98 1.10 0.94

0.15 0.20 0.19 0.14 0.16 0.18 0.16

3 14 11 3 3 6 3

0.72 1.06

0.06 0.09

0 0

4.68 4.20

0.48 0.10 0.96 0.19

0 0

4.67 4.38

0.48 0.10 0.77 0.16

0 0

4.47 4.53

0.64 0.17 0.64 0.17

0 0

4.38 4.04

0.71 0.15 1.00 0.20

0 0

4.27 3.96

0.92 0.18 1.43 0.29

0 0

4.31 4.40

0.83 0.14 1.17 0.19

0 0

4.23 3.68

0.86 1.01

0.07 0.08

0 1

4.48 3.76

0.77 1.01

0.15 0.20

0 0

4.42 3.54

0.58 1.22

0.12 0.25

0 0

4.27 3.80

0.70 0.68

0.18 0.17

0 0

4.13 3.71

1.08 1.12

0.22 0.23

0 0

4.12 3.50

0.99 1.03

0.20 0.21

0 0

4.06 3.79

0.84 0.89

0.14 0.15

0 3

4.34 4.20

0.81 0.83

0.07 0.07

1 1

4.20 4.08

0.87 0.17 0.86 0.17

0 0

4.33 4.17

1.05 0.21 1.05 0.21

0 0

4.07 4.00

0.96 0.25 1.00 0.26

0 0

4.17 4.00

0.70 0.14 0.66 0.13

0 0

4.40 4.40

0.71 0.71

0.14 0.14

4 4

4.63 4.40

0.60 0.74

0.10 0.12

0 0

4.10

0.80

0.07

1

3.84

0.85

0.17

0

4.13

0.85

0.17

0

3.73

0.80

0.21

0

4.33

0.70

0.14

0

4.20

0.76

0.15

4

4.20

0.76

0.13

0

3.75

0.99

0.08

3

3.63

1.13

0.23

4

3.75

1.15

0.24

0

3.60

1.06

0.27

0

3.48

0.95

0.19

4

3.88

0.97

0.19

4

4.00

0.74

0.12

3

3.41

1.13

0.09

1

3.48

1.12

0.22

0

3.75

1.19

0.24

0

3.14

1.10

0.28

7

3.21

1.10

0.23

0

3.23

1.18

0.24

0

3.53

1.08

0.18

3

3.11 3.05

1.00 0.98

0.08 0.08

3 5

2.84 2.96

0.94 0.19 0.86 0.17

0 4

3.09 2.96

1.04 0.21 1.11 0.23

4 4

2.64 2.79

1.28 0.33 1.31 0.34

7 7

3.21 3.25

0.98 0.20 0.94 0.19

0 0

3.32 3.13

0.90 0.18 0.90 0.18

4 8

3.29 3.09

0.94 0.16 0.91 0.15

3 6

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean importance ratings of less than 3.50 are shaded gray.

81

Appendix J — Undergraduate Faculty: Mean Relationship Ratings (Overall and by Subject) Overall

Chemistry

(N = 150) Mean SD SE

(N = 25) Mean SD

2.29

0.59

0.05

2.44

2.02

0.68

0.06

2.26 2.35 2.24

0.69 0.61 0.70

6. Read text material with sufficient care and comprehension to remember major ideas and 2.41 answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 2.38 8. Read text material and outline important ideas and concepts 2.23 9. Distinguish factual information from opinions 2.06

Task statements

Computer science

SE

(N = 24) Mean SD

0.51

0.10

2.25

2.16

0.62

0.12

0.06 0.05 0.06

2.30 2.40 2.12

0.70 0.65 0.60

0.57

0.05

2.32

0.58 0.62 0.73

0.05 0.05 0.06

2.21 2.34

0.62 0.65

2.24

Electrical engineering

SE

(N = 15) Mean SD

0.53

0.11

2.33

2.17

0.70

0.14

0.15 0.13 0.12

2.50 2.30 2.13

0.59 0.63 0.80

0.56

0.11

2.39

2.32 2.12 2.29

0.56 0.67 0.75

0.11 0.13 0.15

0.05 0.06

2.26 2.30

0.54 0.47

0.63

0.05

2.20

2.04 2.12

0.68 0.70

0.06 0.06

2.14

0.66

2.40 2.18

Business/management

SE

(N = 24) Mean SD

0.72

0.19

2.23

1.93

0.62

0.16

0.12 0.13 0.16

2.14 2.43 2.43

0.77 0.76 0.65

0.58

0.12

2.53

2.29 2.09 1.77

0.46 0.60 0.75

0.09 0.12 0.16

0.11 0.10

2.29 2.33

0.62 0.64

0.50

0.10

2.39

2.13 2.24

0.61 0.52

0.13 0.10

0.06

2.16

0.69

0.63 0.67

0.05 0.06

2.12 2.12

2.25 2.02

0.64 0.73

0.05 0.06

2.35

0.66

2.26

0.67

Psychology

SE

(N = 26) Mean SD

0.69

0.15

2.29

1.91

0.68

0.15

0.21 0.20 0.17

2.14 2.55 2.36

0.79 0.51 0.66

0.52

0.13

2.55

2.71 2.50 2.00

0.47 0.65 0.71

0.13 0.17 0.20

0.13 0.13

1.93 2.08

0.62 0.76

0.58

0.12

2.27

2.18 2.22

0.59 0.80

0.13 0.17

0.14

1.96

0.71

0.53 0.60

0.11 0.12

2.48 2.26

2.32 1.96

0.48 0.68

0.10 0.14

0.06

2.28

0.61

0.06

2.36

0.70

History

SE

(N = 36) Mean SD

SE

0.56

0.12

2.22

0.59

0.10

2.00

0.77

0.17

1.94

0.67

0.11

0.17 0.11 0.14

2.00 2.25 2.24

0.65 0.64 0.83

0.15 0.14 0.18

2.31 2.23 2.26

0.63 0.55 0.66

0.11 0.09 0.11

0.60

0.13

2.45

0.60

0.13

2.31

0.58

0.10

2.41 2.32 2.10

0.67 0.57 0.72

0.14 0.12 0.16

2.55 2.45 2.05

0.60 0.51 0.69

0.13 0.11 0.15

2.22 2.09 2.10

0.59 0.63 0.72

0.10 0.11 0.13

0.16 0.21

2.19 2.24

0.75 0.77

0.16 0.17

2.40 2.55

0.60 0.69

0.13 0.15

2.12 2.41

0.59 0.61

0.10 0.10

0.80

0.21

2.45

0.60

0.13

2.00

0.63

0.14

2.17

0.65

0.11

2.07 1.93

0.88 0.80

0.23 0.21

2.19 2.14

0.81 0.77

0.18 0.17

2.00 2.24

0.63 0.62

0.14 0.14

1.80 1.97

0.58 0.72

0.10 0.12

0.15

2.33

0.82

0.21

2.23

0.69

0.15

2.19

0.51

0.11

2.09

0.62

0.11

0.67 0.69

0.14 0.14

2.71 2.21

0.47 0.80

0.13 0.21

2.50 2.14

0.60 0.71

0.13 0.15

2.30 2.24

0.73 0.70

0.16 0.15

2.42 2.17

0.65 0.65

0.11 0.11

2.22 1.91

0.67 0.79

0.14 0.17

2.50 2.14

0.52 0.66

0.14 0.18

2.18 2.23

0.73 0.75

0.16 0.16

2.40 1.90

0.60 0.72

0.13 0.16

2.09 2.03

0.70 0.77

0.12 0.13

0.12

2.61

0.58

0.12

2.27

0.70

0.18

2.32

0.72

0.15

2.33

0.66

0.14

2.28

0.70

0.12

0.14

2.43

0.59

0.12

2.20

0.56

0.14

2.18

0.80

0.17

2.29

0.56

0.12

2.11

0.72

0.12

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

82

Overall

Task statements, continued

Chemistry

(N = 150) Mean SD SE

(N = 25) Mean SD

2.19

0.70

0.06

2.08

2.13 2.12

0.67 0.67

0.06 0.06

2.04 1.94 1.96 2.22 2.19 2.30 2.27

0.62 0.65 0.63 0.64 0.62 0.64 0.63

26. Understand factual information and details 2.41 27. Understand the instructor’s spoken directions regarding assignments and their due dates 2.12

Computer science SE

(N = 24) Mean SD

0.70

0.14

2.29

2.04 2.08

0.61 0.50

0.12 0.10

0.05 0.06 0.05 0.05 0.05 0.05 0.05

2.16 2.04 2.08 2.25 2.32 2.64 2.32

0.55 0.61 0.72 0.61 0.56 0.57 0.63

0.62 0.75

0.05 0.06

2.36 1.96

2.29 2.23

0.66 0.68

0.06 0.06

2.34 2.38

0.61 0.64

2.36

Electrical engineering Business/management

SE

(N = 15) Mean SD

SE

(N = 24) Mean SD

0.69

0.14

2.13

0.64

0.17

2.23

2.08 2.09

0.78 0.67

0.16 0.14

2.20 2.40

0.68 0.74

0.17 0.19

0.11 0.12 0.15 0.12 0.11 0.11 0.13

1.96 2.00 1.87 2.29 2.04 2.22 2.25

0.62 0.66 0.63 0.75 0.69 0.74 0.68

0.13 0.13 0.13 0.15 0.14 0.15 0.14

2.36 2.00 2.07 2.43 2.50 2.29 2.57

0.63 0.78 0.73 0.51 0.52 0.47 0.51

0.57 0.73

0.11 0.15

2.48 2.22

0.59 0.74

0.12 0.15

2.67 2.36

2.24 2.12

0.66 0.67

0.13 0.13

2.22 2.17

0.67 0.65

0.14 0.14

0.05 0.05

2.44 2.40

0.58 0.50

0.12 0.10

2.26 2.43

0.62 0.66

0.64

0.05

2.40

0.65

0.13

2.30

2.10

0.64

0.05

2.13

0.55

0.11

1.98

0.67

0.06

2.13

0.61

1.89 1.85

0.68 0.71

0.06 0.06

1.96 2.08

0.61 0.70

Psychology

History

SE

(N = 26) Mean SD

SE

Mean

(N = 36) SD

SE

0.81

0.17

2.10

0.79

0.18

2.25

0.65

0.11

2.29 2.11

0.72 0.68

0.16 0.16

2.00 2.05

0.65 0.71

0.15 0.16

2.18 2.10

0.63 0.75

0.11 0.13

0.17 0.21 0.20 0.14 0.14 0.13 0.14

2.09 1.90 1.91 2.18 2.18 2.23 2.32

0.68 0.70 0.68 0.73 0.59 0.69 0.65

0.15 0.15 0.15 0.16 0.13 0.15 0.14

1.80 1.80 1.90 2.10 2.00 2.20 2.05

0.52 0.62 0.55 0.55 0.65 0.70 0.60

0.12 0.14 0.12 0.12 0.15 0.16 0.14

2.00 1.91 1.94 2.15 2.21 2.23 2.23

0.64 0.62 0.54 0.61 0.64 0.60 0.65

0.11 0.11 0.09 0.10 0.11 0.10 0.11

0.49 0.84

0.13 0.23

2.36 2.18

0.66 0.73

0.14 0.16

2.38 2.19

0.59 0.75

0.13 0.16

2.33 2.00

0.72 0.76

0.12 0.13

2.71 2.50

0.61 0.65

0.16 0.17

2.18 2.09

0.73 0.81

0.16 0.17

2.33 2.48

0.58 0.51

0.13 0.11

2.25 2.19

0.65 0.71

0.11 0.12

0.13 0.14

2.64 2.36

0.50 0.74

0.13 0.20

2.18 2.23

0.66 0.69

0.14 0.15

2.40 2.48

0.50 0.60

0.11 0.13

2.25 2.36

0.65 0.68

0.11 0.11

0.76

0.16

2.43

0.65

0.17

2.38

0.59

0.13

2.40

0.60

0.13

2.31

0.62

0.10

2.00

0.74

0.15

2.43

0.51

0.14

2.00

0.69

0.15

2.15

0.59

0.13

2.06

0.67

0.11

0.13

1.87

0.63

0.13

1.93

0.70

0.18

1.95

0.67

0.15

2.00

0.73

0.16

1.97

0.71

0.12

0.12 0.14

1.96 1.78

0.77 0.74

0.16 0.15

1.87 1.60

0.74 0.63

0.19 0.16

1.71 1.86

0.64 0.85

0.14 0.19

2.10 1.95

0.72 0.69

0.16 0.15

1.80 1.77

0.63 0.65

0.11 0.11

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts and details

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean relationship ratings of less than 2.00 are shaded gray.

83

Appendix K — Graduate Faculty: Percent Zero and Mean Importance Ratings (Overall and by Subject) Task statements

Overall

Chemistry

Computer science

Electrical engineering

Business/management

Psychology

History

Mean

(N = 200) SD SE

(N = 36) SD SE

(N = 36) SD SE

(N = 33) SD SE

(N = 26) SD SE

3.72

1.13

0.08

1

3.56

1.34

0.22

0

3.92

0.97

3.99

1.02

0.07

1

4.31

0.75

0.12

0

4.06

0.89

3.71 0.99 4.20 0.93 4.29 0.95

0.07 0.07 0.07

1 0 0

3.60 0.95 0.16 4.08 0.91 0.15 4.31 0.86 0.14

4.25

0.91

0.06

2

4.33

0.83

4.40

0.74

0.05

0

4.47

0.65

3.79 1.12 3.88 1.17

0.08 0.09

4.01 4.19

1.02 0.95

4.13

%0

Mean

(N = 31) SD SE

%0

Mean

(N = 38) SD SE

0.16

0

3.57

1.07

0.20

3

3.81

1.12

0.18

3

3.52

1.20

0.21

0

3.96

0.96

0.19

0

0.15

0

4.39

0.67

0.12

0

3.73

1.12

0.18

3

4.12

0.99

0.17

0

3.23

1.27

0.25

0

0 0 0

3.97 0.82 0.14 3.97 1.06 0.18 4.58 0.73 0.12

3 0 0

3.48 1.12 0.20 3.84 1.10 0.20 4.10 1.16 0.21

0 0 0

3.58 0.95 0.15 4.32 0.87 0.14 4.39 0.86 0.14

0 0 0

3.56 4.36 3.82

1.16 0.20 0.78 0.14 1.13 0.20

3 0 0

4.12 0.82 4.69 0.55 4.50 0.71

0.16 0.11 0.14

0 0 0

0.14

0

4.23

0.88

0.15

3

4.00

0.89

0.16

0

4.22

0.93

0.15

5

4.18

1.04

0.18

0

4.60

0.87

0.17

0

0.11

0

4.14

0.72

0.12

0

4.35

0.71

0.13

0

4.51

0.69

0.11

0

4.24

0.97

0.17

0

4.77

0.43

0.08

0

1 3

4.03 1.03 0.17 3.79 1.36 0.24

3 3

3.59 1.08 0.18 3.68 1.05 0.18

0 3

3.93 1.01 0.19 3.48 1.31 0.25

0 7

3.84 1.17 0.19 4.13 1.07 0.18

3 3

3.39 3.93

1.26 0.22 1.20 0.22

0 0

4.00 1.08 4.33 0.82

0.24 0.18

0 0

0.07 0.07

2 1

3.80 3.89

1.13 1.08

0.19 0.18

0 0

3.56 3.80

0.99 0.96

0.17 0.16

3 3

3.31 3.83

0.97 0.99

0.17 0.18

3 0

4.38 4.50

0.82 0.85

0.14 0.14

6 3

4.38 4.44

0.75 0.76

0.13 0.13

0 0

4.73 4.77

0.60 0.51

0.12 0.10

0 0

0.89

0.06

0

3.92

1.02

0.17

0

3.89

0.82

0.14

0

3.94

1.09

0.20

0

4.39

0.82

0.13

0

4.24

0.71

0.12

0

4.42

0.64

0.13

0

3.62 1.02 4.07 1.02

0.07 0.07

2 2

3.50 1.25 0.21 3.94 1.09 0.18

0 0

3.74 0.82 0.14 3.83 0.98 0.16

3 3

3.45 1.06 0.19 3.70 1.12 0.20

6 3

3.74 0.98 0.16 4.43 0.77 0.12

0 3

3.64 4.21

1.06 0.18 1.05 0.18

0 0

3.64 0.91 4.31 0.93

0.18 0.18

4 0

3.86

1.09

0.08

3

3.83

0.21

3

3.71

0.17

3

3.48

0.22

0

3.92

0.16

5

3.97

0.97

0.17

0

4.36

0.86

0.17

4

4.46 0.75 3.85 1.07

0.05 0.08

1 2

4.29 0.99 0.16 3.72 1.26 0.21

0 0

4.17 0.79 0.13 3.51 1.25 0.21

3 3

4.26 0.82 0.15 3.76 0.95 0.17

0 6

4.76 0.43 0.07 4.16 0.89 0.14

0 0

4.55 3.81

0.67 0.12 0.90 0.16

0 0

4.81 4.15

0.40 0.97

0.08 0.19

0 0

4.34 0.81 0.06 3.54 1.13 0.08

1 3

4.11 0.98 0.16 3.31 1.18 0.20

0 3

4.06 0.80 0.13 3.52 1.09 0.18

3 6

4.13 0.96 0.17 3.29 1.24 0.22

0 10

4.61 0.59 3.57 1.04

0.10 0.17

0 0

4.41 3.75

0.71 0.12 0.98 0.17

0 0

4.81 0.40 3.85 1.22

0.08 0.25

0 0

4.06

0.98

0.07

1

3.89

1.14

0.19

0

3.89

1.06

0.18

0

4.07

0.91

0.16

3

4.32

0.84

0.14

0

3.91

1.04

0.18

0

4.35

0.75

0.15

0

3.74

1.03

0.07

0

3.64

1.02

0.17

0

3.53

1.08

0.18

0

3.57

1.17

0.21

0

3.84

0.97

0.16

0

3.91

1.00

0.17

0

4.04

0.87

0.17

0

% 0 Mean

% 0 Mean

% 0 Mean

% 0 Mean

%0

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

1.25

1.05

1.23

1.00

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

84

Overall

Task statements, continued

Chemistry

Computer science

Mean

(N = 200) SD SE

(N = 36) SD SE

4.14

0.82

0.06

0

4.09

0.95

0.16

0

4.11

0.63

3.95 3.96

0.90 0.92

0.06 0.07

0 2

3.83 3.79

0.92 0.15 1.08 0.18

0 6

3.79 3.51 3.44 3.89 3.96 4.12 4.15

0.92 0.97 1.05 0.93 0.91 0.98 0.82

0.07 0.07 0.08 0.07 0.06 0.07 0.06

1 3 2 2 1 2 1

3.94 3.79 3.55 3.71 4.00 4.12 4.24

0.92 1.12 1.28 0.91 0.95 1.01 0.85

0.15 0.19 0.22 0.15 0.16 0.17 0.14

4.41 4.15

0.80 0.98

0.06 0.07

0 0

4.26 3.94

4.32 3.96

0.76 0.89

0.05 0.06

0 0

4.45 4.29

0.74 0.83

0.05 0.06

4.28

0.81

3.91

Electrical engineering

%0

Mean

(N = 31) SD SE

0.11

0

4.06

0.93

3.83 3.88

0.92 0.15 1.02 0.18

0 0

3 3 3 3 3 3 3

3.77 3.54 3.71 3.79 3.91 4.06 4.11

0.94 0.89 1.07 1.04 1.00 0.97 0.80

0.16 0.15 0.18 0.17 0.17 0.16 0.13

0.89 0.15 1.00 0.17

0 0

4.64 4.44

4.17 4.00

0.82 0.92

0.14 0.15

0 0

0 0

4.17 4.11

0.79 0.13 0.80 0.13

0.06

1

4.09

0.89

0.98

0.07

1

3.69

3.70

1.04

0.07

2

3.27 3.16

1.09 1.14

0.08 0.08

2 2

Business/management

Psychology (N = 33) SD SE

History

%0

Mean

(N = 38) SD SE % 0 Mean

0.17

0

4.39

0.79

0.13

0

4.06

0.90

0.16

0

4.08

0.69

0.13

0

3.55 3.75

0.96 0.17 0.97 0.18

0 3

4.39 4.41

0.75 0.12 0.64 0.11

0 0

3.97 3.90

0.85 0.15 0.90 0.17

0 0

4.12 3.96

0.77 0.15 0.72 0.14

0 0

0 0 0 3 3 6 0

3.71 3.53 3.58 3.67 3.68 3.87 4.13

0.94 0.90 0.92 0.96 0.79 1.20 0.81

0.17 0.16 0.17 0.17 0.14 0.22 0.14

0 3 0 3 0 0 0

3.68 3.41 3.30 4.08 4.11 4.11 4.16

0.87 0.83 0.88 0.82 0.80 1.02 0.79

0.14 0.14 0.14 0.13 0.13 0.17 0.13

0 3 3 0 0 3 0

3.82 3.45 3.21 4.12 4.12 4.36 4.15

0.95 0.94 1.08 0.82 0.86 0.78 0.76

0.17 0.17 0.19 0.14 0.15 0.14 0.13

0 0 0 0 0 0 0

3.81 3.29 3.25 3.96 3.92 4.19 4.08

0.94 1.16 1.03 1.00 1.09 0.85 0.98

0.18 0.23 0.20 0.20 0.21 0.17 0.19

0 8 4 0 0 0 0

0.64 0.11 0.91 0.15

0 0

4.35 3.90

0.91 0.16 0.98 0.18

0 0

4.34 4.26

0.94 0.15 0.92 0.15

0 0

4.45 3.97

0.62 0.11 1.05 0.18

0 0

4.38 4.35

0.75 0.15 0.94 0.18

0 0

4.53 4.00

0.65 0.79

0.11 0.13

0 0

4.35 3.90

0.75 0.98

0.14 0.18

0 0

4.26 3.89

0.76 0.89

0.12 0.15

0 0

4.36 4.06

0.82 0.97

0.14 0.17

0 0

4.23 3.88

0.76 0.82

0.15 0.16

0 0

0 0

4.44 4.33

0.73 0.12 0.79 0.13

0 0

4.42 4.32

0.85 0.15 0.87 0.16

0 0

4.53 4.34

0.73 0.12 0.88 0.14

0 0

4.58 4.33

0.61 0.85

0.11 0.15

0 0

4.62 4.28

0.64 0.79

0.12 0.16

0 0

0.15

0

4.40

0.74

0.12

0

3.90

0.91

0.16

0

4.51

0.65

0.11

3

4.45

0.75

0.13

0

4.31

0.84

0.16

0

1.11

0.18

0

4.03

0.82

0.14

0

3.74

0.96

0.17

0

4.03

0.99

0.16

3

3.85

1.03

0.18

0

4.15

0.92

0.18

0

3.62

1.16

0.19

3

4.03

0.76

0.13

3

3.42

1.03

0.18

0

3.73

0.90

0.15

3

3.76

1.20

0.21

0

3.58

1.17

0.23

0

2.97 2.91

1.14 0.19 1.06 0.18

3 3

3.49 3.26

1.17 0.20 1.21 0.20

0 3

2.94 2.55

1.12 0.20 1.18 0.21

0 0

3.46 3.41

0.90 0.15 0.98 0.16

3 3

3.25 3.53

1.11 0.19 1.11 0.19

3 3

3.54 3.27

0.99 0.19 1.12 0.22

0 0

% 0 Mean

(N = 36) SD SE % 0 Mean

% 0 Mean

(N = 26) SD SE

%0

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean importance ratings of less than 3.50 are shaded gray.

85

Appendix L — Graduate Faculty: Mean Relationship Ratings (Overall and by Subject) Overall

Task statements

Electrical engineering

Business/management

SE

(N = 31) Mean SD

SE

(N = 38) Mean SD

0.65

0.11

2.24

0.58

0.11

2.34

1.97

0.77

0.13

1.97

0.67

0.12

0.11 0.11 0.12

2.49 2.51 2.11

0.56 0.66 0.83

0.10 0.11 0.14

2.24 2.30 2.41

0.51 0.60 0.78

0.48

0.09

2.50

0.65

0.11

2.37

2.28

0.52

0.09

2.67

0.53

0.09

0.05 0.06

2.23 2.31

0.57 0.71

0.10 0.13

2.44 1.97

0.61 0.74

0.60 0.62

0.04 0.05

2.25 2.42

0.51 0.56

0.09 0.10

2.57 2.57

2.32

0.70

0.05

2.30

0.64

0.11

2.17

0.65

0.05

2.28

0.52

2.35

0.68

0.05

2.29

2.33

0.68

0.05

2.49 2.35

0.64 0.66

2.29 2.08

(N = 200) Mean SD

2.31

Chemistry SE

(N = 36) Mean SD

0.59

0.04

2.15

2.06

0.76

0.05

2.39 2.36 2.26

0.58 0.63 0.77

2.45

Computer science SE

(N = 36) Mean SD

0.62

0.11

2.47

2.09

0.68

0.12

0.04 0.05 0.06

2.24 2.30 2.21

0.61 0.64 0.69

0.61

0.04

2.34

2.45

0.57

0.04

2.21 2.16

0.65 0.72

2.39 2.45

Psychology

SE

(N = 33) Mean SD

0.59

0.10

2.27

2.11

0.80

0.13

0.09 0.11 0.14

2.46 2.20 2.29

0.66 0.68 0.76

0.67

0.12

2.49

2.28

0.59

0.11

0.10 0.13

2.00 2.11

0.65 0.70

0.61 0.65

0.10 0.11

2.41 2.41

2.39

0.77

0.13

0.09

2.31

0.75

0.58

0.10

2.47

2.32

0.68

0.12

0.05 0.05

2.47 2.26

0.56 0.71

0.67 0.74

0.05 0.05

2.26 2.12

2.37

0.70

0.05

2.27

0.69

0.05

History SE

(N = 26) Mean SD

SE

0.58

0.11

2.36

0.49

0.10

2.00

0.86

0.15

2.24

0.78

0.16

0.11 0.11 0.13

2.44 2.50 2.34

0.56 0.62 0.79

0.10 0.11 0.14

2.44 2.36 2.20

0.51 0.57 0.82

0.10 0.11 0.16

0.61

0.10

2.47

0.62

0.11

2.52

0.65

0.13

2.57

0.56

0.09

2.35

0.55

0.10

2.52

0.59

0.12

0.12 0.13

2.21 2.34

0.64 0.70

0.11 0.12

2.20 2.00

0.61 0.68

0.11 0.13

2.11 2.25

0.81 0.77

0.19 0.19

0.57 0.63

0.11 0.12

2.51 2.54

0.51 0.61

0.09 0.10

2.25 2.34

0.62 0.60

0.11 0.11

2.30 2.38

0.80 0.71

0.18 0.15

2.30

0.75

0.14

2.40

0.69

0.12

2.19

0.70

0.13

2.32

0.69

0.14

0.12

2.10

0.71

0.13

2.14

0.65

0.11

2.06

0.62

0.11

2.08

0.64

0.13

0.65

0.11

2.30

0.65

0.12

2.37

0.73

0.12

2.34

0.70

0.12

2.25

0.79

0.16

2.46

0.70

0.12

2.27

0.58

0.11

2.34

0.76

0.13

2.19

0.69

0.12

2.39

0.66

0.14

0.10 0.12

2.64 2.60

0.64 0.69

0.11 0.12

2.43 2.37

0.68 0.56

0.12 0.10

2.49 2.37

0.66 0.60

0.11 0.10

2.50 2.45

0.67 0.51

0.12 0.09

2.40 1.96

0.65 0.79

0.13 0.16

0.67 0.73

0.11 0.12

2.39 2.17

0.64 0.82

0.11 0.14

2.37 2.07

0.72 0.69

0.13 0.13

2.37 2.00

0.65 0.73

0.11 0.12

2.25 1.97

0.67 0.71

0.12 0.13

2.00 2.17

0.67 0.76

0.14 0.16

2.24

0.70

0.12

2.57

0.65

0.11

2.33

0.76

0.14

2.34

0.64

0.11

2.59

0.61

0.11

2.04

0.79

0.16

2.12

0.64

0.11

2.51

0.74

0.13

2.17

0.71

0.13

2.29

0.62

0.11

2.43

0.68

0.12

2.04

0.69

0.14

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

86

Overall

Task statements, continued

Electrical engineering

Business/management

SE

(N = 31) Mean SD

SE

(N = 38) Mean SD

0.77

0.13

2.20

0.71

0.13

2.29

2.39 2.30

0.80 0.81

0.13 0.14

2.37 2.14

0.67 0.71

0.12 0.13

0.10 0.12 0.10 0.12 0.12 0.11 0.11

2.31 2.21 2.15 2.49 2.54 2.59 2.54

0.72 0.77 0.78 0.66 0.61 0.61 0.56

0.12 0.13 0.13 0.11 0.10 0.10 0.09

2.24 2.34 2.21 2.28 2.28 2.47 2.31

0.69 0.61 0.68 0.70 0.59 0.63 0.66

0.57 0.62

0.10 0.11

2.56 2.17

0.65 0.81

0.11 0.14

2.23 2.17

2.18 2.18

0.64 0.53

0.11 0.09

2.44 2.50

0.69 0.61

0.12 0.10

0.04 0.05

2.33 2.45

0.66 0.62

0.12 0.11

2.69 2.47

0.52 0.70

0.64

0.05

2.28

0.58

0.10

2.58

2.17

0.68

0.05

2.13

0.66

0.12

2.05

0.72

0.05

2.06

0.66

1.95 1.88

0.72 0.68

0.05 0.05

2.09 1.94

0.63 0.61

(N = 200) Mean SD

2.27

Chemistry SE

(N = 36) Mean SD

0.70

0.05

2.03

2.28 2.21

0.68 0.67

0.05 0.05

2.20 2.13 2.07 2.32 2.36 2.44 2.40

0.66 0.69 0.67 0.68 0.63 0.65 0.61

2.40 2.15

Computer science SE

(N = 36) Mean SD

0.72

0.12

2.39

2.06 2.06

0.70 0.61

0.12 0.11

0.05 0.05 0.05 0.05 0.05 0.05 0.04

2.18 2.09 1.91 2.30 2.34 2.39 2.30

0.58 0.68 0.59 0.68 0.65 0.66 0.64

0.66 0.74

0.05 0.05

2.27 2.15

2.39 2.32

0.67 0.63

0.05 0.05

2.46 2.42

0.61 0.65

2.43

Psychology

SE

(N = 33) Mean SD

0.62

0.11

2.45

2.34 2.29

0.48 0.58

0.08 0.10

0.13 0.11 0.13 0.13 0.11 0.11 0.12

2.23 2.06 2.06 2.31 2.40 2.43 2.43

0.73 0.68 0.64 0.63 0.65 0.65 0.61

0.73 0.79

0.13 0.14

2.34 2.06

2.37 2.20

0.67 0.66

0.12 0.12

0.09 0.12

2.30 2.14

0.65 0.69

0.65

0.11

2.33

2.26

0.70

0.12

0.11

2.00

0.79

0.11 0.11

2.00 2.00

0.84 0.72

History SE

(N = 26) Mean SD

SE

0.62

0.11

2.25

0.68

0.14

2.32 2.32

0.65 0.55

0.12 0.10

2.17 2.12

0.70 0.73

0.14 0.15

0.12 0.12 0.11 0.11 0.11 0.11 0.10

2.10 1.97 2.03 2.23 2.23 2.35 2.39

0.65 0.66 0.66 0.67 0.62 0.66 0.62

0.12 0.12 0.12 0.12 0.11 0.12 0.11

2.12 2.16 2.08 2.28 2.32 2.42 2.44

0.60 0.69 0.70 0.79 0.69 0.72 0.58

0.12 0.14 0.14 0.16 0.14 0.15 0.12

0.68 0.78

0.12 0.13

2.53 2.09

0.67 0.82

0.12 0.14

2.48 2.28

0.65 0.61

0.13 0.12

2.37 2.29

0.73 0.67

0.12 0.11

2.48 2.38

0.63 0.61

0.11 0.11

2.56 2.40

0.65 0.71

0.13 0.14

0.12 0.13

2.49 2.54

0.61 0.61

0.10 0.10

2.50 2.47

0.57 0.57

0.10 0.10

2.36 2.40

0.64 0.65

0.13 0.13

0.71

0.13

2.54

0.51

0.09

2.44

0.67

0.12

2.36

0.76

0.15

2.17

0.70

0.13

2.31

0.58

0.10

2.06

0.72

0.13

2.08

0.76

0.15

0.13

2.10

0.71

0.13

2.00

0.73

0.12

2.09

0.64

0.11

2.04

0.84

0.17

0.14 0.12

1.93 1.79

0.75 0.62

0.14 0.12

1.91 1.76

0.70 0.71

0.12 0.12

1.88 2.00

0.66 0.68

0.12 0.12

1.84 1.76

0.75 0.72

0.15 0.14

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean relationship ratings of less than 2.00 are shaded gray.

87

Appendix M — Rank Ordering of Faculty and Student Ratings Faculty

Students

Undergraduate

Task Statements

Importance

Graduate

Relationship

Importance

Relationship

Undergraduate

Graduate

Importance

Importance

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

3.74 3.71

17 20

2.29 2.02

13 37

3.72 3.99

34 21

2.31 2.06

23 39

3.93 3.66

15 29

3.91 3.77

17 26

3.57 4.06 4.39

25 12 3

2.26 2.35 2.24

15 7 18

3.71 4.20 4.29

35 11 8

2.39 2.36 2.26

13 15 28

3.69 4.30 4.24

27 2 3

3.73 4.24 3.89

30 3 18

4.23

6

2.41

2

4.25

10

2.45

5

3.88

17

3.72

32

4.43 3.74 3.66

2 18 23

2.38 2.23 2.06

4 21 33

4.40 3.79 3.88

4 31 28

2.45 2.21 2.16

3 29 34

4.03 3.87 3.56

10 18 38

4.05 4.05 3.53

9 10 36

3.51 3.90

31 13

2.21 2.34

23 9

4.01 4.19

20 12

2.39 2.45

12 4

3.56 3.66

39 30

3.75 3.78

28 25

3.84 3.33 3.51

14 38 30

2.24 2.04 2.12

19 35 30

4.13 3.62 4.07

16 37 18

2.32 2.17 2.35

21 33 18

3.79 3.45 3.65

25 41 33

3.92 3.52 3.82

16 37 22

3.50

32

2.14

27

3.86

29

2.33

19

3.86

20

3.83

20

4.19 3.56

9 26

2.40 2.18

3 26

4.46 3.85

1 30

2.49 2.35

1 17

4.18 3.66

7 31

4.32 3.76

2 27

4.09 3.41

11 36

2.25 2.02

17 36

4.34 3.54

5 38

2.29 2.08

24 37

4.17 3.61

8 36

3.96 3.23

13 42

3.70

21

2.35

8

4.06

19

2.37

14

4.15

9

3.83

19

3.62

24

2.26

16

3.74

33

2.27

26

3.69

28

3.56

35

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

88

Faculty

Students

Undergraduate

Task Statements, continued

Importance

Graduate

Relationship

Importance

Relationship

Undergraduate

Graduate

Importance

Importance

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

3.81

15

2.19

25

4.14

15

2.27

27

4.00

13

4.18

4

3.51 3.38

29 37

2.13 2.12

28 29

3.95 3.96

25 23

2.28 2.21

25 30

3.83 3.93

21 16

4.03 4.12

11 6

3.44 3.12 3.03 3.42 3.54 3.53 3.74

33 39 42 34 27 28 19

2.04 1.94 1.96 2.22 2.19 2.30 2.27

34 40 39 22 24 11 14

3.79 3.51 3.44 3.89 3.96 4.12 4.15

32 39 40 27 22 17 13

2.20 2.13 2.07 2.32 2.36 2.44 2.40

31 36 38 22 16 6 9

3.63 3.62 3.48 3.87 3.71 3.59 4.01

34 35 40 19 26 37 11

3.73 3.50 3.43 3.92 3.79 3.71 3.99

31 38 39 14 24 33 12

4.45 4.24

1 5

2.41 2.12

1 31

4.41 4.15

3 14

2.40 2.15

10 35

4.20 4.37

5 1

4.13 4.08

5 7

4.23 3.68

7 22

2.29 2.23

12 20

4.32 3.96

6 24

2.39 2.32

11 20

4.19 3.81

6 24

4.05 3.83

8 21

4.34 4.20

4 8

2.34 2.38

10 5

4.45 4.29

2 7

2.46 2.42

2 8

4.24 4.01

4 12

4.39 3.92

1 15

4.10 3.75

10 16

2.36 2.10

6 32

4.28 3.91

9 26

2.43 2.17

7 32

3.83 3.97

22 14

3.81 3.73

23 29

3.41 3.11 3.05

35 40 41

1.98 1.89 1.85

38 41 42

3.70 3.27 3.16

36 41 42

2.05 1.95 1.88

40 41 42

3.82 3.44 3.66

23 42 32

3.63 3.23 3.38

34 41 40

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean importance ratings of less than 3.50 are shaded gray; mean relationship ratings of less than 2.00 are shaded gray.

89

Appendix N — Undergraduate Students: Percent Zero and Mean Importance Ratings (Overall and by Educational Status) Overall

Task statements

First- or second-year student

(N = 103)

Third-year student

(N = 29)

Fourth-year student

(N = 35)

(N = 37)

Mean

SD

SE

%0

Mean

SD

SE

%0

Mean

SD

SE

%0

Mean

SD

SE

%0

3.93 3.66

0.96 1.08

0.09 0.11

1 2

3.90 3.68

1.11 1.19

0.21 0.22

0 3

3.91 3.56

0.93 1.21

0.16 0.20

3 3

3.97 3.74

0.87 0.88

0.14 0.14

0 0

3.69 4.30 4.24

0.98 0.83 0.88

0.10 0.08 0.09

1 1 0

3.79 4.14 4.14

0.96 0.76 1.03

0.18 0.14 0.19

3 3 0

3.71 4.37 4.34

0.79 0.73 0.87

0.13 0.12 0.15

0 0 0

3.61 4.36 4.23

1.15 0.96 0.78

0.18 0.15 0.12

0 0 0

3.88

0.99

0.10

1

3.69

1.04

0.19

0

3.74

1.08

0.18

3

4.15

0.81

0.13

0

4.03 3.87 3.56

0.91 1.02 1.08

0.09 0.10 0.11

0 1 1

3.90 3.79 3.41

0.94 1.01 1.19

0.17 0.19 0.22

0 0 4

3.94 3.77 3.57

1.06 1.00 1.12

0.18 0.17 0.19

0 0 0

4.21 4.03 3.67

0.73 1.05 0.98

0.12 0.17 0.16

0 3 0

3.56 3.66

1.04 0.93

0.10 0.09

0 0

3.41 3.72

1.05 0.70

0.20 0.13

0 0

3.74 3.69

0.98 0.93

0.17 0.16

0 0

3.50 3.59

1.08 1.09

0.17 0.17

0 0

3.79 3.45 3.65

0.95 1.07 1.03

0.09 0.11 0.10

1 2 0

3.89 3.44 3.55

1.03 1.19 1.02

0.19 0.22 0.19

3 7 0

3.60 3.20 3.57

1.01 0.93 1.12

0.17 0.16 0.19

0 0 0

3.90 3.67 3.79

0.82 1.08 0.95

0.13 0.17 0.15

0 0 0

3.86

1.08

0.11

0

3.86

1.13

0.21

0

3.74

1.09

0.18

0

3.97

1.04

0.17

0

4.18 3.66

0.93 1.05

0.09 0.10

0 0

4.17 3.62

1.00 1.08

0.19 0.20

0 0

4.11 3.63

1.02 1.11

0.17 0.19

0 0

4.26 3.72

0.79 1.00

0.13 0.16

0 0

4.17 3.61

0.75 0.99

0.07 0.10

1 2

4.10 3.83

0.82 1.10

0.15 0.21

0 0

4.06 3.53

0.78 0.93

0.13 0.16

3 3

4.31 3.53

0.66 0.95

0.10 0.15

0 3

4.15

0.91

0.09

1

4.03

0.94

0.18

0

4.24

0.96

0.16

3

4.15

0.84

0.14

0

3.69

0.95

0.09

1

3.71

1.08

0.20

0

3.74

0.99

0.17

3

3.64

0.81

0.13

0

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

90

Overall

Task statements, continued

First- or second-year student

(N = 103) Mean

SD

SE

4.00 3.83 3.93

0.99 1.02 1.03

3.63 3.62 3.48 3.87 3.71 3.59 4.01

Third-year student

(N = 29) %0

Mean

0.10 0.10 0.10

1 1 1

0.99 0.93 0.99 0.95 0.91 0.98 0.87

0.10 0.09 0.10 0.09 0.09 0.10 0.09

4.20 4.37

0.88 0.85

4.19 3.81

Fourth-year student

(N = 35)

SD

SE

%0

Mean

3.86 3.59 3.74

0.97 1.15 1.26

1 3 2 1 2 2 1

3.45 3.17 3.18 3.52 3.54 3.46 3.76

0.09 0.08

0 0

0.90 0.97

0.09 0.10

4.24 4.01

0.79 0.83

3.83 3.97

3.82 3.44 3.66

SD

0.18 0.21 0.23

3 0 0

4.06 3.97 3.88

0.99 0.93 0.98 1.02 1.07 1.29 0.99

0.18 0.17 0.18 0.19 0.20 0.24 0.18

0 0 3 0 3 3 0

3.59 3.75 3.53 3.91 3.79 3.53 4.26

1.10 0.88 1.08 0.90 0.88 0.96 0.75

3.83 4.24

1.07 0.95

0.20 0.18

0 0

4.23 4.46

0 0

4.00 3.48

0.93 0.95

0.17 0.18

0 0

0.08 0.08

0 0

4.07 3.69

0.92 0.93

0.17 0.17

0.76 0.83

0.07 0.08

0 0

3.69 3.76

0.81 0.87

0.97 0.99 1.04

0.10 0.10 0.10

0 1 0

3.62 3.21 3.48

0.98 1.05 1.12

(N = 37)

SE

%0

Mean

SD

SE

%0

0 3 3

4.05 3.90 4.11

0.19 0.15 0.18 0.15 0.15 0.16 0.13

3 9 3 3 3 3 3

3.79 3.85 3.64 4.10 3.77 3.74 3.97

0.89 0.14 0.87 0.14 0.87 0.14 0.88 0.14 0.81 0.13 0.72 0.11 0.84 0.13

0 0 0 0 0 0 0

0.69 0.66

0.12 0.11

0 0

4.46 4.38

0.79 0.13 0.94 0.15

0 0

4.43 3.86

0.81 1.00

0.14 0.17

0 0

4.13 4.00

0.92 0.15 0.92 0.15

0 0

0 0

4.31 4.09

0.63 0.70

0.11 0.12

0 0

4.31 4.18

0.80 0.13 0.82 0.13

0 0

0.15 0.16

0 0

3.80 3.97

0.83 0.79

0.14 0.13

0 0

3.95 4.13

0.65 0.10 0.83 0.13

0 0

0.18 0.19 0.21

0 0 0

3.86 3.40 3.66

0.88 0.91 0.97

0.15 0.15 0.16

0 0 0

3.92 3.66 3.79

1.04 0.17 0.99 0.16 1.06 0.17

0 3 0

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

1.11 0.94 0.99

0.19 0.16 0.17

0.89 0.14 0.97 0.15 0.88 0.14

0 0 0

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean importance ratings of less than 3.50 are shaded gray.

91

Appendix O — Graduate Students: Percent Zero and Mean Importance Ratings (Overall and by Subject)

Task statements

Overall

Chemistry

Computer science

Electrical engineering

Business/management

Psychology

History

(N = 242)

(N =32)

(N =40)

(N =40)

(N =59)

(N =25)

(N =28)

Mean SD

SE

%0

Mean

SD

SE % 0 Mean

SD

SE

% 0 Mean

SD

SE

%0

Mean

SD

SE

% 0 Mean

SD

SE

3.91

1.05

0.07

0

3.97

1.12

0.20

0

4.08

1.00

0.16

0

3.77

1.05

0.07

2

3.53

1.25

0.22

6

3.74

1.12

0.18

3.73 0.95 0.06 4.24 0.88 0.06 3.89 1.12 0.07

2 1 0

4.09 4.22 3.97

0.86 0.15 0.94 0.17 1.18 0.21

0 0 0

3.53 1.11 4.08 0.92 4.05 0.90

3.72

1.09

0.07

0

3.61

1.12

0.20

3

3.65

4.05

0.90

0.06

0

3.94

0.95

0.17

0

4.20

4.05 0.89 0.06 3.53 1.03 0.07

0 0

4.34 3.78

0.79 0.14 1.01 0.18

3.75 3.78

0.95 1.01

0.06 0.07

1 1

3.78 3.75

0.87 1.11

3.92

1.00

0.06

0

3.97

1.00

3.52 1.09 0.07 3.82 1.02 0.07

2 0

3.83

0.07

% 0 Mean

SD

SE

%0

3.60

1.08

0.17

0

3.95

1.04

0.14

0

3.64

0.91

0.18

0

4.18

1.00

0.17

0

3

3.59

0.99

0.16

3

3.90

1.09

0.14

0

3.76

0.88

0.18

0

4.03

0.97

0.17

0

0.18 0.14 0.14

0 0 0

3.72 0.89 0.14 4.18 0.90 0.14 3.75 1.26 0.20

3 0 0

3.86 0.90 0.12 4.49 0.71 0.09 3.78 1.12 0.15

3 3 0

3.76 0.93 0.19 3.88 0.97 0.19 3.64 1.11 0.22

0 0 0

3.61 4.32 4.15

0.93 0.16 0.91 0.16 1.18 0.20

3 0 0

0.92

0.15

0

3.63

1.15

0.18

0

3.90

1.09

0.14

0

3.48

1.12

0.22

0

3.91

1.19

0.20

0

0.79

0.13

0

3.97

0.99

0.16

3

4.15

0.76

0.10

0

3.64

0.99

0.20

0

4.12

1.04

0.18

0

0 0

3.95 0.96 3.43 1.08

0.15 0.17

0 0

3.80 0.97 0.15 3.18 1.03 0.16

0 0

3.90 0.80 0.10 3.64 0.92 0.12

0 0

3.92 0.95 0.19 3.16 1.18 0.24

0 0

4.35 3.91

0.77 0.13 1.00 0.17

0 0

0.15 0.20

0 0

3.71 3.82

1.01 0.98

0.16 0.16

0 0

3.49 1.07 0.17 3.56 1.10 0.17

3 3

3.95 0.83 0.11 3.84 0.98 0.13

2 2

3.35 0.83 0.17 3.70 0.93 0.19

4 4

4.09 3.94

0.93 0.16 1.01 0.17

0 0

0.18

0

4.18

0.98

0.16

0

3.90

0.17

0

3.90

0.12

0

3.28

1.10

0.22

0

4.06

0.95

0.16

0

3.50 3.91

1.19 0.21 1.12 0.20

0 0

3.98 1.07 3.80 0.85

0.17 0.13

0 0

3.29 0.90 0.14 3.54 1.10 0.17

5 3

3.51 0.95 0.12 4.02 0.84 0.11

0 0

3.04 3.40

1.16 0.23 1.26 0.25

4 0

3.55 4.09

1.20 0.21 1.00 0.17

3 0

0

3.94

0.98

0.17

0

3.62

1.16

0.18

3

3.95

0.16

0

3.81

0.12

0

3.44

1.12

0.22

0

4.06

1.00

0.17

0

4.32 0.74 0.05 3.76 0.94 0.06

0 0

4.16 3.91

0.77 0.14 0.93 0.16

0 0

4.38 0.81 3.75 0.95

0.13 0.15

0 0

4.13 0.80 0.13 3.48 1.11 0.18

3 0

4.41 0.70 0.09 3.92 0.88 0.11

0 0

4.28 0.79 0.16 3.88 0.74 0.15

0 0

4.50 3.56

0.62 0.11 0.93 0.16

0 0

18. Use relevant reasons and examples to support a position or idea 3.96 0.90 0.06 19. Produce sufficient quantity of written text appropriate to the assignment and the 3.23 1.06 0.07 time constraints

0 2

4.06 3.47

0.95 0.17 1.14 0.20

0 3

4.15 0.77 3.20 1.07

0.12 0.17

0 0

3.51 1.05 0.17 2.97 0.99 0.16

0 5

4.07 0.85 3.36 0.96

0.11 0.13

0 0

3.52 0.82 0.16 2.79 0.98 0.20

0 4

4.29 3.62

0.63 0.11 1.04 0.18

0 0

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

1.02

1.06

1.01

0.88

0.92

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

3.83

1.04

0.07

0

3.56

1.05

0.18

0

4.08

1.02

0.16

0

3.72

1.12

0.18

3

3.93

0.98

0.13

0

3.36

1.08

0.22

0

3.94

1.04

0.18

0

3.56

0.99

0.06

0

3.72

1.05

0.19

0

3.58

0.93

0.15

0

3.49

1.00

0.16

3

3.58

1.02

0.13

0

3.16

1.07

0.21

0

3.74

1.02

0.18

0

92

Task statements, continued

Overall

Chemistry

Computer science

Electrical engineering

Business/management

Psychology

(N = 242)

(N =32)

(N =40)

(N =40)

(N =59)

(N =25)

Mean

SD

SE

% 0 Mean

SD

SE

4.18

0.91

0.06

1

4.22

4.03 1.00 0.06 4.12 0.92 0.06

3.73 3.50 3.43 3.92 3.79 3.71 3.99

% 0 Mean

SD

SE

0.91

0.16

0

4.08

1 1

4.13 0.91 0.16 4.28 0.92 0.16

0.06 0.06 0.06 0.06 0.05 0.06 0.06

0 2 2 1 1 2 1

3.84 3.38 3.59 3.78 3.88 3.63 3.81

4.13 0.82 0.05 4.08 1.03 0.07

% 0 Mean

SD

SE

0.94

0.15

0

4.08

0 0

4.03 0.97 0.15 4.28 0.91 0.14

0.14 0.13 0.16 0.17 0.16 0.17 0.16

0 0 0 0 0 0 0

3.78 3.45 3.29 4.08 3.78 3.95 4.05

0 1

4.09 0.96 0.17 4.06 1.19 0.21

4.05 0.92 0.06 3.83 0.93 0.06

0 1

4.39 0.77 0.05 3.92 0.95 0.06 3.81

0.89

3.73

34. Understand the difference among communicative functions such as suggestions, 3.63 advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 3.23 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm) 3.38

% 0 Mean

SD

SE

1.01

0.16

3

4.22

0 0

3.85 1.09 0.17 4.03 0.76 0.12

0.15 0.15 0.16 0.14 0.14 0.18 0.13

0 5 5 0 0 0 0

3.48 3.56 3.28 3.61 3.53 3.50 3.69

0 0

4.15 0.77 0.12 4.00 1.11 0.18

4.06 1.08 0.19 3.88 0.98 0.17

0 0

0 0

4.50 0.80 0.14 4.03 1.06 0.19

0.06

1

3.75

1.11

0.92

0.06

0

3.56

1.03

0.07

1

3.66

1.08 1.07

0.07 0.07

1 2

% 0 Mean

0.86

0.11

2

3.84

3 5

4.19 0.89 4.11 0.87

0.12 0.11

0.16 0.17 0.15 0.14 0.14 0.18 0.19

0 3 3 5 5 5 3

3.79 3.47 3.43 4.02 3.86 3.81 3.93

0.81 0.92 0.97 0.78 0.78 0.96 0.83

0 0

3.83 0.96 0.15 4.05 0.89 0.14

0 3

4.23 0.77 0.12 3.78 0.95 0.15

0 0

3.55 0.96 0.15 3.82 0.90 0.14

0 0

4.43 0.68 0.11 4.13 0.76 0.12

0 0

0.20

0

3.93

0.83

0.13

1.16

0.21

0

3.64

0.90

1.12

0.20

0

3.61

0.95

3.47 1.16 0.21 3.50 1.05 0.19

0 0

History (N =28)

SD

SE

% 0 Mean

SD

SE

%0

1.03

0.21

0

4.50

0.66

0.11

0

2 2

3.40 1.19 0.24 3.74 1.14 0.23

0 0

4.32 0.94 0.16 4.16 1.04 0.18

0 0

0.11 0.12 0.13 0.10 0.10 0.13 0.11

2 2 5 2 2 2 2

3.68 3.20 3.28 3.64 3.64 3.50 4.04

0.21 0.22 0.23 0.22 0.18 0.17 0.18

0 0 0 0 0 4 0

3.76 3.71 3.62 4.21 3.91 3.71 4.44

0.16 0.19 0.18 0.13 0.12 0.14 0.13

0 0 0 0 0 0 0

4.19 0.78 4.05 0.95

0.10 0.12

0 0

4.00 0.76 0.15 4.00 0.93 0.19

0 4

4.56 0.56 0.10 4.26 1.11 0.19

0 0

0 5

4.08 0.86 3.88 0.79

0.11 0.10

0 0

4.04 0.84 0.17 3.56 1.04 0.21

0 0

4.35 0.88 0.15 3.88 1.09 0.19

0 0

4.20 0.97 0.15 3.77 1.25 0.20

0 3

4.29 0.72 3.85 0.83

0.09 0.11

0 0

4.16 3.48

0.90 0.96

0.18 0.19

0 0

4.71 4.09

0.46 0.83

0.08 0.14

0 0

0

3.64

0.99

0.16

3

3.80

0.80

0.10

0

3.58

0.97

0.19

4

4.06

0.69

0.12

0

0.14

0

3.62

0.96

0.15

3

3.75

0.90

0.12

0

3.80

0.91

0.18

0

4.03

0.64

0.11

0

0.15

0

3.63

1.10

0.17

5

3.58

1.05

0.14

0

3.52

1.00

0.20

0

3.76

0.99

0.17

0

3.33 1.07 0.17 3.56 1.02 0.16

0 3

3.18 3.30

0.98 1.10

0.16 0.17

5 8

3.24 1.07 3.52 1.00

0.14 0.13

0 2

2.80 3.00

1.12 1.14

0.22 0.23

0 4

3.29 3.15

1.09 1.13

0.19 0.19

0 0

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

0.91 0.98 1.00 0.90 0.83 0.99 0.93

0.81 0.75 0.91 0.97 0.91 0.94 0.93

0.92 0.98 0.98 0.92 0.86 1.11 0.85

1.04 1.10 0.97 0.92 0.86 1.13 1.22

1.07 1.08 1.14 1.11 0.91 0.83 0.89

0.92 1.12 1.04 0.74 0.71 0.84 0.75

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions

Note . Mean importance ratings of less than 3.50 are shaded gray.

93

Appendix P — Graduate Students: Percent Zero and Mean Importance Ratings (Overall and by Geographic Region of Origin)

Task statements

Overall

Europe

Latin America

(N = 242)

(N =37)

(N =23)

South Asia/East Asia (N =151)

Mean

SD

SE

%0

Mean

SD

SE

%0

Mean

SD

SE

%0

Mean

SD

SE

%0

3.91 3.77

1.05 1.05

0.07 0.07

0 2

3.89 3.46

1.22 1.24

0.20 0.20

0 5

4.30 3.73

0.76 1.12

0.16 0.23

0 4

3.81 3.79

1.02 1.02

0.08 0.08

0 1

3.73 4.24 3.89

0.95 0.88 1.12

0.06 0.06 0.07

2 1 0

3.97 4.24 4.03

0.87 0.89 1.14

0.14 0.15 0.19

0 0 0

3.78 4.09 3.96

0.95 0.73 0.93

0.20 0.15 0.19

0 0 0

3.69 4.21 3.79

0.94 0.89 1.16

0.08 0.07 0.09

3 1 0

3.72

1.09

0.07

0

3.67

1.04

0.17

3

3.61

1.08

0.22

0

3.69

1.11

0.09

0

4.05 4.05 3.53

0.90 0.89 1.03

0.06 0.06 0.07

0 0 0

3.97 4.24 3.73

0.90 0.86 0.96

0.15 0.14 0.16

0 0 0

4.13 3.91 3.22

0.87 0.95 1.17

0.18 0.20 0.24

0 0 0

4.01 3.93 3.43

0.91 0.88 1.03

0.07 0.07 0.08

1 0 0

3.75 3.78

0.95 1.01

0.06 0.07

1 1

3.76 3.75

0.83 1.05

0.14 0.17

0 0

3.64 3.78

1.05 1.00

0.22 0.21

0 0

3.68 3.70

0.97 1.01

0.08 0.08

2 2

3.92 3.52 3.82

1.00 1.09 1.02

0.06 0.07 0.07

0 2 0

3.97 3.57 3.95

1.09 1.14 1.05

0.18 0.19 0.17

0 0 0

4.17 4.17 3.78

0.89 0.98 0.74

0.18 0.21 0.15

0 0 0

3.81 3.34 3.74

1.00 1.08 1.05

0.08 0.09 0.09

0 3 1

3.83

1.02

0.07

0

3.92

1.01

0.17

0

3.50

1.22

0.26

4

3.77

1.01

0.08

0

4.32 3.76

0.74 0.94

0.05 0.06

0 0

4.24 3.86

0.76 1.00

0.12 0.17

0 0

4.48 3.83

0.67 0.83

0.14 0.17

0 0

4.26 3.74

0.78 0.94

0.06 0.08

1 0

3.96 3.23

0.90 1.06

0.06 0.07

0 2

4.14 3.51

0.92 1.07

0.15 0.18

0 3

4.13 3.30

0.76 1.02

0.16 0.21

0 0

3.81 3.14

0.92 1.03

0.08 0.08

0 2

3.83

1.04

0.07

0

3.65

1.09

0.18

0

4.09

0.90

0.19

0

3.79

1.05

0.09

1

3.56

0.99

0.06

0

3.68

1.03

0.17

0

3.61

0.94

0.20

0

3.49

1.02

0.08

1

READING Locating information 1. Locate and understand information that is clearly stated in the text by skimming and scanning 2. Locate and understand information provided in non-prose documents (e.g., charts, graphs, and tables)

Basic comprehension 3. Use contextual cues to establish the meaning of a word in a passage 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas 8. Read text material and outline important ideas and concepts 9. Distinguish factual information from opinions

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 13. Show awareness of audience needs and write to a particular audience or reader 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments 15. Produce writing that effectively summarizes and paraphrases the works and words of others

Organization 16. Organize writing in order to convey major and supporting ideas 17. Use appropriate transitions to connect ideas and information

Development 18. Use relevant reasons and examples to support a position or idea 19. Produce sufficient quantity of written text appropriate to the assignment and the time constraints

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation 21. Demonstrate facility with a range of vocabulary appropriate to the topic

94

Task statements, continued

Mean

Overall

Europe

Latin America

(N = 242)

(N =37)

(N =23)

SD

SE

%0

Mean

SD

SE

4.18

0.91

0.06

1

4.27

0.87

4.03 4.12

1.00 0.92

0.06 0.06

1 1

4.19 4.38

3.73 3.50 3.43 3.92 3.79 3.71 3.99

0.91 0.98 1.00 0.90 0.83 0.99 0.93

0.06 0.06 0.06 0.06 0.05 0.06 0.06

0 2 2 1 1 2 1

4.13 4.08

0.82 1.03

0.05 0.07

4.05 3.83

0.92 0.93

4.39 3.92

South Asia/East Asia (N =151)

%0

Mean

SD

SE

%0

Mean

SD

SE

%0

0.14

0

4.00

0.85

0.18

0

4.13

0.96

0.08

1

0.88 0.89

0.14 0.15

0 0

4.04 4.20

0.82 0.95

0.17 0.20

0 0

3.93 3.99

1.08 0.94

0.09 0.08

1 2

3.89 3.33 3.46 3.81 3.86 3.68 3.84

0.81 0.83 0.99 0.91 0.86 0.91 0.93

0.13 0.14 0.16 0.15 0.14 0.15 0.15

0 3 0 0 0 0 0

3.57 3.30 3.23 4.26 3.65 3.91 4.04

0.84 0.93 0.87 0.75 0.78 1.20 0.82

0.18 0.19 0.18 0.16 0.16 0.25 0.17

0 0 4 0 0 0 0

3.67 3.45 3.39 3.86 3.74 3.64 3.93

0.94 1.01 1.01 0.92 0.84 1.01 0.95

0.08 0.08 0.08 0.07 0.07 0.08 0.08

1 2 3 2 2 3 1

0 1

4.08 4.00

0.92 1.22

0.15 0.20

0 0

4.17 3.96

0.83 1.22

0.17 0.26

0 0

4.07 4.05

0.82 0.97

0.07 0.08

0 1

0.06 0.06

0 1

4.14 3.81

1.03 0.94

0.17 0.15

0 0

4.13 3.48

0.81 0.95

0.17 0.20

0 0

3.94 3.87

0.93 0.90

0.08 0.07

0 1

0.77 0.95

0.05 0.06

0 0

4.49 4.03

0.80 1.01

0.13 0.17

0 0

4.48 4.22

0.67 0.67

0.14 0.14

0 0

4.29 3.81

0.80 0.97

0.06 0.08

0 1

3.81 3.73

0.89 0.92

0.06 0.06

1 0

3.81 3.57

1.05 1.14

0.17 0.19

0 0

3.78 3.43

0.85 0.79

0.18 0.16

0 0

3.75 3.74

0.88 0.91

0.07 0.07

1 1

3.63 3.23 3.38

1.03 1.08 1.07

0.07 0.07 0.07

1 1 2

3.70 3.51 3.57

1.10 1.12 1.04

0.18 0.18 0.17

0 0 0

3.43 3.13 3.68

0.93 1.10 0.89

0.19 0.23 0.19

0 0 4

3.60 3.16 3.33

1.05 1.05 1.08

0.09 0.09 0.09

1 1 3

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 23. Speak clearly and accurately enough to participate in class discussions 24. Speak clearly and accurately enough to make presentations in class 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: a. summarizing information b. giving directions or instructions c. describing objects d. giving and supporting opinions e. making comparisons/contrasts f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts and details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter 29. Use background knowledge and context to understand unfamiliar terminology

Main ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation 33. Understand the parts of lectures, discussions, or conversations, such as the introduction, review of previous information, presentation of new material, summary, and conclusion

Communicative functions 34. Understand the difference among communicative functions such as suggestions, advice, directives, and warnings 35. Recognize the use of examples, anecdotes, jokes, and digressions 36. Recognize the speaker’s attitudinal signals (e.g., tone of voice, humor, sarcasm)

Note . Mean importance ratings of less than 3.50 are shaded gray.

95

Appendix Q — Rank Ordering of Task Statements Selected Using Example Criteria Faculty

Students

Undergraduate

Task Statements

Importance

Graduate

Relationship

Importance

Relationship

Undergraduate

Graduate

Importance

Importance

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

Mean

Rank

4.06 4.39

12 3

2.35 2.24

7 18

4.20 4.29

11 8

2.36 2.26

15 28

4.30 4.24

2 3

4.24 3.89

3 18

4.23

6

2.41

2

4.25

10

2.45

5

3.88

17

3.72

32

4.43

2

2.38

4

4.40

4

2.45

3

4.03

10

4.05

9

3.51 3.90

31 13

2.21 2.34

23 9

4.01 4.19

20 12

2.39 2.45

12 4

3.56 3.66

39 30

3.75 3.78

28 25

3.84 3.51

14 30

2.24 2.12

19 30

4.13 4.07

16 18

2.32 2.35

21 18

3.79 3.65

25 33

3.92 3.82

16 22

4.19

9

2.40

3

4.46

1

2.49

1

4.18

7

4.32

2

4.09

11

2.25

17

4.34

5

2.29

24

4.17

8

3.96

13

3.70

21

2.35

8

4.06

19

2.37

14

4.15

9

3.83

19

3.81

15

2.19

25

4.14

15

2.27

27

4.00

13

4.18

4

3.53 3.74

28 19

2.30 2.27

11 14

4.12 4.15

17 13

2.44 2.40

6 9

3.59 4.01

37 11

3.71 3.99

33 12

4.45 4.24

1 5

2.41 2.12

1 31

4.41 4.15

3 14

2.40 2.15

10 35

4.20 4.37

5 1

4.13 4.08

5 7

4.23

7

2.29

12

4.32

6

2.39

11

4.19

6

4.05

8

4.34 4.20

4 8

2.34 2.38

10 5

4.45 4.29

2 7

2.46 2.42

2 8

4.24 4.01

4 12

4.39 3.92

1 15

4.10

10

2.36

6

4.28

9

2.43

7

3.83

22

3.81

23

READING Basic Comprehension 4. Determine the basic theme (main idea) of a passage 5. Read and understand written instructions/directions concerning classroom assignments and/or examinations

Learning 6. Read text material with sufficient care and comprehension to remember major ideas and answer written questions later when the text is no longer present 7. Read text material with sufficient care and comprehension to remember major ideas

Integration 10. Compare and contrast ideas in a single text and/or across texts 11. Synthesize ideas in a single text and/or across texts

WRITING Content 12. Write in response to an assignment and stay on topic without digressions or redundancies 14. Use background knowledge, reference or non-text materials, personal view points, and other sources appropriately to support ideas, analyze, and refine arguments

Organization 16. Organize writing in order to convey major and supporting ideas

Development 18. Use relevant reasons and examples to support a position or idea

Language 20. Demonstrate a command of standard written English, including grammar, phrasing, effective sentence structure, spelling, and punctuation

SPEAKING 22. Speak clearly and accurately enough so that the instructor can understand and respond to their questions, comments, and suggestions 25. Demonstrate facility with standard spoken English including grammar, word choice, fluency, and sentence structure while performing the following linguistic tasks: f. developing or structuring hypotheses g. explaining or informing

LISTENING Facts And Details 26. Understand factual information and details 27. Understand the instructor’s spoken directions regarding assignments and their due dates

Vocabulary 28. Understand important terminology related to the subject matter

Main Ideas 30. Understand the main ideas and their supporting information 31. Distinguish between important information and minor details

Inferences 32. Make appropriate inferences based on information in a lecture, discussion, or conversation

96

®

Test of English as a Foreign Language P.O. Box 6155 Princeton, NJ 08541-6155 USA 㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭㛭 To obtain more information about TOEFL programs and services, use one of the following:

Phone: 609-771-7100 Email: [email protected] Web site: http://www.toefl.org

58713-003227 • Y101M.750 • Printed in U.S.A.

I.N. 990629

Loading...

The Reading, Writing, Speaking, and Listening Tasks Important for - ETS

Monograph Series MS - 21 NOVEMBER 2001 The Reading, Writing, Speaking, and Listening Tasks Important for Academic Success at the Undergraduate and G...

3MB Sizes 3 Downloads 12 Views

Recommend Documents

TOEIC Listening and Reading Examinee Handbook - ETS
from 5 to 495 for the listening and reading tests, respectively . As each test form will be reused multiple times in dif

TOEIC Listening and Reading Sample Questions - ETS
7. Mark your answer on your answer sheet. 8. Mark your answer on your answer sheet. 9. Mark your answer on your answer s

Writing, Speaking, Listening, Interviewing, Communication
Guidelines for Public Speaking (local copy), by Palkovitz, in Law Enforcement Bulletin, Feb 2013 ..... "Story Model of D

Strategies in Responding to the New TOEFL Reading Tasks - ETS
It also did talk about human activity and global warming. It also talked about soil erosion and how they can absorb wate

TOEIC Listening and Reading Sample Test (PDF) - ETS
Directions: For each question in this part, you will hear four statements about a picture in your test book. ... (C) To

Teaching Listening and Speaking - Finchpark
The bank's downtown branch. Where When is closed today. Is the city office open on Sunday? Where When. I'm going to the

Speaking And Listening | Teaching Ideas
Liven up your speaking and listening activities with a great range of ideas, resources and display materials.

Language Comprehension (Listening and Reading)
surface structure of two sentences may differ, their deep structure can imply the same message, e.g.,. She throws the ba

Free English Study, speaking, listening, grammar lessons, reading
ESLgold.com Grammar Resources Learning English Grammar is fun and useful. This is a full list of grammar lessons includi

Free English Study, speaking, listening, grammar lessons, reading
ESLgold.com Grammar Resources Learning English Grammar is fun and useful. This is a full list of grammar lessons includi