Crowd-Sourced Online Nexus for Developing Assessments of Middle-school Physical Science Disciplinary Core Ideas

用于开发中学物理科学学科核心思想评估的众包在线关系

• 批准号: 2101493
• 负责人: Philip Sadler
• 金额: $ 160.06万
• 依托单位: Harvard University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2101493&HistoricalAwards=false
• 关键词: Crowd; Sourced; Online; Nexus; Developing

This project will develop and test a web-based platform to increase the quality of teacher-administered tests in science classrooms. It draws on classroom teacher knowledge while employing the rigorous statistical methods used in standardized assessment creation and validation. The content focus is on the disciplinary core ideas for grades 6-8 physical science in the Next Generation Science Standards (NGSS). Teachers now spend an estimate 20% of their time in assessment, yet have relatively few tools to draw upon when creating them. Over time, they learn to adapt items from available curriculum materials and textbooks. On the other hand, standardized assessment developers have the benefit of expert item writers, long development cycles, a large and diverse student population, and sophisticated psychometric tools. This project combines these two approaches, drawing upon teachers to contribute their best items, then immediately piloting them using crowdsourced subjects. Psychometric analysis generates measures of item quality and then “recycles” items to participating teachers for improvement. In this way, a large test item bank will be constructed utilizing teacher input with each item possessing: appropriate reading levels, NGSS alignment, scientific accuracy, appropriate difficulty, high statistical discrimination, and negligible difference by gender, race, or ethnicity. Involvement in this project has potential benefits for teachers lacking formal training in assessment, familiarizing participants with the NGSS, and with the elements of high-quality test development.The project will gauge the merits of a novel collaborative system for the development and validation of high-quality test items and assessment instruments. It will measure the degree to which teachers can generate effective items and improve existing items exhibiting problematic issues when given the guidance of rigorous psychometric measures that estimate item quality. It will build on earlier research showing that an adult, crowd-sourced sample works well as an initial proxy for grade 6-8 science students, allowing for extremely rapid feedback on item quality (often overnight), with item response theory computation used to establish item difficulty, item discrimination, guessing levels, and differential item functioning (gender and racial/ethnicity bias). In addition, computed measures of misconception strength, scientific correctness, reading level, and match to the NGSS will help to guide revision by teachers. Use of Bayesian futility analysis will “triage” items, minimizing costly testing of items when deemed unlikely to meet item quality criteria, lowering costs. Field testing with a large sample of grade 6-8 students will provide a final check on item quality. Items will be developed much more inexpensively than by methods used for standardized test development. Two pairs (public-release and secure for chemistry and physics) of assessment instruments will be constructed and be freely available to science teachers for classroom use and by education researchers and curriculum developers. A system that provides quick feedback on item quality could potentially transform university instruction and professional development opportunities in assessment. While starting with selected response (multiple-choice) items, the project will be able to implement a larger variety of formats in the future, incorporating automated approaches as they become available.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将开发和测试一个基于网络的平台，以提高科学课堂中教师管理的测试的质量。它利用课堂教师的知识，同时采用标准化评估创建和验证中使用的严格统计方法。下一代科学标准 (NGSS) 中 6-8 年级物理科学的学科核心思想 教师现在大约花费 20% 的时间进行评估，但随着时间的推移，他们可以使用的工具相对较少。调整现有课程中的项目另一方面，标准化评估开发人员拥有专家项目编写者、较长的开发周期、庞大且多样化的学生群体以及复杂的心理测量工具，该项目结合了这两种方法，充分利用了教师的最佳贡献。项目，然后立即使用众包科目进行试点，生成项目质量测量结果，然后将项目“回收”给参与教师进行改进。这样，将利用教师的输入构建一个大型测试项目库，其中每个项目都具有：阅读水平、NGSS 对齐、科学准确性、适当的难度、高度的统计歧视以及性别、种族或民族的差异可以忽略不计。参与该项目对于缺乏正式评估培训的教师来说具有潜在的好处，可以让参与者熟悉 NGSS 以及高质量的要素。该项目将评估用于开发和验证高质量测试项目和评估工具的新型协作系统的优点。它将衡量教师生成有效项目并改进存在问题的现有项目的程度。严格的心理测量措施的指导它将建立在早期研究的基础上，该研究表明成人众包样本可以很好地作为 6-8 年级理科学生的初始代理，利用项目响应理论可以对项目质量进行极快速的反馈（通常是隔夜）。用于确定项目难度、项目区分、猜测水平和差异项目功能（性别和种族/民族偏见）的计算此外，计算出的误解强度、科学正确性、阅读水平以及与 NGSS 的匹配程度将有助于指导。教师修改。贝叶斯无效性分析将“分类”项目，在被认为不太可能满足项目质量标准时最大限度地减少昂贵的测试，降低对 6-8 年级学生的大量样本进行的现场测试，从而对项目质量进行最终检查。与用于标准化测试开发的方法相比，开发成本要低得多，将构建两对（公开发布且安全的化学和物理）评估工具，并免费提供给科学教师用于课堂使用以及教育研究人员和课程开发人员。 .提供物品快速反馈的系统质量可能会改变评估中的大学教学和专业发展机会。在从选定的回答（多项选择）项目开始的同时，该项目将能够在未来实施更多种类的格式，并在可用时纳入自动化方法。发现研究 preK-12 计划 (DRK-12) 旨在通过创新资源、模型和工具的研究和开发，显着增强 preK-12 学生和教师对科学、技术、工程和数学 (STEM) 的学习和教学。项目在DRK-12 计划建立在 STEM 教育的基础研究和先前的研究和开发工作的基础上，为拟议项目提供理论和实证依据。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响进行评估，被认为值得支持审查标准。