Construct validity of an instrument for assessment of reflective writing-based portfolios of medical students

Purpose: Assessment of reflective writing for medical students is challenging, and there is lack of an available instrument with good psychometric properties. The authors developed a new instrument for assessment of reflective writing-based portfolios and examined the construct validity of this instrument. Methods: After an extensive literature review and pilot testing of the instrument, two raters assessed the reflective writing-based portfolios from years 2 and 3 medical students (n=135) on three occasions. The instrument consists of three criteria: organization, description of an experience and reflection on the experience. We calculated the reliability of scores using generalizability theory with a fully crossed design and two facets (raters and occasions). In addition, we measured criterion validity by testing correlations with students’ scores using other assessment methods. Results: The dependability (Φ) coefficient of the portfolio scores was 0.75 using two raters on three occasions. Students’ portfolio scores represented 46.6% of the total variance across all score comparisons. The variance due to occasions was negligible, while the student– occasion interaction was small. The variance due to student–rater interaction represented 17.7%, and the remaining 27.7% of the variance was due to unexplained sources of error. The decision (D) study suggested that an acceptable dependability (Φ = 0.70 and 0.72) can be achieved by using two raters for one and two occasions, respectively. Finally, we found moderate to large effect-size correlations between students’ scores in reflective writing-based portfolios and communication skills (r = 0.47) and PBL tutorials (r = 0.50). Conclusion: We demonstrated the presence of different sources of evidence that support construct validity of the study instrument. Further studies are warranted before utilizing this instrument for summative assessment of students’ reflective writing-based portfolios in other medical schools.