How teachers assess KB outcomes

Knowledge building through online discussion requires much effort to both follow and evaluate, in preparation for the next step. However, teachers are very busy and often do not have time to read every note contributed to the KB forum. How might they fulfil the requirements of KB?

There are several “layers” to a typical Hong Kong teacher’s concerns:

• Promoting good KB practices:
  • Are students participating in the online discourse?
  • Are there students who made zero participation?

• Insights on pedagogical design:
  • Do ideas progress in the online discourse?
  • Can students raise idea progressing questions?
  • What’s the theme of the discussion threads?
  • Are there notes with potential for further development?
  • Which aspect of the discussion can be further improved?

• Further goals:
  • Is there any advancement in my students over 1 or 2 years of KB activities, in terms of question framing, evaluating and summarising ideas, as reflected from their contributions to the online discourse?

Below is a table that provides suggestions for assessment methods and criteria to help teachers assess how far students have achieved at different stages of the knowledge building process.

Key Pedagogical Focus	Suggested Assessment Criteria	How to Assess	Illustrative Videos & Presentations
Generate inquiry questions	Are students contributing their inquiry questions?		An example of how a primary school Chinese teacher encourages his students to raise questions on the topic of "Hong Kong Kids":
Identify good questions for inquiry	What is a good question? Not asking for descriptive or factual information Not having a clear and simple answer Are broad, ill-structured and multifaceted Are embedded with various concepts and values related to the curriculum (big ideas) Are authentic and related to daily life Can arouse students’ interest Basic level questions VS High level questions Basic level questions Ask to repeat or verify Relate to basic knowledge or event content Only require knowledge of the superficial characteristics of the event or phenomenon Quick to reach conclusion and little room for discussion (e.g. whether, when, where, who, what) High level questions Require understanding and analysis Allow students to demonstrate their ability to extend and evaluate concepts Allow elevation from individual phenomenon to general phenomenon (e.g. why, how) Depth of inquiry Rating scheme: 1. Questions on definitions and simple clarification 2. Questions asking for factual, topical and general information 3. Questions identifying specific gaps and asking for open-ended responses and different viewpoints 4. Explanation-based questions - Focus on problems not topics; identify sources of inconsistencies; generate conjectures and possible explanations	Qualitative methods Group review Classroom discussion	How to use rating scheme to assess depth of inquiry questions:
F2F and online discussion	Are students participating in the discussion? Are students linking and expanding ideas? Examples of measures: # notes created # build-ons made # references created # scaffold supports used # keywords used # words per note % notes linked % notes with keywords % notes read # revisions density of social network	Quantitative methods Applet tool on KF - Contribution Applet tool on KF - Social Network Analytic Toolkit (ATK) statistics	How to interpret ATK statistics: ATK output sample - Class A ATK output sample - Class B
Identify good discussion	What is a good discussion? Criterion 1: Use of authoritative sources & thinking critically Critical evaluation and link appropriate sources to discussion problem at hand Information can be obtained from various sources, including books and webpages Reference to the source of information Criterion 2: Evidence of democratizing knowledge, idea diversity Build on each other’s notes (agreeing, disagreeing, offering opinions) Discussion includes multiple perspectives Everyone contribute their ideas to maximize the effectiveness of the discussion During the discussion, everyone should respect individual differences in culture and knowledge Criterion 3: Evidence of idea improvement Students continuously pursue knowledge and participate in in-depth discussion of the inquiry questions Clarify areas of disagreement and differences Consolidate and summarize the discussion to formulate higher level of problems and solutions After clarification of differences, further extend the direction of exploration and continue to solve problems as well as improve the discussion Criterion 4: Review & reflect regularly for idea progressions and inquiry process Identify problems/aspects that concern one’s natural curiosity to relate to reality and generate real ideas Stimulate constant self-reflection through the discussion to renew knowledge, values, and even, life	Qualitative methods Self evaluation Peer assessment Group review Class discussion	How to assess notes and discussions:
Consolidation	What is a good reflection? Review and consolidate learning Evaluate performance Plan future learning and overcoming learning difficulties based on past learning experience Empower students to take charge of their own learning and to develop into independent life long learners What is a good summary note? Analyze the inter-relatedness of the community’s ideas Reflect on the knowledge building process that the learning community have gone through, which include the progress, collaboration and interaction, challenges and solutions, change of concepts, improvement of knowledge and rise of new ideas, etc., in different phases What is a good electronic portfolio? Criterion 1: Working at the cutting edge Identify knowledge gaps, inconsistencies and ask productive questions Pose problems that extend the edge of understanding of the community Pose problems with potential for continual discussion and inquiry (i.e. interest many people) Criterion 2: Progressive problem solving Show continual efforts to grapple with problems posed by classmates Pose notes aimed at addressing the original problem and questions arising from them Show sustained inquiry: Identify the problem, solve the problem, but keep asking new questions Reinvest efforts to keep solving new problems to improve ideas Criterion 3: Collaborative effort Use various KF functions such as references and rise-above to make knowledge accessible Summarize different ideas and viewpoints and put them together as a better theory Help classmates to extend and improve their understanding Encourage classmates to write notes that follow the other principles Criterion 4: Monitoring own understanding Explain what you did not know and what you have learned Recognize discrepancies and misconceptions and new insights; trace own paths of understanding Show your new ways of looking at things (questions, ideas, issues) after examining other KF notes Criterion 5: Constructive uses of different sources of information Use information from other sources (internet, newspaper, etc) to support or explain your ideas Bring together classroom learning, information from textbook, classmates’ KF notes Provide contrasting or conflicting information to what is printed in the textbook	Qualitative methods Self evaluation Peer assessment Group review	How to assess reflective learning journal, summary note, group review and electronic portfolio:

Technological support for assessing KB

Assessment technologies have been developed to provide teachers and students with useful information about their knowledge building process. Currently, there are three main technologies used in assessing KB outcomes:

• Analytic Toolkit (ATK)
• Applet Tools on Knowledge Forum
• Collaborative Online Discourse Analyzer (COLODA)

Analytic Toolkit (ATK)

The Analytic Toolkit provides summary statistics on activity in Knowledge Forum. It gives an overview of a particular Knowledge Forum database, summary and individual statistics of use of specific features, specific activity logs, analyses of social interactions as well as analyses of note and database history.

Below are a video and a presentation illustrating how to generate ATK statistics.

For examples of how to interpret ATK statistics, please click here.

IKIT has created a reference page explaining the calculation of ATK measures. Click here for the reference page.

Applet Tools on Knowledge Forum

Applet tools also helps analyze different aspects of activity in Knowledge Forum. Unlike ATK, applet tools provides graphical representation of the statistics. Also, applet tools are integrated into KF, so interested students can generate participatory statistics on their own.

Below are a video and a presentation illustrating how to run applet tools on KF.

Collaborative Online Discourse Analyzer (COLODA)

The Centre for Information Technology in Education at the University of Hong Kong, in collaboration with the Research Center of Knowledge Engineering at Beijing Normal University, has developed a pedagogical and assessment tool to assist called the Collaborative Online Discourse Analyzer (COLODA).

COLODA can help tackle the following challenges:

• Are students learning? What have they learnt?
• Are students knowledge building? At what level?
• Building on research literature or discourse analysis to provide quick overview for teachers:
  • Levels of engagement
  • Differences in engagement among students
  • Patterns of interaction
• Help teachers to adjust their facilitation
• Help to build knowledge about KB pedagogy
• Facilitate sharing of expertise among teachers

COLODA's functions include:

• Offer summaries and overviews about the discussions
• Provide numerous displays and analyses automatically for researchers

Outputs from the analyses can be organized into different levels:

• Individual students
• Notes
• Discussion threads
• Discussion views
• Date or week

Real examples of assessing KB outcomes

In the "How do teachers design KB lessons" section, we have included several detailed pedagogies of three teachers. In which, apart from introducing the designs of Mr Wong and Ms Tse, we have listed some graphs which illustate the analyses explained int he above and the students’ learning outcomes. Click here for Mr Wong’s students’ learning outcomes. Also, Click here for Ms Tse’s students’ learning outcomes.