Experimenting with Khan Academy in Diepsloot (Report)

In mid-April 2012, OLICO embarked on a 6-month experiment to gauge the effectiveness of freely-available on-line learning solutions with a group of South African township-school youth. Forty-eight Grade 8 learners from Diepsloot were enrolled on the program. Twenty-nine learners completed the six months with an average increase of 22% on their curriculum content compared to their initial pre-project assessments.

To download the PDF version of this report, click here.

Introduction:

In mid-April 2012 OLICO set out to determine whether there is evidence to suggest that computer-based content such as the freely-available Khan Academy could successfully supplement the mathematics education of South Africa’s township-school youth. Khan Academy is a free online repository of educational videos and exercises across a wide range of disciplines with a particularly extensive collection of mathematics content.

The OLICO project enrolled a group of 48 Grade 8 learners from surrounding Diepsloot high schools. The learners were provided with lesson notes (designed by educators from the mathematics department of St Davids Marist Inanda) that incorporated a selection of Khan Academy videos and exercises covering a range of topics on the Grade 8 curriculum. In total, the lessons covered approximately 15% of the national curriculum. In addition, learners were assigned paper-based homework sheets for each lesson and were required to complete tutorials and an assessment at the end of each module (comprising of up to 5 lessons).

For the first month of the project, learners received computer literacy training which included directions on basic email and internet usage and assistance with registering as a Khan Academy student. In addition, each learner wrote a baseline assessment designed by the teachers from St Davids. The teachers described the assessment as one upon which they would be very concerned if any of their own grade 8 learners scored below 40%. None of the Diepsloot learners managed to pass. The group average was 18% and the highest mark was only 32%.[1]

It was soon apparent that remedial work was required before the learners could continue with curriculum-specific assignments. In general, learners struggled with basic tasks such as “addition with carrying” or “subtraction-with-borrowing” and when asked to complete elementary calculations like multiplying 6 by 7, learners would draw 6 rows of 7 “sticks” and then count them. That such basic concepts related to times-tables knowledge and ‘mental maths activities’ were not previously instilled to these learners despite 8 years of schooling is highly troubling, but is precisely the problem that this experiment is attempting to address.

In formulating our approach we implemented a 10-Step-Process for learners to follow (the details of which can be found below). Local facilitators from Diepsloot were employed to ensure that learners were following this process. In the event that learners needed assistance, the facilitators were given strict instructions to refer learners back to the “Khan Academy” videos or exercise hints, as the facilitators themselves had limited mathematics skills to avoid contaminating the experiment.

It is encouraging that by the end of the 6-month experiment, learners achieved an average increase of 22% on the curriculum content compared to their initial baseline results. While this improvement only lifted the group average to 40% it still reflects significant progress and suggests that computer-based learning solutions like Khan Academy can indeed be effective in supplementing mathematics education.

Of particular interest is that every learner in the group showed a positive improvement on completing the assigned curriculum. No learner achieved less than a 10% increase with the movement of the group as a whole strikingly illustrated in the chart below.

 

 As encouraging as these improvements are, it is unlikely that technology on its own is capable of delivering a solution to SA’s educational challenges, although it seems entirely possible that the proper application of certain kinds of technology has the potential to make a marked positive contribution. At the very least, the growing prevalence of technology in educational contexts across the spectrum offers increasing opportunities for collaboration and shared learning. It is in this spirit that we offer further details of the project below:

Infrastructure Involved

• 1 x computer laboratory with 20-networked computers with individual headsets.

We used NComputing L300-series terminals with a mid-level i7 desktop computer running Windows 7 Enterprise as a server.

• Each learner works on their own computer. On the current model, 1 computer can serve 5 learners attending after school twice a week for 1 hour long sessions in the afternoons. With 20 workstations our current capacity is 100 learners.

• Internet connection is required for Khan Academy exercises and learner progress-tracking.

We used a standard 3G connection since we had good network coverage. It was expensive although Cell C now have a 100GIG prepaid option for R2500 which we are using in 2013. There is also Khan Academy Lite which is designed to work offline as a self-contained solution to low-bandwidth problems.

•  We used approximately 12 GIGs of bandwidth per month.

•  The Khan Academy videos were pre-downloaded onto the server to preserve bandwidth usage.

Learner Selection Process:

•  48 learners from Diepsloot secondary schools enrolled in response to a once-off distribution of pamphlets to surrounding secondary schools.

• The invitation was exclusively for Grade 8 learners. Interested learners were encouraged to self-select onto the programme. We discouraged schools and teachers from pre-selecting learners to avoid learners attending against their will.

• Learners were informed they would be accepted on a first-come, first-served basis although this wasn’t implemented as we accepted all 50 applications we received (2 learners applied and were accepted but did not begin the program).

• There was no prior academic merit requirement and no English comprehension assessment.

 Commitment Required of Learners:

•  Learners were expected to attend at least two sessions per week of at least 1 hour per session.

• The centre was open in the afternoons after school between 2:30pm and 4:45pm.

• Learners booked their places in advance to ensure availability of computers.

• We originally intended to run the pilot for three months and assess the situation thereafter, but extended this by a further three months as a result of the greater than anticipated need for remedial work.

• Learners were required to attend computer literacy training with an emphasis on internet, email, browsing and the skills required to effectively navigate through the Khan Academy website.

• At the end of the experiment, 60% of the learners completed all the modules assigned. The remaining 40% either dropped out or were excluded due to inadequate commitment levels. There was no learner retention strategy applied and those who did not meet the minimum requirements were simply excluded.

Pilot Facilitators:

• The pilot facilitators were relatively low-skilled Diepsloot residents with instructions to follow the 10-Step-Process and direct learners back to the Khan Academy videos or exercise hints in the event that they needed help.

10-Step-Process

1. Learner arrives for pre-booked slot

2. Learner Checks-in with facilitator:

• Marks the Attendance Register
• Hands in Homework

3. Learner logs onto computer with unique username

4. Completes times-tables exercise (5mins)

5. Opens lesson on desktop and proceeds with video/Khan Academy exercises

6. Learner aims to correctly answer approximately 8/10 khan academy exercises in-a-row

7. Facilitator monitors progress and advises next step based on homework submitted & Khan Academy progress

•  Learner either revises lesson homework or begins new lesson

 8. Time up – Learner ends session & logs out

9. Learner Checks Out

• Collects marked homework sheet
• Collects new homework sheet
• Learner books slot for next session and collects sweet

10. Everyone travels home safely

Key Lessons to date:

• Very little extrinsic motivation was offered to keep learners motivated and attending.  None the less, 60% of the learners who enrolled on the project met the minimum attendance requirements with no retention strategy applied.

• Perhaps the most important practical lesson we learnt is that learners need constant supervision to ensure they are on the correct exercise and watching the correct video. Khan Academy is a very open system, which makes it very confusing for our learners to navigate. Left to their own devices, with only instructions about which videos to watch and which exercises to complete, the learners were soon lost within the Khan system and attempting problems that were well beyond their capabilities. They did not have the prerequisite skills to understand where they were within the Khan system itself and how this related to what they needed to learn.

• Learners also required considerable prompting to move on once they understood a concept. Learners had a tendency to remain on topics (or revert back to topics) on which they had achieved success – often completing far more exercises than necessary (sometimes hundreds of exercises and on occasion into the thousands). This may be partly explained by the built-in affirmation the Khan Academy site provides and the concomitant boost to a learner’s self-esteem when repeatedly getting certain concepts right. Their dedication to the task was admirable, but not correctly channeled when considering the amount of ground that has to be covered.

• Learners often don’t have the basic background knowledge to understand the concept that they are currently studying.  Here we need to look at adequate testing to identify prerequisite knowledge before a particular new concept is attempted.

• Since learners are only able to attend two sessions a week at the centre itself, the development of homework material to be used between the computer-based sessions is vital to the success of the program.

• To improve the impact of the Khan Academy videos, learners benefitted from pre-prepared worksheets that corresponded directly to the examples in the Khan videos they were watching. Learners were asked to attempt the exercises prior to watching how they were completed in the video.

• The Khan Academy videos are generally useful, although there are a number of limitations, most notably the following:

• Some videos confused South African learners with references to specifically American themes like dollars, quarters and dimes.

• Certain mathematical mechanisms are different too, for example Khan Academy uses the mnemonic, ‘PEMDAS’ whereas most South African schools use ‘BODMAS’.

• Some of the terminology in the videos is confusing for non-first-language English speakers, e.g. the use of word “parentheses” instead of “brackets”.

• Some of the videos are focused on procedural skills and not on conceptual knowledge. This appears to be a common critique of the Khan Academy videos although possibly overstated.

• A number of the videos are out of sync with the corresponding exercises. For example, certain videos explain a difficult mathematics skill, but the linked exercise is very simple, and vice versa.

• Provision will need to be made for sections of the South African curriculum that are not featured on the Khan Academy platform.

• By registering as a coach on the Khan Academy website, the facilitators are able to monitor the progress of learners in real-time and examine progress on a sum-by-sum basis. The coaching dashboard is very useful but not particularly user-friendly when trying to monitor a number of learners simultaneously. For example, there is no way to filter learners according to those currently logged-on. We’ve since discovered Always Prepped offers a more intuitive layout for higher-level monitoring of individual learner progress.

• For future development, a better pre-assessment is necessary to determine a learner’s prior mathematical ability and to establish an appropriate entry point to the Khan Academy. As stated previously, learners often needed to return to very basic concepts to establish a good grounding.

Translations / Other examples of Khan Academy Projects in South Africa:

The University of the Cape Town, Click Maths and Numeric have translated a number of videos into isiXhosa. Find the Xhosa content here. We used only the English videos in our experiment but it will be interesting to see what impact this has in the future particularly with rural learners.

Numeric is currently using the Khan Academy with a variety of groups and schools in the Western and Eastern Cape. The Numeric website is full of useful tools and tips for anyone wishing to experiment in their own South African context.

The process involved in the OLICO experiment above is slightly different to that of the Numeric approach in that Numeric focuses primarily on Khan Academy as a group activity. This has the potential advantage of fusing a strong social bond amongst learners and is also likely to further promote peer-to-peer learning, a much under-rated educational tool as we discovered during our time with IkamvaYouth. The approach described in the OLICO experiment however is designed specifically for learners to proceed at their own pace. Correspondingly, this has the potential advantage of ensuring that quicker learners are not held back by struggling learners or those who are struggling need not be left behind when a group moves to the next topic.

There is no doubt in our mind that there is considerable merit to both approaches.

Next Phase Questions:

During the next stages of the project there are a number of questions still to be resolved including, but not limited to the following:

• It would be interesting to compare the improvements displayed by the group to the progress (or lack thereof) of a control group.
• Of particular interest is how we design a pathway into algebra that ultimately equips a learner to take Pure Mathematics (as opposed to Maths Literacy) when selecting subjects at the end of Grade 9?
• It remains an open question as to whether there is a minimal level of prerequisite skill required for Khan Academy to be effective; how we measure this; and how we bridge this gap?
• A key element is therefore how we most effectively test a learner’s prerequisite knowledge before proceeding with a particular Khan exercise?
• There are issues pertaining to how best to contextualize the Khan Academy aspects that are not suitable to the South African context?
• And finally, given that we have had learners beginning at extremely low levels of knowledge, can computer-based solutions like Khan Academy also move learners up from 40% to 60% or higher?

Sustainability Factors:

Currently the funding for the project comes from private donations and corporate sponsorship. We have begun experimenting with additional income-generating opportunities, including a low-cost learner fee similar to how we operate our adult education programs.  It’s important to note that this fee is not for access to the Khan Academy (which is free), but to cover the operational costs of the local facilitator, lesson notes and the centre. There will of course be a number of learners who cannot afford even a low-cost option and we invite learner sponsorships to either supplement or fully cover the costs.

In addition, the operating costs of the centre are supplemented by using the computers in the mornings for additional income generating activities.  These include: adult education, post-matric bridging year programmes and IT/internet services for community members.

Commitment to Creative Commons and ‘Open Source’ Content:

Any lessons learnt, notes developed, processes implemented or software built will be published under the creative commons licence and we hope to learn as much from others as from our own experiments. The invitation is thus for everyone to participate, either in an implementation in a local environment or through further development of the programme content itself. The long term vision is to contribute to a network of individuals and groups around the country working together to keep the content current and adapted to the local situation.

Any queries or comments can be directed to Andrew Barrett: andrew@wearegrowing.org.

---