Here is my two cents:
randomized experiments are considered to be the "Gold Standard" of understanding the effects of programs from measuring the effects of new drugs to evaluating the effects of government programs. The idea is to randomly divide people into a treatment and a control group and expose treated group to the program and compare the results between the two group.
In case of Wiki, we divide TAs into two groups randomly. It can be proved (mathematically) that if TAs are selected randomly into these two groups, the groups will have the same characteristics on average and behave similarly (again on average). If we then compare the average evaluation scores in these two groups in semesters before and after wiki we can find the effect. The following picture explains it better:
In the above hypothetical figure, wiki was announced in early Spring 08 semester and the effect of it is 5% ([2.2 – 2.1]/2), which is quite large for a social program.
There are some necessary requirements for this experiment to be convincing:
- There should be two groups, treated and control at the same time. If we do not have a control group then we cannot disentangle the effect of a million other things that could happen simultaneously with wiki and affect TA’s performance. For example, one can argue that the increase in the TA performance is because they gained more experience in teaching and has nothing to do with Wiki. Therefore the results are not convincing.
- One can argue that as we give access to people to use Wiki, not everybody will use it eventually. Therefore, eventually we have a group that never used it and we can consider them as a control group. But this also gives flawed results because people who choose not to use the Wiki may be quite different from those who choose to use it. For example, they are more interested in improving their teaching skills, or they may perceive that Wiki actually helps them while those who don’t use it think it might not help them.They are different in nature and the results are biased (over or under-estimated). Therefore, as said in the gold standard, people should randomly be selected to treated and control group.
- I fully agree that TAs in treated group should not be pushed to use Wiki. On the other hand they should have the full freedom of choice. But suppose instead of one treated groups we have several treated groups. TAs in one of the treated groups have the full freedom of choice without any interference. The other groups could be exposed to a range of marketing schemes in addition, for example to periodic emails, to promotional emails, to workshops regarding Wiki, etc. This way we can not only measure the effect of Wiki, but also the effect of different marketing schemes to sell it in addition.
Thank you very much for reading my long post. Thank you all for the Wiki project. Thank you Bilal for setting up this forum and many other things.
