Sign up for a demo
By clicking the button, you agree to the processing of your personal data
Sign up for a demo
Sign up for a demo
By clicking the button, you agree to the processing of your personal data
Examus Mission+ Asynchronous Proctoring vs Synchronous Proctoring
Thesis 1.
There is no proctoring system that can guarantee 100% detection of cheating attempts; this cannot be done even during an exam in the classroom. Students always were and are inventive and creative.
Thesis 2.
Even if there was a proctoring system that protects against known threats, new write-off methods will constantly appear.
If Thesis #1 and #2 are correct, there is no absolute proctoring system. So why do we need to use proctoring, and what is the most effective system?
We believe that a proctoring system helps reduce the likelihood of cheating attempts due to the following factors:
• the inability to use simple methods, like asking another person to take a test or calling a friend or finding answers on the Internet
• the ability to catch people who are trying to cheat and punish them
The second factor has the same importance level as the first. If students are aware that violating an exam’s rules can result in punishment, a significant portion of them will find it easier to prepare for the exam than to take risks in cheating.
Studies show, and this is confirmed by the practice of Examus, that 85% of people do not tend to violate the rules if they know that they are being watched and a fine can follow (analogue - cameras on the roads: if they are not present, then people often exceed the speed; if there are only signs about cameras, significantly fewer people violate the speed limit). Another 12-14% try the system for strength; they try to violate the system, see that the fine has come, and then behave correctly. And only 2-3% of people tend to risk and hack the system.
Examus’s main goal is to make it possible to regularly find violators and write them fines. We don't have to catch all the violators; simply put, we can't, nor can anyone or anything else. It is more important to catch the student cheating than to prevent violation attempts.
It is much more important to constantly catch new write-off methods, investigate them, and adjust the system to have a larger number of protections in the proctoring database. If the system is stable and does not adapt, students will sooner or later find a key to circumvent it.
Examus’s strategy is that, on the one hand, we are working on new protections and, on the other, we have great expertise in finding new cheating methods in order to further protect ourselves with technical and organizational measures.
There is no proctoring system that can guarantee 100% detection of cheating attempts; this cannot be done even during an exam in the classroom. Students always were and are inventive and creative.
Thesis 2.
Even if there was a proctoring system that protects against known threats, new write-off methods will constantly appear.
If Thesis #1 and #2 are correct, there is no absolute proctoring system. So why do we need to use proctoring, and what is the most effective system?
We believe that a proctoring system helps reduce the likelihood of cheating attempts due to the following factors:
• the inability to use simple methods, like asking another person to take a test or calling a friend or finding answers on the Internet
• the ability to catch people who are trying to cheat and punish them
The second factor has the same importance level as the first. If students are aware that violating an exam’s rules can result in punishment, a significant portion of them will find it easier to prepare for the exam than to take risks in cheating.
Studies show, and this is confirmed by the practice of Examus, that 85% of people do not tend to violate the rules if they know that they are being watched and a fine can follow (analogue - cameras on the roads: if they are not present, then people often exceed the speed; if there are only signs about cameras, significantly fewer people violate the speed limit). Another 12-14% try the system for strength; they try to violate the system, see that the fine has come, and then behave correctly. And only 2-3% of people tend to risk and hack the system.
Examus’s main goal is to make it possible to regularly find violators and write them fines. We don't have to catch all the violators; simply put, we can't, nor can anyone or anything else. It is more important to catch the student cheating than to prevent violation attempts.
It is much more important to constantly catch new write-off methods, investigate them, and adjust the system to have a larger number of protections in the proctoring database. If the system is stable and does not adapt, students will sooner or later find a key to circumvent it.
Examus’s strategy is that, on the one hand, we are working on new protections and, on the other, we have great expertise in finding new cheating methods in order to further protect ourselves with technical and organizational measures.
As for AI proctoring
AI algorithms are not accurate enough for the conditions of online exams; it is impossible to control the quality of lighting, equipment, or running applications in the background. It is impossible to achieve 100% accuracy in student’s behavior interpretation. Therefore, there could be recorded situations on video which AI will mark as a cheating attempt but in which the student was not trying to violate the rules. For example, this can be due to the additional face in the frame (portrait on the wall or a T-shirt), sounds of voices heard from the next room or a student reading test questions aloud, an unevenly installed camera which could be determined as a “wrong gaze direction” alert, etc. In cases like these, the student is guaranteed to win the dispute, and there were even cases of a court decision in favor of the student with the payment of fines.
In addition, GDPR requirements for automated (AI) proctoring are quite stringent in that a student can unilaterally abandon automated video data processing when, for example, life proctoring is not considered as an automated process.
Therefore, we do not recommend using an automated proctoring mode without human control, as the error price is very high.
At the same time, automatic algorithms help significantly increase operational efficiency.
Algorithm recommended by the Exam in practice:
1. Based on AI, we compile a student rating on the cheating probability. We assuredly then check those sessions where the cheating probability score is higher than, for instance, 5%.
2. We compile students' ratings by points scored. We will also check those who passed the exam too quickly (for example, faster than the median value by 20%). This is another 5-10%.
3. We also check some records selectively/randomly.
We constantly update protection work and automatic algorithms, because, in step 2 (captured above), we often find people who use new cheating methods, investigate them in detail, make recommendations on how to minimize these methods, and adjust the system accordingly.
Therefore, we do not recommend using an automated proctoring mode without human control, as the error price is very high.
At the same time, automatic algorithms help significantly increase operational efficiency.
Algorithm recommended by the Exam in practice:
1. Based on AI, we compile a student rating on the cheating probability. We assuredly then check those sessions where the cheating probability score is higher than, for instance, 5%.
2. We compile students' ratings by points scored. We will also check those who passed the exam too quickly (for example, faster than the median value by 20%). This is another 5-10%.
3. We also check some records selectively/randomly.
We constantly update protection work and automatic algorithms, because, in step 2 (captured above), we often find people who use new cheating methods, investigate them in detail, make recommendations on how to minimize these methods, and adjust the system accordingly.
Case Implementation
since 2016
Work
up to 100 thousand per month
Number of exams:
Synchronous / Asynchronous
Type:
LMS:
(multiple instances)
Examus proctoring system is used by the leading university, a top-20 in ТНЕ Emerging Economies 2020 and top-10 in terms of the number of courses on Coursera.
Brief info
The university has been actively using online technologies for more than 10 years. Since 2016, the use of e-learning tools has been chosen as a strategic framework for blended learning, as well as online education. The university placed more than 100 online courses on different platforms, like Coursera, OpenEdu, etc. In addition, since 2018, admission tests have also been moved online, which has significantly increased the number of foreign applicants.
Thus, the proctoring system is a necessary tool for the university's work and one of the critical services. In 2016, the university had to choose between developing its own proctoring system or using a third-party vendor. The second option was chosen, and a strategic partnership was concluded between Examus and the university. Examus made improvements in accordance with the terms of reference of the university, and development of the system continues at the present time, taking into account the needs of the partner university.
Thus, the proctoring system is a necessary tool for the university's work and one of the critical services. In 2016, the university had to choose between developing its own proctoring system or using a third-party vendor. The second option was chosen, and a strategic partnership was concluded between Examus and the university. Examus made improvements in accordance with the terms of reference of the university, and development of the system continues at the present time, taking into account the needs of the partner university.
From the very beginning, Examus provided a full turnkey service; that is, in addition to providing a proctoring system, it provided service for students and teachers, including, for example, setting up exams and providing its proctors for synchronous exams and for record & review. With the increase in exam volumes, the university decided to create its own proctoring center based on Examus’s solution to increase flexibility and reduce costs. Examus handed over to the university all the methodology, technical support documents, conducted trainings for proctors, and provided instructions for proctors and system administrators.
Gradually, the university's proctoring center took over the first line of support for students, recruited a staff of proctors for the live and record and review, conducted trainings for university teachers, and managed the proctoring process through Moodle and Examus administration interfaces. Examus provides a second line of support, training in new functionality, connecting its proctors for peak loads, and, of course, providing and developing the proctoring system itself as a service.
Gradually, the university's proctoring center took over the first line of support for students, recruited a staff of proctors for the live and record and review, conducted trainings for university teachers, and managed the proctoring process through Moodle and Examus administration interfaces. Examus provides a second line of support, training in new functionality, connecting its proctors for peak loads, and, of course, providing and developing the proctoring system itself as a service.
Methodology
Examus proctoring system is used by the leading university, a top-20 in ТНЕ Emerging Economies 2020 and top-10 in terms of the number of courses on Coursera.
The type of proctoring
For important final exams, real-time proctoring is used with live observation of each student. Here there is an emphasis on pre-screening students. Before admitting a student to the exam, the proctor must make sure that the student does not have any prohibited materials or devices. As practice shows, this weeds out most of the cheaters at the beginning and immediately makes it clear to students that the proctors are serious. Not many people risk cheating after that.
For ordinary exams, asynchronous proctoring is used more often with verification of all students or selectively, depending on the results of the scoring, according to the algorithm described below.
For ordinary exams, asynchronous proctoring is used more often with verification of all students or selectively, depending on the results of the scoring, according to the algorithm described below.
Proctors
We are happy and always ready to train the customer's proctors to work with any type of proctoring and provide them with support. In the event that the customer does not have enough proctors of its own, we can agree on a partial use of our proctoring resources to cover mass exams. In this case, a clear division by exams / hours is required, which both sides undertake.
Difficulties and challenges that we face
1. The number of students per live proctor: We have increased the number of students per proctor from 6 to 9 to improve productivity. A larger increase is technically possible, but we consider this amount to be optimal for work without loss of quality.
2. Checking a large number of videos in a short time: To speed up video verification, we use, as the most obvious methods, an increased number of proctors and another, more systemic approach - the formation of ratings by the number of violations and other parameters. We use all available archive tools.
3. Administration and support of a large number of examinations with mixed types of proctoring and proctoring. The university has the ability to independently create slots/proctors, enroll students, and receive reports and analytics without interacting with the Examus support team thanks to expanded access.
2. Checking a large number of videos in a short time: To speed up video verification, we use, as the most obvious methods, an increased number of proctors and another, more systemic approach - the formation of ratings by the number of violations and other parameters. We use all available archive tools.
3. Administration and support of a large number of examinations with mixed types of proctoring and proctoring. The university has the ability to independently create slots/proctors, enroll students, and receive reports and analytics without interacting with the Examus support team thanks to expanded access.
Why you can't use just AI proctor
The accuracy of automatic recognition consists of two parts:
1. The accuracy of the AI: how accurately the AI detects the events. For example, the absence of a student's face, the presence of an additional person in the frame, a voice, switching the working window, etc.
2. Accuracy of event interpretation as an attempt to cheat or false positive
Improving the accuracy of photo/video recognition can be provided by more complex, heavy, and resource-intensive models. For example, services that provide face recognition, which are used in banks for example, cost at least $0.5 per transaction.
1. The accuracy of the AI: how accurately the AI detects the events. For example, the absence of a student's face, the presence of an additional person in the frame, a voice, switching the working window, etc.
2. Accuracy of event interpretation as an attempt to cheat or false positive
Improving the accuracy of photo/video recognition can be provided by more complex, heavy, and resource-intensive models. For example, services that provide face recognition, which are used in banks for example, cost at least $0.5 per transaction.
During one exam, Examus algorithms use facial recognition more than 100 times. Therefore, we are limited by the possibilities of improving the accuracy of our algorithms for economic reasons. Currently, the accuracy of the neural network we use to recognize faces is ~0.9947%.
But even if the accuracy of the algorithms is 100%, many questions about the interpretation of the event remain. For example, an automatically found “no face” event can be triggered if:
• The person covered his face with his hands.
• The person turned away
• The person partially went outside the camera - for example, only the left side of the face with one eye was visible.
What should be done in this case? Should we generate this as an absent student absent event or not? On the one hand, the student is sitting still, and the proctor can see this clearly. On the other hand, it can be suspicious behavior. Even if we decide to exclude such situations from the list of suspicious events, then we have to use more complex neural networks, which is extremely costly for the reason described above.
Thus, we have a situation that the AI proctor sees a suspicious event and marks, but, in order to ascertain whether it is cheating or not, a human must view the record.
Sometimes, by watching a video, a person can clearly determine whether cheating was involved or not. In this case, there is a theoretical possibility of training automatic algorithms to identify such situations. The only question is complexity.
But even a person cannot always confirm that a student is cheating when there is clearly no action on the camera. In such cases, it is even theoretically impossible to catch an AI proctor
But even if the accuracy of the algorithms is 100%, many questions about the interpretation of the event remain. For example, an automatically found “no face” event can be triggered if:
• The person covered his face with his hands.
• The person turned away
• The person partially went outside the camera - for example, only the left side of the face with one eye was visible.
What should be done in this case? Should we generate this as an absent student absent event or not? On the one hand, the student is sitting still, and the proctor can see this clearly. On the other hand, it can be suspicious behavior. Even if we decide to exclude such situations from the list of suspicious events, then we have to use more complex neural networks, which is extremely costly for the reason described above.
Thus, we have a situation that the AI proctor sees a suspicious event and marks, but, in order to ascertain whether it is cheating or not, a human must view the record.
Sometimes, by watching a video, a person can clearly determine whether cheating was involved or not. In this case, there is a theoretical possibility of training automatic algorithms to identify such situations. The only question is complexity.
But even a person cannot always confirm that a student is cheating when there is clearly no action on the camera. In such cases, it is even theoretically impossible to catch an AI proctor
What Examus is doing in this direction
We have qualified proctors who perform video review at an accelerated rate. The cyberproctor's messages help you check your recordings faster without losing quality. These proctors are already trained to detect the behavior of cheaters, and we use their expertise, among other things, to assess the accuracy of the scoring. Depending on the importance of the exams, we sometimes recommend using the synchronous mode.
We are working to improve the accuracy of the AI proctor by improving the accuracy of networks without sacrificing speed or increasing cost. For example, we are presently retraining the network to recognize masked faces and install an automatic light check.
We are conducting research on how to create an algorithm that will look for violations by examining the student's long-term behavior; that is, not by one event, but by their number. For example, looking away every 3-5 seconds indicates cheating. Here we worked with researchers from a partner university. We plan to make the MVP of this functionality in 2021.
We are working to improve the accuracy of the AI proctor by improving the accuracy of networks without sacrificing speed or increasing cost. For example, we are presently retraining the network to recognize masked faces and install an automatic light check.
We are conducting research on how to create an algorithm that will look for violations by examining the student's long-term behavior; that is, not by one event, but by their number. For example, looking away every 3-5 seconds indicates cheating. Here we worked with researchers from a partner university. We plan to make the MVP of this functionality in 2021.
Examples with AI issues
Examus team
November 18, 2020.
Interviews and articles
Examus faces new cheating tricks every academic semester. Let's discuss the newest technologies that we...
In this article, we will describe the crucial factors that hamper students from successful exam taking
The coronavirus has changed the attitude of many employees to remote work.
We use cookies to provide the best site experience. By continuing to use the site, you agree to our cookie policy
I accept
Contact Information
USA: Examus Inc. 2261 Market Street #4280 San Francisco, CA 94114
query@examus.com
EU: Tallinn, Kesklinna linnaosa, Narva mnt 5, 10117
Support
Already our client? If you face a technical problem, our support service will be happy to help you.
© 2013-2022 «Examus»
