Case 1: HSE University

The university has been actively using online technologies for more than ten years. Since 2016, eLearning has been chosen as a strategic tool for blended learning and online education. The university put more than 100 online courses on different platforms, including Coursera, OpenEdu, and others. Since 2018, admission tests have also been moved online, which has significantly increased foreign applications.

Thus, the proctoring system is a necessary tool for and one of the critical services of the university.

since 2016

up to 100 thousand per month

(multiple instances)

Task and Solution

In 2016, the university had to decide whether to develop their proctoring system or use an external vendor service. They chose the second option and started a strategic partnership with Examus. Examus has successfully adjusted the monitoring system to the university's technical requirements and continues to make improvements on the system.

From the very beginning, Examus provided a full ready-to-use service: i.e., in addition to proctoring, it also provided 24/7 technical support for students and teachers, such as setting up an exam system or providing its proctors for synchronous exams and records & reviews.

Since the number of exams rises continuously, the university has decided to create its proctoring center based on Examus’s solution to improve flexibility and reduce costs. Examus handed over the methodology and technical support documents, conducted training for proctors, and provided proctors and system administrators instructions. Gradually, the university's proctoring center has taken over the first line of support for students. They recruit a staff of proctors for synchronous and asynchronous monitoring, train the university teachers, and manage the proctoring process using Moodle and Examus administration interfaces. Examus arranges a second line of support, trains on new features, provides its proctors for overloaded periods, and, of course, maintains and develops the proctoring system itself as a service.

In most scenarios, Examus combines different types of proctoring and proctors.

1. Types of proctoring
For important final exams, synchronous proctoring with live observation of each student is used. Here we focus on pre-exam student checking. Before admitting a student to the exam, the proctor must ensure that they do not have any prohibited materials or devices. As practice shows, this weeds out most of the cheaters and immediately clarifies to students that the proctors are serious. Not many people risk cheating after that.

For regular exams, asynchronous proctoring is used more often. Depending on the scoring results, all or random students will be checked according to the algorithm described below.

2. Types of proctors
We are happy to support our customers and always ready to train their proctors to work with any proctoring type. If our customer does not have enough proctors of its own, we suggest a partial use of our proctoring resources to cover mass exams. In this case, a clear division between both sides by exams/hours is required.

1. The number of students per live proctor: We have increased the number of students per proctor from 6 to 9 to improve productivity. A larger increase is technically possible, but we consider this amount optimal for maintaining quality.

2. Checking a large number of videos in a short time: To speed up video verification, we increase the number of proctors and group ratings by the number of violations and other parameters. We use all available tools om our archive.

3. Administration and support of a large number of examinations with mixed types of proctoring and proctors.: Due to expanded access, the university can independently create slots/proctors, register students, and receive reports and analytics without interacting with the Examus support team.

In general, the accuracy of automatic recognition consists of two parts:

1. The accuracy of the event recognition algorithm, for example, the absence of a student's face on the screen, the presence of another person in the room, unknown voices, switching windows or tabs on the desktop, etc.;
2. The accuracy of interpreting an event as an attempt to cheat.

More complex and resource-intensive models can improve the accuracy of photo/video recognition. For example, face recognition services used in banks cost at least $ 0.5 per transaction. During one exam, Examus algorithms use facial recognition more than 100 times. Therefore, the accuracy improvement of our algorithms is limited for economic reasons. Currently, the accuracy of the neural network we use to recognize faces is ~0.9947%.

Nevertheless, even if the algorithm's accuracy is 100%, there are still many questions about the interpretation of the events. For example, an automatically found "no face" event can be triggered by such cases as:
• A face is covered with hands
• A person turns away from the camera
• A person partially moves out of the camera's view – for example, only the left side of the face is visible.

What should be done in this case? Shall we generate the "Student is absent" event or not? On the one hand, the student is sitting still, and the proctor can see this. On the other hand, it can be suspicious behavior. If we decide to exclude such situations from the list of suspicious events, we have to use more complex neural networks, which is too costly for the reason described above.
Thus, when the AI ​​proctor sees a suspicious event and marks it, a human proctor has to check this event to determine whether it is cheating or not.

Sometimes, by watching a video, a live proctor can determine the fact of cheating. In this case, there is a theoretical possibility of training automatic algorithms to identify such situations. The only question is complexity.
However, when there are no obvious actions in the frame, even a person cannot always confirm if a student is cheating. Therefore, it is almost impossible to teach a cyberproctor to do this.

We have qualified proctors who verify videos at high-speed mode. The cyberproctor's messages help to check recordings faster without losing quality. These proctors are already trained to detect the cheaters' behavior, so we use their expertise to assess the scoring accuracy. Depending on the exam's importance, we sometimes recommend using synchronous mode.

We are working on upgrading the AI ​​proctor efficiency by improving networks' accuracy without sacrificing speed or increasing cost. For example, we are presently retraining the network to recognize masked faces and installing an automatic light check.

We are currently working on creating an algorithm that will look for violations by analyzing the long-term behavior of a student in various events. For example, looking away every 3-5 seconds indicates cheating. Here we collaborate with researchers from the partner university. We plan to present the MVP of this new feature in 2021.

November 16, 2020.

Examus faces new cheating tricks every academic semester. Let's discuss the newest technologies that we...

In this article, we will describe the crucial factors that hamper students from successful exam taking

The coronavirus has changed the attitude of many employees to remote work.