One of the most fun applications of computer vision algorithms is the ability to determine the reaction rate of a person’s nervous system by the reaction of the pupil to light.
This can be used to assess the adequacy of a person, his readiness for difficult, dangerous and responsible work (an airplane pilot, an NPP operator, a subway dispatcher, etc.).
Previously, "ping" was made a simple reaction of the pupil to the light of the flashlight. For example, so does the medical assistant ambulance. But you can also evaluate the reaction rate, which gives much more information. We found a way to automate this by defining pupils on a camera with 40 fps. Computational complexity is low, 3-5 thousand comparisons per frame - full realtime:
Pupillometry data of a healthy personThe rate of passage of the nerve impulse, as well as the rate of its subsequent processing in the nervous system, can be estimated from the speed of the reaction of the pupil to light (contraction and restoration of the diameter of the pupil after the flash).
That is, yes, it is - a kind of ping your CNS. This thing looks like 3D glasses.
Here is our prototype. Usually we build it into
diagnostic terminals (that is, these glasses stick out of the device), but it happens - we use it separately. You can make it more beautiful, but there is no special meaning, this is a pre-production sample:
What does this give?
Even at the end of the 17th century, doctors began to notice that patients in different states react differently to light. After nearly 300 years of research, Master Eckhard Hess picked up, who also concluded that fatigue and types of mental tension affect the pupil's reaction rate before assessment. In general - the rate of expansion of the pupil after a flash of light (adaptation to darkness) can serve as a measure of the degree of excitation of the nervous system. In 1982, Hess also proved that the pupils are revealed more when you look at a subject that interests you. He showed the test-men, of course, photos of girls. Then they gradually moved to more subtle matters — their favorite food, their party candidate, and so on. Then they moved to the measurement of mental load. The more intense a person thinks, the slower the pupils dilate. Then the brain "loaded" with arithmetic. Today we would say that the processes will be parallel, and when the computing power of the brain is occupied with something, the assessment of the surrounding state according to the lighting is slower. It remains unproved the assumption that the dilated pupils serve as one of the attractors in non-verbal communication.
Still later, studies began to appear that are numerically related to the effect on the pupillary reflex of various types of mental stress (careful examination of subjects, performing mental calculations) and nervous disorders (depression, irritability, etc.).
How the device works
These are such big glasses or binoculars. You need to look inside for a few seconds, there are infrared LEDs and video cameras.
To be more precise, they are used: infrared LEDs - to illuminate the eyes, they are invisible to human vision; LEDs in the visible spectrum - with white light for flash; a digital camera for the infrared range with a lens and another small diode tag to focus the gaze during the survey. In the "binoculars" also fits the control board, connectors and other body kit.
First, he recognizes you by the iris, so that you do not bring someone else's eyes. After authentication, the examination itself lasts from 6 to 8 seconds; the subject tightly applies the device mask to the eyes so as to exclude external flare. In the first 1-2 seconds, adaptation to darkness occurs. At this time, the pupils dilate to about 5-6-7 millimeters, trying to give the maximum light to the retina so that at least something can be recognized. We illuminate the eyes with an infrared LED, - for a person this range is not visible, and the pupil does not respond to it. See, the pupils are large and beautiful:
After reaching the maximum pupil diameter for 200 ms, white light diodes (light stimulus) light up. The beginning of the light stimulus is taken as zero on the time scale when building the pupillogram — the plot of the change in pupil diameter versus time. We are interested in the next 5 seconds after the flash. Point Dmin (maximum compression of the sphincter of the pupils) looks like this:
A digital camera records images of two eyes with a frequency of at least 40 frames per second. The control software (SW) finds a pupil at each frame and determines its diameter.
Further the pupil gradually expands.
The main task: we need to get a papillogram, that is, a graph of the reaction of the pupil to light. We recognize the video, remove the noise in the data (we have a lot of reference frames) and we end up with a graph (this is an example of my colleague, a relatively healthy person):
Pupilogram contains three important areas: the latent region, the pupillary constriction region and the recovery region. The key parameters of the latent area are the initial pupil diameter (DN) and the delay time of the pupil reaction from the onset of light exposure (TL). The second area of the graph is characterized by the minimum diameter (Dmin) and the time of narrowing (TC), the third area - by the points of recovery of the pupil diameter at 50, 66 and 75 percent of the narrowing amplitude. The pupillogram may be declared unsuitable for evaluation if a person closed his eyes during a light exposure, blinked too often or drove his eyes especially strongly from side to side. Incorrect results can be obtained for the test in color lenses.
The software then analyzes these parameters, the most important of which are the rate of contraction and the rate of expansion of the pupil. After this, a conclusion is drawn on the adequacy of a person and his readiness for complex and dangerous work.
Charts of people very tired, after a strong lack of sleep or in a state of intoxication, alcoholic or narcotic - will differ from the graphs of people in adequate condition. Especially well the difference will be noticeable if at first we get a standard - we examine a person when he is in a normal state, and later - in an abnormal state.
Tests
We tested the device on approximately 760 employees of the company in different conditions (including corporate events and the days after them). Only 3496 measurements before publication (new ones are added every day). The most difficult thing is not so much recognition as evaluation of the resulting graphics. As part of the test, we also went to various interesting clinics, where patients with a known diagnosis (for example, chronic alcoholism) were asked to voluntarily participate in the study. A drug test for a representative sample is harder to find. As part of the study, we collaborated with several drug treatment clinics, where patients with drug addiction and mental disorders are treated. For medical purposes, they take classes such as tranquilizers, antidepressants, antipsychotics, anticonvulsants and sedatives, so the base has accumulated quite large.
It was necessary to debug the algorithm, collect more data about the patient, and so on. Each person may have individual characteristics of the reaction of the pupil to light. The very first versions of the algorithm sometimes gave out generally funny things because of the difficulties of recognition. So once we almost hospitalized the doctor himself.
In the course of the development of the project and the connection of our beta testers, we have built a system in which the patient cannot consciously deceive the device in the "best" direction. For the worse - maybe with the help of a number of measures, but the application just needs to sift out unsuitable (for example, bus drivers before the flight) from fully combat-ready specialists.
We can carry out an assessment comparing with the standard after the physical examination or with the human standard for this type of people. Therefore, right now we are taking the necessary data from the personnel file of the patient, or we are collecting anamnesis on the field device like this (this is the doctor's interface):
If the device is embedded in a diagnostic terminal, the patient record and current measurement data are sufficient to build a very accurate hypothesis.
It is assumed that the diagnostic terminal does not exclude the medical examination by a specialist personally, but at times accelerates it. And provided that all key indicators are measured on the device (pupillometer, breathalyzer, pressure meter), it is possible to send to a personal medical check-up only those for whom the diagnostic terminal has at least minimal doubts, and this can already be 20 people out of 100.
Where does the card come from? Recall, since we have built cameras into the binoculars, and since we still recognize something from them, it’s foolish not to use the common human-recognition libraries for the iris, since this is a very long-standing task. So we get the patient ID, and we can pull up his previous data.
Here are beta testers selflessly sacrificed themselves:
Having problems with lenses:
Even transparent one-day lenses reduce the severity of the pupillary reflex, which is very similar to fatigue and alcohol intoxication. We assume that this is due to a decrease in the total amount of light that enters the eyes when using lenses. This effect must be taken into account when analyzing such graphs, so our interface contains a mandatory question about lenses before starting an examination.
Now we continue to refine the algorithms and methodology. We need to collect more statistics to improve accuracy. The device is already working as a good drug test, but we want more. Now the statistics does not allow making any reliable diagnostic conclusions. In addition, for a good diagnosis of the condition of a particular person, it is necessary to first carry out several normal measurements of his pupillary reaction, so that after a high likelihood after some impact, catch deviations from the norm - this is a question of the instruction manual for NPP personnel. We strive to improve the algorithm, reduce errors of the first and second kind. The TB safety and security specialists graciously nod as they look at this work.
Shooting "out of sight"
Here's a video:
You also need to recognize mistakes (when a person intentionally blinks, tries to consciously accommodate and generally interferes with the operation of the device) - as long as such cases are visible on the graph and fixed by hand, later there will be a mark about the unreliability of the result.
Links