Organization of the University Signal Processing Course

Pedagogy interested me for a very long time and, for many years, as a student, I was educated, but at the same time tormented and torn by the existing organization of education, I thought about how to improve it. Recently, I have more and more often been given the opportunity to test some of the ideas in practice. In particular, this spring I was given the opportunity to read the course “Signal Processing” at the Polytechnic University (SPBPU). His organization, especially the organization of reporting, is the first experiment, the results of which seem to me to be somewhat successful, and in this article I want to tell about the organization of this course.

I still do not have a clear understanding of what should be read in the course with such a name, but in general it is a course about what and how you can automatically do with images, sound, text, video and other examples of natural and artificially generated signals . By what was read earlier and would be most useful, this is a solution to problems with a semantic gap between the input signal and what one wants to understand. This article is not about course content - even in Russian there are quite a few videos of good courses on similar topics.

The plan of the article is approximately the following: first, the course organization I have chosen is briefly described, then there is a story about the problems that I consider useful to solve, then how I tried to do this while reading the “signal processing” course and how I evaluate the results, what problems I see what ideas to solve them have. All this is nothing more than my thoughts and ideas, and I will be very happy with comments, objections and more ideas! Moreover, this is all written in many respects precisely in the hope of receiving your ideas and comments. Still, perhaps, this text will help someone to find an interest in high-quality teaching, in spite of all that is happening around.

General course organization chart

The course has two components: theoretical and practical. Both parts are very important: a theoretical one gives a great overview of existing algorithms and ideas for constructing them for solving problems with a semantic gap; practical should give at least some review of existing libraries, as well as practice skills in constructing your own algorithms. Accordingly, for both parts, there was a need for stimulating accountability for their study, which sets the main line of work for students.

As usual, the theoretical part consisted of lectures. After each lecture, the students were given a broad list of questions on the lecture, consisting of both routine questions about the details of the narration and creative questions about how and in what cases certain ideas could be improved and where they can be used before students are asked to come up with their own questions. according to the lecture (and you can also answer them). All questions were laid out in a post in the VKontakte group, the answers had to be written in the comments: you could either answer a question that was not yet raised by anyone, or comment on / add an existing answer, including one made by another student. The scope for closely connected with the subject of creativity, in my opinion, was huge!

The addition to the answers to the questions should have been the ranking: after the deadline, the students had to drop me the names of the respondents ranked according to the deserved ratings. Also, comments on rankings were welcomed. After all this, I finally gave points for the lecture. According to the results of these points and a number of additional buns, including those growing from the practical part of the course, marks for the semester were given. Dissenters and idlers could try to raise the mark on the harsh exam (you can use everything, but I strongly ask for understanding).

The general message of the theoretical part was something like this: I try to give insanely much material, hoping that all students will find in it a lot of new and useful things. However, I do not require them to delve into everything, they can either choose interesting / useful moments for themselves and go deep into them, or do a little bit of everything. I take the exam more as a fine for those who worked poorly in the semester than as a norm.

The practical part consisted of

• three mini-labs in which students had to run ready-made code that actively uses different libraries and select data on which it works well or does not work well,
• a term paper in which students were required to independently solve a problem with a semantic gap. They could take the initial task either from the proposed ones, or they could choose and agree with me. Then they had to come up with a solution, encode it, see that it works the first time, it does not work well and then try to improve it, following my own and my advice. The ideal would be to achieve a really good quality, convincing students that even in this area, patience and work in the right direction will grind everything, but, naturally, this can not be hoped for all the time.

All this was necessary to do on the test. The quality of work and the amount of effort expended could vary significantly. Due to the great effort it was possible to get more extra points in addition to the lecture notes.

It happened in the spring semester of the 4th course, when the semester ends a little more than a month earlier due to bachelor's. That is, I had about 10-11 weeks.

I also had an insider in the form of a sister, who studied in one of two groups, in which I lectured. My sister sometimes could stop my crazy ideas with stories about her vision of the real situation in the group and loading in other subjects. In combination with the successful topic of the course, fate really favored the conduct of experiments more than ever!

Reflections on the problems you want to solve

In this section, I try to tell about the problems, reflections on which led me to the described structure of the course. These problems are mainly related to two facts:

• There are creative and active students who are able to independently organize their studies in the direction they really need. Driving everyone under the average level, the existing system of education in universities often creates difficult, nervous, and senseless conditions for such students.
• Many teachers, unfortunately, are not interested in the quality of their work. Often such disinterest is a result of frustration in students. But the bad work of students cannot but be a consequence of the poor work of teachers. The situation can improve if high-quality work will benefit the teachers themselves, not just the students.

Of course, there are still many problems that are not very connected with both the first and the second. For example, what to do with those students who are not able to organize themselves on their own? Or those who seem to be trying, but still can not do anything?

The problems associated with the two facts described above have been most suffered by me, and I thought a lot about solving them. It seems to me that there is at the same time the decisive “silver bullet” of them: if smart students are in comfortable conditions, then they can be of great benefit to teachers.

Teacher motivation

Let's start with the motivation of the teacher. Naturally, it is necessary for a good course. So, from the course, the teacher can receive:

• Pleasure.
• Money. Ours is often symbolic. Especially for those who teach well in IT, this money is absolutely ridiculous. As a rule, these people have or may have several times greater earnings in another job. And they just can not teach well just for the sake of wages.
• The stimulus is much better to immerse yourself in the material. I was very excited about the popularity of my lectures. And, at least for now, I was very afraid of the condemning views of students and their negative opinion: “here’s another one to do nothing but to force us to waste time on some nonsense with which he himself couldn’t or didn’t find it necessary to sort out” .
• The results of student immersion in the material. You can create an atmosphere that will stimulate reasonable questions from students in lectures. Such questions can greatly help the teacher: to point out some mistakes and flaws, to encourage people to look at things from the other side, and maybe even make them understand something new.
• You can stimulate the activities of students, beyond the scope of the material readable in lectures. Then they can collect a lot of new information and produce results in an already somehow processed form. Yes, it is all the same then difficult to understand and verify. But it is precisely in the course of such inspections that the outlook is broadened. And there is another bonus: if something is not clear, sometimes you can ask a student instead of trying to figure it out for yourself. Such a question will also be a test of how well the student figured out.
• Training to communicate with people. Training assessing people, understanding what can be expected from them, including depending on their own actions. You can try to assess in advance which student will cope with the task well and on time, which one is bad, which one will do what is needed, but for a very long time. Train different management approaches (reminders, etc.). Understand how easy it is, and how exactly students (and probably not only them) can manipulate you. The space for experimentation is huge. The results of the experiments can be seen relatively quickly.
• Practice competent presentation of thoughts, lectures and other public speaking skills. Training understanding of poorly formulated answers and questions by students (sometimes all this has to be done on the fly - you can train your own reaction).
• The results of testing simple ideas in practice by the hands of students. Useful can be both the results of testing your own idea, and the idea that occurred to the student. If you find a task that is really interesting for a student, it is likely that the student will generate good ideas and check them qualitatively.
• 'Free' use of students to solve their practical problems.

  It is widely believed that with this teachers receive the greatest benefit. I believed in this for quite a long time, but with each experiment that follows, my belief in it decreases. So far I had only one student, from cooperation with whom I ended up getting exactly what I wanted, on time, and really saved my time. I probably managed to teach the same student better than others. True, and here, then, during the project, it turned out that I needed to solve this task in a slightly different form, but this is definitely my fault.
  For all the other students I came across, I had to run around endlessly, remind them of scientific work, and explain the same thing to them several times. In the end, I received something very strange from them, and often at that moment when I was already solving this task on my own. I don’t understand how useful such a format is for them (like they are training to do something, but somehow it’s very poor quality). In my case, this process eats up a lot of time and nerves. The only plus: sometimes, in the course of discussions, my attention is drawn to certain details of the task, which I had not noticed before.

  
  
  
  

• Fame, prestige - with high-quality teaching
• Visibility of the results of their activities and grateful students. True, it is often difficult to understand the truth, students are often grateful not at all for what they need.
• Meet future specialists in their field. It is better to understand them, to understand what the new generation lives. You can select those who liked, and then invite to work.

That's all that I managed to collect. For myself, I try to realize as clearly as possible what, in addition to pleasure and prestige, I hope to gain from reading the course. What should it be so that I would be ready to pay for it with my time for the entire semester? Without this understanding it is hard to believe in the opportunity to conduct a course well. Own motivation must be considered when thinking through the course structure.

Comfortable conditions for advanced students

The second part of the requirements for the structure of the course is aimed at creative and active students who have a good idea of ​​what they need. Despite the fact that many teachers confidently deny even the possibility of the existence of such students, they definitely exist in advanced universities. By senior courses, their number increases significantly, especially with high-quality training. And it is smart students who are the hope of our fatherland and science.

In almost all universities, training is far from being as effective as it could be. At lectures, students are often told something that might be interesting, but strange: if necessary, then in some world to which the students have not yet grown. It often happens that advanced students have already heard or read about these things, understood, and then forgot - now they are forced to listen again. Often, students have to do strange practical tasks that the teacher came up with just because he thought that students need at least something to download. To write and correct reports that teachers often do not accept from the first time simply because it seems to them unworthy, and you should at least teach something.

If this all falls on people who otherwise would not do anything - this is probably not bad. As practice shows, by the end of training these people understand something, most of them are quite suitable for working in their specialty.

But it happens that such a system is applied to advanced students, who already have their own plan of action, their own work, their own understanding of where to go. Moreover, this understanding is generally correct, and the work can be made very popular, if a little bit corrected. And here these students are lectured with abstract theoretical material, ill-considered practical tasks and reports that need to be written and corrected without end. Even if all this is necessary, it is much more effective to connect this with the scientific interests of the student. So that he understands how this information will help him in practice.

Otherwise, if the student does not understand, only a small part will be learned. Yes, and it will soon be forgotten, if it is not closely used in other courses. There will be only a general idea. As well as from non-core uninteresting school subjects or in nothing interested students. Still, perhaps, there will be an understanding of where to run to figure it out.

But getting this information from students takes quite a lot of personal time. Many advanced students could spend it with great benefit. Such people are ready to absorb the knowledge that they need almost on the fly and with amazing efficiency, especially at senior courses.

Yes, maybe your course is exactly what the advanced student lacks. And he, poor fellow, does not understand. But abstract theoretical lectures are unlikely to help him. If you understand the essence of some interesting work for him and in the right place, advise to apply at least a small piece of knowledge that you give, - the student will surely understand and appreciate. Especially if your proposal for improvement will help to achieve a qualitatively better result.

In fact, of course, everything is somewhat more complicated. Not all useful knowledge can be applied in an area that is interesting to the student. Then, especially if it happens at senior courses, it would be good to try to understand what is more useful for the student: to do what you think is necessary, or what he himself considers necessary for himself. And act in accordance with this.

In this course, I almost did not have such a problem: the course on solving problems with a semantic gap seems to me applicable everywhere and useful to everyone. In fact, this is a course on designing algorithms and models in difficult situations. I believe that it is useful for all to understand that this exists, and how it works at least at the top level. The course also trains well modeling skills and a sensible approach to solving many problems.

Much more I was afraid to tell only what many students already know. I did not want to force them to solve tasks that would not teach anything. I wanted advanced students not to be forced to do tasks for a tick, just to get credit.

To do this, you need to understand good students, understand what they know and what they aspire to. To interrogate them, to get opinions, to look at the results of work, to understand something by them. Make students not afraid of me. Not afraid to incorrectly answer the question. Not afraid to criticize my line.

But you have to be not only not scary, but demanding. Even to advanced students, reasonable demands helps, they are built. The time allotted for the assignment helps to understand which way to choose, how deep to dig in, when to ask for help. Result requirements help to understand what to concentrate on. And all this organizes, helps to prioritize between a lot of fallen cases.

Being not scary and demanding for a teacher is far from easy. Especially if there are many students. Lazy people are more demanding. With them you will suffer to be fair in each specific case. For advanced students - the opposite. They are much more afraid of the teacher self-indulgence than the rest. Because they have more at stake, more depends on the credit and departure. The very first unreasonable demand casts doubt: “Is the teacher intelligent? Will it adequately respond to my criticism? ”. Each following this doubt strengthens, the teacher in the eyes of the student turns into a madcap who needs to be pleased by spending as little time as possible.

It seems that only a reasonable rigid reporting system can solve the problem. Premeditated, which will not change in the semester. Compliance with this system should become more important than the teacher's opinion, no matter how strange it may sound. This dictates a high level of requirements for the reasonableness of the source system. It is clear that it is impossible to foresee everything, and I don’t want to waste time. Therefore, it is possible to clearly indicate the boundaries at which the teacher acts at his discretion. For example, the lab submitted after the deadline will be checked when it is not known, and after two lab failures in time, the consequences can be unpredictable. Further, depending on the reasons that led to this, you can either pardon or punish. But if the done satisfies the requirements, the teacher must do what he promised.

So, it was necessary to come up with a rigid rational reporting system. It needs to be more loyal to reasonable students. Positively taken into account all the useful things that they can come to mind and that will be associated with the course. But she did not give good marks for anything, but rather stimulated precisely to quality work.

It is also important that the reporting system be trusted and calmly felt with it. So that the student could set himself the task of doing everything at the beginning of the semester, get an assessment and feel at ease. Do not be afraid that the teacher in the middle of the semester will think: “he is doing something all too well. Probably, it is possible to give more complex tasks and make the assessment depend on them. ”

Also, as follows from the previous section, the reporting system should take into account the wishes of the teacher. And it turned out that many of the requirements have already been taken into account: they coincided with the requirements of loyalty to reasonable students and to quality work. If advanced students are free to ask questions, they will be asked, including what the teacher does not know. If you can go beyond the course - they will come out and find new information. If they understand what they are doing and why they will do it qualitatively. And information about the results of such experiments, naturally, broadens the teacher’s outlook. Let not immediately, but sooner or later there will be something new, useful for him.

Contented smart student - contented teacher!

Evaluation problems

The reporting system cannot motivate students without a reasonable assessment of their activities. How to evaluate the results of the semester, which student deserves a greater score, and which one less?

The most commonly used criterion with us is the exam grade. The teacher is trying to understand in the course of some communication or written how well the student understands the topic at the time of the exam. This in itself is difficult. Often, almost everyone who understands, but is timid and unable to speak, students get lower marks than students who do not know the subject, but dodgy and arrogant. The written exam allows you to lower the degree of arrogance that a student can use. But interactivity is lost: it is impossible to understand whether a student understands what he did not write (and what he wrote). Another problem is cheating. I am familiar with some masters of pedagogy, whose marks correlated back to the students' knowledge: the tasks covered an insane amount of material and even those who were well prepared could not pass this on to a normal grade. But the copiers received 5 and the teacher based on them confidently concluded that you can cope - if you prepare.

Ideas to solve these problems exist. But even if these problems can be solved, it will still not be possible to assess the residual knowledge of the student.

The probability of increasing the amount of residual knowledge increases if the knowledge in the student’s head is not only at the time of the exam, but also during most of the course. And if knowledge is still supported by practical activities, they will most likely remain. It turns out that a student’s knowledge would be well appreciated several times in a semester. And in the end, put the mark on the machine if the student worked well during the semester. But at the same time, a general overview of the course that the student was supposed to receive during the preparation for the exam is lost.

The problems do not end there: all students are different, it happens that one thing is obvious, and another one has to think long about it. Perhaps, it is fair to evaluate not only their final knowledge, but also the amount of effort expended? How to evaluate them? What is better: overestimate a student or underestimate? Is it advisable when evaluating students to compare their level with the level of a group / stream? On the one hand, it seems yes: if the problem is with the whole flow, it means that the teacher has worked poorly. On the other hand, lowering the bar will contribute to a drop in the level of students.

, : , , CSC , . , , , .

. , , , — , — , . : . , , , , . .

, . , . , , . : , , 4- : ; , . feedback : .

, , 5- . , , . , .

: ? , , , - . , - .

, , — . — . .

, ( , ), . — , , , , , . , . : .

, ( 15, ). , .

— . , .

: , . - — ? , , + , , , . , , . , : . “ ” .

:

Semantic organization of the course

. :

1. — , , + ( )
2. 1-3 , . , canny, , ( / ), ransac, / , , , , .
3. 2-3 ( ) , , . , , , , , , , . ( ), ( , ).
4. , ( , , — , - ). , , , . , PCA LDA ( ), , , .

, . , -, :

1. python, pycharm . : , - .
2. 1-3, , , . , .
3. : : catboost tensorflow , ( , ). , , .
   , 3 — . , . , , , , .

: , - , . , . , . — , . 5 , .

2. : , , , , .
3. : , - , .
4. .
5. , , .

, -, ; .

, 4: , , /. — . , .

Evaluation

Above, I wrote a lot about points, it is time to explain what they gave.

, . “1/< >”. :

• —
• -
• ()

• , , , - , - , . , , .

  
  
  
  

• ,

  , , — . - , , ; , - — , .

  
  

  1 ( 0). 1 ( , ), — 8 . , , , . , 25, — 10, — 5, , - . , , , -, .

  
  

. 8 . , — . , : feedback , . , 5- . , , , . : , , . : , .

, , . , :

( ). , . +1 ( , 0 , ). , , - .

, .

I saw the dynamics of recruiting points during the semester something like this: advanced students will score enough for 5 automatic points for about the first 6-7 lectures. That is, somewhere by the end of March, just when I tell the basic information and move on to examples of setting and solving real problems. With practice, I hoped that the diligent will understand, too, by April, maximum to the middle, if its priority will be reduced by the requirements of other courses. I evaluated it on my own: I think that when I was a student of the 4th course, I would have passed such a course at about the specified time, if nothing unexpected had happened. I expected from less advanced students that at least questions like the possibility of obtaining an automaton would interest many of them, they would read the answers of their colleagues and fragments of lecture presentations. The topics are generally interesting, and it is possible that such students will be able to get hooked, and they will try to sort out deeper.

I would like to make a remark about the selected multiplicative combination of points between directions, and not additive (the root of the work, and not the sum divided by a number). This corresponds to the need to deal with a large number of directions at about the same level; even very, very deep knowledge in a couple of directions will not provide the student with a good grade for the course in the absence of his knowledge in other directions. For example, multiplicativeness protects from being able to get 5, having filled me with suggestions for improving the organization of the course: each next sentence, bringing as many points as the previous one, would have made less and less contribution to the final grade.

One of the immediately noticeable shortcomings of this system is its complexity. But since the course itself is quite complex and solving problems with a semantic gap requires the construction and understanding of complex algorithms, I believe that students should be able to understand this easily. Moreover, this reporting system is somewhat similar in itself to solving a problem with a semantic gap: some problems arose in the course model, the most important were selected from them, approximations were sought for their solution.

Another disadvantage of the system: it can really take a lot of time with students. Therefore, I tried to apply a long-standing idea: to offer students who know the material well and without a course or who consider themselves busy with more important things to contact me in the first month. I am ready to talk with them, and, depending on the level of their knowledge and the reasons for displacing my course, offer them an automatic or simplified, adjusted for them, method of passing the course. After the first month, the offer is removed - otherwise it can be used at the end of the semester by weak students who could not force themselves to do something, but, potentially, would like to.

Approximately it was stated to the students at the first lecture. Then I promised myself not to change it, even if I see that it works poorly and the students do significantly less or worse than expected. The course has begun.

results

The results were much worse than my expectations, although a number of hopes came true. I remember, after the first list of questions on the introductory lecture, I waited in fear: would there be at least some answers and whether they would be meaningful. And here, at last, the first answers began to appear, in the comments some kind of discussion ensued, although, rather, on a philosophical topic. Further, during the semester, the students continued to answer; however, as a rule, there were a couple of dominant students, who brought in about 70% of everything useful that was written.

By the end of the semester, the activity decreased significantly, after the penultimate lecture, they sent me a ranked list consisting of one last name — the only person who answered at least some questions about that lecture. The reasons for this, I think, could be general fatigue, perhaps some disappointment, inadequate assessment, an unsuccessful change in deadlines, which led to the need to wait for 3 weeks to get the final result of the lecture, the increased load on other subjects.

I was also more and more disappointed with the quality of the answers: it often seemed that much from somewhere was torn off without understanding, and the amount of new ideas was far from as large as I had hoped. Even from the students there was a remark that the current system stimulates at least some answers; on the degree of how deeply the student figured out, the scores do not depend so much. But for sure there were those who understood.

Since no one fit into my set plans for scoring, and this threatened that everyone except for a couple or three people would have to take an exam, I began to try to put higher points. It seemed that I overestimate the points too much to those who responded only with examples of tasks and the difference between these formal replies and those who really tried too little. By the end of the semester, I was increasingly embraced by the feeling that there are many students who do not understand almost anything of what was said, although they have relatively acceptable points. This feeling became even more at the last lecture, when I began to try to ask everyone in a row, hoping to better understand the final level and add the points correctly responding - it turned out that many people do not know basic things, for example, what neural networks are or particular points in the image.

The hopes for ranking were not justified either: there were very few comments in the ranked lists, by the end they had disappeared altogether. Often, it seemed to be evaluated visually rather than carefully reading. However, I remember, at least a couple of times, when the ranking really helped and I looked at him and adjusted my ratings. But that it evaluated for me was not the question. The assessment took quite a lot of time, but I could do it on the way to the subway and in the end I was more likely to have timely answers than students.

Did such a system result in more than, for example, the classical system with the exam? The question is complex, I hope that, nevertheless, yes, quite a lot of material has been given, while preparing for the exam, a part of it would certainly have dropped out of consideration even from good students. Although the additions to the course in the answers were not nearly as many as I had hoped.

What I’m sure about is that the course was very successful in terms of advancing and systematizing my knowledge. Of course, most of what I said, I generally represented, and so, I felt many things more deeply. There were algorithms that I knew about and even used, but did not fully understand how they work, did not know many alternatives, or knew only the names. When preparing the course, I had to figure it out. There was also a number of new things that I noticed under the obvious influence of students, for example, on autoencoders. I got a lot, maybe not very often used, but definitely needed for a good orientation in the subject area. I think that the improvement of knowledge that has already happened has even influenced some decisions that I took in my work while thinking over the algorithms, I hope for the better. Pleasure, of course, also brought me a course, but at the same time brought disappointments with disappointments.

Continuation

It may be that I will fall to read this course again, for example, next year. Not for all the problems, I have ideas for a solution, but for some I have, and I will try to describe them.

1. The main problem: the lack of timely advances on a complex task, I think to solve with the help of discussion of similar fragments of other tasks at seminars and clear homework with short deadlines. Each of the homework assignments will require a small fragment of a large laboratory, such as the formulation of the problem statement, the first selection of data, the elaboration of the quality criterion, ... For each fragment performed on time, points will be awarded. If a student is lagging behind, he will have to catch up to start receiving them.
2. I also plan to more clearly and often pronounce the main idea of ​​the course in various contexts. Although not sure that it will help: often, when you say the same thing, it starts the opposite to cause rejection. The main idea, if anything, was that the problem solving skill is not a thoughtless enumeration of various ML models in various configurations, but the manual construction of an individual model for a task using suitable pieces of the existing models with reasonable modifications to the task. For some reason, many people either do not understand this, or diligently make this look. Maybe some people can realize this idea only through practice, through stuffed bumps.
3. I also plan to stop putting 1 point to everyone who came to the lecture; and set, by default, significantly less, for example 0.1. In order to get more points, it will be necessary on the day of the lecture to send or show me the notes of the main lecture theses or their photos. Almost anything can be written, the format and volume do not interest me. But for good notes I am ready to bet significantly more than 1 point.

   I want to add this in order to additionally encourage students to listen to a lecture, rather than sleep and go about their business. Many more essentially remember what they write. Intellectual loading for creation of such notes is not really necessary. It also seems that it will not be too heavy for students who do not have a synopsis, those who lead it - can simply provide it.
   True, all the students surveyed are critics of this idea. In particular, they point out that it is not so difficult to write off these notes from a neighbor at the end of a lecture or simply write something from slides, without paying much attention to the lecture. In addition, the need to write some may distract from understanding.
   So, perhaps, it would be nice to somehow change the shape. But in general, I like this form of reporting, it was used, for example, on the statistics mat course at csc: you need to send a small completed lab to the day - and I think many students were encouraged to sit down for it right away and finish it. Although there were of course those who said that they could not at all that evening and were at a disadvantage. Here, it seems to me, another idea can help: to give each student the opportunity to move the deadlines for several days per semester.

   
   
4. There was a thought to replace the flat structure of answers to woody questions. So that the answers to all the questions would not be a complete list, but would be at least two-level: then the answers to one question would be next to each other, and not mixed up with the answers to other questions. Two-level structure of comments to posts is supported, for example, facebook. But it’s much less likely to enter it and you don’t want to make it the main means of communication. Simultaneously to lead two groups: VKontakte and facebook is strange. I would be glad if someone advises any other solution.

There are many problems that I don’t know how to solve and I don’t know if it is possible at all. The main thing that worries:

• too simple answers of students to my questions
• poor evaluation of answers: my assessment does not always correlate with reality
• ranking, which almost does not help: until the verification of student responses by the students themselves is very far

In general, I definitely do not consider in vain the time spent on preparing and conducting the course; , .

, , .

:

• : , (, ), (, )
• , /
• , - -