In our previous post, we discussed about the data collection techniques and the models we experimented with to classify if a video is depressive or not. Note that this is only an experimental part of our entire research and does not hold a place in the pipeline discussed here. However, we found that it was important to analyze the video content and the kind of comments present on them to get an idea of the kind of viewers that consume it and the kind of affect it has on the viewers.
This post discusses about our validation methodology and our analysis of the comments present on the videos as per their classification.
For the validation and evaluation of the results produced by the classifier, we introduce a scoring method, called CES-D score, for each video to analyze how depressive the video is. The score is based on the calculation of category frequency of each comment in a video, wherein the categories are derived from the various symptoms (Insomnia, Self-hate, Appetite, etc.) considered in the CES-D scale.
Fig.2- shows the set of questions asking about some ways an individual may have felt or behaved. The individual has to indicate how often he/she has felt this way during the time duration of a week. Response options range from 0 to 3 for each item (0 = Rarely or None of the Time, 1 = Some or Little of the Time, 2 = Moderately or Much of the time, 3 = Most or Almost All the Time). Scores range from 0 to 60, with high scores indicating greater depressive symptoms.
The Center for Epidemiologic Studies Depression Scale is a brief self-report questionnaire developed to measure depressive symptoms severity in the general population. For the generation of categories, we use Empath Model (Fast et al., 2016) which combines modern NLP techniques to construct categories on demand using a few seed terms. For e.g., for the seed term ”weight-loss”, the generated set of words were ”weight loss”, ”anemia”, ”stress”, ”malnutrition” etc.
The algorithm here presents how the score is calculated for each comment in the video. The categories generated by each seed-term is merged in a set (categoryTerms). The algorithm iterates through each word of the set and calculates the term-frequency for it and calculates an aggregate by adding the term-frequency for each word. Since the words in the set can be used in both positive and negative connotations, the comment is then sent to the Empath model for analysis of a score for positive and negative emotion. The aggregate generated in the previous step is then multiplied by the difference of positive score form the negative. This is to ensure that only the negative emotions add up to the score. If for any comment, the positive score exceeds the negative score, the CES-D score is calculated as 0. For the calculation of the CES-D score for the video as a whole, in order to determine the intensity of depression in the comments, the aggregate calculated in the first step is added up for all comments and a normalized score is generated by dividing the sum by the total number of comments.
The comments posted on a video is a great way of understanding how the video made the viewers feel. They give us a fair idea of
the kind and content of the video. So, we examined the responses to a video by analyzing the textual features of the comments
considering the idea that the words a person uses reveal information about their psychological state (Fine, 2006). We extracted
the comments posted on videos that fall under one of the following categories:
a) Music b) Depression c) Funny d) Self-Help/Motivational
It was determined that a video belonged to a category if it was in the YouTube search result when the category names were used as keywords. For each keyword (category), we extracted around 200 videos and analyzed the comments in a cumulative manner by calculating their CES-D score (as mentioned in section). On the comparison of various categories mentioned above, the difference between the average CES-D scores of Depressive v/s Non-Depressive videos was quite evident in the figure below.
In the plot, the percent (population count) of non-depressed comments (CES-D = 0) for non-depressive categories (music, funny) was much higher than that in depressive category. Also, the range of CES-D score is the largest in depressive videos which proves that depressed people do loop-back to these videos (and are induced by it). As per different theories, it has been proven that on being sad/depressed, humans tend to be inclined towards groups where they can find similarities to their mental states (Gray et al., 2011). Thus we can deduce that a person watching videos of high CES-D score (Depressive videos) on a frequent basis is likely to have a depressed mental state so he/she needs to be made aware of it and directed to support groups.
To evaluate the classification results produced by our classifier in a real-life setting, we tested the results in a less constructed domain. The model was fed the transcripts of 1500 random videos and the classifications were compared to the CES-D score obtained by the comments. The method mentioned in section 3.1 was used to calculate the CES-D score of the video. A CES-D score greater than 30 was considered depressive. (We came to the conclusion of setting the threshold as 30 after taking the average of CES-D scores for the depressive videos collected for the training purpose). The accuracy of the classifier based on this methodology can be evaluated from the confusion matrix below.
Problem explaination and solution proposal. The path to be followed is explained and pipleines etc. are discussed.
This post explains the model trained for classification of videos as "depressive"
Calculation of Arousal Valence values from a video is discussed here