A Twitter Crawler and News Indexer

TrollHunter is a Twitter Crawler and News Website Indexer. It aims at finding Troll Farmers and Fake News on Twitter.

It is composed of three parts:

• Twint API to extract information about a tweet or a user
• News Indexer which indexes all the articles of a website and extract its keywords
• Analysis of the tweets and news

Installation

You can either run

pip3 install TrollHunter

or clone the project and run

pip3 install -r requirements.txt

Docker

TrollHunter requires many services to run

• ELK ( Elastic Search, Logstash, Kibana)
• InfluxDb & Grafana
• RabbitMQ

You can either launch them individually if you already have them set up or use our docker-compose.yml

• Install Docker
• Run docker-compose up -d

Setup

Change the .env with the required values
Export the .env variables

export $(cat .env | sed 's/#.*//g' | xargs)

Twitter crawler

Twint

For crawling tweets and extracting users' information, we use Twint which allows us to get a lot of information without using the default Twitter API.

Some of the benefits of using Twint vs. Twitter API:

• Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only);
• Fast initial setup;
• Can be used anonymously and without Twitter sign up;
• No rate limitations.

We encountered some problems:

• Bad compatibility with windows and datetime
• We can't set a limit on the recovery of tweets
• Bug with some user agent

So we decided to fork the project.

Which allows us to:

• get tweets
• get user information
• get follow and followers
• search tweet from hashtag or word

API

Open-source framework flask.

Four endpoints are defined:

/tweets/<string:user>

• get all information of a user (tweets, follow, interaction)

/search

• crawl every two hours tweets corresponding to research

/stop

• stop the search

/tweet/origin

• retrieve the origin of a tweet

Some query parameters are available:

• tweet:          set to 0 to avoid tweet (default: 1)
• follow:         set to 0 to avoid follow (default: 1)
• limit:          set the number of tweet to retrieve (Increments of 20, default: 100)
• follow_limit:   set the number of following and followers to  retrieve (default: 100)
• since:          date selector for tweets (Example: 2017-12-27)
• until:          date selector for tweets (Example: 2017-12-27)
• retweet:        set to 1 to retrieve retweet (default: 0)
• search:
• search terms format "i search"
• for hashtag : (#Hashtag)
• for multiple : (#Hashtag1 AND|OR #Hashtag2)
• tweet_interact: set to 1 to parse tweet interaction between users (default: 0)
• depth:          search tweet and info from list of follow

Twitter Storage

Information retrieved with twint is stored in elastic search, we do not use the default twint storage format as we want a stronger relationship.
There are currently three indexes:

• twitter_user
• twitter_tweet
• twitter_interaction

The first and second indexes are stored as in twitter. The third is built to store interaction from followers/following, conversation and retweet.

News Indexer

The second main part of the project is the crawler and indexer of news.

For this, we use the sitemap xml file of news websites to crawl all the articles. In a sitemap file, we extract the tag
sitemap and URL.

The sitemap tag is a link to a child sitemap xml file for a specific category of articles on the website.

The url tag represents an article/news of the website.

The root URL of a sitemap is stored in a postgres database with a trust level of the website (Oriented, Verified, Fake News, ...) and headers. The headers are the tag we want to extract from the URL tag which contains details about the article (title, keywords, publication date, ...).

The headers are the list of fields use in the index pattern of ElasticSearch.

While crawling sitemaps, we insert the new child sitemap in the database with the last modification date or update it for
the ones already in the database. The last modification date is used to crawl only sitemaps which change since the
last crawling.

The data extracts from the url tags are built in a dataframe then sent in ElasticSearch for further utilization with the request in Twint API.

At the same time, some sitemaps don't provide the keywords for their articles. Hence, from ElasticSearch we retrieve the entries without keywords. Then, we download the content of the article and extract the keywords thanks to NLP. Finally, we update the entries in ElasticSearch.

How it works

• Insert a sitemap that you want to crawl with insert_sitemap(loc, lastmod, url_headers, id_trust)
• Then run scheduler_news()which will retrieve all the sitemap that you have inserted in the database.
• You can also run scheduler_keywords() to extract the keywords that are missing from the URL that have been fetched.
• Every URLs found are inserted in elastic.

Run

For the crawler/indexer:

from TrollHunter.news_crawler import scheduler_news

scheduler_news(time_interval)

For updating keywords:

from TrollHunter.news_crawler import scheduler_keywords

scheduler_keywords(time_interval, max_entry)

Or see with the main use with docker.

Grafana

We use grafana for visualizing and monitoring different events with the crawler/indexer as
the insertion of a URL in ElasticSearch and the extraction of keywords in an article.

Create new events.

• Use TrollHunter.loggers.InfluxDBLog()
• Create a new dashboard in grafana, save as json and add it to docker/grafana-provisioning/dashboards

Text analysis

The text Analysis part is under TrollHunter/texto. It aims to process a text or a set of texts to retrieve useful
information that can be used to help determine the "troll" status of a user or link a text to news.

There several classes that do the job:

• Sentiment.py to extract polarity, feeling and subjectivity (integer indicators)
• Keyword.py to extract keywords/topics from a text input (a tweet or maybe news)
• Inicator_average.py to compute tweets from a list of users (in a specific format) and produce a means for all users.
  This was used to detect patterns that could help to qualify a user as a troll or not, by giving a certain trust
  percentage.

Keyword

Keywords extraction is useful because it can help to detect topics of an input text.
To extract keywords from a text, just import "extract" function from keyword and call it with a text as input.
"extract" function is just a wrapper of two extraction functions:

• extract_v1: implements RAKE (Rapid Automatic Keyword Extraction) algorithm. As it name says, it's a faster way to
  extract keywords. Keywords are lemmatized.
• extract_v2: implements TextRank keyword extraction. It produces better results than RAKE but it is slower.

We use both results on the same text and merge them to have keywords from both algorithm. Because of different
algorithms, results are sometimes different so we merge the result into a set of unique keywords to have both visions.
At least 75 keywords are returned: 25 from extract_v1 and 50 from extract_v2 (we can adjust this number by parameter).

Feelings

Feelings extraction is to extract polarity, Feelings and Subjectivity as numerical values from a text or set of text.
To extract them, import from Sentiment.py functions get_sentiment_from_tweets, get_polarity and get_subjectivity.
We use TextBlob for Polarity and Subjectivity analysis.
We use SentimentIntensityAnalyzer from nltk.sentiment.vader (nltk package) for feeling analysis.

Average Indicator

This one is to compute and extract average useful data from a set of tweets for a user (or a set of users).
It consists in one class called "Indicator". You give it one folder with a set of user csv file, and you call
"get_all_indicator_users" function to apply all our algorithms to have an average and detect some patterns.
We can for instance compare a set of troll users and a set of non-troll users.