Description

The Social Media Crawler is a tool that aims at timely social media monitoring by collecting and analyzing social media posts, in particular from Twitter, for topics of interest.

After creating a Twitter developer account and setting explicit search criteria selected by the users, the tool offers real-time collection of tweets, using the Twitter Streaming API in line with the Twitter developer policy.

Apart from a continuous collection of Twitter posts, a set of analysis techniques are built-in, in order to add value to the incoming information:

  • Detection of locations mentioned inside English, Italian or Finnish text and association to coordinates, so as to geotag a tweet
  • Identification of visual concepts that appear in the image of the tweet
  • Detection of inappropriate content in the image of the tweet to blur adult material
  • Text classification to automatically mark the tweets as relevant or irrelevant to the topic of interest

The collected and analyzed information from tweets becomes available in two ways of accessing the data. On the one hand, end users can visit an online web application, while developers can exploit an API to fetch tweets in JSON.

The online web application not only displays the tweets and their analysis, but offers additional visual analytics, such as:

  • Grouping tweets with similar content, which express the trending topics being discussed on Twitter
  • Discovering Twitter user communities through their interaction and identifying the most influencing accounts

The following video presents an introductory tour to the online web application, showing its various functionalities.

Technologies

  • HTML, CSS, JavaScript, jQuery (front-end)
  • PHP, Java, Python, R (back-end)
  • Twitter Streaming API
  • MongoDB

Contact information

Please contact us if you are interested in social media crawling for a live demonstration of the tool!

Stelios Andreadis (andreadisst@iti.gr)
Ilias Gialampoukidis (heliasgj@iti.gr)
Stefanos Vrochidis (stefanos@iti.gr)

References

S. Andreadis, A. Moumtzidou, I. Gialampoukidis, S. Vrochidis and I. Kompatsiaris, “A Flood Monitoring Tool for Urban Areas Using Satellite, Weather and Social Data”, US-Serbia & West Balkan Data Science Workshop 2018 – Big Data and Critical Infrastructures, 26-28 August 2018, Belgrade, Serbia (poster presentation)

G. Vingione, G. Scarpino, L. Marzell, T. Pettengell, I. Gialampoukidis, S. Andreadis, S. Vrochidis, I. Kompatsiaris, B. Valentin, L. Gale, W.-K. Lee, W. Lee, M. Gienger, D. Hoppe, V. Sitokonstantinou, I. Papoutsis, C. Kontoes, F. Baruffi, M. Ferri, H. Yoon, A. Karppinen and A.-M. Harri, “EOPEN: Open interoperable platform for unified access and analysis of earth observation data”, Proc. of the 2019 conference on Big Data from Space (BiDS’19), doi:10.2760/848593, pp. 1–4, Munich, Germany, 2019.

S. Andreadis, I. Gialampoukidis, R. Fiorin, F. Lombardo, D. Norbiato, A. Karakostas, M. Ferri, S. Vrochidis and I. Kompatsiaris, “Social Media Observations for Flood Event Monitoring in Italy over a One-Year Period”, 2nd International Conference Citizen Observatories for natural hazards and Water Management (COWM 2018), 27-30 November 2018, Venezia, Italy

L. Gale, B. Valentin, H. Boulahya, G. Vingione, G. Scarpino, L. Marzell, T. Pettengell, I. Gialampoukidis, S. Andreadis, S. Vrochidis, I. Kompatsiaris, D. Hoppe, L. Zhong, M. Ferri, D. Norbiato, F. Zaffanella, A. Karppinen, J. Tyynela, P. Karsisto, H. Kontoes, I. Papoutsis, V. Sitokonstantinou, T. Drivas, H.-W. Jo, H. Yoon. “H2020 EOPEN Easing Copernicus Data & Services Exploitation”, Asian Conference on Remote Sensing 2019, 14-18 October 2019, Daejeon, Korea.

Related projects