How can researchers capture Twitter ‘conversations’?

We actively welcome comments and queries in relation to this post. Add your comment below, email dun@dunproject.org or tweet us @dunproject.

Problems with Capturing Conversation in Twitter Data

During our social media analysis we have reflected on a number of methodological challenges related to the sampling, collection and ethical use of social media data.

One of the challenges we encountered was systematically capturing dialogue and ‘conversations’ between users on Twitter around defence issues in the UK. In order to collect Twitter data we used search terms such as “British Army”, “Royal Navy” and “RAF”. Although this approach allowed us to collect thousands of relevant tweets, using search terms in this manner did not easily enable us to collect follow up @reply messages if they did not include one of our search terms.

For example, if a Twitter user tweeted a mainstream media article about the Armed Forces which included one of our search terms – e.g. “TwitterUser1: British Army to withdraw from Afghanistan http://ow.ly/Si53G” – that would would be captured in our dataset. But if somebody tweeted an @reply message to TwitterUser1 commenting on the tweeting of the article – e.g. “TwitterUser2: @TwitterUser1: Great article!” – then this tweet would not be captured.

These difficulties mean that conversation and dialogue are under-represented in datasets collected in this manner which, in turn, makes analysing discourse and narrative formation or disruption problematic. In an attempt to mitigate the problem we spent some time using the quantitative data as a starting point and accessing the @replies by performing follow up, in depth, qualitative analysis. Although this ‘manual’ approach has its advantages, it is time-consuming and not comprehensive for larger datasets.

The problem of capturing @replies in Twitter data where collections are based around search terms and hashtags has been identified in previous research (see for example Bruns, How long is a tweet, 2012), but we have not been able to identify tools which can be used by qualitative researchers to capture @replies and further conversations systematically on a large scale.

We would be particularly interested to hear from colleagues who may have encountered similar difficulties and whether there are solutions that have been proposed by researchers studying Twitter. Comments can be posted below or we can be contacted by email: dun@dunproject.org.