Using twitter from the R console

Lately I have been messing around with R and I decided to check out the twitteR package to see if I can post from the R console. In order to use twitter from the R console, we need a couple of things:

• Setup OAuth  authentication for twitter
• Install the twitteR package

Setup OAuth  authentication for twitter
As of March 2013 OAuth authentication is required for all Twitter transactions. If you don't already have a OAuth setup, head over to twitter here: https://apps.twitter.com/app/new

Follow the instructions, once you are done, you will see the following 4 items

Consumer Key (API Key)
Consumer Secret (API Secret)

Access Token
Access Token Secret


Install the twitteR package
Now in you R program install the twitteR package
Once the package is installed, it is time to get busy......


 Load the package by executing the following command


library(twitteR)

Now it is time to setup authentication, you do that by using the setup_twitter_oauth command, below is an example, make sure to replace the keys and tokens below with the values you got back when you setup OAuth on twitter



setup_twitter_oauth("API key", "API secret", "Access token", "Access secret")

[1] "Using direct authentication"

If that is all set, we can send a tweet. To update you twitter status, you can use the updateStatus command, this is very simple to use, you pass your status into the function. Here is what it looks like on twitter

updateStatus('testing Tweeting with twitterR package from witin Revolution R Enterprise')
[1] "DenisGobo: testing Tweeting with twitterR package from witin Revolution R Enterprise"

testing Tweeting with twitterR package from witin Revolution R Enterprise

— Denis Gobo (@DenisGobo) February 20, 2016

Here is what it looks like from the console

Of course nobody is doing all of this to update their status. The reason I am playing around with this is because I want to do twitter searches and then store the results in a file or database. So let's do a simple search for the tag #rstats and let's also limit the search to only return 6 results

tweets <- searchTwitter('#rstats', n=6) 
tweets

Here is what we got back, as you can see some of the results end in ...., those have been truncated

[1] "psousa75: RT @rquintino: @Mairos_B #sqlsatportugal session: all about R in #SqlServer 2016 #rstats https://t.co/DHrqIZrz1e"

[[2]]
[1] "millerdl: a quick script to use imgcat in #rstats https://t.co/fpUlgWNX33 https://t.co/AhCCMLewCH"

[[3]]
[1] "diana_nario: RT @KirkDBorne: Useful packages (libraries) for Data Analysis in R: https://t.co/haRKopFyly #DataScience #Rstats by @analyticsvidhya https:…"

[[4]]
[1] "emjonaitis: Hey #rstats tweeps, do you have any readings to recommend on sensitivity analysis? Books/articles/websites all welcome."

[[5]]
[1] "caryden: RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…"

[[6]]
[1] "ArkangelScrap: RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…"


What I really want is to convert the output to a data frame. Luckily the twitteR package has this built in, you can use twListToDF. Here is how to do that

tweets <- searchTwitter('#rstats', n=6) 
twListToDF(tweets)

The output now has a lot more stuff, you can see if it has been retweeted or favorited as well as the latitude, longtitude and more

1                             RT @rquintino: @Mairos_B #sqlsatportugal session: all about R in #SqlServer 2016 #rstats https://t.co/DHrqIZrz1e
2                                                      a quick script to use imgcat in #rstats https://t.co/fpUlgWNX33 https://t.co/AhCCMLewCH
3 RT @KirkDBorne: Useful packages (libraries) for Data Analysis in R: https://t.co/haRKopFyly #DataScience #Rstats by @analyticsvidhya https:…
4                      Hey #rstats tweeps, do you have any readings to recommend on sensitivity analysis? Books/articles/websites all welcome.
5 RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…
6 RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…
  favorited favoriteCount replyToSN             created truncated replyToSID
1     FALSE             0        NA 2016-02-20 20:29:54     FALSE         NA
2     FALSE             0        NA 2016-02-20 20:24:50     FALSE         NA
3     FALSE             0        NA 2016-02-20 20:16:25     FALSE         NA
4     FALSE             0        NA 2016-02-20 20:11:08     FALSE         NA
5     FALSE             0        NA 2016-02-20 20:11:06     FALSE         NA
6     FALSE             0        NA 2016-02-20 20:02:05     FALSE         NA
                  id replyToUID
1 701141750161784834         NA
2 701140474019577856         NA
3 701138356466483204         NA
4 701137026075140096         NA
5 701137018508722176         NA
6 701134750296227840         NA
                                                                            statusSource
1                Mobile Web (M5)
2 Tweetbot for Mac
3   Twitter for Android
4                     Twitter Web Client
5     Twitter for iPhone
6                     Twitter Web Client
     screenName retweetCount isRetweet retweeted longitude latitude
1      psousa75            3      TRUE     FALSE        NA       NA
2      millerdl            0     FALSE     FALSE        NA       NA
3   diana_nario           50      TRUE     FALSE        NA       NA
4    emjonaitis            0     FALSE     FALSE        NA       NA
5       caryden           41      TRUE     FALSE        NA       NA
6 ArkangelScrap           41      TRUE     FALSE        NA       NA


Now that we have a dataframe, let's dump it into a csv file. Below is what the command is to write the output to a csv file

write.csv(twListToDF(tweets), file = "c:/temp/Tweets.csv")


Here is what it looks like if you open the csv file in Excel

As you can see each column is filled with correct data. How about instead of writing it into a csv file, we write the data into a database?  That is pretty easy as well, we need the RODBC package to accomplish that. You can see that post here: How to store twitter search results from R into SQL Server

What I tweeted this past week, 20151016 edition

Here is some of the interesting stuff I tweeted about this past week

Compile Big Gateway Policy changes in SQL Server 
SQL Server will automatically determine the number of concurrent queries that will be allowed in the Big Gateway depending on the Target Memory. Meaning, SQL Server will allow one query per 25GB of memory up to 80% of target.
The upper limit is only enforced by the medium gateway limit since all queries in big gateway would have first passed through medium gateway. For example, on a system that has 24 physical/48 logical cores and 64GB RAM, the new behavior allows for two large 25GB query compilations up to 48 concurrent compilations (as allowed by the medium gateway) if queries do not need 25GB for compilation. This is a dramatic improvement in number of large compiles compared to earlier versions.


Your first step in dealing with SQLState = S1000 I/O error while writing a BCP data-file 
What should be your first course of action when you encounter this error


Cloud Computing Finally Gets Some Startups
For years, getting into the business of renting out extra computing power through the cloud has been a bit like getting into the business of nuclear power

That formula is changing as cloud startups such as DigitalOcean and Backblaze begin to compete for customers with the likes of Amazon.com, Microsoft, and Google. The startups have managed to underbid the giants in certain markets by keeping expenses relatively low, either by writing their own versions of the software needed to run a cloud or by handcrafting the hardware needed to house one. “All the tools we’re using really pay dividends,” says DigitalOcean Chief Executive Officer Ben Uretsky. “I think that gives us a leg up.”

Computer science is now the top major for women at Stanford University
Computer science has for the first time become the most popular major for female students at Stanford University, a hopeful sign for those trying to build up the thin ranks of women in the technology field.
Based on preliminary declarations by upper-class students, about 214 women are majoring in computer science, accounting for about 30 percent of majors in that department, the California-based university told Reuters on Friday.
Human biology, which had been the most popular major for women, slipped to second place with 208.
If more women majored in technological fields like computer science, advocates say, that could help alleviate the dearth of women in engineering and related professions, where many practitioners draw on computer science backgrounds. 

Some reasons your tSQLt test is not executing

This post explains why some of your tSQLt unit tests might not be running even though it might look correctly.


Finished:  The Devil in the White City: Murder, Magic, and Madness at the Fair That Changed America
This is a book written by Erik Larson about the Chicago World Fair and a serial killer who was active at the same time the fair was happening.


Execution Comparison Tool released with latest SSMS
With the recent release of CTP 2.4, some of you might have noticed something new shipped with SSMS – a Plan Comparison option. The purpose is to provide side-by-side comparison of two different showplans, for easier identification of similarities and changes, that explain the different behaviors for all the reasons stated above, which may include regressions or sudden query performance changes.

The #MH17 investigation points to a revolution in open-source intelligence
A group of Internet sleuths who trawled social media and other open sources in the days after the July 17, 2014, downing and were able to establish that pro-Russian separatists had done the deed. Their remarkable forensic achievements are just the latest example of a citizen-driven open-source intelligence revolution that democratic policy makers must learn to welcome, even when it is a source of embarrassment.

Are you going to watch Ash vs Evil Dead?
Starz is premiering a new show based on the Evil Dead series of movies of the 1980s

BingTweets launched

Microsoft, in cooperation with Federated Media and Twitter, launched its own Twitter search engine today. BingTweets mashes up real-time Twitter search with results from Bing

BingTweets enables you to see deeper, real-time information about the hottest topics on Twitter by fusing Bing search results with the latest tweets. To get started, click on a trending topic, or search for anything that's on your mind in the search box below.



Check it out: http://bingtweets.com/