Sunday, February 21, 2016

Using twitter from the R console

Lately I have been messing around with R and I decided to check out the twitteR package to see if I can post from the R console. In order to use twitter from the R console, we need a couple of things:

  • Setup OAuth  authentication for twitter
  • Install the twitteR package

Setup OAuth  authentication for twitter
As of March 2013 OAuth authentication is required for all Twitter transactions. If you don't already have a OAuth setup, head over to twitter here:

Follow the instructions, once you are done, you will see the following 4 items

Consumer Key (API Key)
Consumer Secret (API Secret)

Access Token
Access Token Secret

Install the twitteR package
Now in you R program install the twitteR package
Once the package is installed, it is time to get busy......

 Load the package by executing the following command


Now it is time to setup authentication, you do that by using the setup_twitter_oauth command, below is an example, make sure to replace the keys and tokens below with the values you got back when you setup OAuth on twitter

setup_twitter_oauth("API key", "API secret", "Access token", "Access secret")
[1] "Using direct authentication"

If that is all set, we can send a tweet. To update you twitter status, you can use the updateStatus command, this is very simple to use, you pass your status into the function. Here is what it looks like on twitter

updateStatus('testing Tweeting with twitterR package from witin Revolution R Enterprise')
[1] "DenisGobo: testing Tweeting with twitterR package from witin Revolution R Enterprise"

Here is what it looks like from the console

Of course nobody is doing all of this to update their status. The reason I am playing around with this is because I want to do twitter searches and then store the results in a file or database. So let's do a simple search for the tag #rstats and let's also limit the search to only return 6 results

tweets <- searchTwitter('#rstats', n=6) 

Here is what we got back, as you can see some of the results end in ...., those have been truncated

[1] "psousa75: RT @rquintino: @Mairos_B #sqlsatportugal session: all about R in #SqlServer 2016 #rstats"

[1] "millerdl: a quick script to use imgcat in #rstats"

[1] "diana_nario: RT @KirkDBorne: Useful packages (libraries) for Data Analysis in R: #DataScience #Rstats by @analyticsvidhya https:…"

[1] "emjonaitis: Hey #rstats tweeps, do you have any readings to recommend on sensitivity analysis? Books/articles/websites all welcome."

[1] "caryden: RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: #MachineLearning #DataScience #Rstats by @Analyti…"

[1] "ArkangelScrap: RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: #MachineLearning #DataScience #Rstats by @Analyti…"

What I really want is to convert the output to a data frame. Luckily the twitteR package has this built in, you can use twListToDF. Here is how to do that

tweets <- searchTwitter('#rstats', n=6) 

The output now has a lot more stuff, you can see if it has been retweeted or favorited as well as the latitude, longtitude and more

1                             RT @rquintino: @Mairos_B #sqlsatportugal session: all about R in #SqlServer 2016 #rstats
2                                                      a quick script to use imgcat in #rstats
3 RT @KirkDBorne: Useful packages (libraries) for Data Analysis in R: #DataScience #Rstats by @analyticsvidhya https:…
4                      Hey #rstats tweeps, do you have any readings to recommend on sensitivity analysis? Books/articles/websites all welcome.
5 RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: #MachineLearning #DataScience #Rstats by @Analyti…
6 RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: #MachineLearning #DataScience #Rstats by @Analyti…
  favorited favoriteCount replyToSN             created truncated replyToSID
1     FALSE             0        NA 2016-02-20 20:29:54     FALSE         NA
2     FALSE             0        NA 2016-02-20 20:24:50     FALSE         NA
3     FALSE             0        NA 2016-02-20 20:16:25     FALSE         NA
4     FALSE             0        NA 2016-02-20 20:11:08     FALSE         NA
5     FALSE             0        NA 2016-02-20 20:11:06     FALSE         NA
6     FALSE             0        NA 2016-02-20 20:02:05     FALSE         NA
                  id replyToUID
1 701141750161784834         NA
2 701140474019577856         NA
3 701138356466483204         NA
4 701137026075140096         NA
5 701137018508722176         NA
6 701134750296227840         NA
1                Mobile Web (M5)
2 Tweetbot for Mac
3   Twitter for Android
4                     Twitter Web Client
5     Twitter for iPhone
6                     Twitter Web Client
     screenName retweetCount isRetweet retweeted longitude latitude
1      psousa75            3      TRUE     FALSE        NA       NA
2      millerdl            0     FALSE     FALSE        NA       NA
3   diana_nario           50      TRUE     FALSE        NA       NA
4    emjonaitis            0     FALSE     FALSE        NA       NA
5       caryden           41      TRUE     FALSE        NA       NA
6 ArkangelScrap           41      TRUE     FALSE        NA       NA

Now that we have a dataframe, let's dump it into a csv file. Below is what the command is to write the output to a csv file

write.csv(twListToDF(tweets), file = "c:/temp/Tweets.csv")

Here is what it looks like if you open the csv file in Excel

As you can see each column is filled with correct data. How about instead of writing it into a csv file, we write the data into a database?  That is pretty easy as well, we need the RODBC package to accomplish that. You can see that post here: How to store twitter search results from R into SQL Server

Monday, February 15, 2016

Started a Today I learned project on Github

I started my own Today I learned project on Github

Today I learned

A collection of concise write-ups on small things I learn day to day across a variety of languages and technologies. These are things that don't really warrant a full blog post. Idea stolen from jbranchaud/til

You can find that project here
The reason I did this is because it gives me an opportunity to use Github, all my stuff is usually backend SQL Server code, I also don't do any web or app programming. The reason I like the Today I Learned project is that you can easily see all the stuff that you have learned over time. I will probably mostly add R and Powershell items in the foreseeable future. I am messing mostly with R on my own time and Powershell at work. Once I start diving deeper into SQL Server 2016, I will probably add that stuff as well to my Today I Learned Github project.

What do you think... cool idea or not?

Again, you can find that project here

Sunday, January 24, 2016

The first 101 stamps issued in the Netherlands

These are the first 101 stamps issued in the Netherlands. These stamps were issued between 1852 and 1913. To determine the first 101 stamps, I used the NVPH (Nederlandsche Vereeniging van Postzegelhandelaren) catalog. I also list the Scott catalog numbers after the NVPH numbers, this way you can quickly look up these stamps if you have either of those two catalogs.

1852 King William III

NVPH 1-3
Scott 1-3

William III (Willem Alexander Paul Frederik Lodewijk, anglicised: William Alexander Paul Frederick Louis; 19 February 1817 – 23 November 1890) was King of the Netherlands and Grand Duke of Luxembourg from 1849 until his death in 1890. He was also the Duke of Limburg from 1849 until the abolition of the duchy in 1866.
William was the son of King William II and Anna Pavlovna of Russia. On the abdication of his grandfather William I in 1840, he became the Prince of Orange. On the death of his father in 1849, he succeeded as King of the Netherlands.

1864 King William III

NVPH 4-6
Scott 4-6

1867 - 1868 King William III

NVPH 7-12
Scott 7-12

1869 - 1871 Coat Of Arms

NVPH 13-18
Scott 17-22

1872 - 1888 King William III

NVPH 19-29
Scott 23-33

1976 - 1894 Numerals

NVPH 30-33
Scott 34-37

1891 - 1894 Princess Wilhelmina

NVPH 34-48
Scott  40-54

1898 Queen Wilhelmina Coronation

Scott 83 type I
This stamp was issued for Queen Wilhelmina's coronation. The number 1 is thicker and wider in this stamp than the stamp issued in 1899.

Wilhelmina (Wilhelmina Helena Pauline Maria; 31 August 1880 – 28 November 1962) was Queen of the Kingdom of the Netherlands from 1890 to 1948. She reigned for nearly 58 years, longer than any other Dutch monarch. Her reign saw World War I and World War II, the economic crisis of 1933, and the decline of the Netherlands as a major colonial power.

1899 - 1913 Numerals

NVPH 50-55
Scott 55-60

1899 - 1921 Queen Wilhelmina

NVPH 56-76
Scott 61-82

1899-1905 Queen Wilhelmina

NVPH 77-80
Scott 83-86

1924 Queen Wilhelmina tête-bêche

NVPH 69b, 69c
Scott 66a

In philately, tête-bêche is a joined pair of stamps in which one is upside-down in relation to the other, produced intentionally or accidentally. Like any pair of stamps, a pair of tête-bêches can be a vertical or a horizontal pair. In the case of a pair of triangular stamps, they cannot help but be linked "head-to-tail".

1922 Queen Wilhelmina

Scott 110
This is a redrawn version of the 1899 edition, the lines behind the Queen's head are wider apart.

1923 Queen Wilhelmina

NVPH 82-83
Scott 111-112
Because of a strike at the printer, these were issued imperforated

1906 Tuberculosis


NVPH 84-86
Scott B1-B3
The design is symbolical of the four chief means for combating tuberculosis: light, water, air and food

1907 De Ruyter

NVPH 87-89
Scott 87-89
Michiel Adriaenszoon de Ruyter and fleet. De Ruyter (24 March 1607 – 29 April 1676) was a Dutch admiral. He is the most famous and one of the most skilled admirals in Dutch history, most famous for his role in the Anglo-Dutch Wars of the 17th century. He fought the English and French and scored several major victories against them, the best known probably being the Raid on the Medway

1913 Centenary of Dutch independence

NVPH 90-101
Scott 90-101
Depicted on these stamps are: William I, William II, William III and Wilhelmina

After the liberation of the Netherlands from France by Prussian and Russian troops in 1813, a provisional government took over the country. It was headed by a triumvirate of three Dutch noblemen, Frans Adam van der Duyn van Maasdam, Leopold of Limburg Stirum and Gijsbert Karel van Hogendorp. This Driemanschap formally took control over the liberated country on 20 November, and declared the Principality of the United Netherlands a day thereafter


Here is also a video with all these stamps.... makes a nice screensaver.....