Using twitter from the R console

Lately I have been messing around with R and I decided to check out the twitteR package to see if I can post from the R console. In order to use twitter from the R console, we need a couple of things:

• Setup OAuth  authentication for twitter
• Install the twitteR package

Setup OAuth  authentication for twitter
As of March 2013 OAuth authentication is required for all Twitter transactions. If you don't already have a OAuth setup, head over to twitter here: https://apps.twitter.com/app/new

Follow the instructions, once you are done, you will see the following 4 items

Consumer Key (API Key)
Consumer Secret (API Secret)

Access Token
Access Token Secret


Install the twitteR package
Now in you R program install the twitteR package
Once the package is installed, it is time to get busy......


 Load the package by executing the following command


library(twitteR)

Now it is time to setup authentication, you do that by using the setup_twitter_oauth command, below is an example, make sure to replace the keys and tokens below with the values you got back when you setup OAuth on twitter



setup_twitter_oauth("API key", "API secret", "Access token", "Access secret")

[1] "Using direct authentication"

If that is all set, we can send a tweet. To update you twitter status, you can use the updateStatus command, this is very simple to use, you pass your status into the function. Here is what it looks like on twitter

updateStatus('testing Tweeting with twitterR package from witin Revolution R Enterprise')
[1] "DenisGobo: testing Tweeting with twitterR package from witin Revolution R Enterprise"

testing Tweeting with twitterR package from witin Revolution R Enterprise

— Denis Gobo (@DenisGobo) February 20, 2016

Here is what it looks like from the console

Of course nobody is doing all of this to update their status. The reason I am playing around with this is because I want to do twitter searches and then store the results in a file or database. So let's do a simple search for the tag #rstats and let's also limit the search to only return 6 results

tweets <- searchTwitter('#rstats', n=6) 
tweets

Here is what we got back, as you can see some of the results end in ...., those have been truncated

[1] "psousa75: RT @rquintino: @Mairos_B #sqlsatportugal session: all about R in #SqlServer 2016 #rstats https://t.co/DHrqIZrz1e"

[[2]]
[1] "millerdl: a quick script to use imgcat in #rstats https://t.co/fpUlgWNX33 https://t.co/AhCCMLewCH"

[[3]]
[1] "diana_nario: RT @KirkDBorne: Useful packages (libraries) for Data Analysis in R: https://t.co/haRKopFyly #DataScience #Rstats by @analyticsvidhya https:…"

[[4]]
[1] "emjonaitis: Hey #rstats tweeps, do you have any readings to recommend on sensitivity analysis? Books/articles/websites all welcome."

[[5]]
[1] "caryden: RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…"

[[6]]
[1] "ArkangelScrap: RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…"


What I really want is to convert the output to a data frame. Luckily the twitteR package has this built in, you can use twListToDF. Here is how to do that

tweets <- searchTwitter('#rstats', n=6) 
twListToDF(tweets)

The output now has a lot more stuff, you can see if it has been retweeted or favorited as well as the latitude, longtitude and more

1                             RT @rquintino: @Mairos_B #sqlsatportugal session: all about R in #SqlServer 2016 #rstats https://t.co/DHrqIZrz1e
2                                                      a quick script to use imgcat in #rstats https://t.co/fpUlgWNX33 https://t.co/AhCCMLewCH
3 RT @KirkDBorne: Useful packages (libraries) for Data Analysis in R: https://t.co/haRKopFyly #DataScience #Rstats by @analyticsvidhya https:…
4                      Hey #rstats tweeps, do you have any readings to recommend on sensitivity analysis? Books/articles/websites all welcome.
5 RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…
6 RT @KirkDBorne: A Complete Tutorial on Time Series Modeling in R: https://t.co/7oI6JKyU4E #MachineLearning #DataScience #Rstats by @Analyti…
  favorited favoriteCount replyToSN             created truncated replyToSID
1     FALSE             0        NA 2016-02-20 20:29:54     FALSE         NA
2     FALSE             0        NA 2016-02-20 20:24:50     FALSE         NA
3     FALSE             0        NA 2016-02-20 20:16:25     FALSE         NA
4     FALSE             0        NA 2016-02-20 20:11:08     FALSE         NA
5     FALSE             0        NA 2016-02-20 20:11:06     FALSE         NA
6     FALSE             0        NA 2016-02-20 20:02:05     FALSE         NA
                  id replyToUID
1 701141750161784834         NA
2 701140474019577856         NA
3 701138356466483204         NA
4 701137026075140096         NA
5 701137018508722176         NA
6 701134750296227840         NA
                                                                            statusSource
1                Mobile Web (M5)
2 Tweetbot for Mac
3   Twitter for Android
4                     Twitter Web Client
5     Twitter for iPhone
6                     Twitter Web Client
     screenName retweetCount isRetweet retweeted longitude latitude
1      psousa75            3      TRUE     FALSE        NA       NA
2      millerdl            0     FALSE     FALSE        NA       NA
3   diana_nario           50      TRUE     FALSE        NA       NA
4    emjonaitis            0     FALSE     FALSE        NA       NA
5       caryden           41      TRUE     FALSE        NA       NA
6 ArkangelScrap           41      TRUE     FALSE        NA       NA


Now that we have a dataframe, let's dump it into a csv file. Below is what the command is to write the output to a csv file

write.csv(twListToDF(tweets), file = "c:/temp/Tweets.csv")


Here is what it looks like if you open the csv file in Excel

As you can see each column is filled with correct data. How about instead of writing it into a csv file, we write the data into a database?  That is pretty easy as well, we need the RODBC package to accomplish that. You can see that post here: How to store twitter search results from R into SQL Server

Started a Today I learned project on Github

I started my own Today I learned project on Github



Today I learned

A collection of concise write-ups on small things I learn day to day across a variety of languages and technologies. These are things that don't really warrant a full blog post. Idea stolen from jbranchaud/til

You can find that project here https://github.com/SQLMenace/til

The reason I did this is because it gives me an opportunity to use Github, all my stuff is usually backend SQL Server code, I also don't do any web or app programming. The reason I like the Today I Learned project is that you can easily see all the stuff that you have learned over time. I will probably mostly add R and Powershell items in the foreseeable future. I am messing mostly with R on my own time and Powershell at work. Once I start diving deeper into SQL Server 2016, I will probably add that stuff as well to my Today I Learned Github project.

What do you think... cool idea or not?


Again, you can find that project here https://github.com/SQLMenace/til

The first 101 stamps issued in the Netherlands

This has been migrated here: http://www.stampio.org/2019/01/the-first-101-stamps-issued-in.html

How I will learn Chinese

I decided to write down what I will use to learn Chinese.in the coming weeks. Here are some of the things that I am currently doing and some things I will be doing in the near future.

Watch videos on Youtube
There are a bunch of videos I have already watched, there are also a bunch that I bookmarked. Here is one of the videos that I have already watched several times

I like Dani Wang's videos and will watch all of them shortly.

Here is a video that I have bookmarked but will watch this week. This video is by Yangyang Cheng, it is a Google hangout showing you the most effective way to learn Mandarin tones, tone pairs

Listen to Podcasts
I downloaded the Learn Chinese | ChineseClass101.com podcast

I have already listened several time to Lesson #1 - What's Your Name in Chinese? It is actually fun and quite interesting. I listen to most podcast at 1.5 speed but always have to remind myself to set it back to 1.0 speed with this language podcast


Visit websites
There are a bunch of sites that I have bookmarked, I have read some of these already and some are bookmarks because those sites were listen on some of these but I did not get enough time to read them all yet. Here is just a small list for you to check out.

Wikihow
How to Learn Mandarin Chinese

Tim Ferriss, 4 hour workweek blog
How to Learn Any Language in 3 Months
How to Learn (But Not Master) Any Language in 1 Hour
12 Rules for Learning Foreign Languages in Record Time — The Only Post You’ll Ever Need

BBC
Real Chinese - For starters
Quick Fix - Essential phrases in Chinese

Lingholic.com
How to learn Chinese

Semanda.com
Printable PDF Mandarin Flashcards


Books
CHINESE in 10 minutes a day
I have the Italian version of this book, it contains the following

• 132-page illustrated workbook
• Full color throughout
• Organized in 25 easy steps, by essential categories
• 150 Sticky Labels for home and office
• Ready-made Flash Cards

This is a real beginners book and is fun to learn from. It will arrive by the end of this week so I will be doing all the other stuff I mentioned in this post instead.

Movies
Watch a movie in Chinese and trying to understand it. I own only 2 movies that are in Chinese: Hero and Crouching Tiger, Hidden Dragon. But luckily for me, my local library has a lot of Chinese movies, here is just a small example of the stuff that they have, there are several of these cases filled with movies.

I won't get to the movies for a couple of months, at this stage there is no point, I probably might recognize 5 words in total. But I will try to watch my first movie in April or so and will let you know how that goes.

Books I read in 2015

This year I read 20 books. Ten of these books were fiction and ten were non-fiction. The Game of Thrones books were a re-read, I already read book 4 and 5 of this series before..

Here  are all the books I read in 2015

A Dance with Dragons (A Song of Ice and Fire) by George R. R. Martin
The most boring book of the series, not much happens, this is also the reason HBO decided to combine book 4 and 5 into one season.

A Feast for Crows: A Song of Ice and Fire (Game of Thrones) by George R.R. Martin
This is better than book 4, but it's not even close to book 3.

The Confident Speaker: Beat Your Nerves and Communicate at Your Best in Any Situation by Harrison Monarth
Got this from work, this book is pretty good, I used some of the techniques from this book to help me prepare for some presentations I had to give.


House of Leaves by Mark Z. Danielewski

The weirdest book I have ever read
One part of this book is about a house which is always changing and somehow the measurements inside are larger  than the outside measurements. There is are video recordings of what happens inside the house to the Navidson family as well as some people who are trying to help. The other part of this book is about Johnny Truant who discovers the manuscript written by Zampanò about the videos mentioned earlier, the Navidson record. I don't want to give you more info about the story, this way I won't spoil anything for you
This book has pages with text in circles, braille, upside down text, mirrored text and much more. Here is also a short video with some of the weirder pages of the book

There are also hidden messages in the book, the word house is always colored blue, some of the references are real, some are made up



To the Last Man: A Novel of the First World War by Jeff Shaara
This is a historical fiction book focusing on World War I. This book is very very good, I would recommend this to anyone. A big part of the book deals with air warfare, you will learn a lot about Red Baron, Manfred von Richthofen and Raoul Lufberry. Another thing you will learn when reading this book is how the soldiers experienced trench warfare.

Becoming Steve Jobs: The Evolution of a Reckless Upstart into a Visionary Leader by Brent Schlender
This book is different than Isaacson book, plenty of quotes and anecdotes that have not been seen before, this book was also more focused on Steve Job's growth over the years.

George Washington's Secret Six: The Spy Ring That Saved the American Revolution by Brian Kilmeade
Learned a lot from this book, for someone who grew up in the Netherlands, this stuff is not taught in school.

Think Like a Freak: The Authors of Freakonomics Offer to Retrain Your Brain by Steven D. Levitt
Similar to freakonomins

Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future by Ashlee Vance
The Edison/Tesla of our time.... This book tells you how Musk grew up, started a bunch of companies and finally wants to be the first human who will travel to Mars. very fascinating character.


Dead Wake: The Last Crossing of the Lusitania by Erik Larson
Excellent book, a non-fiction book that reads like a novel, I couldn't put it down. Amazon's best book of the month for March 2015. Here are just some highlights of what reviewers had to say.

Larson is one of the modern masters of popular narrative nonfiction...a resourceful reporter and a subtle stylist who understands the tricky art of Edward Scissorhands-ing narrative strands into a pleasing story...An entertaining book about a great subject, and it will do much to make this seismic event resonate for new generations of readers."
—The New York Times Book Review

"Larson is an old hand at treating nonfiction like high drama...He knows how to pick details that have maximum soapy potential and then churn them down until they foam [and] has an eye for haunting, unexploited detail."
—The New York Times

"In his gripping new examination of the last days of what was then the fastest cruise ship in the world, Larson brings the past stingingly alive...He draws upon telegrams, war logs, love letters, and survivor depositions to provide the intriguing details, things I didn't know I wanted to know...Thrilling, dramatic and powerful."
—NPR

"This enthralling and richly detailed account demonstrates that there was far more going on beneath the surface than is generally known...Larson's account [of the Lusitania's sinking] is the most lucid and suspenseful yet written, and he finds genuine emotional power in the unlucky confluences of forces, 'large and achingly small,' that set the stage for the ship's agonizing final moments."
—The Washington Post

"Utterly engrossing...Expertly ratcheting up the tension...Larson puts us on board with these people; it's page-turning history, breathing with life."
—The Seattle Times

"Larson has a gift for transforming historical re-creations into popular recreations, and Dead Wake is no exception...[He] provides first-rate suspense, a remarkable achievement given that we already know how this is going to turn out...The tension, in the reader's easy chair, is unbearable..."
—The Boston Globe

Finders Keepers: A Novel by Stephen King
Typical Stephen King story, you can't go wrong with Stephen King.

The Guns of August: The Pulitzer Prize-Winning Classic About the Outbreak of World War I by Barbara W. Tuchman
Very good book about World War I, this book focuses on the first month of the conflict.

The Wright Brothers by David McCullough
Loved this book, anyone who travels by plane should read this book, you will appreciate what these two guys accomplished pretty much by themselves

Seveneves: A Novel by Neal Stephenson
Very interesting book, in my opinion way too long, Neal Stephenson could have cut the book in half and still make a very compelling story. I liked the story a lot, the moon breaks apart and there is only a certain time frame in which to save humanity.

The Devil in the White City: Murder, Magic, and Madness at the Fair That Changed America by Erik Larson
Another book I enjoyed a lot. The most interesting part of the book for me was that you could see what life must have been in Chicago during the time period the book takes place in. The first time we had tall buildings, electricity and light bulbs, the first Ferris wheel. This book really contains two stories, one about the world fair, the other about a mass murdered.

Hitler's Last Day: Minute by Minute by Emma Craigie
Was disapointed in this book because it also covered a bunch of stuff that happened at other time...but I think if this wasn't done, then the book would have been 50 pages or so. Some interesting facts about what happened in the bunker and what it must have been like for the people inside that bunker.

The Girl on the Train by Paula Hawkins
An interesting thriller set in England, this would be a good book to read on the beach or on your commute.


Robopocalypse: A Novel (Vintage Contemporaries) by Daniel H. Wilson
This was pretty good, the robots came alive and decided to eliminate most of the humans. Sounds similar to The Terminator and Skynet but it's nothing like it.

Robogenesis by Daniel H. Wilson
Follows the story of Robopocalypse but I was getting bored towards the end

In the Heart of the Sea: The Tragedy of the Whaleship Essex by Nathaniel Philbrick
Probably my favorite book of the year, Moby Dick was based on this disaster. An excellent story about tragedy and survival at sea. I didn't know much about whaling or whale oil, after reading this book I can only appreciate what a tough job that must have been.


And here are the 5 books I like the most from this list of 20 books

In the Heart of the Sea: The Tragedy of the Whaleship Essex by Nathaniel Philbrick
Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future by Ashlee Vance
To the Last Man: A Novel of the First World War by Jeff Shaara
Dead Wake: The Last Crossing of the Lusitania by Erik Larson
The Wright Brothers by David McCullough

My 2016 new year's resolutions

I decided to put some of my new year's resolutions down in a post. These are non professional goals, here is what my goals are

Learn a new language

I want to learn a new language in 2016. I already know Croatian and Dutch, in addition to those I can understand most of German, I also have some knowledge of Italian and Spanish. The more languages you know, the easier it is to learn another language. To make it more difficult for me, I am picking a non European language. I decided to try to learn Chinese this year. 

When I say learn, what I mean is that I must be able to communicate with a person in Chinese as well as being able to watch a movie in Chinese while understanding 80% of the movie. By Chinese, I mean Mandarin, it would be really funny if I picked a movie in Cantonese and then couldn't understand anything. 

I work with a bunch of people from China and Taiwan, so I can always check with them to see if I am pronouncing words correctly.

If learning the Chinese language goes well, I want to then learn these 4 languages the next 4 years following 2016: Greek, Japanese, Portuguese, Korean. 

Read 30 books

In 2015, I read 20 books, this year I want to read 30 books. I want 5 of those books to deal with American history, I have been living in the US for 20+ years and most of the history books I read were about European history or World War i and World War II history. I also want to read more technical books, so 6 of those books should be technical.

Do a handstand, walk 30 feet on my hands

I can barely hold a handstand for 2 seconds, this year, I want to try to improve on that. My goal is to be able to do a handstand and hold it for 30 seconds. I also want to be able to walk 30 feet on my hands.

Improve on lifts

I would like to be be able to bench press 200 pounds, squat 300 pounds and deadlift 400 pounds by the end of the year. All of the lifts should be raw.

Blog more

I want to write 100 blog posts in 2016, this should not be that difficult, it is only 2 fricking posts per week, but sometimes I don't write anything for 6 months straight. In 2016 this will change, one thing that will make this easier is I will document the progress of my goals, this will be mostly focused on my progress learning the Chinese language.

So there you have it, that is my list, seems very doable and I will keep you and me posted on the progress of these lofty goals.......

What I tweeted this past week, 20151016 edition

Here is some of the interesting stuff I tweeted about this past week

Compile Big Gateway Policy changes in SQL Server 
SQL Server will automatically determine the number of concurrent queries that will be allowed in the Big Gateway depending on the Target Memory. Meaning, SQL Server will allow one query per 25GB of memory up to 80% of target.
The upper limit is only enforced by the medium gateway limit since all queries in big gateway would have first passed through medium gateway. For example, on a system that has 24 physical/48 logical cores and 64GB RAM, the new behavior allows for two large 25GB query compilations up to 48 concurrent compilations (as allowed by the medium gateway) if queries do not need 25GB for compilation. This is a dramatic improvement in number of large compiles compared to earlier versions.


Your first step in dealing with SQLState = S1000 I/O error while writing a BCP data-file 
What should be your first course of action when you encounter this error


Cloud Computing Finally Gets Some Startups
For years, getting into the business of renting out extra computing power through the cloud has been a bit like getting into the business of nuclear power

That formula is changing as cloud startups such as DigitalOcean and Backblaze begin to compete for customers with the likes of Amazon.com, Microsoft, and Google. The startups have managed to underbid the giants in certain markets by keeping expenses relatively low, either by writing their own versions of the software needed to run a cloud or by handcrafting the hardware needed to house one. “All the tools we’re using really pay dividends,” says DigitalOcean Chief Executive Officer Ben Uretsky. “I think that gives us a leg up.”

Computer science is now the top major for women at Stanford University
Computer science has for the first time become the most popular major for female students at Stanford University, a hopeful sign for those trying to build up the thin ranks of women in the technology field.
Based on preliminary declarations by upper-class students, about 214 women are majoring in computer science, accounting for about 30 percent of majors in that department, the California-based university told Reuters on Friday.
Human biology, which had been the most popular major for women, slipped to second place with 208.
If more women majored in technological fields like computer science, advocates say, that could help alleviate the dearth of women in engineering and related professions, where many practitioners draw on computer science backgrounds. 

Some reasons your tSQLt test is not executing

This post explains why some of your tSQLt unit tests might not be running even though it might look correctly.


Finished:  The Devil in the White City: Murder, Magic, and Madness at the Fair That Changed America
This is a book written by Erik Larson about the Chicago World Fair and a serial killer who was active at the same time the fair was happening.


Execution Comparison Tool released with latest SSMS
With the recent release of CTP 2.4, some of you might have noticed something new shipped with SSMS – a Plan Comparison option. The purpose is to provide side-by-side comparison of two different showplans, for easier identification of similarities and changes, that explain the different behaviors for all the reasons stated above, which may include regressions or sudden query performance changes.

The #MH17 investigation points to a revolution in open-source intelligence
A group of Internet sleuths who trawled social media and other open sources in the days after the July 17, 2014, downing and were able to establish that pro-Russian separatists had done the deed. Their remarkable forensic achievements are just the latest example of a citizen-driven open-source intelligence revolution that democratic policy makers must learn to welcome, even when it is a source of embarrassment.

Are you going to watch Ash vs Evil Dead?
Starz is premiering a new show based on the Evil Dead series of movies of the 1980s