A quick and easy way to count the percentage of nulls without a where clause in PostgreSQL

This came up the other day, someone wanted to know the percentage of NULL values in a column

Then I said "I bet you I can run that query without using a NULL in the WHERE clause, as a matter of fact, I can run that query without a WHERE clause at all!!"

I created a SQL Server version of this post here: How to count NULLS without using IS NULL in a WHERE clause, this is the PostgreSQL version


To start, first create this table and verify you have 9 rows

CREATE TABLE foo(bar int);
INSERT INTO foo values(1),(null),(2),(3),(4),
 (null),(5),(6),(7);

SELECT * FROM foo;

Here is what the output should look like

To get the NULL values and NON NULL values, you can do something like this

SELECT COUNT(*) as CountAll FROM foo WHERE bar IS NOT NULL;
SELECT COUNT(*) as CountAll FROM foo WHERE bar IS  NULL;

However, there is another way

Did you know that COUNT behaves differently if you use a column name compared to when you use *

Take a look

SELECT COUNT(*) as CountAll, 
  COUNT(bar) as CountColumn
FROM foo;

If you ran that query, the result is the following

CountAll    CountColumn
----------- -----------
9           7

Here is what it looks like in pgAdmin3

Let's see what the PostgreSQL documentation has to say


COUNT(*) number of input rows

COUNT(expression) number of input rows for which the value of expression is not null

This is thus indeed documented behavior, count(*) counts all rows, count(column) skips the rows with NULLS

So now, lets change our query to return the percentage of non null values in the column

Here is what the query looks like

SELECT COUNT(*) as CountAll, 
  COUNT(bar) as CountColumn, 
  (COUNT(bar)/COUNT(*))*100 as PercentageOfNonNullValues 
FROM foo;

Running that gives us this output

CountAll    CountColumn percentageOfNonNullValues
----------- ----------- ---------------------------------------
9           7                  0

Here is what it looks like in pgAdmin3

As you can see, we have a 0 in the percentage column. The reason is that if you do math with integers, you will get an integer back


We can fix this easy by multiplying one of the integers by 1.0, this will convert the integer to a numeric data type . So instead of this line

 (COUNT(bar)/COUNT(*))*100 as PercentageOfNonNullValues 

We will change it to be like this

(COUNT(bar)*1.0/COUNT(*))*100 as PercentageOfNonNullValues

Here is the changed query

SELECT COUNT(*) as CountAll, 
  COUNT(bar) as CountColumn, 
  (COUNT(bar)*1.0/COUNT(*))*100 as PercentageOfNonNullValues 
FROM foo;

Here is the output

CountAll    CountColumn percentageOfNonNullValues
----------- ----------- ---------------------------------------
9           7           77.7777777777800


Here is what it looks like in pgAdmin3

Instead of doing * 1.0, you can also explicitly cast the integer like this

(CAST (COUNT(bar) as numeric(20,10))/COUNT(*))*100 as PercentageOfNonNullValues

The query now looks like this

SELECT COUNT(*) as CountAll, 
  COUNT(bar) as CountColumn, 
  (CAST (COUNT(bar) as numeric(20,10))/COUNT(*))*100 as PercentageOfNonNullValues
FROM foo;

Here is what it looks like in pgAdmin3

That's it for this short post.. hopefully you knew this, if not, then you know it now  :-)

The Vessel, a honeycomb shaped structure that will be a great tourist destination at Hudson Yards in New York City

I took a stroll at the high-line in NYC yesterday and noticed that the Vessel was open to the public


The Vessel is a  copper-colored steel structure comprises 54 interconnecting flights of stairs, 2,500 steps, and 80 landings, it was designed by Heatherwick Studio's. The Vessel rises to a height of 150 ft (45 m) offering views of the city and the Hudson River.

Here is what it looks like




Here it is on the map, it's right at the end/start of the highline and also just a block or so from the Jacob K. Javits Convention Center. I think a lot of visitors to the Jacob Javits Center will stop by

The first thing you will notice is how shiny the Vessel is... in place when the sun hits it..it's like hitting a mirror. I wonder how soon before it turns green because of oxidation. I wouldn't want to be the person who has to polish this copper behemoth




The second thing you need to know is that you can't go inside the Vessel without a ticket. Tickets are free, I picked one up in front of the Vessel near the building right in front of it. I did have to wait 30 minutes to get in, the tickets are only valid for 1 hour

The third thing you will notice once you enter the Vessel is a blue light and a crowd of people near it

You will see something like this

 

Or something like this


 

That spot with the blue light is the selfie station, you put your phone down in the middle, then you stand there while a selfie is taken

I decided not to do that since I don't take selfies, but here is the view from the bottom when you take a normal shot






Here you can see all the stairs, it's nice to walk all over the structure, you get nice views of the city.


 

If for some reason, you can't take the stairs, there is an elevator as well. The thing you see in the middle in the pic below is where the elevator travels

Here is a pic I took of people on the top of the Vessel, it was very windy yesterday, I stayed on the one below the top



I took a picture of these path trains

 

TWID March 30, 2019: Tsundoku, DevOps, robots, turing award,gps drifting

This is a post detailing some stuff I did, learned, posted and tweeted this week, I call this TWID (This week in Denis). I am doing this mostly for myself... a kind of an online journal so that I can look back on this later on. Will use the label TWID for these

This Week I Learned

Tsundoku
Tsundoku (積ん読) is acquiring reading materials but letting them pile up in one's home without reading them. The term originated in the Meiji era (1868–1912) as Japanese slang. It combines elements of tsunde-oku (積んでおく, to pile things up ready for later and leave) and dokusho (読書, reading books). It is also used to refer to books ready for reading later when they are on a bookshelf. As currently written, the word combines the characters for "pile up" (積) and the character for "read" (読).

OK, so I have a bunch of books that I bought years ago and never finished. Some of these I never started. So what I decided to do is take 1 hour each day before going to work, grab 3 books and read 20 minutes per book.  I also have some books in the office, I am reading 20 minutes each time when I go to the office, this is currently twice a week. The book I am reading in the office is Pragmatic Thinking and Learning, I am 40% done with that book

I already finished the book War Of Art, am 25% done with Tools of Titans and just started Python Tricks

Here is a pic of the books

The books in that picture are the following

Tools of Titans: The Tactics, Routines, and Habits of Billionaires, Icons, and World-Class Performers (started)
Tribe of Mentors: Short Life Advice from the Best in the World
War Of Art  (finished)
Wired To Eat
Python Tricks the book: A Buffet of Awesome Python Features (started)
Fluent Python: Clear, Concise, and Effective Programming
Book of Five Rings
How Linux Works, 2nd Edition: What Every Superuser Should Know


I continued my DevOps training and like last week, I decided to put my note at the end of this post, this way I can easily look it up again

This Week I Tweeted

Three ‘Godfathers of Deep Learning’ Selected for Turing Award

Three computer scientists who laid the foundations for many of the recent advances in artificial intelligence are being honored with this year’s Turing Award, considered the field’s highest accolade.

Geoff Hinton, an emeritus professor at the University of Toronto and a senior researcher at Alphabet Inc.’s Google Brain, Yann LeCun, a professor at New York University and the chief AI scientist at Facebook Inc., and Yoshua Bengio, a professor at the University of Montreal as well as co-founder of AI company Element AI Inc., will share this year’s award, which is given annually by the Association for Computing Machinery.

The three winners will split a $1 million prize that comes with the award, which is currently underwritten by Google

For a list of previous winners see here: https://en.wikipedia.org/wiki/Turing_Awardhttps://en.wikipedia.org/wiki/Turing_Award



Was MongoDB Ever the Right Choice?

I was reading a post recently about Red Hat removing MongoDB support from Satellite (and yes, some folks say it is because of the license changes). It made me think how often over the last few years I’ve seen post after angry post about how terrible MongoDB is and how no one should ever use. However, in all this time, MongoDB has become a much more mature product. So what happened? Did all of the hate truly come from mistakes made in the early implementation/marketing of MongoDB? Or is the problem that people are blaming MongoDB for their own lack of efforts when evaluating if it was a good fit?

Read and find out.....



Australia Is Drifting So Fast GPS Can't Keep Up

Australia is not quite where you think it is. The continent has shifted by 4.9 feet since the last adjustment was made to GPS coordinates in 1994, reports the New York Times.

A significant correction must be made by the end of the year for navigation technology to keep working smoothly.

This is interesting, I  about drift because undersea cables will break, but never had an idea that it was this fast that they have to adjust GPS

Boston Dynamics’ new robot stacks boxes 

It’s a “mobile manipulation robot” designed for the logistics sector. It can autonomously stack and unstack boxes onto and off pallets, and shift them onto conveyor belts. It uses an onboard vision system to track which objects go where, and to judge how to grasp and place each box. It uses a robotic technique called “force control” to nestle each box up against its neighbors. It can handle weights of up to 15 kilograms (33 pounds.)

And that is how it all starts... wondering when they will rename the company to SkyNet  :-)

Some cool stuff you might enjoy

Awesome Newsletters
A curated list of awesome newsletters.

Inspired by the awesome-* trend on GitHub.

There is something here for everyone




The most populous cities in the world from 1500 to 2018   

This is amazing, make sure to watch the whole animation..it's like a race....

Some DevOps notes

This is for me.. not for you.... but feel free to take a look...although it won't probably make a lot of sense if you see these terms for the first time



Top 10 books  from 10 to number 1
Visible Ops
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
Release It!: Design and Deploy Production-Ready Software
Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale
Lean Software Development: An Agile Toolkit 
Web Operations: Keeping the Data on Time
The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services
The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations
Leading the Transformation: Applying Agile and DevOps Principles at Scale
The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win


Devopsweekly.com mailinglist

Wall of Confusion
Impedance mismatch between teams

Blameless postmortems
---------------------------------------
It's a meeting
within 48 hours of incident
run by a 3rd party

Have everything in a timeline. Not here to blame a person but making sure this doesn't happen again

Trasnparent Uptime
-------------------------
Admit failure
Sound like a human.. no doublespeak
Have a communication channel
Be authentic


Trust blockers
--------------------
Lack of Context
Conflicting goals


Open it Up
-----------------------
Chat rooms
Wiki pages
Source code (to read)
Infrastructure
Monitoring Tools
Ticket Tracker

Chatops
USe something like chat or teams instead of emails


Shared responsibility
-------------------------------
Feedback
Automation
Team culture
No Silos



Shadow IT/Bimodal IT
Groups deliberate bypassing processes to get things done

Devs responsibe for failed deployment after code checkin

Changing your ways of how you do work.... might be hard..but it will pay off


Conway's law
----------------------------
organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.

Eric S. Raymond, an open-source advocate, restated Conway's law in The New Hacker's Dictionary
James O. Coplien and Neil B. Harrison stated:

If the parts of an organization (e.g., teams, departments, or subdivisions) do not closely reflect the essential parts of the product, or if the relationship between organizations do not reflect the relationships between product parts, then the project will be in trouble ... Therefore: Make sure the organization is compatible with the product architecture.


Westrum Model
--------------------------------
Pathological (Power Oriented)
Bureaucratic (Rule Oriented)
Generative (Performance Oriented)



Management Best Practices
-----------------------------------
Independent, cross-functional teams
People first
Agile, lean processes



Kaizen
---------------------------
Small change  aka continues improvement
Good processes bring good results
Gemba (Go see for yourself).. (the real pace or Locus).. meaning look at the code or the system or factory floor
Speak with data, manage by facts  (scientific method)
Take action to contain and correct root causes (The 5 whys, people don't fail processes do)
Work as a team
Kaizen is everubody's business



======================================================
------------------------------------------------------
======================================================


Waterfall --> Iterative
DevOps has strong roots in Agile but is not just agile


Lean, 7 principles
1 Eliminate Waste
2 Amplify Learning
3 Decide as late as possible
4 Decide as fast as possible
5 Empower the team
6 Build in integrity
7 See the whole picture


Waste
Muda: work that absorbs resources but adds no value
Muri: Unreasonable work imposed on workers and machines
Mura: Work coming in unevenly instead of a constant or regular flow


The 7 Wastes Of Software
1 Partially work done
2 Extar features
3 Relearning
4 Handoffs
5 Delays
6 Task Switching
7 Defects


Build Measure Learn
-----------------------
Build minimum viable product
Measure the outcome and internal metrics
Learn about you problem and your solution
Repeat, go deep where it's needed


Concept to cash: from idea to it's realization, including everything needed to get it to customer



ITSM (Information Technology Service Management)

ITIL (Information Technology Infrastructure Library)
1 Service  Strategy
2 Service  Design
3 Service  Transition
4 Service  Operation


ITIL 2007... 2000 pages long...  The Visible OPS handbook (only 100 pages)




Use ITIL but make it lean



======================================================
------------------------------------------------------
======================================================



Infrastructure as code
Everything programmatically, no UI, same code runs on test, stage, UAT and production



Servers are cattle not pets.. deploy in mass, not handcrafted



Provisioning: Making a server ready for operation..including OS, network connectivity and system services
Deployment: Automatically deploying and upgrading applications on a server
Orchestration: Performing coordinated operations across multiple systems
Configuration Management: Management of change control after initial provision, maintaining and upgrading apps and app dependencies

Imperative/procedural: Commands necessary to produce a desired state are defined and executed
Declaritive/Functional: A desired state is defined, relying on the tool to configure a system to match that state
Idempotent: The ability to execute repeatedly, resulting in the same outcome
Self Service: The ability for an end user to initiate a process without having to go through other people



Canary (staged) Deployment Pattern
Upgrade 1 server, see how it works before upgrading the rest

Blue Green deployment
Two environment, code is pushed to one, then environments are swapped, variant like cluster immune system deployment

Containers making things easier to deploy, golden image is coming back now since you can just packe it up as a container.

Terraform.. used so that you can use setting across Azure and AWS



---------------------------------------------------------------

Continuous Deploy
Continuous Delivery
Continuous Integration


Time to market goes down... high performing IT orgs can deploy on demand, some 10 or 100 times a day

“Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.” quote by W. Edwards Deming


With Continuous Delivery you can much easier see what change caused a degradation of an application, with a quarterly release there are so many changes, it is difficult to find out what caused the degradation

The goal of continuous integration is that working software is in a working state all the time quote by Jez Humble

6 practices
Builds should pass the coffee test (should be less than 5 minutes)
Commit really small bits
Don't leave the build broken
Use a trunk based development flow
No crappy tests, fix your tests
The build should return a status, a log and an artifact


Only build artifacts once..immutable, should not change... this creates trust.. underlying bits did not change
Stop deploy if a step fails, notify teams


Test Unit testing
Code hygiene  linters, formatters, banned functions check
Integration testing



TDD/BDD/ATDD

TDD
State desired outcome as test
write code to pass test
Repeat

BDD
Work with stakeholders
Describe business functionality
Tests are based on natutal language descriptions

ATDD (acceptance test driven development)
End user perpective
Use case automated testing
testing is continous during development


3 ways to get around slow tests... so you don't violate your coffee rule
use nonblocking tests
use time scheduled tests
use monitoring


MTTR
Mean Time To Recovery


Cascading failure pattern: Cascading failure is number one threat to stability, one layer can choke out other layers
Circuit breaker pattern: If app see an issue with one specific set of calls redirect to other part.. minimizes outage risk



The twelve-factor methodology
https://12factor.net/

I. Codebase
One codebase tracked in revision control, many deploys

II. Dependencies
Explicitly declare and isolate dependencies

III. Config
Store config in the environment

IV. Backing services
Treat backing services as attached resources

V. Build, release, run
Strictly separate build and run stages

VI. Processes
Execute the app as one or more stateless processes

VII. Port binding
Export services via port binding

VIII. Concurrency
Scale out via the process model

IX. Disposability
Maximize robustness with fast startup and graceful shutdown

X. Dev/prod parity
Keep development, staging, and production as similar as possible

XI. Logs
Treat logs as event streams

XII. Admin processes
Run admin/management tasks as one-off processes



If it hurts, do it more often. The best way to avoid failure is to fail constantly

Monitoring
------------------
Service performance/uptime
software component metrics
system metrics
app metrics
performance
security

5 Principles of log data
Do not collect log data if you never plan to use it
Retain log data for as long as it can be used
Log all you can, alert only what you must respond to, define log levels
Don't make logging more available than production stack
Make log changes as needed




Future
Containers and Serverless (serverless might only run 1 invocation)

Security
100:10:1  (Dev:Ops:Sec)
100 developers, 10 Ops, 1 Security person


The Rugged Manifesto: https://ruggedsoftware.org/

I am rugged and, more importantly, my code is rugged.
I recognize that software has become a foundation of our modern world.
I recognize the awesome responsibility that comes with this foundational role.
I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended.
I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic, and national security.
I recognize these things - and I choose to be rugged.
I am rugged because I refuse to be a source of vulnerability or weakness.
I am rugged because I assure my code will support its mission.
I am rugged because my code can face these challenges and persist in spite of them.
I am rugged, not because it is easy, but because it is necessary and I am up for the challenge.

TWID March 24, 2019: Gout, Music Biz, Space 2.0, Luke Starkiller, space angels, DevOps Notes

This is a post detailing some stuff I did, learned, posted and tweeted this week, I call this TWID (This week in Denis). I am doing this mostly for myself... a kind of an online journal so that I can look back on this later on. Will use the label TWID for these


Didn't do a TWID post for a couple of weeks, will continue regularly from this week on. Had an issue with my foot... still not sure if I sprained my toe or if it was gout. Didn't run for 3 weeks but started to run again this week. It feels good to run again in nature. This morning I heard so many woodpeckers

This Week I Learned

Finished the book Space 2.0: How Private Spaceflight, a Resurgent NASA, and International Partners are Creating a New Space Age
Some interesting stuff, I like how the cost went down a lot after SpaceX, Blue Horizon and others made the government contractor

Started on  the book Loonshots: How to Nurture the Crazy Ideas That Win Wars, Cure Diseases, and Transform Industries

Listened to the Tim Ferriss podcast with Safi Bahcall and in that podcast Safi mentioned that the 1st draft of Star Wars was so bad, no studio wanted it
It had these characters: Luke and Windy Starkiller.  Lol can you imagine those names having made it into the movie

Summary of the Original Script of “The Star Wars”: https://www.starwarz.com/starkiller/summary-of-the-original-script-of-the-star-wars/




Listened tot the final episode of East Coast vs West Coats Business Wars

Learned some stuff about how it is changing how artists get paid in the music industry. Artists used to be paid by album, now they are paid by the song, 1500 streams is about $9, this is cut between label/produces/songwriter. Since artists are paid by song.. , songs are getting shorter and an album will have more songs. So if you listen through an album, you are listening to more songs.  Artist are also putting the chorus at the start of the song to hook you in, because if you skip before 30 seconds or so, the play doesn't count.



Gompertz function
The Gompertz curve or Gompertz function, is a type of mathematical model for a time series and is named after Benjamin Gompertz (1779-1865). It is a sigmoid function which describes growth as being slowest at the start and end of a given time period

One example: Mobile phone uptake, where costs were initially high (so uptake was slow), followed by a period of rapid growth, followed by a slowing of uptake as saturation was reached.


The Fertility Doctor’s Secret
Her husband had given her a DNA test for Christmas because she was interested in genealogy. Her heritage turned out to be exactly what she had thought—Scottish, with English, Irish, and Scandinavian mixed in—and she never bothered to click on the link that would show whether anyone on the site shared her DNA.

Apparently she did have relatives on Ancestry.com—and not just distant cousins. The people now sending her messages said they were Cline’s secret biological children. They said their parents had also been treated by Cline. They said that decades ago, without ever telling his patients, Cline had used his own sperm to impregnate women who came to him for artificial insemination.

According to her DNA, Woock, too, was one of his children.

This so messed up, and with more and more people doing DNA testing, I think this will only increase. 

But there is a person who has 600 offspring, his name is Bertold_Wiesner


Space Angels

Space Angels is Venture Capital firm, they invest in SpaceX as well as the ones in the list below and more

Accion Systems
Accion Systems is developing revolutionary ion beam propulsion technologies for satellites, that are light, powerful, and affordable.


Analytical Space
Analytical Space is developing a cost-effective, high-throughput satellite data relay service using Cubesats with laser downlink capabilities.

Astrobotic Technology
Astrobotic will be the first private company to regularly deliver customer payloads and communication services to the Moon's surface.


Atlas Space Operations
ATLAS' network of RF satellite ground stations will offer reliable delivery of big data from LEO satellites at one-third of the cost of legacy providers.


Because Learning
Because Learning is an interactive STEM platform that enables any school to run experiments from Earth to space, through the Spire satellite network.


Bridgesat
BridgeSat is developing an optical communications network that offers secure delivery of big data from LEO at low cost and high speeds.

NanoRacks
NanoRacks is the leading commercial provider of hardware and services in low-Earth orbit for microgravity research and space station utilization.


Took some DevOps trainging this week... wanted to save the notes I took.. so I put them at the end of this post :-)

This Week I Tweeted

The lawyers who took on Big Tobacco are aiming at Realtors and their 6% fee 

Homeowners who are ready to sell their properties usually hire a real-estate agent to represent them by staging the home, photographing it, adding it to the MLS, marketing it, and showing it to prospective buyers. Sellers agree to pay that person a commission on the selling price of the home. That commission has traditionally been known as the “6%,” but it’s a little more complicated than that.

Sellers can really only negotiate with the agent they’ve hired, while agents representing buyers are generally assured of a standard 3% commission. That means that a seller’s agent who’s willing to negotiate, or one that works for a discount brokerage like Redfin RDFN, +2.93%  , will be paid less than a buyer’s agent.

Buyers can choose to be represented by an agent, or to go without one – but in any case, all commission money for both sides of the deal is always paid by the seller, thanks to a 1996 NAR rule known as the “Buyer Broker Commission Rule.”

I always thought it was strange that listing a 100K house and a 2 Million house would give you so much more money for essentially the same amount of work. Of course selling a million dollar home takes more time. Then there are people who will talk to an agent...do the house tours and not put a bid down with that agent.  If this prevails.. I wonder if the million dollar listing tv show will still be around?

Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years

Hundreds of millions of Facebook users had their account passwords stored in plain text and searchable by thousands of Facebook employees — in some cases going back to 2012, KrebsOnSecurity has learned. Facebook says an ongoing investigation has so far found no indication that employees have abused access to this data.

Facebook is probing a series of security failures in which employees built applications that logged unencrypted password data for Facebook users and stored it in plain text on internal company servers. That’s according to a senior Facebook employee who is familiar with the investigation and who spoke on condition of anonymity because they were not authorized to speak to the press.

This was maybe some logging, but still, this is bad


New Jersey becomes second state to ban cashless shops and restaurants

On Monday, New Jersey Governor Phil Murphy signed a bill banning cashless retail stores and restaurants in the Garden State. Murphy's signature makes New Jersey the second state in the US to ban cashless stores, after Massachusetts banned them in 1978.

More recently, New Jersey's move follows that of Philadelphia, which banned cashless stores earlier this month. Philadelphia's legislation was a reaction to a growing number of stores that only accept credit cards or require customers to pay with an app, like Amazon's new Amazon Go stores.

Ha, I ran into this the other day in Manhattan. Left the office with a $20 in my pocket, went to this place named Dos Torros with my co-worker, ordered a burrito and then couldn't pay, had to have my co-worker pay for me and then had to pay him back.  My main reason yo pay with cash in restaurants and smaller shops is that I don't want my credit card to be hacked, my next reason is privacy.

Also some people with bad credit might now have credit cards.. how would they pay?

I welcome this law.

Some cool stuff you might enjoy

Ten Lessons I Learned While Teaching Myself to Code

The following is a guest post by Clive Thompson (@pomeranian99), a journalist who’s written about technology and science for two decades. Clive is a longtime contributing writer for the New York Times Magazine and a columnist for Wired.

In his guest post, Clive outlines the most important lessons he learned teaching himself to code after interviewing 200+ programmers for his new book Coders: The Making of a New Tribe and the Remaking of the World.

So, you want to learn to code.

Join the club! We live in a time when, as the venture capitalist Marc Andreessen famously put it, “…software is eating the world.” So the people who know how to program are in a catalytic spot; they can make things happen. Maybe you’ve watched this from the sidelines and thought: Huh. Could I learn to do that? Perhaps you’re out of school; maybe you can’t afford either the money or the time to go back and do a four-year degree in computer science. You’ve seen a zillion of these online tutorials in coding. Could you just sort of, well, teach yourself?

The short answer is: Sure you can.

The longer answer is… the rest of this essay.

Some interesting tidbits


Some numbers that you will know by heart if you have been working with SQL Server for a while

I was troubleshooting a deadlock the other day and it got me thinking.... I know the number 1205 by heart and know it is associated to a deadlock.  What other numbers are there that you can associate to an event or object or limitation. For example 32767 will be known by a lot of people as the database id of the ResourceDb, master is 1, msdb is 4 etc etc.

A fun and quick posts with some numbers that most SQL Server peeps probably know



Some DevOps notes I took
Had some DevOps training, took some notes, didn't know where to save them, so put it here


CAMS
-------------
Culture
Automation
Measurement
Sharing


DevOps Principles 3 ways
----------------------------------------------
System Thinking  (concept to cash)  aka overall view
Amplified feedback loop  (later you find the bug.. the more it costs to fix)
Work culture to allow for learning and continues experimentation (fail fast, working code wins..sharing..)


Five Methodologies
------------------------------------
People over process over tools
Continues Delivery
Lean management (small batches, progress limits, feedback loops, visualization)
Visible Ops Change Control (repeatable build process, manage dependencies, eliminate fragile artifacts, continues improvement)
Infrastrucure as code




10 practices for DevOps success
------------------------
Incident command system
Developers on call
Status pages (transparent uptime)
Blameless postmortems
Embedded teams
The Cloud
Andon Cords (anyone can stop the release, this way something doesn't go to prod)
Dependency Injection (Inversion of Control)
Blue/Green deployment (2 system, 1 live, the other is not..deploy changes and shift traffic towards it, if problems shift back to prev version)
Chaos Monkey (trash servers occasional so that you can code for it and be prepared in case it happens out of your control.. came from Netflix)


DevOps Tools ... cart or horse?
----------------------------
Toolchain... that works together
Should be programmable.. no UI tools
Verifiable
Well behaved (from dev and operations point of view)
Write own tools if you need to....

TWID March 3, 2019: Death by GPS, Python tool vs code, Alexa + Jenkins, Job Red Flags, Game of Thrones Fantasy League

This is a post detailing some stuff I did, learned, posted and tweeted this week, I call this TWID (This week in Denis). I am doing this mostly for myself... a kind of an online journal so that I can look back on this later on. Will use the label TWID for these

This Week I Learned

Finished the Pluralsight course Applying SQL Server 2016 Features to Real-world Data Problems by Ana Voicu

Death by GPS refers to the death of people attributable, in part, to following GPS directions or GPS maps. Death by GPS has been noted in several deaths in Death Valley, California; a lost hiker at Joshua Tree National Park in southeastern California; and incidents in Washington State, Australia, England, Italy and Brazil.

Got to the wikipedia entry after reading some comments on this link on hackernews: The Hunt for the Death Valley Germans

This Week I Tweeted

Python in Visual Studio Code – February 2019 Release

Test Explorer
This release includes the ability to visualize, navigate and run unit tests through a test explorer, a feature that is commonly requested by our users.

Validated breakpoints when debugging
This release also includes an enhancement to the debugger: validation of breakpoints targets. If a breakpoint is set on a line where breakpoints are invalid (e.g. blank lines, pass statement, lines in the middle of a multi-line statement), then it’s automatically moved to the nearest preceeding valid line

The team keeps cranking out more awesomeness every month


Jenkins + Alexa: Say Hello to Voice Controlled CI/CD 

Alexa (named after the ancient library of Alexandria) is Amazon’s Artificial Intelligence (AI) powered intelligent voice assistant that runs in the cloud. Software engineers make Alexa smarter by creating apps, called skills. From the time that I developed my first Alexa skill, I dreamed of deploying my Java projects to the cloud via voice. For me, telling Alexa to deploy my code is the ultimate level of cool! I recently made my dream a reality when I devoted a weekend to developing my newest Alexa skill, DevOps Pal. In this blog, I will show you how I developed DevOps Pal and hopefully inspire you to build your own version.


LOL, I have Alexa at home.. but I mostly use it to listen to Music. I am just wondering if they did this because they could and not because this is actually useful. Our builds build upon checkin (on dev, qa/test is scheduled), not sure why you would want to initiate it


Red Flags in Software Developer Job Descriptions

For some reason I’ve been reading a lot of job descriptions for junior devs lately and that has naturally left me with an inordinate number of opinions. I have, with great effort, condensed them here for you, dear reader.

The following aren’t pulled from any specific listings because these types of awfulness transcend the individual - and because that would be dickish. But their spirit should ring true to anyone in the trenches of the job search process.

“Technologies: Coldfusion + jQuery”
Your job search shouldn’t be driven by what’s fashionable in tech, but popularity and the market for a skill are important factors, since they influence a career track’s employability and profitability. The market for Coldfusion devs isn’t too hot (or even a thing) and jQuery simply doesn’t have the complexity or potential power of a full JS MVC. Companies advertising outdated stacks open you up to the risk of building the wrong sorts of skills, which can have far-reaching effects on your career.


One thing that caught my eye was  Coldfusion + jQuery, this is the stack that DJindexes website was built on so had to go and troll my ex co-workers  haha. One reply I got was: All of these jobs sound very interesting to me. Do they have any COBOL and JCL openings?

See also the comments on hackernews here: https://news.ycombinator.com/item?id=19268989

Motorola confirms its foldable phone is coming

By now, it's not much of a secret that Motorola is working on a folding phone of its own: A patent for such a device surfaced late last year, and a subsequent report in The Wall Street Journal basically confirmed the company's plans. Despite the ensuing hype, Motorola has mostly kept quiet about its progress, but in an interview with Engadget this week, Motorola VP of Global Product Dan Dery shed some additional light on the company's ambitions.

"We started to work on foldables a long time ago," Dery said. "And we have been doing a lot of iteration."

Oh man.. now everyone is jumping on this bandwagon??  What happens when you only crack one screen? This still seems idiotic to me, especially at that price point that Samsung has.

SQL Server 2019 community technology preview 2.3 is now available

This CTP 2.3 preview brings the following new features and capabilities to SQL Server 2019:

Big data clusters

Submit Spark jobs on SQL Server big data clusters from IntelliJ

Application deployment and management experience for deploying a variety of data-related apps including operationalizing machine learning models using R and Python, running SQL Server Integration Services (SSIS) jobs, and more

Database engine

Accelerated database recovery to provide constant time recovery and instant rollback for long running transactions

Performance enhancements in query plan recompilations, transaction log management, and Query Store

SQL Graph enhancement to enable cascaded deletion of edges upon the deletion of nodes

SQL Server Analysis Services (SSAS)

Calculation groups in tabular models that reduce the number of measures by reusing calculation logic

Installed on Sunday morning..checking out what is new and shiny. One thing that Brent Ozar noticed was the is_result_set_caching_on as a new column in sys.databases. So far I haven't heard yet how to turn it on

Some cool stuff you might enjoy

Finished The Annotated Turing: A Guided Tour Through Alan Turing's Historic Paper on Computability and the Turing Machine by Charles Petzold. a little dry at times.. but if you like math or want to learn about Turing machines, this book is for you


There is a Game Of Thrones fantasy league? 

I signed up and created a league with a bunch of co-workers
Here is how some of the scoring works


Violence:

10 Kill Random Character / White Walker / Wight (capped at 50pts)
15 Kill White Walker
25 Kill Known or Drafted Character
150 Kill Dragon
5 Incapacitate Random Character (not killed)
10 Incapacitate Known or Drafted Character (not killed)
25 Exit Bonus for Drafted Character Dying Memorably


Status/Power:

200 Take the Iron Throne
25 Sack a City / Lead a Battle Victory (must be present at victory)
25 Get a Seat on the High Council
50 Magic Use (1x Episode / Character)
50 Gain ownership of Valyrian Steel (Must keep through end of episode)
20 Have a Vision/Prophecy (1x / Episode / Character)
15 Take up a Weird/New Religion
10 Get Engaged
20 Get Married
15 Get Pregnant/Get Someone Pregnant
-20 Lose a Baby
25 Official Promotion
-25 Official Demotion
50 Come Back from the Dead
15 Act of Betrayal
15 Form / Join existing alliance
20 Ride or Control Dragon (1x / Episode)
20 MVP of Episode (Scorekeeper's Choice)
50 Revealed as Azor Ahai


I just love Times Square, here is a panorama I took'

Cool day trip in February if you live in New Jersey.. tree tapping at Howell Living History Farm

Our family went to Howell Living History Farm for the tree tapping event 6 years or so ago, the kids were little and didn't really remember the event. We decided to repeat the event this year so that they would remember this.  From our home in Princeton it's about a 25 minute drive, not too bad

It was a cloudy day, so the pics look a little gloomy but it was not too cold

Here you can see some benches that are next to the main entrance where the shop is located



At the shop, you can see a board with all the activities and the time when the activities start. Ask someone for directions if you need to know where to go


While we were walking towards the area where they had the sap collecting buckets, these sheep came running towards us, my kids freaked out but they were very exciting to see the sheep

 



A little later we saw this big duck, looked like it weighed at least 10 pounds to me, it was about the size of one of those Canada geese birds




The farm had this stack of neatly piled wood




You could actually get busy and cut the wood with this huge saw




These orange roots looked really weird and much better in person




Here you can see the buckets that they use to collect the sap. If you stand still and don't make any noise, you can here the drops making noise when they hit the bucket. It's a pretty interesting sound



A close up shot of the buckets

I tasted the sap as did my wife and the kids. It doesn't taste as syrup at all, it is like water with a very tiny hint of sweetness, the sap is also clear, it does not have a color


This is the evaporator, it is used to make syrup from the sap



Open pan evaporation methods have been streamlined since colonial days, but remain basically unchanged. Sap must first be collected and boiled down to obtain pure syrup without chemical agents or preservatives. Maple syrup is made by boiling between 20 and 50 volumes of sap (depending on its concentration) over an open fire until 1 volume of syrup is obtained, usually at a temperature 4.1 °C (7.4 °F) over the boiling point of water

Boiling the sap for too long will create crystals so you have to be on top of the process and check it.  While they were doing the explaining, they also mentioned that the indigenous people would warm up stones and then drop those hot stones in the sap to create syrup.

After you are done with the presentation, you can go to the house where they will make you some old fashioned pancakes.



Farm house

Chickens

These chickens come inside at night because foxes and hawks will snatch them and eat them. They told us to come back in about 4 weeks because that is when they will have the little chicks.



My daughter Catherine petting this horse




After the farm, we drove to Nomad Pizza in Hopewell. If you want to eat there you will probably have to wait since they only have 10 tables or so. In the summer there is an outside area as well. This is why if we go there during the colder months we make sure to get there by 5

I had the chorizo with onion, pepper and mozzarella pizza, it was delicious

TWID Feb 25, 2019: Galaxy Fold, Huawei Mate X, Hololens 2, Juventus, Linux Fsync Issue fix for PostgreSQL, syrup

This is a post detailing some stuff I did, learned, posted and tweeted this week, I call this TWID (This week in Denis). I am doing this mostly for myself... a kind of an online journal so that I can look back on this later on. Will use the label TWID for these

This Week I Learned

Almost finished with the book The Annotated Turing: A Guided Tour Through Alan Turing's Historic Paper on Computability and the Turing Machine by Charles Petzold

It takes a lot of maple sap to create maple syrup.  The higher the sugar content of the sap, the smaller the volume of sap is needed to obtain the same amount of syrup. 57 units of sap with 1.5 percent sugar content will yield 1 unit of syrup, but only 25 units of sap with a 3.5 percent sugar content are needed to obtain one unit of syrup

Those are my youngest two trying the sap. I tried it as well, I must say there really is no taste to it.

I will make a separate post about our visit to the farm

This Week I Tweeted

Samsung’s foldable phone is the Galaxy Fold, available April 26th starting at $1,980

Samsung first teased its foldable phone back in November, and at the company’s Galaxy Unpacked event today, it’s further detailing its foldable plans. Samsung’s foldable now has a name, the Samsung Galaxy Fold, and the company is revealing more about what this unique smartphone can do. Samsung is planning to launch the Galaxy Fold on April 26th, starting at $1,980, through AT&T and T-Mobile in the US, with a free pair of Samsung’s new wireless earbuds. There will be both an LTE and 5G version of the Galaxy Fold, and Samsung is even planning on launching the device in Europe on May 3rd, starting at 2,000 euros.

Overpriced and if you damage a screen how much to repair this. I rather have a phone like the showed in the Expanse or even better a phone you can roll up so it is the size and shape of a pen. This thing is just too bulky as well. A better design would have been if there was a screen that you could slide out instead.

If you thought the Galaxy Fold was not expensive enough.. no worried The Mate X is Huawei’s 5G foldable phone... the price $2600.  Fitting name Mate X, as in mate you will need eXtra money for this one... 

Pass on both from me


Juventus share price is down 9% after their champions league result

  Ouch, that is not good, but Atletico played a much better 2nd half and converted their chances. Let's see if Juve can advance by scoring at least 2 at home in the return game.


Falsehoods Programmers Believe About Phone Numbers 

Some interesting things you might already know, still good to revisit this list

Some cool stuff you might enjoy

Microsoft’s HoloLens 2: a $3,500 mixed-reality headset for the factory

The Microsoft HoloLens 2 is available for preorder today for $3,500, and it’s expected to ship later this year. However, Microsoft has decided that it is only going to sell to enterprise customers who want to deploy the headset to their workers. As of right now, Microsoft isn’t even announcing a developer kit version of the HoloLens 2.

Compared to the HoloLens we first saw demonstrated four years ago, the second version is better in nearly every important way. It’s more comfortable, it has a much larger field of view, and it’s better able to detect real physical objects in the room. It features new components like the Azure Kinect sensor, an ARM processor, eye-tracking sensors, and an entirely different display system.

It has a couple of speakers, the visor flips up, and it can see what your hands are doing more accurately than before. There’s an 8-megapixel front-facing camera for video conferencing, it’s capable of full 6 degrees of tracking, and it also uses USB-C to charge. It is, in short, chock-full of new technology. But after four years, that should be no surprise.


Linux Fsync Issue for Buffered IO and Its Preliminary Fix for PostgreSQL

One of the common fixes applied to all the supported PostgreSQL versions is on – panic instead of retrying after fsync () failure. This fsync failure has been in discussion for a year or two now, so let’s take a look at the implications.

A fix to the Linux fsync issue for PostgreSQL Buffered IO in all supported versions
PostgreSQL performs two types of IO. Direct IO – though almost never – and the much more commonly performed Buffered IO.

PostgreSQL uses O_DIRECT when it is writing to WALs (Write-Ahead Logs aka Transaction Logs) only when wal_sync_method is set to : open_datasync or to  open_sync with no archiving or streaming enabled. The default  wal_sync_method may be fdatasync that does not use O_DIRECT. This means, almost all the time in your production database server, you’ll see PostgreSQL using O_SYNC / O_DSYNC while writing to WAL’s. Whereas, writing the modified/dirty buffers to datafiles from shared buffers is always through Buffered IO

Starting from kernel 4.13, we can now reliably detect such errors during fsync. So, any open file descriptor to a file includes a pointer to the address_space structure, and a new 32-bit value (errseq_t) has been added that is visible to all the processes accessing that file. With the new minor version for all supported PostgreSQL versions, a PANIC is triggered upon such error. This performs a database crash and initiates recovery from the last CHECKPOINT. There is a patch expected to be released in PostgreSQL 12 that works for newer kernel versions and modifies the way PostgreSQL handles the file descriptors. A long term solution to this issue may be Direct IO, but you might see a different approach to this in PG 12.

Some more info that I found in this hackernews comment thread that might interest you:
If you want an overview of the issue, here's a presentation from Tomas Vondra at FOSDEM 2019: https://youtu.be/1VWIGBQLtxo
Or an early recap of the "fsyncgate" issue in textual form: https://lwn.net/Articles/752063/

Related (also listed by Tomas Vondra): Linux's IO errors reporting https://youtu.be/74c19hwY2oE


As always, I will leave you with a pic I took this past week. This one is a pic of some orange roots over a stream

Calculating Sexy Primes, Prime Triplets and Sexy Prime Triplets in PostgreSQL

The other day I was reading something on Hackernews and someone posted a link to a Sexy Primes wikipedia article.  I looked at that and then decided to do this in SQL Server because.. why not? Then I decided to see how different this would be to do in PostgreSQL.  For the first method to create the prime numbers it's different. For the method with the CTE it is very similar


From the Sexy Primes wikipedia link: https://en.wikipedia.org/wiki/Sexy_prime


In mathematics, sexy primes are prime numbers that differ from each other by six. For example, the numbers 5 and 11 are both sexy primes, because 11 minus 5 is 6.

The term "sexy prime" is a pun stemming from the Latin word for six: sex.

If p + 2 or p + 4 (where p is the lower prime) is also prime, then the sexy prime is part of a prime triplet.

Ok I did a couple of versions of this and over the weekend.. here is what I ended up with

So first we need a table that will just have the prime numbers

I decided to populate a table with numbers from 2 till 500 and then use the sieve of Eratosthenes method to delete the non primes

This will look like this

Create this table

CREATE  TABLE  PrimeNumbers  (N INT);

In one window run this to create the function/proc

CREATE OR REPLACE FUNCTION MakePrime() RETURNS void AS $$
DECLARE I integer := 2;
BEGIN
WHILE I <= SQRT(500) LOOP
    DELETE FROM PrimeNumbers WHERE N % I = 0 AND N > I;
    I := I + 1 ; 
END LOOP;

END;

$$ LANGUAGE plpgsql;

In a another window populate the table by making the call to the function
 

INSERT  INTO PrimeNumbers(n)
SELECT N
 FROM (SELECT generate_series(2,500) as n) x;


SELECT MakePrime() ; -- Yes that is a proc call

SELECT * FROM PrimeNumbers

Thinking about it a little more I decided to do it with a CTE instead of a loop with delete statements, if your tables will be big then the delete method is probably better... it's for you to test that out :-)

What we are doing is a NOT EXISTS query against the same cte and we are filtering out numbers that are greater than the number in the current row and are not divisible by the current number

CREATE TABLE IF NOT EXISTS  PrimeNumbers  (N INT);


;WITH cte AS (
  SELECT * FROM generate_series( 2, 500 )  n
  )

INSERT INTO PrimeNumbers
SELECT n
FROM cte
WHERE NOT EXISTS (
  SELECT n FROM  cte as cte2
WHERE cte.n > cte2.n AND cte.n % cte2.n = 0)
;

SELECT * FROM PrimeNumbers;

If we run that last select statement, we should have 95 rows

2
3
5
7
 .....
 .....
463
467
479
487
491
499

Now that we have our table filled with prime numbers till 500, it's time to run the queries

Sexy prime pairs
The sexy primes (sequences OEIS: A023201 and OEIS: A046117 in OEIS) below 500 are:

(5,11), (7,13), (11,17), (13,19), (17,23), (23,29), (31,37), (37,43), (41,47), (47,53), (53,59), (61,67), (67,73), (73,79), (83,89), (97,103), (101,107), (103,109), (107,113), (131,137), (151,157), (157,163), (167,173), (173,179), (191,197), (193,199), (223,229), (227,233), (233,239), (251,257), (257,263), (263,269), (271,277), (277,283), (307,313), (311,317), (331,337), (347,353), (353,359), (367,373), (373,379), (383,389), (433,439), (443,449), (457,463), (461,467).


Here is that query for the sexy prime pairs

-- 46 rows.. sexy primes
SELECT t1.N,t2.N 
 FROM PrimeNumbers t1
join PrimeNumbers t2 on t2.N - t1.N = 6 
order by 1

It's very simple.. a self join that returns rows where the number from one table alias and the number from the other table alias differ by 6




Prime triplets
The first prime triplets below 500 (sequence A098420 in the OEIS) are

(5, 7, 11), (7, 11, 13), (11, 13, 17), (13, 17, 19), (17, 19, 23), (37, 41, 43), (41, 43, 47), (67, 71, 73), (97, 101, 103), (101, 103, 107), (103, 107, 109), (107, 109, 113), (191, 193, 197), (193, 197, 199), (223, 227, 229), (227, 229, 233), (277, 281, 283), (307, 311, 313), (311, 313, 317), (347, 349, 353), (457, 461, 463), (461, 463, 467)

A prime triplet contains a pair of twin primes (p and p + 2, or p + 4 and p + 6), a pair of cousin primes (p and p + 4, or p + 2 and p + 6), and a pair of sexy primes (p and p + 6).

So we need to check that the 1st and 3rd number have a difference of 6, we also check that that difference between number 1 and 2 is 2 or 4.  That query looks like this

-- 22 rows.. Prime Triplets
SELECT t1.N AS N1,t2.N AS N2, t3.N AS N3
 FROM PrimeNumbers t1
join PrimeNumbers t2 on t2.N > t1.N 
join PrimeNumbers t3 on t3.N - t1.N = 6
and t3.N > t2.N
and t2.n - t1.n IN (2,4)
order by 1

Here is what it looks like from pgAdmin

Sexy prime triplets
Triplets of primes (p, p + 6, p + 12) such that p + 18 is composite are called sexy prime.  p p, p+6 and p+12 are all prime, but p+18 is not

Those below 500 (sequence OEIS: A046118) are:

(7,13,19), (17,23,29), (31,37,43), (47,53,59), (67,73,79), (97,103,109), (101,107,113), (151,157,163), (167,173,179), (227,233,239), (257,263,269), (271,277,283), (347,353,359), (367,373,379)


The query looks like this.. instead of a self join, we do a triple self join, we also check that p + 18 is not a prime number in the line before the order by

-- 14 rows.. Sexy prime triplets
SELECT t1.N AS N1,t2.N AS N2, t3.N AS N3
 FROM PrimeNumbers t1
join PrimeNumbers t2 on t2.n - t1.n = 6
join PrimeNumbers t3 on t3.N - t1.N = 12
and t3.N > t2.N
AND NOT EXISTS( SELECT null FROM PrimeNumbers p WHERE p.n = t1.n +18)
order by 1

And that's it for this post.  If you are interested in the SQl Server version, you can find it here: Calculating Sexy Primes, Prime Triplets and Sexy Prime Triplets in SQL Server


More PostgreSQL posts can be found here:  /label/PostgreSQL