Saturday, March 30, 2019

TWID March 30, 2019: Tsundoku, DevOps, robots, turing award,gps drifting


This is a post detailing some stuff I did, learned, posted and tweeted this week, I call this TWID (This week in Denis). I am doing this mostly for myself... a kind of an online journal so that I can look back on this later on. Will use the label TWID for these



This Week I Learned

Tsundoku
Tsundoku (積ん読) is acquiring reading materials but letting them pile up in one's home without reading them. The term originated in the Meiji era (1868–1912) as Japanese slang. It combines elements of tsunde-oku (積んでおく, to pile things up ready for later and leave) and dokusho (読書, reading books). It is also used to refer to books ready for reading later when they are on a bookshelf. As currently written, the word combines the characters for "pile up" (積) and the character for "read" (読).

OK, so I have a bunch of books that I bought years ago and never finished. Some of these I never started. So what I decided to do is take 1 hour each day before going to work, grab 3 books and read 20 minutes per book.  I also have some books in the office, I am reading 20 minutes each time when I go to the office, this is currently twice a week. The book I am reading in the office is Pragmatic Thinking and Learning, I am 40% done with that book

I already finished the book War Of Art, am 25% done with Tools of Titans and just started Python Tricks

Here is a pic of the books

Tsundoku (積ん読) is acquiring reading materials but letting them pile up in one's home without reading them


The books in that picture are the following

Tools of Titans: The Tactics, Routines, and Habits of Billionaires, Icons, and World-Class Performers (started)
Tribe of Mentors: Short Life Advice from the Best in the World
War Of Art  (finished)
Wired To Eat
Python Tricks the book: A Buffet of Awesome Python Features (started)
Fluent Python: Clear, Concise, and Effective Programming
Book of Five Rings
How Linux Works, 2nd Edition: What Every Superuser Should Know


I continued my DevOps training and like last week, I decided to put my note at the end of this post, this way I can easily look it up again


This Week I Tweeted

Three ‘Godfathers of Deep Learning’ Selected for Turing Award

Three computer scientists who laid the foundations for many of the recent advances in artificial intelligence are being honored with this year’s Turing Award, considered the field’s highest accolade.

Geoff Hinton, an emeritus professor at the University of Toronto and a senior researcher at Alphabet Inc.’s Google Brain, Yann LeCun, a professor at New York University and the chief AI scientist at Facebook Inc., and Yoshua Bengio, a professor at the University of Montreal as well as co-founder of AI company Element AI Inc., will share this year’s award, which is given annually by the Association for Computing Machinery.

The three winners will split a $1 million prize that comes with the award, which is currently underwritten by Google

For a list of previous winners see here: https://en.wikipedia.org/wiki/Turing_Awardhttps://en.wikipedia.org/wiki/Turing_Award



Was MongoDB Ever the Right Choice?

I was reading a post recently about Red Hat removing MongoDB support from Satellite (and yes, some folks say it is because of the license changes). It made me think how often over the last few years I’ve seen post after angry post about how terrible MongoDB is and how no one should ever use. However, in all this time, MongoDB has become a much more mature product. So what happened? Did all of the hate truly come from mistakes made in the early implementation/marketing of MongoDB? Or is the problem that people are blaming MongoDB for their own lack of efforts when evaluating if it was a good fit?

Read and find out.....



Australia Is Drifting So Fast GPS Can't Keep Up

Australia is not quite where you think it is. The continent has shifted by 4.9 feet since the last adjustment was made to GPS coordinates in 1994, reports the New York Times.

A significant correction must be made by the end of the year for navigation technology to keep working smoothly.

This is interesting, I  about drift because undersea cables will break, but never had an idea that it was this fast that they have to adjust GPS



Boston Dynamics’ new robot stacks boxes 

It’s a “mobile manipulation robot” designed for the logistics sector. It can autonomously stack and unstack boxes onto and off pallets, and shift them onto conveyor belts. It uses an onboard vision system to track which objects go where, and to judge how to grasp and place each box. It uses a robotic technique called “force control” to nestle each box up against its neighbors. It can handle weights of up to 15 kilograms (33 pounds.)

And that is how it all starts... wondering when they will rename the company to SkyNet  :-)

Some cool stuff you might enjoy

Awesome Newsletters
A curated list of awesome newsletters.

Inspired by the awesome-* trend on GitHub.

There is something here for everyone




The most populous cities in the world from 1500 to 2018   

This is amazing, make sure to watch the whole animation..it's like a race....





Some DevOps notes

This is for me.. not for you.... but feel free to take a look...although it won't probably make a lot of sense if you see these terms for the first time



Top 10 books  from 10 to number 1
Visible Ops
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
Release It!: Design and Deploy Production-Ready Software
Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale
Lean Software Development: An Agile Toolkit 
Web Operations: Keeping the Data on Time
The Practice of Cloud System Administration: DevOps and SRE Practices for Web Services
The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations
Leading the Transformation: Applying Agile and DevOps Principles at Scale
The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win


Devopsweekly.com mailinglist


Wall of Confusion
Impedance mismatch between teams

Blameless postmortems
---------------------------------------
It's a meeting
within 48 hours of incident
run by a 3rd party

Have everything in a timeline. Not here to blame a person but making sure this doesn't happen again

Trasnparent Uptime
-------------------------
Admit failure
Sound like a human.. no doublespeak
Have a communication channel
Be authentic


Trust blockers
--------------------
Lack of Context
Conflicting goals


Open it Up
-----------------------
Chat rooms
Wiki pages
Source code (to read)
Infrastructure
Monitoring Tools
Ticket Tracker

Chatops
USe something like chat or teams instead of emails


Shared responsibility
-------------------------------
Feedback
Automation
Team culture
No Silos



Shadow IT/Bimodal IT
Groups deliberate bypassing processes to get things done

Devs responsibe for failed deployment after code checkin

Changing your ways of how you do work.... might be hard..but it will pay off


Conway's law
----------------------------
organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.

Eric S. Raymond, an open-source advocate, restated Conway's law in The New Hacker's Dictionary
James O. Coplien and Neil B. Harrison stated:

If the parts of an organization (e.g., teams, departments, or subdivisions) do not closely reflect the essential parts of the product, or if the relationship between organizations do not reflect the relationships between product parts, then the project will be in trouble ... Therefore: Make sure the organization is compatible with the product architecture.


Westrum Model
--------------------------------
Pathological (Power Oriented)
Bureaucratic (Rule Oriented)
Generative (Performance Oriented)



Management Best Practices
-----------------------------------
Independent, cross-functional teams
People first
Agile, lean processes



Kaizen
---------------------------
Small change  aka continues improvement
Good processes bring good results
Gemba (Go see for yourself).. (the real pace or Locus).. meaning look at the code or the system or factory floor
Speak with data, manage by facts  (scientific method)
Take action to contain and correct root causes (The 5 whys, people don't fail processes do)
Work as a team
Kaizen is everubody's business



======================================================
------------------------------------------------------
======================================================


Waterfall --> Iterative
DevOps has strong roots in Agile but is not just agile


Lean, 7 principles
1 Eliminate Waste
2 Amplify Learning
3 Decide as late as possible
4 Decide as fast as possible
5 Empower the team
6 Build in integrity
7 See the whole picture


Waste
Muda: work that absorbs resources but adds no value
Muri: Unreasonable work imposed on workers and machines
Mura: Work coming in unevenly instead of a constant or regular flow


The 7 Wastes Of Software
1 Partially work done
2 Extar features
3 Relearning
4 Handoffs
5 Delays
6 Task Switching
7 Defects


Build Measure Learn
-----------------------
Build minimum viable product
Measure the outcome and internal metrics
Learn about you problem and your solution
Repeat, go deep where it's needed


Concept to cash: from idea to it's realization, including everything needed to get it to customer



ITSM (Information Technology Service Management)

ITIL (Information Technology Infrastructure Library)
1 Service  Strategy
2 Service  Design
3 Service  Transition
4 Service  Operation


ITIL 2007... 2000 pages long...  The Visible OPS handbook (only 100 pages)




Use ITIL but make it lean



======================================================
------------------------------------------------------
======================================================



Infrastructure as code
Everything programmatically, no UI, same code runs on test, stage, UAT and production



Servers are cattle not pets.. deploy in mass, not handcrafted



Provisioning: Making a server ready for operation..including OS, network connectivity and system services
Deployment: Automatically deploying and upgrading applications on a server
Orchestration: Performing coordinated operations across multiple systems
Configuration Management: Management of change control after initial provision, maintaining and upgrading apps and app dependencies

Imperative/procedural: Commands necessary to produce a desired state are defined and executed
Declaritive/Functional: A desired state is defined, relying on the tool to configure a system to match that state
Idempotent: The ability to execute repeatedly, resulting in the same outcome
Self Service: The ability for an end user to initiate a process without having to go through other people



Canary (staged) Deployment Pattern
Upgrade 1 server, see how it works before upgrading the rest

Blue Green deployment
Two environment, code is pushed to one, then environments are swapped, variant like cluster immune system deployment

Containers making things easier to deploy, golden image is coming back now since you can just packe it up as a container.

Terraform.. used so that you can use setting across Azure and AWS



---------------------------------------------------------------

Continuous Deploy
Continuous Delivery
Continuous Integration


Time to market goes down... high performing IT orgs can deploy on demand, some 10 or 100 times a day

“Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.” quote by W. Edwards Deming


With Continuous Delivery you can much easier see what change caused a degradation of an application, with a quarterly release there are so many changes, it is difficult to find out what caused the degradation

The goal of continuous integration is that working software is in a working state all the time quote by Jez Humble

6 practices
Builds should pass the coffee test (should be less than 5 minutes)
Commit really small bits
Don't leave the build broken
Use a trunk based development flow
No crappy tests, fix your tests
The build should return a status, a log and an artifact


Only build artifacts once..immutable, should not change... this creates trust.. underlying bits did not change
Stop deploy if a step fails, notify teams


Test Unit testing
Code hygiene  linters, formatters, banned functions check
Integration testing



TDD/BDD/ATDD

TDD
State desired outcome as test
write code to pass test
Repeat

BDD
Work with stakeholders
Describe business functionality
Tests are based on natutal language descriptions

ATDD (acceptance test driven development)
End user perpective
Use case automated testing
testing is continous during development


3 ways to get around slow tests... so you don't violate your coffee rule
use nonblocking tests
use time scheduled tests
use monitoring


MTTR
Mean Time To Recovery


Cascading failure pattern: Cascading failure is number one threat to stability, one layer can choke out other layers
Circuit breaker pattern: If app see an issue with one specific set of calls redirect to other part.. minimizes outage risk



The twelve-factor methodology
https://12factor.net/

I. Codebase
One codebase tracked in revision control, many deploys

II. Dependencies
Explicitly declare and isolate dependencies

III. Config
Store config in the environment

IV. Backing services
Treat backing services as attached resources

V. Build, release, run
Strictly separate build and run stages

VI. Processes
Execute the app as one or more stateless processes

VII. Port binding
Export services via port binding

VIII. Concurrency
Scale out via the process model

IX. Disposability
Maximize robustness with fast startup and graceful shutdown

X. Dev/prod parity
Keep development, staging, and production as similar as possible

XI. Logs
Treat logs as event streams

XII. Admin processes
Run admin/management tasks as one-off processes



If it hurts, do it more often. The best way to avoid failure is to fail constantly

Monitoring
------------------
Service performance/uptime
software component metrics
system metrics
app metrics
performance
security

5 Principles of log data
Do not collect log data if you never plan to use it
Retain log data for as long as it can be used
Log all you can, alert only what you must respond to, define log levels
Don't make logging more available than production stack
Make log changes as needed




Future
Containers and Serverless (serverless might only run 1 invocation)

Security
100:10:1  (Dev:Ops:Sec)
100 developers, 10 Ops, 1 Security person


The Rugged Manifesto: https://ruggedsoftware.org/

I am rugged and, more importantly, my code is rugged.
I recognize that software has become a foundation of our modern world.
I recognize the awesome responsibility that comes with this foundational role.
I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended.
I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic, and national security.
I recognize these things - and I choose to be rugged.
I am rugged because I refuse to be a source of vulnerability or weakness.
I am rugged because I assure my code will support its mission.
I am rugged because my code can face these challenges and persist in spite of them.
I am rugged, not because it is easy, but because it is necessary and I am up for the challenge.

No comments: