Summer of code 2017: Python, Day 44 Frequency of words in the book Moby Dick

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 44. Today I wanted to see if I could get a python script to run and return me all the words and their occurrences in the book Moby Dick

Some interesting things you might want to know:

How many time is Moby used in the book?
How many distinct words in total?
How many words are used only once?
What are the top 20 most used words?


If you are interested in how many times Moby is in the book... here is the answer

As you can see Moby is in the book 90 times.

Ok so let's get started, if you want to follow along, you will need the Moby Dick book. Since Moby Dick is in the public domain, you can download the book for free. You can get it from project Gutenberg, the link is here:  http://www.gutenberg.org/ebooks/2701

Make sure to grab the  Plain Text UTF-8 version

In Python, what do we need to get the words an their counts? We need a function that will store the text of the book in a variable

It will look like the following

with open(r'C:\Downloads\MobyDick.txt', 'r', encoding="utf8") as myfile:
    doc=myfile.read().replace('\n', ' ')

As you can see we are also stripping off the line feed by replacing \n with ''. Make sure encoding is in utf8 otherwise you will get errors like the one below

Traceback (most recent call last):
  File "c:\MapReduce.py", line 11, in
    doc=myfile.read().replace('\n', ' ')
  File "C:\Program Files\Python36\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 7237: character maps to

Now that we have the file in a variable, we need to count the words, here is what that function will look like

def CountWords(text):
    output =''.join(c.lower() if c.isalpha() else ' ' for c in text)
    frequencies = {}
    for word in output.split():
        frequencies[word] = frequencies.get(word, 0) + 1
    return frequencies

What the function does is strips all non alpha characters, after that it loops through all the words created by the split function and increments the counter. The function then returns this key value pair


In order to print the output on more than 1 line, we will use pprint,  I already posted about print here: Summer of code 2017: Python, Pretty printing with pprint in Python

from pprint import pprint as pp

Finally, we need to reverse the order, sort the output and limit the output to n numbers,  I have chosen 500 here

pp(sorted(CountWords(doc).items(), key=lambda x: (-x[1], x[0]))[:500])

All of this together will look like this, make sure to change the path to the file to match your computer's path

def CountWords(text):
    output =''.join(c.lower() if c.isalpha() else ' ' for c in text)
    frequencies = {}
    for word in output.split():
        frequencies[word] = frequencies.get(word, 0) + 1
    return frequencies
 
 
with open(r'C:\Downloads\MobyDick.txt', 'r') as myfile:
    doc=myfile.read().replace('\n', ' ')
 
 
from pprint import pprint as pp
 
pp(sorted(CountWords(doc).items(), key=lambda x: (-x[1], x[0]))[:500])

Now let's ask those questions again, but this time we will have the answers as well

What are the 20 most used words?
Here we go

('the', 14718),
 ('of', 6743),
 ('and', 6518),
 ('a', 4807),
 ('to', 4707),
 ('in', 4242),
 ('that', 3100),
 ('it', 2536),
 ('his', 2532),
 ('i', 2127),
 ('he', 1900),
 ('s', 1825),
 ('but', 1823),
 ('with', 1770),
 ('as', 1753),
 ('is', 1751),
 ('for', 1646),
 ('was', 1646),
 ('all', 1545),
 ('this', 1443)

How many distinct words in total?
17,148 distinct words (you need to remove the limit in order to get the full set back, just remove [:500])


How many words are used only once?
There are 7416 words used only once, here are some of them starting with the letter z (you need to remove the limit in order to get the full set back, just remove [:500])

('zag', 1),
 ('zay', 1),
 ('zealanders', 1),
 ('zephyr', 1),
 ('zeuglodon', 1),
 ('zig', 1),
 ('zip', 1),
 ('zogranda', 1),
 ('zoroaster', 1)


How many times does captain Ahab's name appear in the book?
Captain Ahab's name appears 517 times in the book


And I will leave you with the first 100 most used words in the book

 ('the', 14718),
 ('of', 6743),
 ('and', 6518),
 ('a', 4807),
 ('to', 4707),
 ('in', 4242),
 ('that', 3100),
 ('it', 2536),
 ('his', 2532),
 ('i', 2127),
 ('he', 1900),
 ('s', 1825),
 ('but', 1823),
 ('with', 1770),
 ('as', 1753),
 ('is', 1751),
 ('for', 1646),
 ('was', 1646),
 ('all', 1545),
 ('this', 1443),
 ('at', 1335),
 ('whale', 1244),
 ('by', 1227),
 ('not', 1172),
 ('from', 1105),
 ('on', 1073),
 ('him', 1068),
 ('so', 1066),
 ('be', 1064),
 ('you', 964),
 ('one', 925),
 ('there', 871),
 ('or', 798),
 ('now', 786),
 ('had', 779),
 ('have', 774),
 ('were', 684),
 ('they', 670),
 ('which', 655),
 ('like', 647),
 ('me', 633),
 ('then', 631),
 ('their', 620),
 ('are', 619),
 ('some', 619),
 ('what', 619),
 ('when', 607),
 ('an', 600),
 ('no', 596),
 ('my', 589),
 ('upon', 568),
 ('out', 539),
 ('man', 527),
 ('up', 526),
 ('into', 523),
 ('ship', 519),
 ('ahab', 517),
 ('more', 509),
 ('if', 501),
 ('them', 474),
 ('ye', 473),
 ('we', 470),
 ('sea', 455),
 ('old', 452),
 ('would', 432),
 ('other', 431),
 ('been', 415),
 ('over', 409),
 ('these', 406),
 ('will', 399),
 ('though', 384),
 ('its', 382),
 ('down', 379),
 ('only', 378),
 ('such', 376),
 ('who', 366),
 ('any', 364),
 ('head', 348),
 ('yet', 345),
 ('boat', 337),
 ('long', 334),
 ('time', 334),
 ('her', 332),
 ('captain', 329),
 ('here', 325),
 ('do', 324),
 ('very', 323),
 ('about', 318),
 ('still', 312),
 ('than', 311),
 ('chapter', 308),
 ('great', 307),
 ('those', 307),
 ('said', 305),
 ('before', 301),
 ('two', 298),
 ('has', 294),
 ('must', 293),
 ('t', 291),
 ('most', 285)

Summer of code 2017: Python, Day 42 args and kwargs

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 42.  Today I looked at args and kwargs, these notes are mostly for me but who knows, they might be helpful for someone else in the future as well


Args and kwargs? WTF is that? I had the same thought, it turns out these are magic variables  :-)

From the docs:

args

A tuple of positional arguments values. Dynamically computed from the arguments attribute.

kwargs

A dict of keyword arguments values. Dynamically computed from the arguments attribute.

Basically this is a way to pass in a unknown number of variables into a function.

Args are prefixed with an asterisk, kwargs are prefixed with 2 asterisks

The name of these  variables does not have to be *args and **kwargs, you can name it anything

For example, here is an args named bars

def test_args(foo, *bars):

And here is a kwargs also named bars

def test_kwargs(foo,**bars):

The only difference between these two is the single and double asterisk



Let's make a very simple function, that will loop through the *args and print them out. This function accepts a normal variable foo and an args variable named *bars

def test_args(foo, *bars):
    print ('first normal argument:', foo)
    for bar in bars:
        print ("another looping through all the *bars :", bar)
 
    print ('all *bars on one line', bars)

Now let's call this function like this

>>> test_args('args','Denis','likes','playing','with','Python')

Here is the output

>>> test_args('args','Denis','likes','playing','with','Python')
first normal argument: args
another looping through all the *bars : Denis
another looping through all the *bars : likes
another looping through all the *bars : playing
another looping through all the *bars : with
another looping through all the *bars : Python
all *bars on one line ('Denis', 'likes', 'playing', 'with', 'Python')
>>> 

Let's now call this function like this

>>> test_args('args','enough','Python')

Here is the output of that call

>>> test_args('args','enough','Python')
first normal argument: args
another looping through all the *bars : enough
another looping through all the *bars : Python
all *bars on one line ('enough', 'Python')
>>> 

As you can see, you can pass a variable number of values into the function by using args

Now let's take a look at kwargs, here is our function, we now named the variable **bars to denote that this is a variable of type kwargs

def test_kwargs(foo,**bars):
    print ('first normal argument:', foo)
    for bar in bars:
        print ("another looping through all the **bars :", bar)
 
    print ('all **bars on one line', bars)

Calling this function requires a change

If you try calling it like we did with args you will get an error

>>> test_kwargs('args','enough','Python')
Traceback (most recent call last):
  File "", line 1, in <module>
TypeError: test_kwargs() takes 1 positional argument but 3 were given
>>> 

What you have to do instead is use named arguments

If we change it to this it will work fine

>>> test_kwargs('kwargs',name ='Denis1', age = 200)

Here is the output

>>> test_kwargs('kwargs',name ='Denis1', age = 200)
first normal argument: kwargs
another looping through all the **bars : name
another looping through all the **bars : age
all **bars on one line {'name': 'Denis1', 'age': 200}
>>> 

Here is another example

>>> test_kwargs('kwargs',month ='July', day = 29)

Here is the output

>>> test_kwargs('kwargs',month ='July', day = 29)
first normal argument: kwargs
another looping through all the **bars : month
another looping through all the **bars : day
all **bars on one line {'month': 'July', 'day': 29}
>>> 

You might have noticed that we didn't print the value of the month or the value of the date. Let's change our function so it looks a little different, now we will print both the key and the value

def test_kwargs(foo, **bars):
    print ('first normal argument:', foo)
    if bars is not None:
        for key, value in bars.items():
            print ("%s : %s" %(key,value))
 
    print ('all *bars on one line', bars)

Now we can make the same call

test_kwargs('kwargs',name ='Denis1', age = 200)

And here is the output

>>> test_kwargs('kwargs',name ='Denis1', age = 200)
first normal argument: kwargs
name : Denis1
age : 200
all *bars on one line {'name': 'Denis1', 'age': 200}
>>> 

As you can see we now have the name as well as the value of the key printed


If you want to use args and kwargs in a function,  you need to have the args before the kwargs

def test_args(foo, **bars, *namedbars):
                               ^
SyntaxError: invalid syntax

This is an error because we have kwargs before args

def test_args(**bars, foo, *namedbars):
                            ^
SyntaxError: invalid syntax

This is also an error because we still have kwargs before args

This is how it should be, forst normal variables, then args and finally kwargs

def test_args(foo, *bars, **namedbars):

Here is an example of such a signature

def test_args(foo, *bars, **namedbars):
    print ('first normal argument:', foo)
    for bar in bars:
        print ("another looping through all the *bars :", bar)
 
    print ('all *bars on one line', bars)
    print ('all **namedbars on one line', namedbars)

Calling the function above gives us the following output

>>> test_args('args','Denis','likes','playing','with','Python')
first normal argument: args
another looping through all the *bars : Denis
another looping through all the *bars : likes
another looping through all the *bars : playing
another looping through all the *bars : with
another looping through all the *bars : Python
all *bars on one line ('Denis', 'likes', 'playing', 'with', 'Python')
all **namedbars on one line {}

As you can see, there is nothing printed for kwargs, this is because we did not pass anything in.
Let's make a change and add a value for the kwargs

Here is the output from that call

>>> test_args('args','enough','Python', namedbars='Denis')
first normal argument: args
another looping through all the *bars : enough
another looping through all the *bars : Python
all *bars on one line ('enough', 'Python')
all **namedbars on one line {'namedbars': 'Denis'}
>>> 

There you have it.. a rather simple blog post that that explains the difference between args and kwargs

Summer of code 2017: Python, Day 37 Pandas, Spyder, IPython, Numpy and more

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 37. You might be thinking I have been slacking off since I have not posted anything. This is actually not true, it is true that I have not posted anything but I have been quite busy with Python. I watched a couple of sections of the Python – Beyond the Basics course by Austin Bingham and Robert Smallshire on Pluralsight.

I learned about packages, subpackages, lambdas, local functions, decorators and more. I will probably need to watch it again since this course is more intense than the previous course. Expect some posts about these things I learned this week

I also listened to a bunch of podcasts on Python the last couple of days

Talk Python To Me podcast

Here are the episodes I listened to

#121 2017-07-19 Microservices in Python Miguel Grinberg

#120 2017-07-12 Python in Finance Yves Hilpisch

#119 2017-07-06 Python in Engineering Allen Downey

#101 2017-03-03 Adding a full featured Python environment to Visual Studio Code Don Jayamanne

#100 2017-02-22 Python past, present, and future with Guido van Rossum Guido van Rossum

PODCAST.__INIT_ podcast

Here are the episodes I listened to

Moving to MongoDB with Michael Kennedy – Episode 119

Zulip Chat with Tim Abbott – Episode 118

Pandas with Jeff Reback – Episode 98

PyTables with Francesc Alted – Episode 97

I also messed around with different projects in Visual Studio.

I created a bunch of web projects in Django as well as in Flask to understand  what the differences were. I then looked at some documentation about these web frameworks to see which one looks easier to use and setup

Finally I also messed around with Spyder, iPython, pandas and numpy while following some examples from the Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython book

Since the book is a bit dated I ran into some problems with the pandas.io.data module. But a helpful error message informed me that the pandas.io.data module got replaced by pandas-datareader package. So i then installed the pandas-datareader package and the code worked fine

Summer of code 2017: Python, Day 32 unit testing in Python with unittest

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 32. today I decided to take a look at unit testing. I decided to look at the unittest unit testing framework that ships with Python

The unittest unit testing framework was originally inspired by JUnit and has a similar flavor as major unit testing frameworks in other languages. It supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework.

So let's see what it all looks like. First we are going to create a simple method and save it in a file named utils.py

This method will return what was passed in if it was none or a string, it will convert to utf-8 if bytes were passed in, for everything else a type error will be returned

def ToString(data):
    if isinstance(data, str):
        return data
    elif isinstance(data, bytes):
        return data.decode('utf-8')
    elif data is None:
         return data
    else:
        raise TypeError('Must supply string or bytes,'
                        ' found: %r' % data)

To unit test this method, we are creating our test class and we will save this in a file named UnitTestSample.py

Here is what it looks like

from unittest import TestCase, main
from utils import ToString
 
 
class UtilsTestCase(TestCase):
    def test_ToString_bytes(self):
        self.assertEqual('hello', ToString(b'hello'))
 
    def test_ToString_str(self):
        self.assertEqual('hello', ToString('hello'))
 
    def test_ToString_bad(self):
        self.assertRaises(TypeError, ToString, object())
 
    def test_ToString_none(self):
        self.assertIsNone(  ToString(None))
 
    def test_ToString_not_none(self):
        self.assertIsNotNone(  ToString('some val'))
 
if __name__ == '__main__':
   main()

We need to import unittest as well as our ToString method

As you can see, we have a couple of AssertEqual calls,  AssertEqual tests that first and second are equal. If the values do not compare equal, the test will fail.

We also have assertIsNone and assertIsNotNone, these test that something is none or is not none

Finally we use assertRaises. AssertRaises tests that an exception is raised when callable is called with any positional or keyword arguments that are also passed to assertRaises(). The test passes if exception is raised, is an error if another exception is raised, or fails if no exception is raised.

Running the code above will give us this output, all 5 tests have passed

Running C:\Python\Projects\UnitTestSample\UnitTestSample.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.000s

To print out the name of each test, we need to change main and pass in verbosity level 2

In the code main will now look like this

main(verbosity=2)

Running the same test class again will give us also the test methods that were called, here is the output

The interactive window has not yet started.
Running C:\Python\Projects\UnitTestSample\UnitTestSample.py
test_ToString_bad (__main__.UtilsTestCase) ... ok
test_ToString_bytes (__main__.UtilsTestCase) ... ok
test_ToString_none (__main__.UtilsTestCase) ... ok
test_ToString_not_none (__main__.UtilsTestCase) ... ok
test_ToString_str (__main__.UtilsTestCase) ... ok

----------------------------------------------------------------------
Ran 5 tests in 0.016s

OK
The interactive Python process has exited.
>>> 

That is all for this post, if you want to know more about unit testing with Python, I suggest you start here: https://docs.python.org/3/library/unittest.html#module-unittest

You might also want to look at Nose2, you can find that here: http://nose2.readthedocs.io/en/latest/

nose2 is the next generation of nicer testing for Python, based on the plugins branch of unittest2. nose2 aims to improve on nose by:

• providing a better plugin api
• being easier for users to configure
• simplifying internal interfaces and processes
• supporting Python 2 and 3 from the same codebase, without translation
• encouraging greater community involvement in its development

Summer of code 2017: Python, Day 29 Else after For and While loops

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 29. today I decided to take a look at how an Else block works with For and While loops in Python. This is not how you would expect it to work if you are coming to Python from another language.


Take a look at the following code

for i in range(5):
    print(i)
else:
    print('else')

What do you think will happen? Will the else part be printed?

Let's run it

    >>> for i in range(5):
...     print(i)
... else:
...     print('else')
... 
0
1
2
3
4
else
>>> 

So after the for loop finished, the else block gets executed. To me that was very surprising, I had not expected this.


What will happen if we put a break in the loop, now the code looks like this

for i in range(5):
    print(i)
    if i == 2:
        break
else:
    print('else')

What do you think happens now? Let's executed it

>>> for i in range(5):
...     print(i)
...     if i == 2:
...         break
... else:
...     print('else')
... 
0
1
2
>>> 

Interestingly the else block did not get executed, this is because the else will only execute if the loop completed

From the docs(I have made the relevant part bold)

The for statement is used to iterate over the elements of a sequence (such as a string, tuple or list) or other iterable object:
for_stmt ::= "for" target_list "in" expression_list ":" suite ["else" ":" suite]
The expression list is evaluated once; it should yield an iterable object. An iterator is created for the result of the expression_list. The suite is then executed once for each item provided by the iterator, in the order of ascending indices. Each item in turn is assigned to the target list using the standard rules for assignments, and then the suite is executed.

When the items are exhausted (which is immediately when the sequence is empty), the suite in the else clause, if present, is executed, and the loop terminates.
A break statement executed in the first suite terminates the loop without executing the else clause's suite. A continue statement executed in the first suite skips the rest of the suite and continues with the next item, or with the else clause if there was no next item.

Let's look at what happens when the sequence is empty now

>>> for i in []:
...     print(i)
... else:
...     print('else')
... 
else
>>> 

As you can see the else block get executed

If you have a while loop that is initially false, the else block will also get executed

>>> while False:
...     print('it is false')
... else:
...     print('else')
... 
else
>>> 

So why does Python have this? One way is to know that you have executed the loop all the way to the end of the loop without having run a break statement.

Take a look at this silly example

>>> x = 10
>>> for i in range(5):
...     if i == x:
...         print('x found')
...         break
... else:
...     print('x not found')
... 
x not found
>>> 

As you can see the else block is executed and we can see that x was not found. Now normally you would store the state in a variable or you would have a helper function that would return false instead.

So that is all for today. Next up is unit testing in Python

Summer of code 2017: Python, Day 23 Files

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 23. today I will take a look at files

First lets take a look at the open() function, you need to pass in the path, everything else is optional

You should always pass in the mode for clarity

Here are all the open() modes

Character	Meaning
'r'	open for reading (default)
'w'	open for writing, truncating the file first
'x'	open for exclusive creation, failing if the file already exists
'a'	open for writing, appending to the end of the file if it exists
'b'	binary mode
't'	text mode (default)
'+'	open a disk file for updating (reading and writing)
'U'	universal newlines mode (deprecated)

Writing to a file in Python

Here is an example of how to write to a file

f = open('boo.txt', mode ='wt', encoding='utf=8')
f.write('the first line')
f.write('line 2\n')
f.write('line 3')
f.write('line 4')
f.close()

Here is what the file looks like after running the example above

As you can see if you don't supply line breaks, there won't be any line breaks in the file, this is why there is only 1 line break in the file

If you run the same example again, the same exact file will be created, this is because mode w will truncate the file first

If you change the mode from wt to at, then the text will be appended

f = open('boo.txt', mode ='at', encoding='utf=8')
f.write('the first line')
f.write('line 2\n')
f.write('line 3')
f.write('line 4')

Reading a file in Python

Reading files is similar to writing files, you still use open and specify mode and encoding
To read the first 10 characters, you can specify read(10)
If you want to read the whole file, you can just specify read(), if you specify read() again, you will get back an empty string. To start reading from the beginning again, you can use seek(0)
To read a line at a time, use readline()

Here is how all this above looks inside the REPL

>>> f = open('boo.txt', mode ='rt', encoding='utf=8')
>>> f.read(10)
'the first '
>>> f.read()
'lineline 2\nline 3line 4the first lineline 2\nline 3line 4the first lineline 2\nline 3line 4'
>>> f.read()
''
>>> f.seek(0)
0
>>> f.readline()
'the first lineline 2\n'
>>> f.readline()
'line 3line 4the first lineline 2\n'
>>> f.readline()
'line 3line 4the first lineline 2\n'
>>> f.readline()
'line 3line 4'
>>> f.readline()
'' 

That is all for today,  short and sweet....

Summer of code 2017: Python, Day 20 Classes

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 20. today I will take a look at classes

All types in Python have a class
Class instances are created by calling the class as if it were a function

Here is the simplest form of a class

class Test:
    """description of class"""
 
   
 
    def val(self, val):
        return val

If we now instantiate the class and pass in the string "denis' to the val method, the string is returned

>>> f = Test()
>>> print(f.val("denis"))
denis
>>> 

In Python self is similar to this in Java or c#

Implementation details are denoted by a leading underscore, in Python there are no public, protected or private access modifiers like in Java or c#

Here is an example of the class where we added __init__ and we are also returning _val instead of val

class Test:
    """description of class"""
 
    def __init__(self, val):
        self._val = val
 
    def val(self):
        return self._val

Here is what the output looks like now

>>> f = Test("denis")
>>> print(f.val())
denis
>>> f = Test("yo")
>>> print(f.val())
yo
>>> 

As you can see it is all pretty simple stuff at this point

To do inheritance, you add the base class name in parentheses after the class name. So if a class SomeTest inherits from Test, you would specify it like this:  class SomeTest(Test)

Here is what the code looks like for both classes

class Test:
    """description of class"""
 
    def __init__(self, val):
        self._val = val
 
    def val(self):
        return self._val
 
 
class SomeTest(Test):
    """description of class"""
 
 
    def someVal(self):
        return self._val

Now we can call both the val and someVal methods from the SomeTest class because the val method exists in the test class from which the SomeTest class inherits

Here is the output

>>> f = Test("yo")
>>> print(f.val())
yo
>>> g = SomeTest("Den")
>>> print(g.val())
Den
>>> print(g.someVal())
Den
>>> 

I am sure I will be revisiting and adding to my notes when I start messing around with classes more

That is it.. a short post for a short Holiday week in the US

Summer of code 2017: Python, Day 18 Iterables

As explained in my Summer of code 2017: Python post I decided to pick up Python

This is officially day 18. today I will take a look at iterables


Comprehensions

A comprehension is a list, a set or a dictionary that is computed via a set of looping and filtering instructions. The comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses

List Comprehensions

The basic syntax for list comprehensions looks like this

[expr(item) for item in iterable]

Here is an example in code, the code below returns the length for each item in the list

>>> words= "A comprehension is a list, a set or a dictionary that is computed via a set
 of looping and filtering instructions".split()
>>> words
['A', 'comprehension', 'is', 'a', 'list,', 'a', 'set', 'or', 'a', 'dictionary', 'that',
 'is', 'computed', 'via', 'a', 'set', 'of', 'looping', 'and', 'filtering', 'instructions']
>>> [len(word) for word in words]
[1, 13, 2, 1, 5, 1, 3, 2, 1, 10, 4, 2, 8, 3, 1, 3, 2, 7, 3, 9, 12]
>>> 

Pretty simple stuff, nothing too complicated here


Set Comprehensions

The basic syntax for set comprehensions looks like this

{expr(item) for item in iterable}

Here is an example in code, the code below returns the length as well as the max character for each item of the set

>>> x = {'abc','def','vvv','nnn'}
>>> type(x)
<class 'set'>
>>> {len(str(x)) for x in x}
{3}
>>> {max(str(x)) for x in x}
{'f', 'v', 'n', 'c'}
>>> 

As you can see the set is distinct, the number 3 is only returned once. Also the set is unordered



Dictionary Comprehensions


The basic syntax for dictionary comprehensions looks like this

{ key_expr:value_expr  for item in iterable }

Here is an example in code, the code below will swap the key value pair

>>> stocks ={
'TGT' :  '51.33', 
'AAPL': '147.48',
'MSFT':  '71.49',
'F'   :  '11.12', 
'INTC':  '34.37'}
>>> stocks
{'TGT': '51.33', 'AAPL': '147.48', 'MSFT': '71.49', 'F': '11.12', 'INTC': '34.37'}
>>> price_to_stock = {ticker: price for price, ticker in stocks.items()}
>>> price_to_stock
{'51.33': 'TGT', '147.48': 'AAPL', '71.49': 'MSFT', '11.12': 'F', '34.37': 'INTC'}
>>> 

We can make it more interesting by using a condition and only return the key value pair if the price is the max price for that ticker. I added a price of 1000 for AAPL and a price of 5000 for F. As you can see in the code below, I have used a filtering predicate if max(price)

>>> stocks ={
'TGT' :  '51.33', 
'AAPL': '147.48',
'AAPL': '1000',
'MSFT':  '71.49',
'F'   :  '11.12', 
'F'   :  '5000',
'INTC':  '34.37'}
>>> price_to_stock = {price: ticker for ticker, price in stocks.items() if max(price)}
>>> price_to_stock
{'51.33': 'TGT', '1000': 'AAPL', '71.49': 'MSFT', '5000': 'F', '34.37': 'INTC'}
>>> 

As you can see for Apple (AAPL), we return the pair with the price of 1000, for F (Ford), we return the pair with the price of 5000, the other prices for AAPL and F are not returned

Generators

Iterator

An object representing a stream of data. Repeated calls to the iterator’s __next__() method (or passing it to the built-in function next()) return successive items in the stream.

>>> i = ['a', 'b', 'c', 'd' ]
>>> iterator =iter(i)
>>> next(iterator)
'a'
>>> next(iterator)
'b'
>>> next(iterator)
'c'
>>> 

When no more data is available a StopIteration exception is raised instead. At this point, the iterator object is exhausted and any further calls to its __next__() method just raise StopIteration again.

Here is what that looks like

>>> next(iterator)
'd'
>>> next(iterator)
Traceback (most recent call last):
  File "", line 1, in 
    next(iterator)
StopIteration
>>> 

We can trap this by checking for the StopIteration exception, here is an example

>>> try:
 next(iterator)
except(StopIteration):
    print ('max value reached')

    
max value reached
>>> 

I also looked at itertools, itertools  implements a number of iterator building blocks inspired by constructs from APL, Haskell, and SML. Each has been recast in a form suitable for Python.

Here are some of these iterators

count()

cycle()

repeat()

accumulate()

chain()

chain.from_iterable()

compress()

dropwhile()

filterfalse()

groupby()

islice()

starmap()

takewhile()

tee()

zip_longest()

That was all for today, next I will look at classes