Tuesday, April 18, 2017

Now You Have Two Problems

Now You Have Two Problems

There’s a programming joke that goes like this
You have a problem and you think “I’ll use a
regular expression.” Now you have 2 problems.
Some people misuse regular expressions, but sometimes they are just what you need.
This Stack Overflow question involves parsing a string containing math operators and
integers. The answers ranged from simple to complex, and at least one answer required reading The Dragon Book.
In the spirit of “doing the simplest thing that can possibly work”, I present this regular expression
[()*/+-]|\d+
  • [()*/+-] means match one character from the set ()*/+-
  • the \d stands for a single digit and the + means appearing one or more times
  • the | is the logical OR
So the expression means "match one of these punctuation characters OR an integer number". It will silently ignore anything else such as whitespace, and this is exactly the behavior we want.

Sunday, April 16, 2017

Top 3 Reasons You Need a GitHub Account

Top 3 Reasons You Need a GitHub Account

Reason #1 : GitHub is your new resume

Having a public GitHub repository lets potential employers see samples of your work.

Reason #2 : GitHub lets you work in teams

GitHub supports team projects in 3 ways
  1. it provides a common workspace for the team
  2. it saves a complete copy of all your files
  3. it lets you see what changes were made and who made them

Reason #3 : GitHub has great resources for students

The Student Developer Pack gives you free access to over 20 professional software development tools.
So don't wait. Go create your GitHub account today.

Tuesday, April 4, 2017

Reading Python error messages

What every beginner needs to know

It's normal to make mistakes when you first start learning to program. Python uses exceptions to report errors. Usually, your program will just print an error message and abort. These error messages contain a lot of information, but you have to know how to read them.


Here’s a sample error message




The message has 2 parts
  1. A traceback, showing the path taken by the program when the error occurred.
  2. The name of the exception followed by a short description.


The traceback is important because it shows the exact line of code containing the error. In this case, we can see the problem is in line 9 of the “C” method. We can also see how we got there
  • method A (line 5), which called
  • method B (line 7), which called
  • method C (line 9), where the error occurred

The last line of the message reads “ZeroDivisionError: float division by zero”. This lets us know that dividing by zero is not allowed and we need to fix it.

So don’t panic the next time one of your programs fails. Read the error message carefully and you can probably figure out what’s wrong.

Friday, January 20, 2017

The misunderstood max() function

The built-in functionmax is one of the most misunderstood functions in Python. That’s because everyone who tries to explain it uses the same stupid example.
What example am I talking about?
Taking a list of numbers and finding the largest number in the list.
That’s just about the most useless example I can think of. Nobody is going to give you a list of numbers and say “find the biggest one”. That just doesn’t happen in real life.
Here’s the problem: nobody tells you how you’re really supposed to use the max function.

It’s a searching function

Searching and sorting are two of the most fundamental algorithms in programming, so it makes sense that Python would give you an easy way to search for things. The way you search with max function is by specifying a key function through the “key=key_function” parameter.

The key function

The key function returns something interesting about an object. This is usually one of the object’s attributes or it’s something that can be easily calculated from the object.
The key function gets called for every object in the list. Once the max function has generated a key value for each object, it searches for the object that maximizes that key value.
For example
  • if the key value is a weight, max finds the heaviest object
  • if the key value is an age, max finds the oldest object
  • if the key value is a distance, max finds the object that’s farthest away
  • if the key value is a length, max finds the longest object
  • if the key value is a price, max finds the most expensive object
In database terms, it would look something like this
SELECT object
FROM list_of_objects
WHERE object.whatever = MAX(object.whatever)
So your challenge is to come up with good key functions. Here are some examples to help you get started.
def key_longest_sequence(x):
    ‘’’ Let’s you search for the longest string, list, or tuple ‘’’
    return len(x)

def key_most_words(text):
    ‘’’ Let’s you search for the string containing the most words ‘’’
    return len( text.split() )

def key_most_important(x):
    ‘’’ Let’s you search for the highest priority object ‘’’
    return x.priority
Your own creativity is the only limit!

Sunday, January 1, 2017

Writing a Small Game From Scratch (video)

Bulls And Cows


Designing The Game

Here’s the basic strategy for writing any computer program:
  1. Understand the problem you are trying to solve.
  2. Break the problem into smaller pieces.
  3. Combine the pieces into a finished program.

Understanding the game

Start by reading the Wikipedia article that explains the Bulls and Cows game. Try to imagine how you would play this game against the computer.

Break up the problem

Next, try to think of all the different actions your program will have to perform. Here’s the list I came up with:
  • create the secret number
  • get the player’s guess
  • count the number of bulls and cows
  • determine if a digit is a bull or a cow
  • display the number of bulls and cows in the player’s guess

Draw a structure chart

A structure chart will help you visualize how the pieces of the program fit together. Sketch out a structure chart using paper and pencil. Don’t waste time trying to make it pretty. Try to organize the activities into top-level, mid-level, and low-level functions.

Implementing The Game

You should design top-down, but you should implement bottom-up. There are several reasons for this:
  1. The lowest level functions are the easiest to write and test because they don’t depend on anything else.
  2. Just like an Egyptian pyramid, each higher level is built on the level below it.

Determining if a digit is a bull or a cow

The is_bull and is_cow functions are the foundation of the game and everything else depends on them.
A digit is a bull if it appears at the same position in both the secret number and the guess.
I could write the function like this
def is_bull(digit, secret, guess):
    ''' Return True if the digit appears at the same position in both numbers '''
    if (digit in secret) and (digit in guess):
        return secret.index(digit) == guess.index(digit)
    else:
        # the digit is missing from one or both numbers
        return False
But I’m not going to. Instead, I’ll write it like this:
def is_bull(digit, secret, guess):
    ''' Return True if the digit appears at the same position in both numbers '''
    try:
        return secret.index(digit) == guess.index(digit)
    except ValueError:
        # the digit is missing from one or both numbers
        return False
I’m not testing for digit in secret or digit in guess because index will raise an exception if the value isn’t found. Don’t repeat functionality that Python gives you for free. Your code will be simpler and easier to understand.
The is_cow function is similar to is_bull. A digit is a cow if it appears in both numbers, but at different positions.
def is_cow(digit, secret, guess):
    ''' Return True if the digit appears at different positions in both numbers '''
    try:
        return secret.index(digit) != guess.index(digit)
    except ValueError:
        # the digit is missing from one or both numbers
        return False

Counting bulls and cows

Now it’s time to count the number of bulls and cows in the player’s guess.
def count_bulls(secret, guess):
    count = 0
    for digit in guess:
        if is_bull(digit, secret, guess):
            count += 1
    return count
def count_cows(secret, guess):
    count = 0
    for digit in guess:
        if is_cow(digit, secret, guess):
            count += 1
    return count
Do you see how I was able to build these new functions from the is_bull and is_cow functions I’ve already written? Let’s keep going.
def print_bulls_and_cows(secret, guess):
    ''' Print a hint to the player. '''
    bulls = count_bulls(secret, guess)
    cows = count_cows(secret, guess)
    message = 'There are {0} bulls and {1} cows.'.format(bulls, cows)
    print message
The game is over when you find 4 bulls. Don’t be afraid to write short functions if they help you express the logic of your program.
def game_over(secret, guess):
    ''' Return True if the game is over. 
    
    The game is over when the player guesses all 4 bulls.
    '''
    return count_bulls(secret, guess) == 4

Generating the secret number and getting the player’s guess

Spend some time getting familiar with Python’s standard modules.
Generating the secret number requires you to know a little bit about Python’s random number module. You can’t just pick a random number between 1000 and 9999 because some of the digits might repeat. For example, you might get
  • 1212
  • 3333
and so forth. Fortunately, the random.sample function does exactly what we want.
import random

def generate_secret_number():
    ''' Return a string of 4 unique digits. '''
    digits = random.sample('0123456789', 4)
    return ''.join(digits)
You’ll also need to get the player’s guess. You should clean up the input by stripping off any leading or trailing spaces.
def new_guess():
    ''' Return the player's guess as a string. '''
    return raw_input('Enter your guess: ').strip()
At this point you’ve written everything except the top level function.
def bulls_and_cows():
    secret_number = generate_secret_number()
    players_guess = ''
    while not game_over(secret_number, players_guess):
        print_bulls_and_cows(secret_number, players_guess)
        players_guess = new_guess()
That’s it!

Tuesday, April 12, 2016

The Shocking Truth About The Del Statement


The del statement doesn’t actually delete objects


Wait … what?

question-mark-1020165_1280.jpg

To understand this, you need to know a little bit about how Python handles variables and assignment.


Objects, references, and the = operator


An assignment statement looks like this:
<variable-name> = <object>
The assignment statement has 3 parts
  1. The variable name.
  2. The object being assigned to the variable.
  3. The = operator, which creates a reference from the variable name to the object.
Point #3 is key to understanding how all this works. It might help if you imagine drawing an arrowhead on the = sign, like this
<variable-name> ⇒ <object>
This emphasizes the fact that variables don’t contain objects, they refer to objects.

If I assign the same object to two variables, they each contain a reference to that object.


Then if I delete the "X" variable here’s what happens

Clearly, the int(29) object didn’t get deleted. You know that because you can still refer to it through the "Y" variable. It was the reference from X to 29 that got deleted. And because you can’t have a variable that doesn’t refer to anything, the variable “X” also got deleted.

So how do you delete an object?


Well, you really can’t. Python does something called “reference counting”. It keeps track of how many references exist to an object. When the number of references drops to zero, the object is marked for “garbage collection”. Exactly how and when garbage collection happens is outside your control.

References

Wednesday, March 16, 2016

The Most Important Programming Skill



Learn to type.

I mean REAL typing ... not hunt-and-peck-like-a-chicken typing. Typing without looking at your hands.

If you can't type, take a look at typing.com or do a Google search for "learn to type".

Why is this important? Because if you can't type, writing code will be harder ... writing documentation will be harder ... writing unit tests will be harder ... in fact, EVERYTHING will be harder.


Tuesday, March 8, 2016

Refactoring And Testing With Doctest

"If you can't test it, don't build it. If you don't test it, rip it out."
Boris Beizer

Python comes with a variety of testing frameworks which are dead simple to use. You really have no excuse not to test your code as you're writing it.

I've put together a short video where you get to look over my shoulder as I refactor the FizzBuzz program. Even though this problem is completely artificial, it still has enough complexity to make refactoring a worthwhile exercise.


Wednesday, March 2, 2016

Mini lesson: De Morgan's Laws

The any() and all() functions

You can apply the logic operators "and" and "or" to a whole series of values using Python's any() and all() functions.


Given a sequence of values S
  • all(S) returns True if every value in S is True
  • any(S) returns True if at least one value in S is True

This means
  • all(S) == (S[0] and S[1] and S[2] ...)
  • any(S) == (S[0] or S[1] or S[2] ...)

De Morgan's Laws

De Morgan's laws define the relationship between the and, or, and not operators.
  • (not A and not B and not C) == not (A or B or C)
  • (not A or not B or not C) == not (A and B and C)
By knowing these laws you can sometimes rewrite complicated logical expressions in a simpler form.

Applying De Morgan's Laws

The all() function lets you test if every value is True, but how would you test if every value is False? You could negate each value (possibly with a list comprehension)

all( [not s for s in S] )

But it's easier to apply De Morgan's law

not any(S)

Similarly, you could test for at least one False value by negating each value

any( [not s for s in S] )

But De Morgan's law allows you to rewrite this as

not all(S)


Summary

So, to summarize
  • all(S) is True when every value in S is True 
  • not any(S) is True when every value in S is False
  • any(S) is True when at least one value in S is True
  • not all(S) is True when at least one value in S is False

References

https://en.wikipedia.org/wiki/De_Morgan%27s_laws

Tuesday, February 23, 2016

Functional Python (UPDATE)

I had some very interesting responses to my previous post.


One concern some people had was the fact that my solution didn't scale for large files because I was reading the entire file into memory all at once. That's certainly a valid point.

It took me a while to come up with an alternate implementation, but I think I've got it. The trick is using reduce() to sum the 3 columns in parallel.


from functools import reduce

def wc_func(filename):

    '''
    Description:

    1. use a generator to read the file and form

       a (1, word_count, char_count) tuple from each line

    2. use reduce to calculate a running total of each column

    '''

    with open(filename, 'r') as f:
        data = ((1, len(line.split()), len(line)) for line in f)
        return reduce (lambda x,y: (x[0]+y[0], x[1]+y[1], x[2]+y[2]), data)

print (wc_func('bigdata.txt'))


What about performance?

It depends. Using the standard CPython, the performance is painfully slow (basically unusable). But using pypy tells a very different story. With a 1.7Gb test file, performance is actually a bit better than the native "wc" Linux command. 


18:54 ~$ which wc /usr/bin/wc   18:54 ~$ which pypy /usr/local/bin/pypy   18:54 ~$ pypy --version Python 2.7.9 (9c4588d731b7, Mar 23 2015, 16:30:30) [PyPy 2.5.1 with GCC 4.6.3]   18:54 ~$ ls -lh bigdata.txt -rw-rw-r-- 1 mpoulin registered_users 1.7G Feb 22 23:48 bigdata.txt   18:54 ~$ time pypy wc_func.py ; time wc bigdata.txt (100000000, 400000000, 1800000000) real 0m39.695s user 0m30.376s sys 0m1.051s 100000000 400000000 1800000000 bigdata.txt real 0m40.433s user 0m31.336s sys 0m0.533s 18:56 ~$


Monday, February 22, 2016

Functional Python

Python allows you to write code in a variety of styles: procedural, object oriented, and functional. What is functional programming and why should you care?

What is functional programming?

Functional programming is a declarative style of programming that tries to minimize side effects and the use of variables. You might already be doing functional programming without realizing it.
  • Have you ever written an SQL query for a database? 
  • Have you ever piped a series of commands in a Linux shell?
If so, you've done functional programming.

A Python example

Linux has a "wc" command that counts lines, words, and characters in a text file. Let's compare two versions, one written in a procedural style and the other in a functional style.

Procedural wc

def wc_1(filename):
   '''
   Description:

   1. initialize some counter variables
   2. loop through a file line by line
   3. for each line, increment the counters
   '''

   line_count = 0
   word_count = 0
   char_count = 0

   with open(filename, 'r') as f:
       for line in f:
           line_count += 1
           char_count += len(line)
           word_count += len(line.split())

   return (line_count, word_count, char_count)


Functional wc

def wc_2(filename):
    '''
    Description:

    1. read the file into a list
    2. get the total number of lines
    3. sum the number of characters in each line
    4. sum the number of words in each line
    '''
    
    with open(filename, 'r') as f:
        lines = f.readlines()

    line_count = len(lines)
    char_count = sum(len(line) for line in lines)
    word_count = sum(len(line.split()) for line in lines)

    return (line_count, word_count, char_count)

Comparing the two versions

In the procedural version
  • you have to create 3 separate counter variables
  • you have to initialize them correctly
  • you have to update the counts for each line in the file
In the functional version
  • there are no counter variables
  • you let Python's sum() function tally the totals
I prefer the functional version. I can say "sum the number of characters in each line" or "sum the number of words in each line" and Python knows how to do that. And I know that Python won't make any stupid mistakes like forgetting to initialize a variable or increasing the count by the wrong amount.

Performance

The profiler shows us that performance is about the same for both versions. So with this particular input file there's no performance penalty for reading the entire file into memory.


prun wc_1('python_kwic.txt') 6046 function calls in 0.020 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.011 0.011 0.020 0.020 wc.py:21(wc_1) 2014 0.004 0.000 0.004 0.000 {method 'split' of 'str' objects} 4028 0.004 0.000 0.004 0.000 {len} 1 0.001 0.001 0.001 0.001 {open} 1 0.000 0.000 0.020 0.020 <string>:1(<module>) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}


prun wc_2('python_kwic.txt') 6050 function calls in 0.017 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.008 0.008 0.017 0.017 wc.py:1(wc_2) 4029 0.004 0.000 0.004 0.000 {len} 2014 0.004 0.000 0.004 0.000 {method 'split' of 'str' objects} 1 0.001 0.001 0.001 0.001 {method 'readlines' of 'file' objects} 1 0.001 0.001 0.001 0.001 {open} 2 0.000 0.000 0.000 0.000 {sum} 1 0.000 0.000 0.017 0.017 <string>:1(<module>) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}


Additional Resources


Functional Programming HOWTO -- https://docs.python.org/2/howto/functional.html

Functional Programming in Python -- http://www.oreilly.com/programming/free/functional-programming-python.csp

Mastering Python Lists -- http://amzn.com/B01BIHNWI0