Yet Another Data Science Python Tutorial¶

This is a basic tutorial to teach a beginner the Python needed to perform Data Science. It assumes some knowledge of getting around a computer and the Internet, and a basic background in mathematics ideally with some statistics. By the end you should have learned how to set up a Python environment, prepare and run a significant Python program, and be ready to learn and use basic Data Science libraries such as NumPy, Pandas, SciKit Learn, and Matplotlib to do real Data Science.

Data Science¶

Data science uses mathematics and statistics to enable organizations to extract insights and guide operational decisions from raw data. It combines mathematics, computing and domain knowledge to solve real-world problems and uncover hidden patterns. Companies are generating vast amounts of data that data scientists can clean, statistically analyze, transform, and visualize to discover patterns and predict events in the future. These can be presented as actionable insights to business decision makers.

Why Python?¶

Python is the dominant programming language for data science because its simplicity makes it easy to learn and use and it is currently the most popular programming language in the world. It was created by Guido van Rossum in 1991, named for the BBC comedy series Monty Python’s Flying Circus. It's usefulness for data science is because of the large ecosystem of open-source third-party packages for Data Science and the large community of developers who create libraries and tools to make Python easier to use.

Data¶

Humans are the third most intelligent species on the planet (after mice and dolphins, as anyone who has seen or read Hitchhikers Guide to Galaxy knows). Humans are distinguished by their ability to create symbols, or abstract signs that mean some concept or object in the world. The oldest human created object symbols include 67,800 year old handprints and 45,500 year old painted animals found in caves in Indonesia. Animal symbols have been found that include 20,000 year old markings that tracked their mating behavior as an aid to hunters. The oldest evidence of counting has been found in 44,000 year old bones with notches on a bone that may have marked a lunar cycle.

Tally marks were a form of concrete counting where symbols formed a one-to-one correspondence with physical objects or concepts. Eventually shepherds used stones to count their flock and tokens were used in the Middle East to count agricultural goods and livestock. Over time, the tokens evolved to represent abstract quantities of what was actually counted. These tokens became marks inscribed on tablets that led to the development of cuneiform as a method of writing letters, words and text.

Data consists of symbols for descriptive facts, figures, observations, symbols, and other useful information. These can be electronically stored as numbers, letters, and text for automatic processing in the computer. The purpose of data science is to inspect, clean, transform, and interpret raw data to support data-driven decisions.

Basic Python Data Types¶

Python has four basic built-in types of data:

• Strings: sequences of text characters
• Integers: negative and positive whole numbers, without fractions
• Floats: continuous numbers with fractions
• Booleans: validity that conditions hold

Note in the following that Python code, lines beginning with '#' are comments for descriptive purposes only and are ignored.

Strings¶

Strings are dynamic sequences of text characters inside matching single or double quotes.
These are examples of creating strings.

String characters can be accessed by position counting from position 0 as the first character. Characters can be accessed from the end of the string with negative positions. The string "STRING" can be visualized as:

String slices can be accessed by referencing two positions separated by a colon :. The characters are selected up to, but not including, the second position.

Slices can include negative positions. Again, the characters are selected up to, but not including, the second position.

Either the first or second position may be omitted.

• If the first position is omitted, the slice starts at the beginning of the string, up to but not including the second position.
• If the second position is omitted, the slice ends at with the last character of the string.

There are a variety of operations that can be performed on strings.

Integers¶

Integers are negative and positive whole numbers, without fractions. The result of any integer operation is always an integer. As anyone familiar with Hitchhikers Guide to Galaxy would know, the ultimate answer is always 42.

Floats¶

Floats are negative and positive real numbers with fractions. The result of any floating operation is always an float.

Booleans¶

Booleans are truth values, and always have one of two values True and False. There are a number of comparison operations that return booleans. They work for integer, float, as well as string types. For integers and floats, they work as expected. For strings, characters are compared from the left to find the result.

There are a number of string functions that return booleans.

Operator Precedence¶

The table below shows the precedence and associativity of arithmetic operators

Python Data Structures¶

Python data structs collect group data value. These include:

• Lists: an ordered sequence of any values
• Sets: a group of values with no duplicates
• Dictionaries: a group where values are associated with a key

Lists¶

Lists are dynamic sequences of values that can be accessed by position. Lists are formed with enclosing values of any type between square brackets []. List values are accessed by position counting from 0:

• list[0] is the first element
• list[1:3] are the first up to the third element (i.e., elements 1 and 2)
• list[-1] is the last element

and so on. The list ["text", 1, True, 4.2] can be visualized as:

There are a variety of operations that can be performed on lists.

Sets¶

Sets are an unordered group of values with no duplicates. Sets have rich set of operations as defined by set theory, of which a useful set are implemented in Python.

Dictionaries¶

Dictionaries are a group where you look up a value with another value, a key, like looking up a word definition in a real dictionary. Dictionaries are formed with enclosing pairs of values of any type between braces {}. Dictionary values are accessed by key using square brackets. The dictionary {'leaf': 'green', 'sky': 'blue', 'apple': 'red'} can be visualized as:

Python RegEx¶

A RegEx, or Regular Expression, is a sequence of characters that forms a search pattern.

RegEx can be used to check if a string contains the specified search pattern.

RegEx Module¶

Python has a built-in package called re, which can be used to work with Regular Expressions. To use it, we import the re module.

import re

RegEx Functions¶

The re module offers a set of functions that allows us to search a string for a match:

Metacharacters¶

Metacharacters are characters with a special meaning:

Special Sequences¶

A special sequence is a \ followed by one of the characters in the list below, and has a special meaning:

Sets¶

A set is a set of characters inside a pair of square brackets [] with a special meaning:

The findall() Function¶

The `findall() function returns a list containing all matches.

The search() Function¶

The search() function searches the string for a match.

• If there is more than one match, only the first occurrence of the match will be returned.

The split() Function¶

The split() function returns a list where the string has been split at each match.

You can control the number of occurrences by specifying the maxsplit parameter.

The sub() Function¶

The sub() function replaces the matches with the text of your choice.

You can control the number of replacements by specifying the count parameter.

Matching with groups¶

Text matching a pattern can be captured into a group by putting the match pattern between parentheses (). The text matching each group can be accessed separately.

Placeholders¶

A dataset is the collection of the data being processed in a data analytics session. The current values for all that data at any point in time is a data state. The initial data state are all the available raw data. This data is modified step by step as it is cleaned, analyzed, transformed, and visualized. Placeholders are needed to hold the data from step to step. When all steps are completed, the data is no longer needed and is released. Any data needed for later must be saved before it is removed.

The placeholders are called variables because the data they hold may be changed from step to step. This is distinct from mathematical variables, whose value never changes once it is associated with them. The processing proceeds through a series of data states which are the values contained by all existing variables.

Programs¶

A program automates the processing of data through data states, from the initial raw data until the last data state is handled and the session is over. A program is a recipe and consists of a series of steps where some data is needed for each step, the data is modified, and the changed data results in a new data state. A recipe for making bread gives an example.

• make the dough
  • input: flour, water, and yeast
  • processing: mix and let rise
  • output: raw dough
• knead the dough
  • input: raw dough
  • processing: roll, press, and fold the dough
  • output: smooth dough
• bake the dough:
  • input: smooth dough
  • processing: heat in the over
  • output: bread

The initial data state is (flour, water, yeast) and the final data state is (bread). In Python the data state is stored as the types of data and data structures above. Each data item is assigned to a variable that holds the data through the next steps, until that data is no longer needed and the variable and data are released (called garbage collection). A variable's data may be changed or combined with other variables' data and assigned to a new variable. The data state is the set of values assigned to all variables at a given point in time.

Program Statements¶

A program statement is responsible for each step of the processing that changes the data state. Program statements may:

• assign data to a variable
• modify the data associated with a variable
• pass data to a function that uses the data to cause some external effect such as:
  • saving the data
  • creating a report from the data
  • visualizing the data
  • giving the data to a machine learning algorithm to predict future effects

Each statement modifies the data state. The progression of a program as it is run is:

(initial data state)
program statement 1
(data state 1)
program statement 2
(data state 2)
...
last program statement
(last data state)
(all data is released and the program ends)

Writing Python programs¶

Program statements, for our purposes, are written in Python and first some discussion of writing Python programs is required. Python programs are plain text files. They are created in plain text editors such as:

• Notepad++ (Windows)
• TextEdit (MacOS)
• Sublime (Windows and MacOS)
• Visual Studio Code (Windows and MacOS)
• Applications, such as PyCharm and Visual Studio Code, and websites such as Google Colab and w3schools, have built in text editors The files names usually end with “.py”.

Python program structure¶

Comments are included to give text descriptions of what the code is doing in two ways:

• Text following "#" on one line is ignored through the end of the line
• Text between two single lines containing three double-quotes (“””) is ignored:

“””
This is an ignored comment about the following program code.
“””

Long program lines can be continued on the next line, they are usually indented using tabs or spaces

# A program line too long to fit on a page
foo = long_function_name(var_one, var_two,
                         var_three, var_four)

Occasionally a group of statements must be processed together under some condition, called a block. These lines have exactly equal indentation (any size) to indicate processing them together:

# two lines run together if a condition is true
if (x < 3 and y == 4):
    z = x + y
    print(x, y, z)

Blank lines can be added anywhere between program lines to make it more readable. At the beginning of files, programs usually indicate any standard libraries of code they use with “import”

import os
import sys
# rest of the program 

Python program statements¶

Programs consist of a series of lines containing statements that are processed from the beginning of the file to the end. When all lines of the program have been processed, all values in the data state are released. All data needs to have been processed by that point, and once it is released the computer is available to run another program. These are various types of statements available.

Variable assignment statements¶

The most basic program statement assigns a value to a variable with "=". Variables are assigned with expressions. An expression is a combination of programming language variables, constants, operators, and functions that produce a single value. Expressions follow the rules of standard mathematical statements. Any place a variable or function appears in an expression, its value is substituted. You have seen many examples of Python expressions above. Each are in a code block where the expression is printed after it is evaluated.

In the following, code blocks will now contain a series of Python statements. The statements are processed one at a time, and the expression listed on the last line is printed.

The data state at this point is:

• (p, value 42) Any variable then can be used and have its value substituted in an expression in any following statement.

A variable initially is empty. It must be assigned a value before it is used, else it gives an error. This commonly occurs when a variable name is misspelled.

Statements that modify the value of variables¶

Once variables are assigned, other statements can modify the values of variables. After assigning a variable in a statement, the variable can be assigned a new value in a later statement.

Statements that modify lists¶

The following are operations that modify lists assigned to variables

Statements that modify sets¶

The following are operations that modify sets assigned to variables

Statements that modify dictionaries¶

The following are operations that modify dictionaries assigned to variables

Statements that control the flow of processing statements¶

Once variables are assigned, other statements can modify the values of variables. After assigning a variable in a statement, the variable can be assigned a new value in a later statement.

if Statements¶

Perhaps the most well-known statement type is the if statement. For example:

There can be zero or more elif parts, and the else part is optional. The keyword ‘elif’ is short for ‘else if’, and is useful to avoid excessive indentation. An if … elif … elif … sequence is a substitute for the switch or case statements found in other languages.

for Statements¶

The for statement in Python differs a bit from what you may be used to in C or Pascal. Rather than always iterating over an arithmetic progression of numbers (like in Pascal), or giving the user the ability to define both the iteration step and halting condition (as C), Python’s for statement iterates over the items of any sequence (a list or a string), in the order that they appear in the sequence. For example (no pun intended):

Code that modifies a collection while iterating over that same collection can be tricky to get right. Instead, it is usually more straight-forward to loop over a copy of the collection or to create a new collection:

The range() Function¶

If you do need to iterate over a sequence of numbers, the built-in function range() comes in handy. It generates arithmetic progressions:

The given end point is never part of the generated sequence; range(10) generates 10 values, the legal indices for items of a sequence of length 10. It is possible to let the range start at another number, or to specify a different increment (even negative; sometimes this is called the ‘step’):

To iterate over the indices of a sequence, you can combine range() and len() as follows:

break and continue Statements¶

The break statement breaks out of the innermost enclosing for or while loop:

The continue statement continues with the next iteration of the loop:

else Clauses on Loops¶

In a for or while loop the break statement may be paired with an else clause. If the loop finishes without executing the break, the else clause executes.

In a for loop, the else clause is executed after the loop finishes its final iteration, that is, if no break occurred.

In a while loop, it’s executed after the loop’s condition becomes false.

In either kind of loop, the else clause is not executed if the loop was terminated by a break. Of course, other ways of ending the loop early, such as a return or a raised exception, will also skip execution of the else clause.

This is exemplified in the following for loop, which searches for prime numbers:

(Yes, this is the correct code. Look closely: the else clause belongs to the for loop, not the if statement.)

One way to think of the else clause is to imagine it paired with the if inside the loop. As the loop executes, it will run a sequence like if/if/if/else. The if is inside the loop, encountered a number of times. If the condition is ever true, a break will happen. If the condition is never true, the else clause outside the loop will execute.

When used with a loop, the else clause has more in common with the else clause of a try statement than it does with that of if statements: a try statement’s else clause runs when no exception occurs, and a loop’s else clause runs when no break occurs. For more on the try statement and exceptions, see Handling Exceptions.

match Statements¶

A match statement takes an expression and compares its value to successive patterns given as one or more case blocks. This is superficially similar to a switch statement in C, Java or JavaScript (and many other languages), but it’s more similar to pattern matching in languages like Rust or Haskell. Only the first pattern that matches gets executed and it can also extract components (sequence elements or object attributes) from the value into variables. If no case matches, none of the branches is executed.

The simplest form compares a subject value against one or more literals:

Note the last block: the “variable name” _ acts as a wildcard and never fails to match.

You can combine several literals in a single pattern using | (“or”):

Study that one carefully! The first pattern has two literals, and can be thought of as an extension of the literal pattern shown above. But the next two patterns combine a literal and a variable, and the variable binds a value from the subject (point). The fourth pattern captures two values, which makes it conceptually similar to the unpacking assignment (x, y) = point.

Defining Functions¶

We can create a function that writes the Fibonacci series to an arbitrary boundary:

The keyword def introduces a function definition. It must be followed by the function name and the parenthesized list of formal parameters. The statements that form the body of the function start at the next line, and must be indented.

The first statement of the function body can optionally be a string literal; this string literal is the function’s documentation string, or docstring. (More about docstrings can be found in the section Documentation Strings.) There are tools which use docstrings to automatically produce online or printed documentation, or to let the user interactively browse through code; it’s good practice to include docstrings in code that you write, so make a habit of it.

The execution of a function introduces a new symbol table used for the local variables of the function. More precisely, all variable assignments in a function store the value in the local symbol table; whereas variable references first look in the local symbol table, then in the local symbol tables of enclosing functions, then in the global symbol table, and finally in the table of built-in names. Thus, global variables and variables of enclosing functions cannot be directly assigned a value within a function (unless, for global variables, named in a global statement, or, for variables of enclosing functions, named in a nonlocal statement), although they may be referenced.

The actual parameters (arguments) to a function call are introduced in the local symbol table of the called function when it is called; thus, arguments are passed using call by value (where the value is always an object reference, not the value of the object). [1] When a function calls another function, or calls itself recursively, a new local symbol table is created for that call.

A function definition associates the function name with the function object in the current symbol table. The interpreter recognizes the object pointed to by that name as a user-defined function. Other names can also point to that same function object and can also be used to access the function:

Coming from other languages, you might object that fib is not a function but a procedure since it doesn’t return a value. In fact, even functions without a return statement do return a value, albeit a rather boring one. This value is called None (it’s a built-in name). Writing the value None is normally suppressed by the interpreter if it would be the only value written. You can see it if you really want to using print():

It is simple to write a function that returns a list of the numbers of the Fibonacci series, instead of printing it:

This example, as usual, demonstrates some new Python features:

The return statement returns with a value from a function. return without an expression argument returns None. Falling off the end of a function also returns None.

The statement result.append(a) calls a method of the list object result. A method is a function that ‘belongs’ to an object and is named obj.methodname, where obj is some object (this may be an expression), and methodname is the name of a method that is defined by the object’s type. Different types define different methods. Methods of different types may have the same name without causing ambiguity. (It is possible to define your own object types and methods, using classes, see Classes) The method append() shown in the example is defined for list objects; it adds a new element at the end of the list. In this example it is equivalent to result = result + [a], but more efficient.

More on Defining Functions¶

It is also possible to define functions with a variable number of arguments. There are three forms, which can be combined.

Default Argument Values¶

The most useful form is to specify a default value for one or more arguments. This creates a function that can be called with fewer arguments than it is defined to allow. For example:

This function can be called in several ways:

giving only the mandatory argument: ask_ok('Do you really want to quit?')

giving one of the optional arguments: ask_ok('OK to overwrite the file?', 2)

or even giving all arguments: ask_ok('OK to overwrite the file?', 2, 'Come on, only yes or no!')

This example also introduces the in keyword. This tests whether or not a sequence contains a certain value.

The default values are evaluated at the point of function definition in the defining scope, so that

Keyword Arguments¶

Functions can also be called using keyword arguments of the form kwarg=value. For instance, the following function:

accepts one required argument (voltage) and three optional arguments (state, action, and type). This function can be called in any of the following ways:

but all the following calls would be invalid:

parrot()                     # required argument missing
parrot(voltage=5.0, 'dead')  # non-keyword argument after a keyword argument
parrot(110, voltage=220)     # duplicate value for the same argument
parrot(actor='John Cleese')  # unknown keyword argument

In a function call, keyword arguments must follow positional arguments. All the keyword arguments passed must match one of the arguments accepted by the function (e.g. actor is not a valid argument for the parrot function), and their order is not important. This also includes non-optional arguments (e.g. parrot(voltage=1000) is valid too). No argument may receive a value more than once. Here’s an example that fails due to this restriction:

When a final formal parameter of the form **name is present, it receives a dictionary (see Mapping Types — dict) containing all keyword arguments except for those corresponding to a formal parameter. This may be combined with a formal parameter of the form name (described in the next subsection) which receives a tuple containing the positional arguments beyond the formal parameter list. (name must occur before **name.) For example, if we define a function like this:

It could be called like this:

Generators¶

There are statements that combine loops and generate data structures.

List Comprehensions¶

List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.

For example, assume we want to create a list of squares, like:

Note that this creates (or overwrites) a variable named x that still exists after the loop completes. We can calculate the list of squares without any side effects using:

A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. The result will be a new list resulting from evaluating the expression in the context of the for and if clauses which follow it. For example, this listcomp combines the elements of two lists if they are not equal:

and it’s equivalent to:

Note how the order of the for and if statements is the same in both these snippets.

If the expression is a tuple (e.g. the (x, y) in the previous example), it must be parenthesized.

List comprehensions can contain complex expressions and nested functions:

Nested List Comprehensions¶

The initial expression in a list comprehension can be any arbitrary expression, including another list comprehension.

Consider the following example of a 3x4 matrix implemented as a list of 3 lists of length 4:

The following list comprehension will transpose rows and columns:

As we saw in the previous section, the inner list comprehension is evaluated in the context of the for that follows it, so this example is equivalent to:

which, in turn, is the same as:

In the real world, you should prefer built-in functions to complex flow statements. The zip() function would do a great job for this use case:

print()¶

print(*objects, sep=' ', end='\n', file=None, flush=False) Print objects to the text stream file, separated by sep and followed by end. sep, end, file, and flush, if present, must be given as keyword arguments.

All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no objects are given, print() will just write end.

The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used. Since printed arguments are converted to text strings, print() cannot be used with binary mode file objects. For these, use file.write(...) instead.

Output buffering is usually determined by file. However, if flush is true, the stream is forcibly flushed.