Python for Data Science

From Sinfronteras
Revision as of 19:58, 18 October 2019 by Adelo Vieira (talk | contribs) (→‎DataFrames)
Jump to: navigation, search

For a standard Python tutorial go to Python



Anaconda

Anaconda is a free and open source distribution of the Python and R programming languages for data science and machine learning related applications (large-scale data processing, predictive analytics, scientific computing), that aims to simplify package management and deployment. Package versions are managed by the package management system conda. https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)

En otras palabras, Anaconda puede ser visto como un paquete (a distribution) que incluye no solo Python (or R) but many libraries that are used in Data Science, as well as its own virtual environment system. It's an "all-in-one" install that is extremely popular in data science and Machine Learning.Creating sample array for the following examples:



Installation

https://linuxize.com/post/how-to-install-anaconda-on-ubuntu-18-04/

https://www.digitalocean.com/community/tutorials/how-to-install-the-anaconda-python-distribution-on-ubuntu-18-04



Anaconda comes with a few IDE

  • Jupyter Lab
  • Jupyter Notebook
  • Spyder
  • Qtconsole
  • and others



Anaconda Navigator

Anaconda Navigator is a GUI that helps you to easily start important applications and manage the packages in your local Anaconda installation

You can open the Anaconda Navigator from the Terminal: 

anaconda-navigator



Jupyter

Jupyter comes with Anaconda.

  • It is a development environment (IDE) where we can write codes; but it also allows us to display images, and write down markdown notes.
  • It is the most popular IDE in data science for exploring and analyzing data.
  • Other famoues IDE for Python are Sublime Text and PyCharm.
  • There is Jupyter Lab and Jupyter Notebook



Online Jupyter

There are many sites that provides solutions to run your Jupyter Notebook in the cloud: https://www.dataschool.io/cloud-services-for-jupyter-notebook/

I have tried:

https://cocalc.com/projects/595bf475-61a7-47fa-af69-ba804c3f23f9/files/?session=default
Parece bueno, pero tiene opciones que no son gratis


https://www.kaggle.com/adeloaleman/kernel1917a91630/edit
Parece bueno pero no encontré la forma adicionar una TOC


Es el que estoy utilizando ahora



Courses

  • Udemy - Python for Data Science and Machine Learning Bootcamp
https://www.udemy.com/course/python-for-data-science-and-machine-learning-bootcamp/



Most popular Python Data Science Libraries=

  • NumPy
  • SciPy
  • Pandas
  • Seaborn
  • SciKit'Learn
  • MatplotLib
  • Plotly
  • PySpartk



NumPy

  • NumPy (or Numpy) is a Linear Algebra Library for Python, the reason it is so important for Data Science with Python is that almost all of the libraries in the PyData Ecosystem rely on NumPy as one of their main building blocks.



Installation

It is highly recommended you install Python using the Anaconda distribution to make sure all underlying dependencies (such as Linear Algebra libraries) all sync up with the use of a conda install.


If you have Anaconda, install NumPy by: 

conda install numpy


If you are not using Anaconda distribution: 

pip install numpy


Then, to use it:

import numpy as np
arr = np.arange(0,10)


Arrays

Method/Operation Description/Comments Example 
import numpy as np
Methods for creating NumPy Arrays
From a Python List
 array() We can create an array by directly converting a list or list of lists. my_list = [1,2,3]

np.array(my_list)


my_matrix = [[1,2,3],[4,5,6],[7,8,9]]

np.array(my_matrix)

From Built-in NumPy Methods
 arange() Return evenly spaced values within a given interval. np.arange(0,10)

np.arange(0,11,2)

zeros() Generate arrays of zeros. np.zeros(3)

np.zeros((5,5))

ones() Generate arrays of ones. np.ones(3)

np.ones((3,3))

linspace() Return evenly spaced numbers over a specified interval. np.linspace(0,10,3)

np.linspace(0,10,50)

eye() Creates an identity matrix. np.linspace(0,10,50)
random rand() Create an array of the given shape and populate it with random samples from a uniform distribution over [0, 1). 
np.random.rand(2)
np.random.rand(5,5)


# Another way to invoke a function:
from numpy.random import rand
# Then you can call the function directly
rand(5,5)

randn() Return a sample (or samples) from the "standard normal" distribution. Unlike rand which is uniform. np.random.randn(2)

np.random.randn(5,5)

randint() Return random integers from low (inclusive) to high (exclusive). np.random.randint(1,100)

np.random.randint(1,100,10)

seed() sets the random seed of the NumPy pseudo-random number generator. It provides an essential input that enables NumPy to generate pseudo-random numbers for random processes. See s1 and s2. for explanation. np.random.seed(101)
Others Array Attributes and Methods
 reshape() Returns an array containing the same data with a new shape. arr.reshape(5,5)
max(), min(), argmax(), argmin() Finding max or min values. Or to find their index locations using argmin or argmax. arr.max()

arr.argmax()

shape() Shape is an attribute that arrays have (not a method). NO LO ENTENDI.. REVISAR!


#Length of array

arr_length = arr2d.shape[1]

dtype() You can also grab the data type of the object in the array. arr.dtype
- - - - -
Indexing and Selection
  • How to select elements or groups of elements from an array.
  • The general format is arr_2d[row][col] or arr_2d[row,col]. I recommend usually using the comma notation for clarity.

Creating sample array for the following examples: 

import numpy as np
arr = np.arange(0,10)
# 1D Array:
arr = np.arange(0,11)
#Show
arr
Output: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

# 2D Array
arr_2d = np.array(([5,10,15],[20,25,30],[35,40,45]))
#Show
arr_2d
Output: 
array([[ 5, 10, 15],
       [20, 25, 30],
       [35, 40, 45]])
Bracket Indexing and Selection (Slicing)
 Note: When we create a sub-array slicing an array (slice_of_arr = arr[0:6]), data is not copied, it's a view of the original array! This avoids memory problems! To get a copy, need to use the method copy(). See important note below. 
#Get a value at an index
arr[8]

#Get values in a range
arr[1:5]

slice_of_arr = arr[0:6]

#2D
arr_2d[1]
arr_2d[1][0]
arr_2d[1,0] # The same that above

#Shape (2,2) from top right corner
arr_2d[:2,1:]
#Output: 
array([[10, 15],
       [25, 30]])

#Shape bottom row
arr_2d[2,:]

Fancy Indexing:

Fancy indexing allows you to select entire rows or columns out of order.

Example:
# Set up matrix
arr2d = np.zeros((10,10))

# Length of array
arr_length = arr2d.shape[1]

# Set up array
for i in range(arr_length):
    arr2d[i] = i
    
arr2d
# Output:
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [3., 3., 3., 3., 3., 3., 3., 3., 3., 3.],
       [4., 4., 4., 4., 4., 4., 4., 4., 4., 4.],
       [5., 5., 5., 5., 5., 5., 5., 5., 5., 5.],
       [6., 6., 6., 6., 6., 6., 6., 6., 6., 6.],
       [7., 7., 7., 7., 7., 7., 7., 7., 7., 7.],
       [8., 8., 8., 8., 8., 8., 8., 8., 8., 8.],
       [9., 9., 9., 9., 9., 9., 9., 9., 9., 9.]])

# Fancy indexing allows the following
arr2d[[6,4,2,7]]
# Output:
array([[6., 6., 6., 6., 6., 6., 6., 6., 6., 6.],
       [4., 4., 4., 4., 4., 4., 4., 4., 4., 4.],
       [2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
       [7., 7., 7., 7., 7., 7., 7., 7., 7., 7.]])

Broadcasting


(Setting a value with index range)

Setting a value with index range:

Numpy arrays differ from a normal Python list because of their ability to broadcast.

arr[0:5]=100
#Show

arr

Output: array([100, 100, 100, 100, 100, 5, 6, 7, 8, 9, 10])

#Setting all the values of an Array

arr[:]=99

Get a copy of an Array
 copy() Note: When we create a sub-array slicing an array (slice_of_arr = arr[0:6]), data is not copied, it's a view of the original array! This avoids memory problems! To get a copy, need to use the method copy(). See important note below. arr_copy = arr.copy()
Important notes on Slices
 
slice_of_arr = arr[0:6]
#Show slice
slice_of_arr
Output: array([0, 1, 2, 3, 4, 5])

#Making changes in slice_of_arr
slice_of_arr[:]=99
#Show slice
slice_of_arr
Output: array([99, 99, 99, 99, 99, 99])

#Now note the changes also occur in our original array!
#Show
arr
Output: array([99, 99, 99, 99, 99, 99, 6, 7, 8, 9, 10])

#When we create a sub-array slicing an array (slice_of_arr = arr[0:6]), data is not copied, it's a view of the original array! This avoids memory problems!

#To get a copy, need to use the method copy()
Using brackets for selection based on comparison operators and booleans
 
arr = np.arange(1,11)
arr > 4
# Output:
array([False, False, False, False,  True,  True,  True,  True,  True,
        True])

bool_arr = arr>4
bool_arr
# Output:
array([False, False, False, False,  True,  True,  True,  True,  True,
        True])

arr[bool_arr]
# Output:
array([ 5,  6,  7,  8,  9, 10])

arr[arr>2]
# Output:
array([ 3,  4,  5,  6,  7,  8,  9, 10])

x = 2
arr[arr>x]
# Output:
array([ 3,  4,  5,  6,  7,  8,  9, 10])
- - - - -
Arithmetic operations
 arr + arr

arr - arr

arr * arr

arr/arr

1/arr

arr**3

Warning on division by zero, but not an error!

0/0 -> nan

1/0 -> inf 

import numpy as np
arr = np.arange(0,10)

arr + arr
# Output:
array([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18])

arr**3
# Output:
array([  0,   1,   8,  27,  64, 125, 216, 343, 512, 729])
Universal Array Functions
 np.sqrt(arr) Taking Square Roots 
np.sin(arr)
# Output:
array([ 0.        ,  0.84147098,  0.90929743,  0.14112001, -0.7568025 ,
       -0.95892427, -0.2794155 ,  0.6569866 ,  0.98935825,  0.41211849])
np.exp(arr) Calcualting exponential (e^)
np.max(arr)

same as arr.max()

Max
np.sin(arr) Sin
np.log(arr) Natural logarithm



Pandas

You can think of pandas as an extremely powerful version of Excel, with a lot more features. In this section of the course, you should go through the notebooks in this order:



Series

A Series is very similar to a NumPy array (in fact it is built on top of the NumPy array object). What differentiates the NumPy array from a Series, is that a Series can have axis labels, meaning it can be indexed by a label, instead of just a number location. It also doesn't need to hold numeric data, it can hold any arbitrary Python Object.

Method/Operator Description/Comments Example 
import pandas as pd

Creating Pandas Series


You can convert a list, numpy array, or dictionary to a Series.

From a List
 pd.Series(my_list) 
# Creating some test data:
labels = ['a','b','c']
my_list = [10,20,30]
arr = np.array([10,20,30])
d = {'a':10,'b':20,'c':30}


pd.Series(data=my_list)
pd.Series(my_list)
pd.Series(arr)
# Output:
0    10
1    20
2    30
dtype: int64

pd.Series(data=my_list,index=labels)
pd.Series(my_list,labels)
pd.Series(arr,labels)
pd.Series(d)
# Output:
a    10
b    20
c    30
dtype: int64
From a NumPy Array
 pd.Series(arr)
From a Dectionary
 pd.Series(d)

Data in a Series

 A pandas Series can hold a variety of object types. Even functions (although unlikely that you will use this)
pd.Series(data=labels)
# Output:
0    a
1    b
2    c
dtype: object

# Holding «functions» into a Series
# Output:
pd.Series([sum,print,len])
0      <built-in function sum>
1      <built-in function print>
2      <built-in function len>
dtype: object

Index in Series

 The key to using a Series is understanding its index. Pandas makes use of these index names or numbers by allowing for fast look ups of information (works like a hash table or dictionary).
ser1 = pd.Series([1,2,3,4],index = ['USA', 'Germany','USSR', 'Japan'])
ser1
# Output:
USA        1
Germany    2
USSR       3
Japan      4
dtype: int64

ser2 = pd.Series([1,2,5,4],index = ['USA', 'Germany','Italy', 'Japan'])

ser1['USA']
# Output:
1

# Operations are then also done based off of index:
ser1 + ser2
# Output:
Germany    4.0
Italy      NaN
Japan      8.0
USA        2.0
USSR       NaN
dtype: float64



DataFrames

DataFrames are the workhorse of pandas and are directly inspired by the R programming language. We can think of a DataFrame as a bunch of Series objects put together to share the same index. Let's use pandas to explore this topic! 


import pandas as pd
import numpy as np

from numpy.random import randn
np.random.seed(101)

df = pd.DataFrame(randn(5,4),index='A B C D E'.split(),columns='W X Y Z'.split())

df
# Output: 
           W           X           Y           Z
A   2.706850    0.628133    0.907969    0.503826
B   0.651118   -0.319318   -0.848077    0.605965
C  -2.018168    0.740122    0.528813   -0.589001
D   0.188695   -0.758872   -0.933237    0.955057
E   0.190794    1.978757    2.605967    0.683509


DataFrame Columns are just Series:

type(df['W'])
# Output: 
pandas.core.series.Series
Method/

Operator

Description/Comments Example
Selection and Indexing


Let's learn the various

methods to grab data

from a DataFrame

Standard systax df['W'] 
# Pass a list of column names:
df[['W','Z']]

           W           Z
A   2.706850    0.503826
B   0.651118    0.605965
C  -2.018168   -0.589001
D   0.188695    0.955057
E   0.190794    0.683509
SQL syntax (NOT

RECOMMENDED!)

df.W
Creating a new column 
df['new'] = df['W'] + df['Y']
Removing Columns df.drop() 
df.drop('new',axis=1)
# Output:
           W           X           Y           Z
A   2.706850    0.628133    0.907969    0.503826
B   0.651118   -0.319318   -0.848077    0.605965
C  -2.018168    0.740122    0.528813   -0.589001
D   0.188695   -0.758872   -0.933237    0.955057
E   0.190794    1.978757    2.605967    0.683509

# Not inplace unless specified!
df
# Output:
           W           X           Y           Z         new
A   2.706850    0.628133    0.907969    0.503826    3.614819
B   0.651118   -0.319318   -0.848077    0.605965   -0.196959
C  -2.018168    0.740122    0.528813   -0.589001   -1.489355
D   0.188695   -0.758872   -0.933237    0.955057   -0.744542
E   0.190794    1.978757    2.605967    0.683509    2.796762

df.drop('new',axis=1,inplace=True)
df
# Output:
           W           X           Y           Z
A   2.706850    0.628133    0.907969    0.503826
B   0.651118   -0.319318   -0.848077    0.605965
C  -2.018168    0.740122    0.528813   -0.589001
D   0.188695   -0.758872   -0.933237    0.955057
E   0.190794    1.978757    2.605967    0.683509


# Can also drop rows this way:
df.drop('E',axis=0)
# Output:
           W           X           Y           Z
A   2.706850    0.628133    0.907969    0.503826
B   0.651118   -0.319318   -0.848077    0.605965
C  -2.018168    0.740122    0.528813   -0.589001
D   0.188695   -0.758872   -0.933237    0.955057




Missing Data


GroupBy


Merging,Joining,and Concatenating


Operations


Data Input and Output


Retrieved from "http://wiki.sinfronteras.ws/index.php?title=Python_for_Data_Science&oldid=12041"