pandas #pandas Table of Contents About 1 Chapter 1: Getting started with pandas 2 Remarks 2 Versions 2 Examples 3 Installation or Setup 3 Install via anaconda 5 Hello World 5 Descriptive statistics 6 Chapter 2: Analysis: Bringing it all together and making decisions 8 Examples 8 Quintile Analysis: with random data 8 What is a factor 8 Initialization 8 pd.qcut - Create Quintile Buckets 9 Analysis 9 Plot Returns 9 Visualize Quintile Correlation with scatter_matrix 10 Calculate and visualize Maximum Draw Down 11 Calculate Statistics 13 Chapter 3: Appending to DataFrame 15 Examples 15 Appending a new row to DataFrame 15 Append a DataFrame to another DataFrame 16 Chapter 4: Boolean indexing of dataframes 18 Introduction 18 Examples 18 Accessing a DataFrame with a boolean index 18 Applying a boolean mask to a dataframe 19 Masking data based on column value 19 Masking data based on index value 20 Chapter 5: Categorical data 21 Introduction 21 Examples 21 Object Creation 21 Creating large random datasets 21 Chapter 6: Computational Tools 23 Examples 23 Find The Correlation Between Columns 23 Chapter 7: Creating DataFrames 24 Introduction 24 Examples 24 Create a sample DataFrame 24 Create a sample DataFrame using Numpy 24 Create a sample DataFrame from multiple collections using Dictionary 26 Create a DataFrame from a list of tuples 26 Create a DataFrame from a dictionary of lists 26 Create a sample DataFrame with datetime 27 Create a sample DataFrame with MultiIndex 29 Save and Load a DataFrame in pickle (.plk) format 29 Create a DataFrame from a list of dictionaries 30 Chapter 8: Cross sections of different axes with MultiIndex 31 Examples 31 Selection of cross-sections using .loc and slicers 32 Chapter 9: Data Types 34 Remarks 34 Examples 34 Checking the types of columns 35 Changing dtypes 35 Changing the type to numeric 36 Changing the type to datetime 37 Changing the type to timedelta 37 Selecting columns based on dtype 37 Summarizing dtypes 38 Chapter 10: Dealing with categorical variables 39 Examples 39 One-hot encoding with `get_dummies()` 39 Chapter 11: Duplicated data 40 Examples 40 Select duplicated 40 Drop duplicated 40 Counting and getting unique elements 41 Get unique values from a column. 42 Chapter 12: Getting information about DataFrames 44 Examples 44 Get DataFrame information and memory usage 44 List DataFrame column names 44 Dataframe's various summary statistics. 45 Chapter 13: Gotchas of pandas 46 Remarks 46 Examples 46 Detecting missing values with np.nan 46 Integer and NA 46 Automatic Data Alignment (index-awared behaviour) 47 Chapter 14: Graphs and Visualizations 48 Examples 48 Basic Data Graphs 48 Styling the plot 49 Plot on an existing matplotlib axis 50 Chapter 15: Grouping Data 51 Examples 51 Basic grouping 51 Group by one column 51 Group by multiple columns 51 Grouping numbers 52 Column selection of a group 53 Aggregating by size versus by count 54 Aggregating groups 54 Export groups in different files 55 using transform to get group-level statistics while preserving the original dataframe 55 Chapter 16: Grouping Time Series Data 57 Examples 57 Generate time series of random numbers then down sample 57 Chapter 17: Holiday Calendars 59 Examples 59 Create a custom calendar 59 Use a custom calendar 59 Get the holidays between two dates 59 Count the number of working days between two dates 60 Chapter 18: Indexing and selecting data 61 Examples 61 Select column by label 61 Select by position 61 Slicing with labels 62 Mixed position and label based selection 63 Boolean indexing 64 Filtering columns (selecting "interesting", dropping unneeded, using RegEx, etc.) 65 generate sample DF 65 show columns containing letter 'a' 65 show columns using RegEx filter (b|c|d) - b or c or d: 65 show all columns except those beginning with a (in other word remove / drop all columns sa 66 Filtering / selecting rows using `.query()` method 66 generate random DF 66 select rows where values in column A > 2 and values in column B < 5 66 using .query() method with variables for filtering 67 Path Dependent Slicing 67 Get the first/last n rows of a dataframe 69 Select distinct rows across dataframe 70 Filter out rows with missing data (NaN, None, NaT) 71 Chapter 19: IO for Google BigQuery 73 Examples 73 Reading data from BigQuery with user account credentials 73 Reading data from BigQuery with service account credentials 74 Chapter 20: JSON 75 Examples 75 Read JSON 75 can either pass string of the json, or a filepath to a file with valid json 75 Dataframe into nested JSON as in flare.js files used in D3.js 75 Read JSON from file 76 Chapter 21: Making Pandas Play Nice With Native Python Datatypes 77 Examples 77 Moving Data Out of Pandas Into Native Python and Numpy Data Structures 77 Chapter 22: Map Values 79 Remarks 79 Examples 79 Map from Dictionary 79 Chapter 23: Merge, join, and concatenate 80 Syntax 80 Parameters 80 Examples 81 Merge 81 Merging two DataFrames 82 Inner join: 82 Outer join: 83 Left join: 83 Right Join 83 Merging / concatenating / joining multiple data frames (horizontally and vertically) 83 Merge, Join and Concat 84 What is the difference between join and merge 85 Chapter 24: Meta: Documentation Guidelines 88 Remarks 88 Examples 88 Showing code snippets and output 88 style 89 Pandas version support 89 print statements 89 Prefer supporting python 2 and 3: 89 Chapter 25: Missing Data 90 Remarks 90 Examples 90 Filling missing values 90 Fill missing values with a single value: 90 Fill missing values with the previous ones: 90 Fill with the next ones: 90 Fill using another DataFrame: 91 Dropping missing values 91 Drop rows if at least one column has a missing value 91 Drop rows if all values in that row are missing 92 Drop columns that don't have at least 3 non-missing values 92 Interpolation 92 Checking for missing values 92 Chapter 26: MultiIndex 94 Examples 94 Select from MultiIndex by Level 94 Iterate over DataFrame with MultiIndex 95 Setting and sorting a MultiIndex 96 How to change MultiIndex columns to standard columns 98 How to change standard columns to MultiIndex 98 MultiIndex Columns 98 Displaying all elements in the index 99 Chapter 27: Pandas Datareader 100 Remarks 100 Examples 100 Datareader basic example (Yahoo Finance) 100 Reading financial data (for multiple tickers) into pandas panel - demo 101 Chapter 28: Pandas IO tools (reading and saving data sets) 103 Remarks 103 Examples 103 Reading csv file into DataFrame 103 File: 103 Code: 103 Output: 103 Some useful arguments: 103 Basic saving to a csv file 105 Parsing dates when reading from csv 105 Spreadsheet to dict of DataFrames 105 Read a specific sheet 105 Testing read_csv 105 List comprehension 106 Read in chunks 107 Save to CSV file 107 Parsing date columns with read_csv 108 Read & merge multiple CSV files (with the same structure) into one DF 108 Reading cvs file into a pandas data frame when there is no header row 108 Using HDFStore 109 generate sample DF with various dtypes 109 make a bigger DF (10 * 100.000 rows) 109 create (or open existing) HDFStore file 110 save our data frame into h5 (HDFStore) file, indexing [int32, int64, string] columns: 110 show HDFStore details 110 show indexed columns 110 close (flush to disk) our store file 111 Read Nginx access log (multiple quotechars) 111 Chapter 29: pd.apply 112 Examples 112 pandas.apply Basic Usage 112 Chapter 30: Read MySQL to DataFrame 114 Examples 114 Using sqlalchemy and PyMySQL 114 To read mysql to dataframe, In case of large amount of data 114 Chapter 31: Read SQL Server to Dataframe 115 Examples 115 Using pyodbc 115 Using pyodbc with connection loop 115 Chapter 32: Reading files into pandas DataFrame 117 Examples 117 Read table into DataFrame 117 Table file with header, footer, row names, and index column: 117 Table file without row names or index: 117 Read CSV File 118 Data with header, separated by semicolons instead of commas 118 Table without row names or index and commas as separators 118 Collect google spreadsheet data into pandas dataframe 119 Chapter 33: Resampling 120 Examples 120 Downsampling and upsampling 120 Chapter 34: Reshaping and pivoting 122 Examples 122 Simple pivoting 122 Pivoting with aggregating 123 Stacking and unstacking 126 Cross Tabulation 127 Pandas melt to go from wide to long 129 Split (reshape) CSV strings in columns into multiple rows, having one element per row 130 Chapter 35: Save pandas dataframe to a csv file 132 Parameters 132 Examples 133 Create random DataFrame and write to .csv 133 Save Pandas DataFrame from list to dicts to csv with no index and with data encoding 134 Chapter 36: Series 136 Examples 136 Simple Series creation examples 136 Series with datetime 136 A few quick tips about Series in Pandas 137 Applying a function to a Series 139 Chapter 37: Shifting and Lagging Data 141 Examples 141 Shifting or lagging values in a dataframe 141 Chapter 38: Simple manipulation of DataFrames 142 Examples 142 Delete a column in a DataFrame 142 Rename a column 143 Adding a new column 144 Directly assign 144 Add a constant column 144 Column as an expression in other columns 144 Create it on the fly 145 add multiple columns 145 add multiple columns on the fly 145 Locate and replace data in a column 146 Adding a new row to DataFrame 146 Delete / drop rows from DataFrame 147 Reorder columns 148 Chapter 39: String manipulation 149 Examples 149 Regular expressions 149 Slicing strings 149 Checking for contents of a string 151 Capitalization of strings 151 Chapter 40: Using .iat to access a DataFrame 154 Examples 154 Using .loc 155 Chapter 41: Working with Time Series 157 Examples 157 Creating Time Series 157 Partial String Indexing 157 Getting Data 157 Subsetting 157 Credits 159 Chapter 1: Getting started with pandas Remarks Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python.
The official Pandas documentation can be found here. Versions Pandas Version Release Date 0.com/ 2 Version Release Date 0.0 2013-07-23 Examples Installation or Setup Detailed instructions on getting pandas set up or installed can be found here in the official documentation. Installing pandas with Anaconda Installing pandas and the rest of the NumPy and SciPy stack can be a little difficult for inexperienced users. The simplest way to install not only pandas, but Python and the most popular packages that make up the SciPy stack (IPython, NumPy, Matplotlib, .) is with Anaconda, a cross-platform (Linux, Mac OS X, Windows) Python distribution for data analytics and scientific computing.
After running a simple installer, the user will have access to pandas and the rest of the SciPy stack without needing to install anything else, and without needing to wait for any software to be compiled. Installation instructions for Anaconda can be found here. A full list of the packages available as part of the Anaconda distribution can be found here. An additional advantage of installing with Anaconda is that you don’t require admin rights to install it, it will install in the user’s home directory, and this also makes it trivial to delete Anaconda at a later date (just delete that folder).
Installing pandas with Miniconda The previous section outlined how to get pandas installed as part of the Anaconda distribution. However this approach means you will install well over one hundred packages and involves downloading the installer which is a few hundred megabytes in size. If you want to have more control on which packages, or have a limited internet bandwidth, then installing pandas with Miniconda may be a better solution. Conda is the package manager that the Anaconda distribution is built upon.
