book

High Performance Python, 2nd Edition

by Micha Gorelick, Ian Ozsvald

April 2020

Intermediate to advanced

468 pages

12h 52m

English

O'Reilly Media, Inc.

Book available

Start your free trial

Related skills

Python

Associated roles

AI engineer
Analytics engineer
Backend developer
Business analyst

Business intelligence analyst
Data analyst
Data architect
Database administrator
Data engineer
Data scientist
ML/AI architect
Penetration tester
Python developer
QA engineer

+10 more

Who This Book Is ForWho This Book Is Not ForWhat You’ll LearnPython 3Changes from Python 2.7LicenseHow to Make an AttributionErrata and FeedbackConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
The Fundamental Computer SystemComputing UnitsMemory UnitsCommunications LayersPutting the Fundamental Elements TogetherIdealized Computing Versus the Python Virtual MachineSo Why Use Python?How to Be a Highly Performant ProgrammerGood Working PracticesSome Thoughts on Good Notebook PracticeGetting the Joy Back into Your Work
Profiling EfficientlyIntroducing the Julia SetCalculating the Full Julia SetSimple Approaches to Timing—print and a DecoratorSimple Timing Using the Unix time CommandUsing the cProfile ModuleVisualizing cProfile Output with SnakeVizUsing line_profiler for Line-by-Line MeasurementsUsing memory_profiler to Diagnose Memory UsageIntrospecting an Existing Process with PySpyBytecode: Under the HoodUsing the dis Module to Examine CPython BytecodeDifferent Approaches, Different ComplexityUnit Testing During Optimization to Maintain CorrectnessNo-op @profile DecoratorStrategies to Profile Your Code SuccessfullyWrap-Up
A More Efficient SearchLists Versus TuplesLists as Dynamic ArraysTuples as Static ArraysWrap-Up
How Do Dictionaries and Sets Work?Inserting and RetrievingDeletionResizingHash Functions and EntropyDictionaries and NamespacesWrap-Up
Iterators for Infinite SeriesLazy Generator EvaluationWrap-Up
Introduction to the ProblemAren’t Python Lists Good Enough?Problems with Allocating Too MuchMemory FragmentationUnderstanding perfMaking Decisions with perf’s OutputEnter numpyApplying numpy to the Diffusion ProblemMemory Allocations and In-Place OperationsSelective Optimizations: Finding What Needs to Be Fixednumexpr: Making In-Place Operations Faster and EasierA Cautionary Tale: Verify “Optimizations” (scipy)Lessons from Matrix OptimizationsPandasPandas’s Internal ModelApplying a Function to Many Rows of DataBuilding DataFrames and Series from Partial Results Rather than ConcatenatingThere’s More Than One (and Possibly a Faster) Way to Do a JobAdvice for Effective Pandas DevelopmentWrap-Up
What Sort of Speed Gains Are Possible?JIT Versus AOT CompilersWhy Does Type Information Help the Code Run Faster?Using a C CompilerReviewing the Julia Set ExampleCythonCompiling a Pure Python Version Using CythonpyximportCython Annotations to Analyze a Block of CodeAdding Some Type AnnotationsCython and numpyParallelizing the Solution with OpenMP on One MachineNumbaNumba to Compile NumPy for PandasPyPyGarbage Collection DifferencesRunning PyPy and Installing ModulesA Summary of Speed ImprovementsWhen to Use Each TechnologyOther Upcoming ProjectsGraphics Processing Units (GPUs)Dynamic Graphs: PyTorchBasic GPU ProfilingPerformance Considerations of GPUsWhen to Use GPUsForeign Function Interfacesctypescffif2pyCPython ModuleWrap-Up
Introduction to Asynchronous ProgrammingHow Does async/await Work?Serial CrawlerGeventtornadoaiohttpShared CPU–I/O WorkloadSerialBatched ResultsFull AsyncWrap-Up

An Overview of the multiprocessing ModuleEstimating Pi Using the Monte Carlo MethodEstimating Pi Using Processes and ThreadsUsing Python ObjectsReplacing multiprocessing with JoblibRandom Numbers in Parallel SystemsUsing numpyFinding Prime NumbersQueues of WorkVerifying Primes Using Interprocess CommunicationSerial SolutionNaive Pool SolutionA Less Naive Pool SolutionUsing Manager.Value as a FlagUsing Redis as a FlagUsing RawValue as a FlagUsing mmap as a FlagUsing mmap as a Flag ReduxSharing numpy Data with multiprocessingSynchronizing File and Variable AccessFile LockingLocking a ValueWrap-Up
Benefits of ClusteringDrawbacks of Clustering$462 Million Wall Street Loss Through Poor Cluster Upgrade StrategySkype’s 24-Hour Global OutageCommon Cluster DesignsHow to Start a Clustered SolutionWays to Avoid Pain When Using ClustersTwo Clustering SolutionsUsing IPython Parallel to Support ResearchParallel Pandas with DaskNSQ for Robust Production ClusteringQueuesPub/subDistributed Prime CalculationOther Clustering Tools to Look AtDockerDocker’s PerformanceAdvantages of DockerWrap-Up
Objects for Primitives Are ExpensiveThe array Module Stores Many Primitive Objects CheaplyUsing Less RAM in NumPy with NumExprUnderstanding the RAM Used in a CollectionBytes Versus UnicodeEfficiently Storing Lots of Text in RAMTrying These Approaches on 11 Million TokensModeling More Text with Scikit-Learn’s FeatureHasherIntroducing DictVectorizer and FeatureHasherComparing DictVectorizer and FeatureHasher on a Real ProblemSciPy’s Sparse MatricesTips for Using Less RAMProbabilistic Data StructuresVery Approximate Counting with a 1-Byte Morris CounterK-Minimum ValuesBloom FiltersLogLog CounterReal-World Example
Streamlining Feature Engineering Pipelines with Feature-engineFeature Engineering for Machine LearningThe Hard Task of Deploying Feature Engineering PipelinesLeveraging the Power of Open Source Python LibrariesFeature-engine Smooths Building and Deployment of Feature Engineering PipelinesHelping with the Adoption of a New Open Source PackageDeveloping, Maintaining, and Encouraging Contribution to Open Source LibrariesHighly Performant Data Science TeamsHow Long Will It Take?Discovery and PlanningManaging Expectations and DeliveryNumbaA Simple ExampleBest Practices and RecommendationsGetting HelpOptimizing Versus ThinkingAdaptive Lab’s Social Media Analytics (2014)Python at Adaptive LabSoMA’s DesignOur Development MethodologyMaintaining SoMAAdvice for Fellow EngineersMaking Deep Learning Fly with RadimRehurek.com (2014)The Sweet SpotLessons in OptimizingConclusionLarge-Scale Productionized Machine Learning at Lyst.com (2014)Cluster DesignCode Evolution in a Fast-Moving Start-UpBuilding the Recommendation EngineReporting and MonitoringSome AdviceLarge-Scale Social Media Analysis at Smesh (2014)Python’s Role at SmeshThe PlatformHigh Performance Real-Time String MatchingReporting, Monitoring, Debugging, and DeploymentPyPy for Successful Web and Data Processing Systems (2014)PrerequisitesThe DatabaseThe Web ApplicationOCR and TranslationTask Distribution and WorkersConclusionTask Queues at Lanyrd.com (2014)Python’s Role at LanyrdMaking the Task Queue PerformantReporting, Monitoring, Debugging, and DeploymentAdvice to a Fellow Developer

Overview

Your Python code may run correctly, but you need it to run faster. Updated for Python 3, this expanded edition shows you how to locate performance bottlenecks and significantly speed up your code in high-data-volume programs. By exploring the fundamental theory behind design choices, High Performance Python helps you gain a deeper understanding of Python’s implementation.

How do you take advantage of multicore architectures or clusters? Or build a system that scales up and down without losing reliability? Experienced Python programmers will learn concrete solutions to many issues, along with war stories from companies that use high-performance Python for social media analytics, productionized machine learning, and more.