Even if you have numpy which uses all available cores of your processor your programs can still be pretty slow. This is especially the case if you process massive datasets, performing a lot of data transformations and calculations.
Switch to vector operations
The very first hint would be to quit pythonic-way of programming and get rid of list comprehensions and all other loops as these make your programs as fast as snails.
Instead, use numpy’s vector operations which bring you a significant performance improvement. For instance, you have a simple vector each element of which contains a timestamp and need to know all time intervals. Let’s write a simple code containing both a list comprehension and a vector operation:
But compare the performance:
This is an enormous speed up - 176 times faster. Those who have mastered vector operations and never write loops and list comprehensions anymore believe that this is the fastest way of data processing. And they can’t be more wrong…
Numba makes it even faster
To put it simply, Numba is a just-in-time compiler that makes your Python code as fast as a code written in C. All you need is add a decorator to your usual python function (you have to comply to certain constraints, though, as not any code can be made fast). Let me demonstrate it on the simplest example:
And now execution times:
The first call of a numba function includes compilation, that is why it takes longer. But when compiled it is 1.5 times faster than highly optimized numpy code. This is just unbelievable! The simplest summation gives you an immensive performance increase. But this is not all numba can do for you.
While you can perform a vector-wide operations and data transformations, numpy is great. However, eventually you bump into a situation when you need some element-wise logic… and numpy just stops delivering here.
You might be eager to see the performance, don’t you? Here it is:
Now you see it - numba is 127 times faster than numpy.
Show your types
As you could notice, the first call to numba functions takes much longer as it includes a compilation phase. Fortunately, you can avoid this with a so-called eager, or decoration time, compilation when you pass type signatures to the decorator.
You can pass several signatures if your work with different data types in the same function:
Now your first call will be fast as well, since compilation will be performed in advance. However, if you call this function with data of some other types you will get a run-time error. So be cautious and think carefully about your data and how to process it.