How do I filter with a condition?

Turn a comparison into an array of True and False, use it to select elements, count them, and switch it into a position of 1 or 0.

A mask is an array of True and False, one entry per element, and you put it inside square brackets to keep only the elements where it is True. You get one by comparing an array: the comparison answers once per element, not once for the whole array.

In this lesson I build masks on an array of daily returns, count and average with them, combine two conditions with & and |, and use np.where to turn a condition into a position of 1 or 0.

Step 1. A comparison returns an array

import numpy as np

x = np.array([3, 8, 2, 9, 5])

print(x > 4)        # -> [False  True False  True  True]
print(x[x > 4])     # -> [8 9 5]
[False  True False  True  True]
[8 9 5]

x > 4 compares every element and reports five answers. x[x > 4] reads the five answers and keeps the three elements that scored True.

Step 2. Down days

Ten daily returns. The mask picks out the negative ones.

returns = np.array([0.012, -0.004, 0.007, -0.021, 0.003,
                    0.015, -0.009, 0.002, -0.013, 0.006])

mask = returns < 0                  # one True or False per day

print(mask)
# -> [False  True False  True False False  True False  True False]
print(returns[mask])                # -> [-0.004 -0.021 -0.009 -0.013]
[False  True False  True False False  True False  True False]
[-0.004 -0.021 -0.009 -0.013]

mask has ten entries, one per day. returns[mask] has four, because four days were negative.

True counts as 1 and False counts as 0, so summing a mask counts the days and averaging it gives the fraction of days.

print(mask.sum())     # -> 4      <- number of down days
print(mask.mean())    # -> 0.4    <- fraction of days that were down
print(len(returns))   # -> 10
4
0.4
10

~ flips a mask, so ~mask is the up days. Select with each and average what comes back.

print(f"down days: {returns[mask].mean():.4%}")     # -> down days: -1.1750%
print(f"up days:   {returns[~mask].mean():.4%}")    # -> up days:   0.7500%
down days: -1.1750%
up days:   0.7500%

Step 3. Two conditions at once

& means and, | means or. Each one compares two masks element by element and gives a third mask.

Every condition needs its own brackets. & binds tighter than < and >, so without the brackets Python tries 0 & returns first.

mild_down = (returns < 0) & (returns > -0.010)      # negative, but not by much

print(mild_down)
# -> [False  True False False False False  True False False False]
print(returns[mild_down])     # -> [-0.004 -0.009]
print(mild_down.sum())        # -> 2
[False  True False False False False  True False False False]
[-0.004 -0.009]
2
big_move = (returns > 0.010) | (returns < -0.010)   # a large move either way

print(returns[big_move])      # -> [ 0.012 -0.021  0.015 -0.013]
print(big_move.sum())         # -> 4
[ 0.012 -0.021  0.015 -0.013]
4

Here is what dropping the brackets does:

try:
    returns < 0 & returns > -0.010
except TypeError as e:
    print(type(e).__name__)   # -> TypeError
    print(e)
    # -> ufunc 'bitwise_and' not supported for the input types, and the inputs
    #    could not be safely coerced to any supported types according to the
    #    casting rule ''safe''
TypeError
ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

The and and or of Lesson 3 do not work here either. They ask for one True or False, and an array of ten answers cannot supply it.

try:
    (returns < 0) and (returns > -0.010)
except ValueError as e:
    print(e)
    # -> The truth value of an array with more than one element is ambiguous.
    #    Use a.any() or a.all()
The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

So: and and or for single values, & and | for arrays, and brackets round each condition.

Step 4. From a condition to a position

np.where(condition, a, b) walks the mask and takes a where it is True and b where it is False. Feed it 1 and 0 and the mask becomes a position: in the market or out of it.

position = np.where(returns > 0, 1, 0)

print(position)          # -> [1 0 1 0 1 1 0 1 0 1]
print(position.dtype)    # -> int64
print(position.sum())    # -> 6      <- days holding
print(position.mean())   # -> 0.6    <- fraction of days holding
[1 0 1 0 1 1 0 1 0 1]
int64
6
0.6

The two outputs can be any numbers, not just 1 and 0. Half size on the big-move days:

sized = np.where(big_move, 0.5, 1.0)

print(sized)
# -> [0.5 1.  1.  0.5 1.  0.5 1.  1.  0.5 1. ]
[0.5 1.  1.  0.5 1.  0.5 1.  1.  0.5 1. ]

position here sits on the same day as the return that produced it, which is a position decided with that day’s return already known. Moving a signal forward one day before it earns anything is shift, in Lesson 20.

Your turn

Take r = np.array([0.004, -0.011, 0.022, -0.003, 0.000, -0.018, 0.009]). Build a mask for days where the move was bigger than 0.5% in either direction. Print those returns, how many there were, and what fraction of the week they made up. Then use np.where to build an array holding 0 on those days and 1 on the rest.

import numpy as np

r = np.array([0.004, -0.011, 0.022, -0.003, 0.000, -0.018, 0.009])

m = (r > 0.005) | (r < -0.005)      # brackets round each condition

print(m)              # -> [False  True  True False False  True  True]
print(r[m])           # -> [-0.011  0.022 -0.018  0.009]
print(m.sum())        # -> 4
print(m.mean())       # -> 0.5714285714285714

print(np.where(m, 0, 1))   # -> [1 0 0 1 1 0 0]

mask.any() is True when at least one element is True. mask.all() is True only when every element is.

print((returns < 0).any())    # -> True
print((returns < 0).all())    # -> False
True
False

These give one True or False, so they are what an if can read.