The story
Named after the rectified linear unit, where “leaky” denotes how the negative half of the function becomes slightly negative, being “leaky” in comparison to the default ReLU, which is zero.
Etymology from Wiktionary, CC BY-SA 4.0.
leaky rectified linear unit
Today it means: A variation of the rectified linear unit, defined as f(x)=x when x≥0 and f(x)=αx for some small α, e.g. 0.01, when x<0. Full definition →