Blackman Filter - The Smoother The BetterIntroduction
Who doesn't like smooth things? I'd like a smooth market price for christmas! But i can't get it, instead its so noisy...so you apply a filter to smooth it, such filters are called low-pass filters, they smooth and its great but they have lag, so nobody really use them, but they are pretty to look at.
Its on a childish note that i will introduce this indicator, so what it is all about? I propose a new FIR filter using a blackman function as filter kernel for financial time-series smoothing, do you prefer the childish tone ? Fear not its surprisingly easy!
The Blackman Function
The blackman function look like a bell shaped curve, look:
The blackman function will produce such curve. This function is called a cosine sum function because she is based on the sum of cosine functions, here only 2.
0.42 - 0.5 * cos(2 * pi * k) + 0.08 * cos(4 * pi * k)
Originally you use this function for windowing , what does it means? In signal processing you have a function called sync function , if you use this function as filter kernel you would get the ideal frequency domain response filter, sometime called brickwall filter, it would be extremely smooth.
Above the optimal low pass filter frequency response.
However the sync function has no ending values and goes on forever, therefore we can't use it for convolution, expect if we apply windowing. Filters using windowing are called windowed-sinc filters, i will describe the procedure below :
1 - Create a sync function = sin(pi*n)/(pi*n)
2 - Truncate it = I only keep the first length points of the sync function.
This create a abrupt end, the frequency of a filter using step 1 as kernel would contain ripples in the pass band and stop band, this is bad! The frequency response would look like this :
3 - I multiply my values of step 2 by a window function, it can the blackman window, i no longer have an abrupt end, its smooth!
The frequency response of the filter using this kernel would no longer have ripples! This is the power of windowing functions.
Here we are not using such thing, but we could in the future. Here instead we use the blackman function as filter kernel, because this function is bell shaped this mean that the filter will certainly be smooth (symmetrical weighting is a rule of thumb for kernels when we want really smooth filters).
The Filter
This filter is quite smooth, unlike the gaussian filter this filter give less weights to recent and past values, this is because the blackman function has fatter tails than the gaussian one. I could make a comparison of both, however they are quite alike, if you often use a gaussian filter its up to you to decide which one you prefer.
The filter can do a better job than the moving average when it comes to preserve the frequency components that constitute the cycles/trend.
We can see that the filter has a greater performance when it comes to keep the shape of the market price, thus it has a slightly better fit.
Conclusion
Ok so in this post you learned a bit about the sync function and windowing, those are basic subjects in signal processing, they allow us to approximate the filter with the ideal frequency response, i also showed you that those windowing function could be used as kernel and that they where pretty smooth on their own, there are many others, but the one i prefer is the blackman windowing function.
I know what you are thinking, "we want trailing stops, alerts, colors, arrows!", and i understand you pal, but sometimes its cool to take a break from all this stuff. However i can tell that i'am working on a side project that aim to estimate rolling maximum/minimum as fast as possible, any experiments will be published here, and i can ensure you that those indicators will make your day quite brighter, we will see that soon.
I hope you learned something from this post! I'am a bit tired (look i'am disappearing !)
Thanks for reading !
Indicator
Setting-Less Trend-Step FilteringIntroduction
Indicators settings have been a major concern in trading strategies, in order to provide the best results each indicators involved in the strategy must have its settings optimized, when using only 1 indicator this task can easily be achieved, but an increasing number of indicators involve more slower computations, lot of softwares will use brute force for indicators settings optimization, this involve testing each indicator settings and see which setting/combination maximize the equity, in order to fasten this process softwares can use a user defined range for the indicator settings. Nonetheless the combination that maximize the equity at time t might be different at time t+1...n .
Therefore i propose an indicator without any numerical setting that aim to filter small price variations using the architecture of the T-step lsma, such indicator can provide robust filtering and can therefore be used as input for other indicators.
Robustness Vs Non Robustness
Robustness is often defined as the ability of certain statistical tools to be less affected by outliers, outliers are defined as huge variations in a data-set, high volatility movements and large gaps might be considered as outliers. However here we define robustness as the ability of an indicator to be non affected by price variations that are not correlated with the main trend, which can be defined in technical analysis as pullbacks.
Some small pullbacks in INTEL, the indicator is not affected by them, which allow the indicator to filter the price in a "smart" way.
This effect is made possible by using exponential averaging in the indicator, exponential averaging is defined as y = sc*x + (1-sc)*y , with 1 > sc > 0 . Here sc is calculated in a similar way as the kalman gain, which is in the form of a/(a + b) , in our case this is done with :
sc = abs(input - nz(b ))/(abs(input - nz(b )) + nz(a ))
Non Robust Version Of The Indicator
The user is allowed to use the non robust version of the indicator by unchecking "robust" in the setting panel, this allow a better fit with the price at the cost of less filtering.
robust checked
robust unchecked
Conclusion
I proposed a technical indicator that aim to filter short frequencies without the use of parameters, the indicator proven to be robust to various pullbacks and therefore was able to follow the main trend, although using the term trend for such small price variations might be wrong. Removing high frequencies is always beneficial in trading, noisy series are harder to manipulate, this is why you'll see a lot of indicators using median price often defined as hl2 instead of the closing price.
Like previous settings-less indicators i published this one can behave differently depending on the time frame selected by the user, lower time frames will make the indicator filter more. I'll try to make more setting-less indicators that will correct this effect.
Acknowledgements
The support and interest of the community is only thing that allowed me to be where i'am today, i'am thankful. Special thanks to the tv staff, LucF, and my family who may not have believed in this project but are still proud of their son.
Indicator
Indicator
Template For Custom FIR Filters - Make Your Moving AverageIntroduction
FIR filters (finite impulse response) are widely used in technical analysis, there is the simple or arithmetic moving average, the triangular, the weighted, the least squares...etc. A FIR filter is characterized by the fact that its impulse response (the output of a filter using an impulse as input) is finite, this mean that the impulse response won't have infinite outputs unlike IIR filters.
They are extremely simple to design to, even without the Fourier transform, this is why i post this template that will let you create custom filters from step responses. Don't hesitate to post your results.
How It Works
Originally you create your filters from the frequency response you want your filter to have, this is because the inverse Fourier transform of the frequency response is the filter impulse response.
After that step you use convolution (convolution is the sum of the product between the signal and the impulse response) and you will have your filter. But we don't have Fourier transforms in pine so how can we possibly make FIR filters from convolution ? Well here the thing, the impulse response is the derivative of the step response and the step response is the sum of the impulse response, this mean we can create filters from step responses.
Step response of a moving average.
Step responses are easy to design, you just need a function that start at 0 and end up at 1.
How To Use The Template
All the work is done for you, the only thing you need to do is to enter your function at line 5 :
f(x)=> your function
For example if you want your filter to have a step response equal to sqrt(x) just enter :
f(x)=> sqrt(x)
This will give the following filter output :
You can create custom step responses from online graphing tools like fooplot or wolfram alpha, i recommend fooplot.
You can also design your filter step response from the line 14/15/16, b will be your filter step response, just use a , for example b = pow(a,2) , then replace output in plot by b and use overlay false, you can also plot step , if you like your step response copy the content of b and paste after f(x) => .
Filter Characteristics
The impulse response determine how many of a certain signal you want in your filter, this is also called weighting, you can think of filter design as cooking where your ingredients are the the signal at different periods and the impulse response determine how many of an ingredient you must include in the recipe. The step response can also tell you about your filter characteristics, for example :
This one converge faster to the step function, this mean that the filter will have less lag.
However this one converge slower to the step function, this mean the filter might have more lag but could be smoother.
Be aware that you must find a good weighting balance, else you can have output equals to the signal or just a delayed version of the signal without smoothing.
Real Case
Lets design a sine weighted moving average (swma), this FIR filter use the first 180 degrees of a sine wave function as impulse response.
Impulse response of the swma.
We can design it from the step response without much problems, remember that the impulse response is the derivative of the step response, therefore the derivative of the step response is equal to the first 180 degrees of a sine wave, the derivative of the cosine function is a sine function, therefore :
f(x)=> .5*(1 - cos(x*pi))
And voila.
Designing A BandPass Filter
The bandpass filter like a low-pass and high pass filter, you can think of it as a smooth oscillator.
To design a bandpass filter your step response must be bell shaped, or starting at 0 and ending at 0, for example :
f(x)=>sin(x*pi) give :
Conclusion
Just use fooplot and experiment, you could get nice filters, i will try to post some using this template but it would be really nice to have other people use it. If you need further help pm me.
Thanks for reading !
Indicator
Indicator
Indicator
Well Rounded Moving AverageIntroduction
There are tons of filters, way to many, and some of them are redundant in the sense they produce the same results as others. The task to find an optimal filter is still a big challenge among technical analysis and engineering, a good filter is the Kalman filter who is one of the more precise filters out there. The optimal filter theorem state that : The optimal estimator has the form of a linear observer , this in short mean that an optimal filter must use measurements of the inputs and outputs, and this is what does the Kalman filter. I have tried myself to Kalman filters with more or less success as well as understanding optimality by studying Linear–quadratic–Gaussian control, i failed to get a complete understanding of those subjects but today i present a moving average filter (WRMA) constructed with all the knowledge i have in control theory and who aim to provide a very well response to market price, this mean low lag for fast decision timing and low overshoots for better precision.
Construction
An good filter must use information about its output, this is what exponential smoothing is about, simple exponential smoothing (EMA) is close to a simple moving average and can be defined as :
output = output(1) + α(input - output(1))
where α (alpha) is a smoothing constant, typically equal to 2/(Period+1) for the EMA.
This approach can be further developed by introducing more smoothing constants and output control (See double/triple exponential smoothing - alpha-beta filter) .
The moving average i propose will use only one smoothing constant, and is described as follow :
a = nz(a ) + alpha*nz(A )
b = nz(b ) + alpha*nz(B )
y = ema(a + b,p1)
A = src - y
B = src - ema(y,p2)
The filter is divided into two components a and b (more terms can add more control/effects if chosen well) , a adjust itself to the output error and is responsive while b is independent of the output and is mainly smoother, adding those components together create an output y , A is the output error and B is the error of an exponential moving average.
Comparison
There are a lot of low-lag filters out there, but the overshoots they induce in order to reduce lag is not a great effect. The first comparison is with a least square moving average, a moving average who fit a line in a price window of period length .
Lsma in blue and WRMA in red with both length = 100 . The lsma is a bit smoother but induce terrible overshoots
ZLMA in blue and WRMA in red with both length = 100 . The lag difference between each moving average is really low while VWRMA is way more precise.
Hull MA in blue and WRMA in red with both length = 100 . The Hull MA have similar overshoots than the LSMA.
Reduced overshoots moving average (ROMA) in blue and WRMA in red with both length = 100 . ROMA is an indicator i have made to reduce the overshoots of a LSMA, but at the end WRMA still reduce way more the overshoots while being smoother and having similar lag.
I have added a smoother version, just activate the extra smooth option in the indicator settings window. Here the result with length = 200 :
This result is a little bit similar to a 2 order Butterworth filter. Our filter have more overshoots which in this case could be useful to reduce the error with edges since other low pass filters tend to smooth their amplitude thus reducing edge estimation precision.
Conclusions
I have presented a well rounded filter in term of smoothness/stability and reactivity. Try to add more terms to have different results, you could maybe end up with interesting results, if its the case share them with the community :)
As for control theory i have seen neural networks integrated to Kalman flters which leaded to great accuracy, AI is everywhere and promise to be a game a changer in real time data smoothing. So i asked myself if it was possible for a neural networks to develop pinescript indicators, if yes then i could be replaced by AI ? Brrr how frightening.
Thanks for reading :)
Indicator
Edge-Preserving FilterIntroduction
Edge-preserving smoothing is often used in image processing in order to preserve edge information while filtering the remaining signal. I introduce two concepts in this indicator, edge preservation and an adaptive cumulative average allowing for fast edge-signal transition with period increase over time. This filter have nothing to do with classic filters for image processing, those filters use kernels convolution and are most of the time in a spatial domain.
Edge Detection Method
We want to minimize smoothing when an edge is detected, so our first goal is to detect an edge. An edge will be considered as being a peak or a valley, if you recall there is one of my indicator who aim to detect peaks and valley (reference at the bottom of the post) , since this estimation return binary outputs we will use it to tell our filter when to stop filtering.
Filtering Increase By Using Multi Steps Cumulative Average
The edge detection is a binary output, using a exponential smoothing could be possible and certainly more efficient but i wanted instead to try using a cumulative average approach because it smooth more and is a bit more original to use an adaptive architecture using something else than exponential averaging. A cumulative average is defined as the sum of the price and the previous value of the cumulative average and then this result is divided by n with n = number of data points. You could say that a cumulative average is a moving average with a linear increasing period.
So lets call CMA our cumulative average and n our divisor. When an edge is detected CMA = close price and n = 1 , else n is equal to previous n+1 and the CMA act as a normal cumulative average by summing its previous values with the price and dividing the sum by n until a new edge is detected, so there is a "no filtering state" and a "filtering state" with linear period increase transition, this is why its multi-steps.
The Filter
The filter have two parameters, a length parameter and a smooth parameter, length refer to the edge detection sensitivity, small values will detect short terms edges while higher values will detect more long terms edges. Smooth is directly related to the edge detection method, high values of smooth can avoid the detection of some edges.
smooth = 200
smooth = 50
smooth = 3
Conclusion
Preserving the price edges can be useful when it come to allow for reactivity during important price points, such filter can help with moving average crossover methods or can be used as a source for other indicators making those directly dependent of the edge detection.
Rsi with a period of 200 and our filter as source, will cross triggers line when an edge is detected
Feel free to share suggestions ! Thanks for reading !
References
Peak/Valley estimator used for the detection of edges in price. Indicator
Zero Phase Filtering [Repaint] - ExperimentalImportant !
The indicator is for experimental purpose only, it must not be used as a decisional tool but only as a visual one (like Zig-Zag, Fractal etc). The information this indicator display is uncertain and subject to drastic changes over time. If you have further question feel free to pm me.
Introduction
Most of the filters you will find are causal, this mean that they depend on present and past input values, this explain the lag they produce. Non causal filters however will use future input values. A well know way to get a zero-phase filter is by using the forward backward method, but this is not possible in pinescript as i recall. So we have to use some kind of function that will display future values, this is possible using the security function in version 2 or the one in version 3 using barmerge.lookahead_on .
The Use Of A Repainting Indicator
Its always better to filter data in order to have a clearer view of what is happening, this can be useful when doing some forecasting or doing less formal kind of analysis. However since it repaint you cant use it as a signal provider or use signals of other indicators using this filter as source.
For example if you want to forecast a smooth indicator, the forecast of this indicator under normal circumstances could still have lag associated with it, so you would have to react before your forecast, this wont happen if you apply this filter as your indicator source.
The Filter
We smooth with a simple moving average the price provided by the security function twice, length control the smoothing level. Since security depend on the time frame you are in you must select your time frame in the indicator parameter selection window.
Filtering using 45 minutes time frame close price in a 5 minutes chart, we fix this by selecting our time frame.
Consider the fact that the input of the indicator is just periodic price, so sometimes the lag can sometimes be less or more than 0 and the estimation not centered.
The indicator can work on time frames up to 1h, after that the filter have some lag, i tried fixing this and i ended up having data errors.
Applying our filter as source for the rsi oscillator.
Conclusion
It is possible to have a kind of zero-phase filters, but it would be better if pinescript could support backward indexing thus making us able to do forward backward filtering.
Since noise can affect our analysis, applying smoothing without having to use offset in plot can be considered useful.
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
