Weighted Fourier Transform: Spectral Gating & Main Frequency🙏🏻 This drop has 2 purposes:
1) to inform every1 who'd ever see it that Weighted Fourier Tranform does exist, while being available nowhere online, not even in papers, yet there's nothing incredibly complicated about it, and it can/should be used in certain cases;
2) to show PulseWire users how they can use it now in dem endevours, to show em what spectral filtering is, and what can they do with all of it in diy mode.
... so we gonna have 2 sections in the description
Section 1: Weighted Fourier Transform
It's quite easy to include weights in Fourier analysis: you just premultiply each datapoint by its corresponding weight -> feed to direct Fourier Transform, and then divide by weights after inverse Fourier transform. Alternatevely, in direct transform you just multiply contributions of each data point to the real and imaginary parts of the Fourier transform by corresponding weights (in accumulation phase), and in inverse transform you divide by weights instead during the accumulation phase. Everything else stays the same just like in non-weighted version.
If you're from the first target group let's say, you prolly know a thing or deux about how to code & about Fourier Transform, so you can just check lines of code to see the implementation of Weighted Discrete version of Fourier Transform, and port it to to any technology you desire. Pine Script is a developing technology that is incredibly comfortable in use for quant-related tasks and anything involving time series in general. While also using Python for research and C++ for development, every time I can do what I want in Pine Script, I reach for it and never touch matlab, python, R, or anything else.
Weighted version allows you to explicetly include order/time information into the operation, which is essential with every time series, although not widely used in mainstream just as many other obvious and right things. If you think deeply, you'll understand that you can apply a usual non-weighted Fourier to any 2d+ data you can (even if none of these dimensions represent time), because this is a geometric tool in essence. By applying linearly decaying weights inside Fourier transform, you're explicetly saying, "one of these dimensions is Time, and weights represent the order". And obviously you can combine multiple weightings, eg time and another characteristic of each datum, allows you to include another non-spatial dimension in your model.
By doing that, on properly processed (not only stationary but Also centered around zero data), you can get some interesting results that you won't be able to recreate without weights:
^^ A sine wave, centered around zero, period of 16. Gray line made by: DWFT (direct weighted Fourier transform) -> spectral gating -> IWFT (inverse weighted Fourier transform) -> plotting the last value of gated reconstructed data, all applied to expanding window. Look how precisely it follows the original data (the sine wave) with no lag at all. This can't be done by using non-weighted version of Fourier transform.
^^ spectral filtering applied to the whole dataset, calculated on the latest data update
And you should never forget about Fast Fourier Transform, tho it needs recursion...
Section 2: About use cases for quant trading, about this particular implementaion in Pine Script 6 (currently the latest version as of Friday 13, December 2k24).
Given the current state of things, we have certain limits on matrix size on PulseWire (and we need big dope matrixes to calculate polynomial regression -> detrend & center our data before Fourier), and recursion is not yet available in Pine Script, so the script works on short datasets only, and requires some time.
A note on detrending. For quality results, Fourier Transform should be applied to not only stationary but also centered around zero data. The rightest way to do detrending of time series
is to fit Cumulative Weighted Moving Polynomial Regression (known as WLSMA in some narrow circles xD) and calculate the deltas between datapoint at time t and this wonderful fit at time t. That's exactly what you see on the main chart of script description: notice the distances between chart and WLSMA, now look lower and see how it matches the distances between zero and purple line in WFT study. Using residuals of one regression fit of the whole dataset makes less sense in time series context, we break some 'time' and order rules in a way, tho not many understand/cares abouit it in mainstream quant industry.
Two ways of using the script:
Spectral Gating aka Spectral filtering. Frequency domain filtering is quite responsive and for a greater computational cost does not introduce a lag the way it works with time-domain filtering. Works this way: direct Fourier transform your data to get frequency & phase info -> compute power spectrum out of it -> zero out all dem freqs that ain't hit your threshold -> inverse Fourier tranform what's left -> repeat at each datapoint plotting the very first value of reconstructed array*. With this you can watch for zero crossings to make appropriate trading decisions.
^^ plot Freq pass to use the script this way, use Level setting to control the intensity of gating. These 3 only available values: -1, 0 and 1, are the general & natural ones.
* if you turn on labels in script's style settings, you see the gray dots perfectly fitting your data. They get recalculated (for the whole dataset) at each update. You call it repainting, this is for analytical & aesthetic purposes. Included for demonstration only.
Finding main/dominant frequency & period. You can use it to set up Length for your other studies, and for analytical purposes simply to understand the periodicity of your data.
^^ plot main frequency/main period to use the script this way. On the screenshot, you can see the script applied to sine wave of period 16, notice how many datapoints it took the algo to figure out the signal's period quite good in expanding window mode
Now what's the next step? You can try applying signal windowing techniques to make it all less data-driven but your ego-driven, make a weighted periodogram or autocorrelogram (check Wiener-Khinchin Theorem ), and maybe whole shiny spectrogram?
... you decide, choice is yours,
The butterfly reflect the doors ...
∞ Indicator
Indicator
Indicator
[blackcat] L2 Hann Ehanced DMILevel: 2
Background
Among the many indicators, it can be said that DMI is the only "super turning" indicator. This indicator can alone send out risk warning signals when extreme market conditions occur in the stock market, helping us to solve some problems.
If we can operate according to the instructions of DMI, firstly, we can avoid the mistake of buying stocks at the head. Secondly, in the process of falling fear of the market, we can follow the direction signal sent by DMI and catch every time on the way down. Opportunity to rebound to unwind.
If you look at the diagram of the DMI, you will think it is very complicated, because there are four lines in its diagram, and they are intertwined, and it is difficult to distinguish the complex signals in it. But don't worry about its complex structure, we will fully dissect this indicator.
Function
These four lines are: PDI, MDI, ADX and ADXR. The scale of the table is from 0-100, which means from very weak to very strong. The PDI curve and MDI curve on some software are called +DI curve and -DI curve , all have the same meaning.
PDI: Represents the position of multiple parties in the market.
In market movements, the higher the PDI, the stronger the current market. On the contrary, it is a weak market. The A-share market is easy to go to extremes. Therefore, we can see that in the past A-share market, the PDI sometimes fell to near zero, and at this time, it often indicated that a rebound and uptrend was about to start.
MDI: Represents the position of the bears in the market.
In the market movement, the higher the MDI goes, the weaker the current market is, and vice versa, it is a strong market. Before a big bull market comes, we can see the MDI drop to a position close to zero, and at this time, the bears in the market have no power to fight back.
The relationship between PDI and MDI:
In the operation of the market, PDI and MDI are intertwined with each other. If the PDI is above the MDI, the market at this time is a strong market. The MDI is above the PDI, which is a bear market. The closer the distance between the two, the market is in a stalemate of consolidation. On the contrary, the further apart the two lines are, the more obvious the unilateral nature of the market is, whether it is a bull market or a bear market. The so-called unilateral market means that there is no midway adjustment when it rises, and there is no rebound correction when it falls.
ADX: Fast steering pullback.
The difference between ADX and other analysis indicators is that whether it is rising or falling, as long as there is a unilateral market, it runs upwards, not like other indicators, the strong market runs upwards and the weak market runs downwards.
The thread is almost entwined with PDI and MDI in general market movement, which makes no sense at this time. However, once the market breaks out of the market and starts to go to extremes, whether the market is rising or falling, ADX will start to run upwards. At this time, ADX has a clear meaning, because DMI has begun to issue early warning of impending turn!
ADXR: slow pull back.
This line is matched to ADX and is a moving average of ADX values. When ADX goes up, ADXR goes up with it, just slower.
When a round of rapid decline ends, it usually needs to be corrected by a rebound, and ADX will take the lead in turning up. Once it crosses with ADXR, it is regarded as an effective breakthrough.
Numerical division. I set an input threshold for HEDMI, and users can set the optimal threshold to buy and sell according to different TFs.
When PDI crosses the threshold, no matter how strong the bull market is, we must beware of risks from happening at any time.
In order to distinguish more clearly, I slightly modified the formula of the system, and when this happens, the indicator will issue a green warning label, so as to avoid risks in time.
Comprehensive use of four lines:
If the four lines in the steering indicator DMI are intertwined below 50, it usually means that the market is in a state of mild consolidation at this time. The DMI indicator at this time is useless because it does not generate a strong pullback force. Don't worry about an unexpected turnaround in the market. As for the consolidation, it's not a turnaround, it's a breakout.
When PDI and MDI gradually separate, at this time, ADX and ADXR will also rise. At this time, the DIM that is usually messy like twine will be clearly separated. When rising, PDI rises along with ADX and ADXR, while MDI sinks weakly. On the contrary, when the market starts to fall, MDI will rise along with ADX and ADXR, and PDI will sink helplessly. At this time, the DMI will be like a "tiger's mouth", gradually opening its bloody mouth. The bigger the opening, the more lethal the bite.
Here comes a tactic, or technical trend, called double hooves, that is, PDI and MDI split, ADX and ADXR upward to produce golden forks, PDI and MDI are like the double front hooves of a horse, ADX and ADXR The golden fork is like the rear hooves of a steed ready to take off, and this trend of the four lines is like the four legs of a steed that is about to run.
If you think it is too complicated to look at DMI like this, then I can tell you the easiest way to judge, that is, just look at the PDI line. When the PDI line falls below 10, boldly buy the dip, because it is a dip, so you need to calculate the rebound At this time, combined with the golden section theory I often talk about, you can easily find the selling point by making the golden section of the downward trend for the previous trend.
This kind of bottom-hunting method uses the golden section theory, and basically there will be no losses. Remember that one thing is not to be greedy and strictly enforce discipline. This is bottom-hunting, and advancing with both hooves is chasing up. The two styles are different, and the operation styles are different. You also need to explore more in actual combat. Any kind of trick, if you practice it proficiently, it is a unique trick.
Remark
Hanning Window Enhanced DMI
Free and Open Source Indicator Indicator
windowing_taAll Signals Are the Sum of Sines. When looking at real-world signals, you usually view them as a price changing over time. This is referred to as the time domain. Fourier’s theorem states that any waveform in the time domain can be represented by the weighted sum of sines and cosines. For example, take two sine waves, where one is three times as fast as the other–or the frequency is 1/3 the first signal. When you add them, you can see you get a different signal.
Although performing an FFT on a signal can provide great insight, it is important to know the limitations of the FFT and how to improve the signal clarity using windowing. When you use the FFT to measure the frequency component of a signal, you are basing the analysis on a finite set of data. The actual FFT transform assumes that it is a finite data set, a continuous spectrum that is one period of a periodic signal. For the FFT, both the time domain and the frequency domain are circular topologies, so the two endpoints of the time waveform are interpreted as though they were connected together. When the measured signal is periodic and an integer number of periods fill the acquisition time interval, the FFT turns out fine as it matches this assumption. However, many times, the measured signal isn’t an integer number of periods. Therefore, the finiteness of the measured signal may result in a truncated waveform with different characteristics from the original continuous-time signal, and the finiteness can introduce sharp transition changes into the measured signal. The sharp transitions are discontinuities.
When the number of periods in the acquisition is not an integer, the endpoints are discontinuous. These artificial discontinuities show up in the FFT as high-frequency components not present in the original signal. These frequencies can be much higher than the Nyquist frequency and are aliased between 0 and half of your sampling rate. The spectrum you get by using a FFT, therefore, is not the actual spectrum of the original signal, but a smeared version. It appears as if energy at one frequency leaks into other frequencies. This phenomenon is known as spectral leakage, which causes the fine spectral lines to spread into wider signals.
You can minimize the effects of performing an FFT over a noninteger number of cycles by using a technique called windowing. Windowing reduces the amplitude of the discontinuities at the boundaries of each finite sequence acquired by the digitizer. Windowing consists of multiplying the time record by a finite-length window with an amplitude that varies smoothly and gradually toward zero at the edges. This makes the endpoints of the waveform meet and, therefore, results in a continuous waveform without sharp transitions. This technique is also referred to as applying a window.
Here is a windowing_ta library with J.F Ehlers Windowing functions proposed on Sep, 2021.
Library "windowing_ta"
hann()
hamm()
fir_sma()
fir_triangle() Library
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Indicator
Blackman Filter - The Smoother The BetterIntroduction
Who doesn't like smooth things? I'd like a smooth market price for christmas! But i can't get it, instead its so noisy...so you apply a filter to smooth it, such filters are called low-pass filters, they smooth and its great but they have lag, so nobody really use them, but they are pretty to look at.
Its on a childish note that i will introduce this indicator, so what it is all about? I propose a new FIR filter using a blackman function as filter kernel for financial time-series smoothing, do you prefer the childish tone ? Fear not its surprisingly easy!
The Blackman Function
The blackman function look like a bell shaped curve, look:
The blackman function will produce such curve. This function is called a cosine sum function because she is based on the sum of cosine functions, here only 2.
0.42 - 0.5 * cos(2 * pi * k) + 0.08 * cos(4 * pi * k)
Originally you use this function for windowing , what does it means? In signal processing you have a function called sync function , if you use this function as filter kernel you would get the ideal frequency domain response filter, sometime called brickwall filter, it would be extremely smooth.
Above the optimal low pass filter frequency response.
However the sync function has no ending values and goes on forever, therefore we can't use it for convolution, expect if we apply windowing. Filters using windowing are called windowed-sinc filters, i will describe the procedure below :
1 - Create a sync function = sin(pi*n)/(pi*n)
2 - Truncate it = I only keep the first length points of the sync function.
This create a abrupt end, the frequency of a filter using step 1 as kernel would contain ripples in the pass band and stop band, this is bad! The frequency response would look like this :
3 - I multiply my values of step 2 by a window function, it can the blackman window, i no longer have an abrupt end, its smooth!
The frequency response of the filter using this kernel would no longer have ripples! This is the power of windowing functions.
Here we are not using such thing, but we could in the future. Here instead we use the blackman function as filter kernel, because this function is bell shaped this mean that the filter will certainly be smooth (symmetrical weighting is a rule of thumb for kernels when we want really smooth filters).
The Filter
This filter is quite smooth, unlike the gaussian filter this filter give less weights to recent and past values, this is because the blackman function has fatter tails than the gaussian one. I could make a comparison of both, however they are quite alike, if you often use a gaussian filter its up to you to decide which one you prefer.
The filter can do a better job than the moving average when it comes to preserve the frequency components that constitute the cycles/trend.
We can see that the filter has a greater performance when it comes to keep the shape of the market price, thus it has a slightly better fit.
Conclusion
Ok so in this post you learned a bit about the sync function and windowing, those are basic subjects in signal processing, they allow us to approximate the filter with the ideal frequency response, i also showed you that those windowing function could be used as kernel and that they where pretty smooth on their own, there are many others, but the one i prefer is the blackman windowing function.
I know what you are thinking, "we want trailing stops, alerts, colors, arrows!", and i understand you pal, but sometimes its cool to take a break from all this stuff. However i can tell that i'am working on a side project that aim to estimate rolling maximum/minimum as fast as possible, any experiments will be published here, and i can ensure you that those indicators will make your day quite brighter, we will see that soon.
I hope you learned something from this post! I'am a bit tired (look i'am disappearing !)
Thanks for reading !
Indicator
