Learn how to detect dominant cycles with spectrum analysis

Name: Learn how to detect dominant cycles with spectrum analysis
Rating: 4.3 (27 reviews)

Using the Fast Fourier Transform and the DFT-Goertzel algorithm to detect cycles in noisy data sets (financial markets)

Created byLars von Thienen

Last updated 11/2023

English

What you'll learn

This course explains the key elements of a Fourier-based spectrum analysis.
Understanding the basic computations involved in FFT-based or Goertzel-algorithm-based measurement.
Explaining the core background of FFT in layman terms and concentrate on the important aspects on “how to read a spectrum” plot.
Learn why the Goertzel algorithm outperforms classical Fourier transforms for the purpose of cycles detection in financial markets
Get the source code to implement the generalized Goertzel transform

Course content

5 sections • 11 lectures • 49m total length

Introduction - Example dataset with 3 cycles2:02
We generate our sample dataset based on 3 cycles with different length and phase offsets. We keep the dataset as simple as possible to compare the basic resolution of each cycle detection method. We will compare the following algorithms in regards to their cycle length detection accuracy:

a) FFT,
b) improved, interpolated FFT,
c) Goertzel alogorithm and
d) generaltized Goertzel algorithm
Applying the Fast Fourier Transform “FFT” for cycle detection12:36
We learn the basic terms required to read a spectrum plot. Using our test input signal, we perform a standard Fast Fourier Transform and plot the magnitude of each Fourier coefficient. We get N/2 Fourier integer coefficient. Each Fourier coefficient "k" represents a frequency bin. The FFT returns all k coefficients in one run. We can look for the peaks in the spectrum and use the k-th coefficient number/frequency to convert the frequency back to the time domain and derive the cycle length.
The Fourier index coefficient – time / frequency conversion1:10
In our Market Analysis, we are interested in the length (and phase) of the cycle measures in the time domain of our input data. Therefore we always need to convert forth and back between the frequency domain. As we learned that the Fourier analysis works in the Frequency domain and delivers integer-spaced k-coefficient. The k-coefficient is also known as wavenumber. The wave number, k, is a measure of the periodicity of a cycle, i.e. the number of oscillations per length unit. In our case, the length unit is "bars" in the time resolution we used, e.g. days, hours or minutes. So a k-index 6 means that the cycle has 6 full oscillations in our measured input data with a total length of 800 bars.

Therefore we can calculate the cycle length via Frequency conversion directly from the k index:

cycle length = N / k

Improving FFT resolution using "zero padding" (a)4:37
As the math shows, the resolution, or delta-frequency, will improve with increasing N. However, if we analyze a fixed time-range from our time domain, we can not add more data to the input data. So what to do?
The method is called “zero-padding”. We just add zeros to the end of the (de-trended) dataset which increases the amount of datapoints “N”. Zero-Padding is a well known approach in digital-signal processing.
Improving FFT resolution: Using interpolation (b)6:22
In this lesson, we zoom into the spectrum of our previous FFT spectrum and have a close look at the coefficients 12 and 13. The spectrum peak appears at the 12th FFT coefficient while we know that the real peak is located between the 12-th and 13-th coefficient.
We use the cubic spine interpolation to improve the resolution.
Improving FFT resolution: Weighted average around cycle peaks (c)1:23
This is another method of improving the frequency resolution:
We can estimate the actual frequency of a discrete frequency component to a greater resolution than the ∆f given by the FFT by performing a weighted average of the frequencies around a detected peak in the amplitude spectrum.
Examples are given in this lecture.

The Goertzel algorithm to detect cycles3:33
Most engineers are familiar with the fast Fourier transform (FFT) and would have little problem using an FFT routine. What many don't know, however, is that there is a much faster method if you only need to detect a few frequencies. It is called the Goertzel algorithm.
It's a subset of the discrete Fourier transform. So why should we use the Goertzel algorithm?
In short, here are the main reasons:
* The GA is faster if you limit the search on specific cycles or frequencies instead of calculating the full spectrum
* The GA can detect cycles in between the 1/N spacing of the FFT
* The GA is better in detecting cycles in environments with lot of noise and trending data
When the number of frequencies searched is small, the Goertzel algorithm can even be faster than the FFT. In addition, the Goertzel algorithm can interpolate between the 1/N frequency divisions, so the frequency search is not limited to just the 1/N intervals of the DFT.
Generalized Goerzel algorithm to detect non-integer coefficients6:30
Generalized Goertzel Algorithm - A Discrete Time Fourier Transform
The novel approach allows to employ also the non-integer-valued multiples of the fundamental frequency spacing of the FFT, making it possible to compute the Fourier transform in discrete-time (DTFT) this way. The main advantage consists in that in various applications, it is no longer necessary to round the frequencies of desire, thus obtaining more accurate results.
In other words, the discrete-time Fourier transform (DTFT) is a form of Fourier analysis that is applicable to a sequence of values. That is exactly what we are looking for - not discrete frequency elements with a limit to an 1/N spacing. We can now measure discrete time cycles.
Even thought if some engineers are aware of the Goertzel algorithm, only a few a aware of the generalized second order version which is able to calculate discrete time cycles in the sequence of given values (e.g. market data).
The lecture deals with the Goertzel algorithm, used to establish the modulus and phase of harmonic components of a signal. The advantages of the Goertzel approach over the DFT and the FFT are explained.
An article about how to code the generalized Goertzel algorithm is included in the ressource section.

Requirements

Basic cycle and/or spectrum analysis knowledge is helpfull, but not mandatory.

Description

At the heart of almost every cycle analysis platform is a spectrum module.

Various derivatives of the Fourier transform are available. But which application of Fourier is the "best" for use in economic markets? This course tries to provide an answer.

Therefore, the course focuses on explaining the essential aspects in layman's terms:

Fundamental aspects on "How to read a spectrum diagram" are at the center of the course.
Different Fourier spectrum analysis methods are compared in terms of their performance in detecting exact cycle lengths ("frequency" components).
Learn what is important in detecting cycles in the financial markets.
Get the source code to implement for your own usage

Understanding the basic calculations involved in measuring cycle length, knowing the correct scaling, correct non-integer interpolation, converting different units (frequency vs. time), and learning how to read spectral plots are all critical to the success of cycle analysis and related projection.

Being equipped with this knowledge will allow you to have more success with your custom cycle analysis application.

There are many issues to consider when analyzing and measuring cycles in financial markets. Unfortunately, it is easy to make incorrect spectral measurements resulting in inaccurate cycle projections either on wrong phase or length gathered from the spectrum plot.

This course explains the key elements of a Fourier-based spectrum analysis.

You will learn why the Goertzel algorithm outperforms classical Fourier transforms for the purpose of cycles detection in financial markets.

Compared to an FFT, the Goertzel algorithm is simple and much more efficient for detecting cycles in data series related to financial markets. You will learn and understand why in this course.

Who this course is for:

Data-science and financial market analysts interested in applying digital signal processing to analyzing and measuring cycles in financial markets
Experts who want to understand the differences between standard Fourier and Goertzel algorithm (FFT vs. G-DFT)

Learn how to detect dominant cycles with spectrum analysis

What you'll learn

Explore related topics

Course content

Introduction3 lectures • 16min

Improving the Fast-Fourier-Transform3 lectures • 12min

The Goertzel algorithm2 lectures • 10min

Comparison & Impacts FFT vs. Goertzel-DFT2 lectures • 6min

Source Code - generalized Goertzel Transform1 lecture • 5min

Requirements

Description

Who this course is for: