Demystifying Financial Forecasting: Validation Methods Benchmark for Time-Series Models
Project description
The project aims to research the best evaluation method for time-series forecasting models in the financial sector. In the research, various validation techniques were evaluated in order to identify the most effective metric for assessing model performance in the financial domain, which is characterized by dynamic trends. The first part of the project was to identify which are the best methods, from various research papers and after a list was made with the most effective one, they were tested on the existing models from BDO, evaluated and concluded which is the best to use overall.
Context
The financial sector plays a pivotal role in global economic stability and growth, encompassing activities like asset management, insurance, banking, and advisory services. It is characterized by rapid technological advancements and reliance on precise data-driven insights to make informed decisions. BDO, a leading international consultancy firm, operates in this dynamic domain, offering a spectrum of services including audit, tax, advisory, and digital transformation. Within its BDO Digital department, the ClaireVoyance project represents a sophisticated AI-driven solution designed to enhance predictive accuracy in financial metrics such as sales, revenue, and profit forecasts.
This project, embedded in ClaireVoyance, aims to address the challenges associated with evaluating time-series forecasting models. Time-series data in finance is inherently complex, often exhibiting volatile patterns and trends that require nuanced validation methods. Current practices, such as the use of Root Mean Squared Error (RMSE), may not adequately reflect predictive reliability, prompting the need for research into optimized evaluation techniques. Leveraging Python and Azure resources, this initiative investigates the most reliable validation methods, aiming to refine ClaireVoyance’s performance. By improving prediction quality, the project seeks to empower financial analysts with tools that provide actionable insights, ensuring BDO clients can navigate their strategic decisions with greater confidence. This research aligns with BDO’s commitment to innovation and precision in delivering client-centric digital solutions.
Results
The most important outcomes of this project include both the practical implementation and the critical insights gained through the evaluation of machine learning validation methods for time-series forecasting in the financial sector. The research initially indicated that sMAPE (Symmetric Mean Absolute Percentage Error) could be the optimal evaluation metric due to its theoretical alignment with time-series data characteristics. However, when put into practice within the ClaireVoyance framework, the results were quite the opposite. sMAPE consistently yielded the worst results in terms of aligning predictions with real-world outcomes. This unexpected outcome underscores the gap that can exist between theoretical considerations and practical applications in data science.
On the other hand, the practical test identified R-Squared as the most effective evaluation method for ClaireVoyance. Unlike sMAPE, R-Squared demonstrated superior performance in capturing the predictive reliability of models, particularly in scenarios involving complex patterns and trends common in financial data. This finding reshapes the understanding of validation metrics in this context and provides a clear direction for future enhancements to the ClaireVoyance tool.
The practical value of these findings lies in their direct application within BDO’s services. By implementing R-Squared as the preferred validation metric, ClaireVoyance will deliver more accurate and reliable forecasts, supporting better decision-making for BDO’s clients. The project highlights the importance of empirical validation, where theoretical assumptions are rigorously tested against data, ensuring that solutions are both effective and actionable in practice.