Best practices for feature engineering in time-series data
I've been working on my submission for the Q1 competition and wanted to share some techniques I've found useful for feature engineering with financial time-series data.
Lag Features
Creating lagged versions of price and volume data (1d, 5d, 21d) captures momentum signals.
Rolling Statistics
- Rolling mean/std over 5, 10, 21 day windows
- Rolling correlation between assets
- Exponentially weighted variants
Calendar Features
- Day of week, month, quarter
- Days until options expiry
- Earnings announcement proximity
19 Replies
Has anyone tried using attention mechanisms for this? The temporal attention weights could tell you which historical periods are most relevant.
The competition scoring docs could definitely be clearer. I spent 2 hours debugging what turned out to be a normalization issue.
I disagree about the GARCH approach. In my experience, realized volatility estimators (like the Rogers-Satchell estimator) outperform parametric models.
Great discussion! This is why I love this community - knowledge sharing makes everyone better.
Anyone else noticing that momentum factors have been working particularly well in the last month of competition data?
For those new to the platform: start with the tutorial competition. It has a smaller dataset and more forgiving scoring.
One more thing: the scoring engine uses a held-out test period that you never see. So your validation score is the best you can do.
The biggest mistake I see newcomers make: optimizing for the wrong metric. Sharpe != best trading strategy. Consider Calmar, Sortino, and max drawdown.
For factor models, I'd strongly recommend the Fama-French 5-factor model as a starting point. It captures most systematic risk.
Can confirm this approach works. I implemented something similar and jumped from rank 150 to rank 23 in two weeks.
This is a common pitfall. Make sure your features are computed before the prediction date, not on it. That's subtle look-ahead bias.
I ran a quick backtest on this idea and got a Sharpe of about 1.2 before costs. Not bad for a simple strategy.
Turnover control is crucial. My best performing model has a turnover of only 8% daily. High turnover strategies rarely survive transaction costs.
The biggest mistake I see newcomers make: optimizing for the wrong metric. Sharpe != best trading strategy. Consider Calmar, Sortino, and max drawdown.
I ran a quick backtest on this idea and got a Sharpe of about 1.2 before costs. Not bad for a simple strategy.
Market regimes are the elephant in the room. A strategy that works in a trending market will fail in mean-reverting conditions.
Can confirm this approach works. I implemented something similar and jumped from rank 150 to rank 23 in two weeks.
Be careful with the Kelly criterion for position sizing. Full Kelly is way too aggressive. I use quarter-Kelly in practice.
import numpy as np
from scipy import optimize
def max_sharpe_portfolio(returns, rf=0.0):
n = returns.shape[1]
init_w = np.ones(n) / n
bounds = [(0.0, 0.1)] * n
constraints = {'type': 'eq', 'fun': lambda w: np.sum(w) - 1.0}
result = optimize.minimize(
lambda w: -(np.mean(returns @ w) - rf) / np.std(returns @ w),
init_w, bounds=bounds, constraints=constraints
)
return result.x
Have you considered using PCA to reduce the dimensionality of the feature space? I found that the first 10 components capture 80%+ of the variance.