In my “Man vs. Machine” paper, I undertake an intriguing exercise.¹ The analysis required a lengthy sample of hedge funds. Half of the sample declared whether they were systematic or discretionary. The other half made no declaration but did provide detailed descriptions of what the fund did. We set out to do the following natural language processing exercise: we would look for words and phrases that distinguished systematic from discretionary in our training sample (where we knew the truth) and then apply this to the thousands of unclassified funds.

Certain words made a lot of sense, such as algorithmic. We were also keen on the word quant or quantitative. To our surprise, the word quant did not separate systematic from discretionary. Indeed, it was more likely that “quant” was associated with discretionary fund descriptions!

What does this mean? It is simple: quantitative analysis is a crucial part of the investment process for both discretionary and systematic funds. While in the past, the discretionary portfolio manager might have provided a spreadsheet with valuation models for her favorite names, today’s information environment demands the use of quantitative tools. Thousands of databases are now available to investment professionals and it is implausible that any single manager can manually process all of the data demanded by these quantitative tools.

In today’s environment, we are all quants, but we do not all run systematic portfolios. In systematic investment, trades are generated by rules or algorithms, which are of course designed by humans. These algorithms operate independently when used in live trading. In discretionary portfolios, managers make the final trading decision even though they may use plenty of quantitative tools to assist their decision process. In the end, however, a person generates the trade idea—not an algorithm.

Thirty-five years ago, systematic investing was a niche investment style, mainly focused on trend-following systems. The initial algorithms operationalized a century-old investing approach called technical analysis. Although technical analysis has many flavors, the identification and extrapolation of trends is its cornerstone. One drawback is the inevitable turning point. At some moment in time, the trend will reverse. Algorithms evolved so that after an extended trend (or a very strong trend signal), risk was reduced. This capability effectively allowed for reversals and reduced the losses suffered at turning points.

The next wave was quantitative stock-selection models. These models used an algorithmic approach to identify stocks the strategy should buy or sell. For long-only portfolios, these models determined over- and underweighting of securities. These models typically went beyond price data and included fundamental information like valuation, growth, profitability and quality metrics.

The next significant innovation in systematic investment was the emergence of so-called smart beta strategies. These low cost products might focus on a particular factor or strategy, such as value.  The name – typically applied to a wide array of formulaic or algorithmic strategies, often with impressive backtest results – plants the impression that the strategies are smart. However, there are plenty of strategies that are not smart offered under this rubric. The smart beta strategies create an index using an algorithmic approach. Investors can access the strategy in many forms, such as exchange-traded funds or mutual funds. Smart beta strategies also have multifactor versions.

Simultaneously as more capital entered the market, many managers realized the easiest way to increase alpha was to reduce costs. One way to reduce costs was through improved execution. Hence, the third wave was the emergence of systematic high-frequency trading. Such trading can produce stand-alone profitability to funds such as Renaissance Technologies—or it can be part of the execution strategies of both systematic and discretionary funds.

Currently, we are at the beginning of the era of using artificial intelligence (AI) tools in both systematic and discretionary strategies. For example, large language models hold the possibility of helping researchers analyze a vast amount of financial information and isolate risk factors.

In recent years, machine-learning tools have emerged to drive systematic investment strategies. These tools have been around for quite a while. Indeed, I tried to implement some deep-learning tools on equity returns almost 25 years ago. The model failed because it was too simple. It was simple because of computational constraints.

Three specific factors have led to the surge in machine-learning applications. First, computing speed greatly increased. In 1990, a Cray 2 supercomputer cost $32 million (today’s dollars), weighed 5,500 pounds, and needed a cooling unit. It was able to do 1.9 billion floating-point operations per second.  Today, your mobile phone is 500 times faster than the Cray 2.

The second factor is data. In the time of the Cray 2, a cost of a gigabyte of storage was $10,000. Today, the cost of a gigabyte is less than one cent. This allows for the cheap collection and storage of vast amounts of data. In addition to cheap storage, the scope of data expanded beyond financial and price information to include unstructured data from a multitude of sources (text, voice, web, geosat, pictures, etc.).

The third factor is open-source software. In the past, software development was siloed. Today, we have a completely different situation. Development is much more efficient, because engineers do not have to reinvent the wheel: they go to GitHub and find many others have dealt with the same problem they are facing, and the solutions are freely available to them. 

The main advantage is discipline. Using an agreed-upon set of rules, the algorithm implements those rules. The machine, obviously, does not have direct behavioral biases and will not fall prey to human emotion.  Indeed, the best algorithmic strategies will observe, learn from, and profit from others’ emotional choices. For example, the algorithm will not necessarily hold on to loser trades (the disposition bias). The machine is unable to feel regret. Further, in periods of heightened market tensions, the machine “has a cool head.”

A second advantage is the machine’s ability to process information. Two components enhance the value of this processing. The first is that, given the explosion of big data, it is feasible for the machine to process large datasets – a task that is infeasible for the manager operating without a model. The other component is speed. Whether by processing large datasets or quickly reacting to news in the market, the machine has distinct advantages.

The main disadvantage is the loss of flexibility. Algorithms are a simplification of the world and often highly parameterized. They are optimized on past behavior, but the world changes. This time is always different. It is a challenge—even with today’s technology—to construct a reliable algorithm that evolves through time. Fitting a stationary algorithm to a nonstationary market is a recipe for failure.

A second disadvantage is a tendency to overfit in the model development stage.² Algorithms and parameterizations are optimized to the past data. Given that the signal-to-noise ratio is so low for financial assets, researchers tend to optimize the noise. An overfit algorithm will look great in the backtest and perform poorly in live trading.

Third, algorithms usually fail to account for market structure. For example, an algorithm may successfully detect a market mispricing. In such a case, the model is not overfit (i.e., the backtest performance is not exaggerated). When applied in real-time trading, however, the model fails. The failure is not due to a deficiency in the research process. Instead, the market has evolved as others entered to take advantage of what is (now) a temporary mispricing.

Fourth, some investors fall into the trap of the “black box.” This often occurs with machine-learning implementations that are purely data driven, rather than based on a solid economic foundation. Investors should beware of managers’ statements such as “We can’t reveal how the model works because it is proprietary information.” All algorithms need to be explainable. Even the most complex machine-learning algorithms can be reverse-engineered to some extent by feeding in shocks to the inputs to determine how the model’s outputs change.

Finally, new AI tools pose risk. For example, the generative AI can learn from past data and might inherit some behavioral biases. GPTs are relatively new and sometimes the extrapolations made by the current generation of algorithms could be problematic. To successfully deploy these tools, it is likely best that asset managers fine tune the open-source GPT tools to their needs rather than relying on off-the-shelf commercial products that suffer from the black box problem.

I believe we are at a tipping point in the investment management landscape. Investors correctly realize that systematic investing tools are not just here to stay but will likely grow in prominence. Investment managers are increasingly adopting these tools. Their implementation, however, requires considerable skill and collaboration across investment, technology and quantitative capabilities.

In the current transition period, some asset managers need to check the box and thus will offer some machine-learning and AI enhancement to their current strategies. These asset managers are operating at a considerable disadvantage, however. First, they may not have expertise in systematic model development or machine learning/AI. A summer intern with a single machine-learning course on her resume does not constitute “expertise.” Second, with hundreds of machine-learning algorithms as well as an increasing number of AI tools, selecting the approach that best suits the problem at hand requires skill. Third, inexperienced firms mistakenly believe “big data is free.” Even if the data are readily available on the internet, the data are not free. Considerable care (and cost) is involved in cleaning the data. If the data are uncleaned, they can deliver misleading results when the algorithm is applied. Fourth, many investment firms are unlikely to have the specialized computing power necessary to successfully train and run these algorithms in real time.

Putting all of these points together, many investors will be disappointed because they thought they were investing in the latest technology. Technology alone, however, does not increase the probability of outperformance: outperformance depends on the skills of the team who are applying the technology to the investment problem. As such, I see a shakeout in investment management. Some smaller companies that cannot afford to hire a machine-learning team, invest in the IT resources for reliable database management, or acquire the necessary computing assets, will fail or will be acquired by larger companies that have both the experience and the necessary technological assets.

One forecast is easy to make. Research projects that were just theoretical decades ago are now feasible to execute. We are already seeing a surge in research on systematic investing, which is only the tip of the iceberg. Systematic strategies are being increasingly utilized by institutional investors in tail hedging and defensive overlay programs, as well as in asset allocation solutions to replace part of equity and fixed income portfolios.  In the last year, we have seen the emergence of a new class of promising AI tools. As I have pointed out, there are risks that need to be mitigated. New systematic strategies hold the promise of improving portfolio selection and well as reducing risks. It is an exciting time to be in the asset management business.