“In-Context Learning”, Synthetic Data and Evaluation Issues: An (Extended) Comment on Recent…

“In-Context Learning”, Synthetic Data and Evaluation Issues: An (Extended) Comment on Recent Advancements in Time Series Foundation Models.

Entering 2026, we are now roughly two and a half years into the era of Time Series Foundation Models (TSFM). Arguably, the feelings in the community remain somewhat mixed. Since the first release [43], we saw many new models, each accompanied by promising results, with several claiming state-of-the-art performance (please refer to [44] for a survey).

Yet, in contrast to Large Language Models (LLMs), their widespread usage is far to be fully established. While they perform very well on some datasets, there are frequent cases where they fail to beat simpler models (see [38, 45, 46, 47]). In my opinion, these discrepancies are heavily influenced by the inherent difficulties in evaluating such models — a topic I discuss in the third part of this article.

Much like the (early) evaluation challenges with LLMs, the most reliable benchmark at the moment isn’t a public dataset, but rather community feedback derived from closed production environments and non-public test data like competition datasets (which is the closest thing to closed datasets like ARC-AGI for LLMs [48]).

Recently, though, we have seen a spark that has renewed confidence in the potential of these models. This optimism centers on one of the most persistent questions in forecasting: Knowing there is a theoretical (information compression) limit to forecasting a single univariate series, how can we effectively leverage information from related time series to improve our predictions?

Yes, I know, global learning. Training a single model on multiple related time series. Keep in mind that a few years back, the dominance of global learning was not widely recognized. I will skip the details on the major benefits of global learning. Hopefully, by now, they are clear to everyone.

While standard global learning implicitly shares patterns across a dataset through shared weights, distinct architectures have been engineered to explicitly leverage cross-series context. For instance, methods like Smyl’s Context Neural Networks [49,50] or Inter-Series Attention [51] allow models to actively “attend” to related items, dynamically capturing hidden dependencies.

These innovations, despite being effective, were largely confined to dataset-specific learning. Now, we are (slowly) witnessing these mechanisms being rigorously scaled to “Foundation Model levels.” This is what approaches like Chronos-2 [9], the updated TimesFM (via in-context fine-tuning) [53], and similar works [52] are attempting to do.

Interestingly, the authors are borrowing the term In-Context Learning (ICL) from NLP to refer to their approach.

For those familiar with LLM literature, ICL is that remarkable, emergent property where a model adapts to a new task simply by processing examples in its prompt — essentially “learning” without any parameter updates. The current trend in TSFMs is an attempt to replicate this zero-shot adaptation. The primary motivation is to break from univariate constraints, allowing the model to explicitly exploit information from related time series or covariates within the context window to refine its forecasts on the fly.

But there is a catch.

Unlike LLMs, where ICL emerges spontaneously from the sheer scale and inherent task diversity of internet text, this is not the case for time series. Simply feeding a model massive amounts of standard univariate sequences will not magically teach it to recognize cross-series dependencies or adapt to unseen tasks.

Instead, achieving this requires deliberate dataset engineering. This involves creating synthetic datasets with mathematically imposed multivariate constraints (as seen in Chronos-2) or restructuring real-world data into multi-task input-output contexts (as demonstrated by the In-Context Time-series Pre-training, or ICTP, used for TimesFM). In both cases, the strategy shifts from passive data consumption to explicitly teaching the model how to learn from context, rather than waiting for the capability to emerge from scale alone.

But let me slow down a bit.

While this article focuses heavily on Chronos-2 due to its popularity and performance, I will also touch upon other approaches leveraging In-Context Learning (or in-context fine-tuning). My apologies if I focus more on Chronos or miss some other similar approaches.

Below is the roadmap for this article:

1. The Nature of ICL in Forecasting: I will discuss why I believe the ICL as described in models like Chronos-2 is fundamentally different from the original ICL observed in LLMs.
2. The Role of Synthetic Data: Given that the Chronos ICL mechanism relies on Group Attention built entirely using synthetic data, we must discuss the implications of training Foundation Models on generated vs. real-world data.
3. The Evaluation Crisis: Finally, I will address the challenges in evaluating TSFMs. Despite the promising efforts of authors to create “fair” evaluations, I think we are still miles away from benchmarks that truly investigate whether these models perform better than established baselines in messy, real-world settings. I also share my thoughts on how to approach this topic.

Disclaimer: The aim of this article is definitely not to criticize the approach followed by these researchers. On the contrary, I believe this is a significant step in the right direction. My goal is simply to initiate a discussion around these topics to drive the field forward.

In Context Learning for Time Series Forecasting.

Conventionally, a model “learns” some behavior through an iterative training process — spanning pre-training, fine-tuning, or post-training — where internal parameters are explicitly updated via an optimization algorithm. This process relies on minimizing a loss function computed against pairs of inputs and their corresponding ground-truth targets to adjust the model’s weights.

In-Context Learning (ICL), in the context of LLMs, involves teaching models what to do via prompts without any weight updates. It was introduced in the groundbreaking GPT-3 paper [1], where the authors showed that a model can learn from examples in the prompt even if they differ from the data the model was trained on [2]. Basically, the model leverages the vast and diverse knowledge acquired during pre-training to complete a task based on instructions or a few input-output demonstrations [3].

Two key points:

• Emergence from Scale: As framed early in the LLM literature by researchers at Google and reiterated by others, ICL is not a feature that was explicitly designed. Rather, it is an emergent property of scale [6.7].
• Shot-Based Prompting: First defined by OpenAI in the GPT-3 paper, ICL is most commonly realized via "shot-based" prompting (e.g., one-shot, few-shot). In this context, a "shot" refers to a single input-output demonstration of the task at hand.

Beyond these definitions, there is compelling research into how this works. In an excellent article from Anthropic, authors used mechanistic interpretability — attempting to reverse-engineer the model’s internal computations — to investigate the mechanics of ICL [4]. They presented strong evidence that this ability is not just a vague, high-level phenomenon but is driven by a specific, identifiable circuit they named “induction heads”. While I won’t delve deep into the literature here, the key takeaway is that we are finding concrete evidence pinpointing how ICL emerges.

To simplify, we can use an analogy famously shared by Andrej Karpathy [5]: the “context window” serves as the equivalent of RAM for a Foundation Model. If the model’s learned weights represent compressed, static knowledge from training (a “hazy recollection” of data) , the context window allows for dynamic, short-term “working memory” that influences immediate behavior. Using this analogy, in-context learning is simply the model’s ability to adapt its behavior by utilizing this dynamically provided context in “RAM” rather than altering its underlying weights.

Consider this example:

You are a “Linear Trend Forecaster.” You forecast by identifying the constant step-size (slope) in the sequence and adding it to the last value to predict the next step.

Example 1:

- Input: [10, 12, 14, 16]

- Forecast: 18

- Reasoning: The series increases by +2 at every step. 16 + 2 = 18.

Example 2:

- Input: [100, 110, 120, 130]

- Forecast: 140

- Reasoning: The series increases by +10 at every step. 130 + 10 = 140.

Your Task:

Input: [50, 55, 60, 65]

Forecast: ____

There are multiple theories on why ICL works so well for LLMs but are out of scope. My favourite, described by Google Research in [8], is that Transformers perform in-context learning by essentially simulating an optimization algorithm — specifically gradient descent — within their own forward pass. The idea is that during its training, the Transformer meta-learns how to learn.

So, why am I saying this and what does this background matter for forecasting?

In Chronos 2 paper [9], the authors explicitly claim that their “group attention mechanism facilitates in-context learning through efficient information sharing across multiple time series within a group.” They lean heavily on this concept, dedicating substantial discussion to how these ‘in-context learning capabilities’ enable the model to handle multivariate forecasting and covariates.

First things first. To fairly assess this claim, we must first understand how their methodology actually works.

Chronos-2 introduces a “group attention” layer” that operates alongside traditional time attention. During both training and inference, time series in a batch are assigned group IDs; series sharing an ID can attend to one another at each specific patch index. For instance, in a batch with IDs [1, 1, 2], the first two series share information, while the third remains isolated

During training, they explicitly create different grouping scenarios:

• Single series (each gets unique ID) for univariate forecasting
• Multiple variates of the same multivariate series (same ID) for multivariate forecasting
• Targets with their covariates (same ID) for covariate-informed forecasting

At inference, users control these interactions by specifying the Group IDs.

The key insight here is that the model was explicitly trained to handle these different grouping structures. It is an architecturally hardcoded behavior using different grouping patterns, not emergent learning from context.

While the “group attention” mechanism in Chronos-2 is both clever and empirically effective, labeling it “in-context learning” requires a nuanced critique. In the broader landscape of foundation models, ICL is typically conceptualized as an implicit “meta-optimization” process — essentially a meta-dataset-specific-optimizer that simulates gradient descent within a single forward pass. In Chronos-2, however, the model appears to be leveraging its trained attention mechanism to selectively aggregate information across pre-designated series.

This behavior might be more accurately described as “in-context inference” or “group inference,” as it fundamentally differs from true ICL in two critical dimensions:

1. In Large Language Models (LLMs), ICL is documented as an emergent property of scale, where models spontaneously develop the ability to learn from examples without explicit supervision for that specific behavior. In contrast, Chronos-2’s group attention is an explicitly engineered architectural feature. The model does not “discover” how to group data; it is deliberately trained with specific objectives to handle these predefined grouping structures
2. Original ICL relies on shot-based learning, where a model adapts its internal logic based on input-output demonstrations (examples) provided within the prompt. Chronos-2 does not infer new relationships or patterns from such contextual demonstrations. Instead, it executes a pre-learned information aggregation strategy, relying on fixed attention patterns to combine related series rather than adapting its behavior to novel, unseen patterns on the fly.

Google’s updated TimesFM adopts a similar philosophy but a different mechanism via “in-context fine-tuning” [53]. Like Chronos, this is not an emergent behavior but a learned one: the model undergoes a dedicated second training stage where it is explicitly taught to process prompts containing sequences of related time series (separated by special tokens) alongside the target history. Taking this concept further, the ICTP framework [52] restructures the entire pre-training corpus into multi-task input-output examples, explicitly teaching the model to dynamically adapt to unseen objectives — such as imputation or backtracing — solely from context. While Chronos-2 relies on parallel “Batch Inference” (Group Attention), these approaches rely on sequential prompting — stacking related series or task demonstrations linearly into the context window.

Both, however, are engineered solutions relying on deliberate data restructuring and training not some emergent behavior. Approaches like ICTP and TimesFM-ICF largely rely on reorganizing real-world repositories — such as the standard ETT or Weather datasets — into multi-task prompts. While this feels “closer to reality,” it faces a critical bottleneck: naturally occurring, high-quality “grouped” data is scarce and fragmented, often lacking the diverse inter-series dependencies needed for robust generalization. Relying purely on real data risks overfitting to specific corpus biases or suffering from negative transfer when retrieved examples are noisy. In contrast, Chronos-2 bypasses this hurdle by engineering its own training curriculum via synthetic data.

But more on that later.

My point is that it’s critical to distinguish between these two ICL mechanism and how they work, especially since influential research groups like the Amazon folks can shape entire paths of future work. While developing models that adapt internal logic to novel, unseen datasets is an exceptionally promising direction, we must be carefull when we import terminologies (and methodologies) from LLMs literature. Time series and natural language have clear, fundamental differences, and applying a term like “ICL” imprecisely risks oversimplifying the challenge and diverting focus from the true core issues that still need to be solved by simply trying to “borrow” ICL techniques from LLMs. (maybe rephrase last sentence)

This distinction between engineered and emergent capabilities is critical and extends beyond time series forecasting. We see a similar pattern with foundation models for tabular data [10]. In this case, “in-context learning” describes the model’s ability to process an entire training set as its context, from which it makes zero-shot predictions for test points. While impressive, this capability is the designed outcome of its specific architecture, which was pre-trained on a massive corpus of synthetic datasets to execute this single, pre-defined task. This represents a powerful form of algorithmic approximation, but it fundamentally differs from the spontaneous, shot-based task adaptation that defines ICL in language models.

To summarize, Chronos-2’s (and TimesFM-ICF & ICTP) capability is not an emergent behavior but an explicitly engineered and trained mechanism. It is not a product of pure scaling alone, but rather stems from carefully engineered synthetic datasets designed to approximate desired behaviors.

The model’s ability to “learn tasks” is a direct consequence of its pre-training on a purpose-built synthetic dataset where these grouping structures were methodically constructed. While this heavy reliance on synthetic generation is not necessarily a weakness, it represents a critical dependency that requires careful attention.

(Grouped) Synthetic Data for Time Series Forecasting.

Synthetic data is at the core of the batch-inference mechanism (I will refer to ICL as “Batch-Inference” in this work). According to the authors: “These results underscore the importance of synthetic data, suggesting that with further research, real data may not even be required for effective pretraining”.

I find this a rather ambitious perspective. Yet, the field remains open, and such possibilities are worth considering.

Arguably, synthetic data has become a foundational component of Time Series Foundation Models (TSFMs). This is mainly due to the challenges in finding high-quality pre-training collections of time series to scale Transformers. In my opinion, the biggest challenge for highly curated corpora is diversity, hence the need for synthetic data. As a consequence, modern TSFMs rely on hybrid approaches, mixing massive real-world corpora with synthetic data.

Literature contains many approaches for generating synthetic data. There are several comprehensive surveys (such as [11]) that detail the most popular and highly recommended approaches.

Despite some promising findings — such as ablation studies in TSFM papers like [10], [12], [13], [14], and models trained exclusively on synthetic data like [15] and [16], I was a bit skeptical of this over-enthusiasm/reliance on synthetic data. Not because of some conflicting evidence claiming that for TSFM, real-world data consistently outperforms synthetic, even when only a limited number of real examples are available [17]. The main reason is that claims such as “synthetic data will revolutionize pretraining” existed initially in LLM literature as well. Since then, its use has been stabilized, typically as a portion of the pre-training dataset or as a recipe for high-quality data in self-supervised learning and reasoning traces for thinking models. Ah, and my personal favorite: using distillation to train lightweight or micro-models. However, that level of success requires extensive data cleaning and an in-depth understanding of what it takes to build an extremely curated synthetic dataset (aimed for specific use cases and not as universal problem solvers)

To my (very big) surprise, and to mitigate my own skepticism, I tested Chronos batch inference (both with and without covariates) when it was released, and I was amazed. It was obvious that this approach could lead to significant accuracy gains. I share some of those findings below. Of course, because these are closed datasets, I cannot provide more details. However, I am working on a follow-up article that will detail the specific approach I followed.

Despite these results (and the increased usage of synthetic data for reasoning LLMs), I remain a bit skeptical. Maybe it's personal bias.

But let's take it step by step.

The key contribution regarding synthetic data is the introduction of “multivariatizers.” The aim is to enable the model to “attend” to other related time series within a group. First, the authors generate univariate synthetic time series using a variety of techniques (AR & ETS models, TSI, KernelSynth, etc.). Then, they impose dependencies among them: either by applying linear and nonlinear transformations at identical time steps to simulate instantaneous correlations (cotemporaneous) or by weaving dependencies across time steps to induce richer properties like lead-lag relationships and cointegration (sequential).

The contribution around the synthetic data are the multivariatizers. The aim is for the model to “attend” to other (related) time series within a collection. First they generate univariate synthetic time series using a number of techniques(AR & ETS models, TSI, KernelSynth etc). Then they impose dependencies among them either by eapplying linear and nonlinear transformations at identical time steps to simulate instantaneous correlations, or by weaving dependencies across time steps to induce richer properties like lead-lag relationships and cointegration.

This process yields a corpus of synthesized, interactive signals specifically designed to train the group attention mechanism. It is worth noting that this reliance on synthetic priors to define model behavior mirrors the philosophy behind tabular foundation models (TabPFN), effectively adapting that paradigm to the temporal domain.

In my view, this specifically targets a critical limitation of earlier univariate TSFMs: superficial pattern matching. When a model is trained on a massive, undifferentiated “soup” of time series — mixing crypto volatility with retail inventory — it risks learning spurious correlations, similar to the classic statistical fallacy linking ice cream sales to drownings. Group attention constrains this behavior — at least partially, as superficial matching is still possible at the group level — by explicitly forcing the model to focus on the local context of related time series and covariates.

Intuitively, one can think of the Multivariatizer and Group Attention combination as a massive, probabilistic lookup table of grouped dependencies. By manufacturing mathematical relationships, the authors have essentially created a finite “vocabulary” of dependencies. During inference, the model performs a form of “soft-matching” against this synthetic library, activating the relevant attention patterns to exploit the information. In this sense, the group attention mechanism acts as an associative memory of mathematical relationships.

Ok, so what is the issue with this approach?

It is by now common knowledge that LLMs (and Transformers in general) perform very well on the data they were trained on. Findings in the TSFM literature point to this phenomenon as well [19]. The core idea is that TSFMs excel specifically within the domains they were exposed to during training. No surprises there.

Where am I getting at?

My point here is twofold. First, since TSFMs perform well on datasets containing common dependencies — like those that have been mathematically synthesized — it is safe to assume that these patterns are also identifiable by dataset-specific global models. It is not magic, after all. I expect that (under conditions) current Global Models (both GBDTs and established NN architectures) can effectively capture relationships created via such transformations, lead-lag effects, and cointegration.

Second, a limitation identified in the synthetic data literature for LLMs is the “reality gap” between the data used to approximate a distribution and the actual distribution [20, 21]. Synthetic generators impose structured dependencies that tend to be cleaner and more mathematically consistent than the chaotic, non-stationary dynamics of real-world systems. This constraint does not apply to dataset-specific models engaged in “pattern discovery.” Such models aim to capture the manifold of the specific dataset at hand, even if that shape is weird, ugly, or physically impossible to generate via standard synthetic equations.

When you perform batch inference with Chronos, you are essentially projecting your real data onto this “synthetic manifold” — a smooth, high-dimensional surface representing all the synthetic data the model absorbed during training. If your real data lies off this manifold (i.e., it contains a relationship not covered by the synthetic generator), Chronos must “approximate” it by snapping to the nearest point on that surface. The distance between your real data’s true structure and the synthetic manifold represents the foundation model’s irreducible error. Crucially, the model cannot overcome this gap because its weights are frozen.

Ironically, true ICL would assist here. As mentioned, Transformers performing ICL are effectively executing an optimization algorithm — an implicit gradient descent — within their own forward pass [8, 22] This “meta-optimization” hypothesis suggests that the model uses the provided context examples to construct a transient, internal loss function and then dynamically updates its internal state to minimize this error, effectively “fine-tuning” itself on the fly to the specific quirks of the input without changing its permanent weights.

On the other hand, attempting to sidestep this “reality gap” by avoiding synthetic data introduces its own distinct set of challenges. Alternative approaches, such as ICF for TimesFM [53] or the ICTP framework [52], bet on restructuring massive real-world corpora to teach the model adaptability. While this strategy ensures the model encounters realistic noise and non-stationarity — avoiding the smoothness of the “synthetic manifold” — it hits a hard ceiling: the finite pool of high-quality, public time series. Unlike text, where the internet provides a practically infinite stream of diverse tokens, public time series repositories are limited. Consequently, models trained solely on real data might struggle to scale to true “foundation” levels because they may simply run out of diverse patterns to learn.

This “diversity barrier” is further compounded by a fundamental theoretical hurdle recently highlighted by [54]. In Computer Vision, datasets scale effectively because objects possess semantic invariance — an “apple” remains an “apple” regardless of lighting or rotation. Time series lack this clear-cut invariance; a pattern representing “demand” in one dataset might be mathematically identical to “noise” in another depending on the sampling frequency or domain. This makes building diverse, scalable datasets incredibly difficult, as we cannot easily curate “distinct concepts” to ensure the model learns a universal representation. Thus, we are left with a difficult trade-off: rely on synthetic data (like Chronos) to mathematically manufacture diversity at the cost of a reality gap, or rely on real data (like TimesFM) to ensure realism at the cost of hitting a diversity ceiling that limits true zero-shot generalization.

Given this specific trade-off, my argument is that for any given dataset, there likely exists a domain-specific global model that performs equally well or better. This is because a domain-specific model bypasses the need for “universal” generalization entirely. Through direct optimization such a model compresses the predictable patterns (both univariate and shared) that a foundation model has merely approximated via a mathematical synthetic data schema.

Of course, this assumes specific conditions (I am working out the maths) and requires that the model possesses “good” priors (architectural design and/or engineered features), often dependent on the modeler’s expertise. However, the architectural flexibility of domain-specific models to adapt to the task at hand remains a significant advantage.

This raises a philosophical question: Is it better to memorize every possible problem in the world, or to learn the solution on the fly with the appropriate model capacity? This question echoes the “AGI haunting” of current LLMs, the scaling barriers, and the need for some continual learning module.

To be fair, in practice, discovering this “optimal” domain-specific model can be incredibly difficult, especially in sparse or noisy data regimes. This is precisely why I believe Chronos-2 is a valuable tool in a forecaster’s toolbox. Having a model that generates good off-the-shelf performance for virtually any task is both a gift and a challenge for experienced practitioners.

I also believe that with a few breakthrough papers, the current limitations of on-the-fly optimization could be overcome. Additionally, smart innovations will likely narrow the “reality gap” of synthetic approaches, while new curation methods might unlock more value from real-world corpora despite the diversity barrier. But this leads to a fundamental question: squeezed between artificial smoothness and finite real-world variance, how much more “information” can really be extracted from a time series (or a collection of them) without the actual underlying data to explain the variations? Is the true bottleneck really a lack of “better architectures” or the inherent limits of the data itself?

To conclude this section, I am not trying to be adversarial toward synthetic data or group attention. My goal is simply to deconstruct how these models work and share my perspective on the difficult trade-off required to scale them. Understanding these underlying mechanics and data-specific limitations is arguably more critical than simply engineering more synthetic generation methods or blindly ingesting more real-world tokens. Or maybe not. Maybe brute force and massive scaling really is the solution to force invariance. But learning from LLMs and the unique challenges of time series, this might not be the case.

Evaluating Time Series Foundation Models.

As if we didn’t already have enough issues with evaluating forecasts in general. You know: non-stationarities, non-normalities, non-independences, metrics that break down with zeros, metrics that favor over/under-forecasting, averaging issues, cross-validation issues, and leakage issues. Please refer to [23] or [24] for details.

In my opinion, the biggest problem here is the misalignment between Intrinsic and Extrinsic metrics. You know, metrics that measure forecasting performance versus metrics that measure the success of the downstream decision-making task. After all, in practice, we forecast to support a decision. Lately, we tend to forget this. But thats a talk for another day. This is an interesting read on the topic [25], and some extra references can be found here [26].

The problem with TSFMs is that we now have to worry about issues connected to “Universal” or “Foundational” evaluations. These are well-documented in the LLM literature: Data contamination (answering based on memorization), benchmark saturation, reliability, robustness, and so on. For the details, see this article from Anthropic [28] and this well-executed survey [27].

Recently, driven by skepticism about whether LLMs are truly taking us to the “heaven of AGI”, the conversation has shifted to the misalignment between benchmarks and reality. We see models achieving terrific scores on leaderboards, yet users rarely experience those same gains in daily decision-making. This phenomenon is often referred to as “benchmaxxing” (overfitting to the test set). And if you think this is a terrifying issue for LLMs, where we have thousands of evaluation datasets for different tasks, (yeah 15–20 are “main ones”), in TSFM we basically have … one.

GIFTeval [29]. The holy bible of TSFM. Sure, there are others: Monash forecasting repository [30], BasicTS+ [31] TFB [32], ProbTS [33], and Chronos Benchmarks[34]. But GIFT-Eval is the undisputed king.

And GIFT-Eval is recent. Before that, we had multiple distinct topics. Labs were training and reporting accuracy on different datasets or splits, benchmarks had very narrow scopes (like the Long Sequence Forecasting setup popularized by the Informer paper [35]), and leakage was hard to quantify. There was no way to trust a model’s performance. Smaller models were outperforming larger variations (like in [34]), and papers were using inconsistent datasets to claim SOTA performance.

It was a mess.

Despite being a step in the right direction, GIFT-Eval certainly did not resolve all the issues with model evaluation. To their credit, the Chronos authors acknowledged this gap and tried to create a more comprehensive evaluation framework to test their model [36]. This is undoubtedly another crucial step.

In my opinion, fev-bench [36] is arguably one of the (if not the) most robust attempts up to date. It moves beyond the single-split method by enforcing rolling window evaluations, integrates principled aggregation via win rates and skill scores, and, crucially, applies bootstrapped confidence intervals to separate genuine signal from statistical noise. This directly addresses the statistical significance issue found in previous attempts.

Yet, it does not solve all the issues. For instance, fev-bench relies on a self-reporting “honor system” to handle direct data leakage, but it fails to account for what [37] terms “Global Pattern Memorization” (basically, indirect leakage). Because Foundation Models are trained on massive datasets confounded by global events, they may “remember” and predict patterns influenced by these events — even on time series technically never seen during training. Okay, [37] also reports issues like the confused lineage of benchmark datasets (e.g., “Electricity” vs “ECL” vs “LD2011”), where renaming and remixing make it difficult to detect if a model has already trained on a different version of the test data. But I think with some careful work, this can be mitigated

The second issue I suspect might happen (again, not sure — experiments are needed here to verify this) is that the evaluation potentially suffers from a “synthetic bias” and a disconnect from business reality. Chronos-2 dominates on covariate tasks using synthetic data generation that likely aligns well with the curated, “clean” academic datasets (potential benchmaxing?), but this performance may not translate to the chaotically “dirty” real-world data we face in production.

The point is that truly evaluating these models is extremely hard, and we currently lack the proper depth of research in this domain. This creates a skepticism that is further reinforced by conflicting findings. For instance, according to the Chronos authors using fev-bench, models like MOIRAI, TimesFM, and Chronos Bolt outperform a statistical ensemble. Yet, in another Amazon publication [38] using a different evaluation methodology, that very same statistical ensemble performed better than all three foundation models. (Note: I am not 100% certain if the exact same model versions were used, but the citations are identical in both papers).

This does not mean that 1) Evaluation A is flawed and statistical models are bad, nor that 2) Evaluation B is flawed and foundation models are bad. It means that something might be wrong with how we are evaluating. Looking back a bit, there was a lot of skepticism regarding Global Learning as well, before the results became undisputed. There is a reason battle-tested models like ETS are still around. There is a reason why GBDTs are preferred in practice — after all, they were the winning method (or amongh them) in the latest competition that included forecasting (included the most recent VN2). And there is a reason models like DeepAR, TFT, and N-BEATS are preferred over a many other NN alternatives.

Honestly, regardless of these results, I believe that even if every new benchmark in the world is saturated, people will still be skeptical — and with good reason. And this is not a problem unique to TSFMs but a common issue with LLMs too. Despite benchmarks being saturated one after another, people realize we are still far from the big claims (you know: ASI, AGI) using the current architecture (self-attention Transformers). Many influential voices have recently stated that having a continual learner — one that can adapt to any task — might be better than training a huge static model and expecting it to deliver state-of-the-art performance across every task in the world [39, 40]

Which, I guess, in the case of TSFMs, is basically a generalized version of my argument for why a dataset-specific model appeals more to me (again, no hard evidence yet, just intuition)

Let me explain further. Consider Simpson’s Paradox [41], where Model A performs better than Model B on aggregated data but worse than Model B on every subset of that data. It makes sense, right? Optimizing a global loss across all tasks tends to converge on a global statistic (like a mean or median depending on the loss). Yet, in practice, we often need top performance solely on Task A, not “average” performance across every task. Current LLMs seem to struggle with this specific depth, which is why in NLP we rely on approaches like fine-tuning when prompt engineering and ICL don’t work.

TSFMs also minimize a global loss across all available datasets. However, things there are a bit different.

First, we have no clear evidence yet that fine-tuning is beneficial in this domain. Second, in contrast to LLMs, performance across datasets is not interchangeable. With LLMs, if you train a model (say, a “Opus 4.5”) that scores 80% on SWE-bench, you can be reasonably confident it will solve most coding problems. The skill transfers. In forecasting, however, performing well on Electricity tasks does not mean you will perform well on Retail tasks. The patterns are not identical, even if they might spuriously appear so. They are driven by different factors. Hence, in many cases, we are not interested in the global minimum across multiple datasets. Which is precisely why I think the approach on fev-bench is a good idea.

Recent research strongly supports this intuition [46]. They demonstrated that a simple, task-specific model — combining PCA for feature extraction with a linear head — could effectively match or even outperform complex, pre-trained foundation models. Similarly, [45] argued that current benchmarks often set the bar too low. They showed that a carefully tuned vanilla Linear Regression could serve as a “litmus test” frequently outperforming popular foundation models like Chronos Base on standard benchmarks. These finding challeng the assumption that “universal” pre-training is always necessary, suggesting instead that capturing the specific low-rank structure of the target dataset is often far more effective than applying a generic global prior.

Third, forecasting is a means to an end. We forecast to support decisions. Unlike in NLP, where the generated text (or, in the case of Opus 4.5, code) is often the final product, an accurate prediction here is rarely the end task itself. And no, I am not referring to the well-known problem that intrinsic accuracy metrics (like MASE or ScaledQL) are often poor proxies for the success of downstream decision-making tasks. That is a general issue. The specific problem here is that forecasting is a decision-support tool that often demands accuracy at the individual item level (e.g., operational reordering decisions per SKU, or strategic long-term decisions), yet global optimization tends to mask local failures.

As Kolassa’s analysis of the M5 competition [42] demonstrates, complex methods outperformed simple baselines (like Exponential Smoothing) on only ~58% of individual series, while actually losing to them on nearly 20% of the data. Now, one can argue that for dataset-specific Global Learning, this trade-off might be acceptable. If a retailer trains a single model across all their SKUs, minimizing the aggregate error directly aligns with the business objective: reducing total operational costs across the entire portfolio. In this context, a model that performs better “on average” for that specific company is a win, even if it loses on some individual items. But optimizing a global loss across unrelated datasets? That is meaningless to decision-makers (unless we have evidence of constant, superior performance everywhere). We simply cannot assume that a lower loss on a universal benchmark translates to better decision support for a specific, isolated domain.

Does it really matter for my downstream, item-level decision-making task to use a model trained on (and performing well on average across) unrelated domain datasets, given that performance is not interchangeable? I would vote “no,” though I might be wrong or overly sticking to operational forecasts and ignoring other use cases. This is exactly why I believe we need more research on these foundational questions instead of just (over)engineering new models.

To conclude this section, you might accuse me of having some “boomer” nostalgic bias for traditional methods (yeah, I refer to global models as traditional now). That is a fair criticism, but an incorrect one. I trust science. Every time I test a new Foundation Model on closed data, I am genuinely anxious to see if it will dethrone production-ready models. And if it does, I will be the first to adopt it.

While evaluation methodologies are definitely improving, extraordinary claims, like a single model one-shot forecasting every time series in the world without refitting, require extraordinary evidence. Again, this is not criticism towards the researchers behind Chronos or other TSFM. After all, they are doing the heavy lifting to push the field forward. I am simply arguing that we haven’t yet reached the level of evidence required to blindly trust these “universal” capabilities in production.

I believe there are three concrete ways to bridge this trust gap:

1. Community Validation: We need open-minded scientists and engineers to test these models on their messy, proprietary datasets and transparently share the results — good or bad.
2. Rigorous, Closed Challenges: We need more “hard” benchmarks similar to the ARC-AGI prize or competitions like the M/VN series that test generalization on truly unseen data, preventing “benchmark leakage.”
3. Deeper Mechanistic Research: We need more ablation studies that dissect exactly where the performance gains come from — separating true generalization from simple pattern memorization.

Until then, healthy skepticism remains a very good tool.

Conclusions:

In my opinion, Chronos-2 represents a significant leap forward in the field of TSFM. Whether we refer to the usage of group attention as “in-context learning” or as a highly sophisticated form of batch inference, it is undeniably a very clever solution to the problem of handling covariates and cross-series correlations. We should also thank the authors for their transparency and for sharing this work with the community. Having high-quality, open-weight models to experiment with, deconstruct, and learn from is a privilege we should not take for granted.

In terms of practical application, I strongly believe that Chronos-2 is a great solution for scenarios demanding fast, off-the-shelf results without the overhead of building complex training pipelines. It provides a high-quality “quick and dirty” baseline when there is no time or expertise to build a specialized, dataset-specific solution.

However, we must remain grounded regarding hyped claims. In this not-so-short article, I tried to share my main concerns regarding the model — not to criticize, but to create further discussion to drive the field forward. The shift toward foundation models is exciting, but it shouldn’t come at the cost of critical thinking. We should embrace these tools with enthusiasm, while maintaining the healthy skepticism required to distinguish between genuine generalization and sophisticated pattern matching.

Ultimately, the field may be shifting towards a hybrid view: rather than relying on a single static model for everything, we might see a move towards better post-training techniques or efficient, dataset-specific adaptations. However, simple, optimized dataset-specific models often still hold the edge. To conclude, in my eyes, the field is progressing rapidly, but a true “universal forecaster” might not be this close.

Thanks for reading!