Imagine this: It’s a typical day. Your team is running efficiently, relying on accurate forecasts to plan shipments, restock shelves, and confidently meet customer demand. Inventory is flowing, sales trends are clear, and operations feel predictable. Suddenly, your forecasting system breaks without warning. You have no visibility into recent sales or current store inventory levels.
Your warehouse doesn’t know how much product to ship or where to send it. Supply chains stall, fresh produce rots, and customer expectations go unmet. This nightmare became a reality for several major companies in late 2024 when their third-party forecasting provider, Blue Yonder, was hit with a ransomware attack.
This incident sheds light on a growing risk: reliance on external platforms for critical operational intelligence. Outsourced forecasting may offer quick access to “out of the box” functionality, but it also introduces fragility, generic insights, and a loss of ownership over a core business process.
It’s time to rethink the model. Bringing forecasting in-house is more than a technical upgrade – it’s a strategic investment. It strengthens operational resilience, enables custom and proprietary insights, and empowers cross-functional collaboration between business and IT.
Critical operations require resilient systems
The Blue Yonder incident proved that even mature third-party platforms aren’t immune to outages or cyberattacks. When a third-party forecasting system is down, there’s nothing to do but sit and wait for service to be restored, regardless of how critical that system is to your operations.
Relying solely on a third-party forecasting system is like depending entirely on the city’s power grid. When the power goes out, you’re in the dark — literally. But if you’ve invested in a backup generator, the lights stay on and operations keep running.
When weighing these options, it’s important to consider the criticality of the operation. Most individual households don’t have generators, but hospitals do because they can’t afford downtime. It might be inconvenient for the lights to go out at home or for the food in your fridge to spoil, but it’s catastrophic when life-saving equipment stops working in a healthcare facility. The same principle applies to forecasting. When a system directly affects your customers, your supply chain, or your revenue, resilience isn’t optional — it’s essential.
While third-party tools can be useful in lower-risk scenarios, the core forecasting that drives supply chain decisions and shapes the customer experience carries too much strategic weight to outsource and is best served by an in-house solution.
“By bringing forecasting in-house and hosting it through a modern technical architecture, you’re not just building a tool, you’re building resilience.”
Proprietary insights drive better results
Most off-the-shelf forecasting solutions are designed for broad applicability, not for the unique dynamics of your business. You know the intricacies of your business better than anyone. Even companies in commoditized markets have their own “secret sauce” – a mix of customer preferences, local knowledge, and strategic nuance that drives performance.
Take grocery retailers, for example. Even in the same metro area, different chains operate with vastly different approaches. One might lean into organic produce, another might specialize in private-labels, and a third may thrive during game-day weekends or local events (such as the NFL Draft). These differences aren’t captured by a generic model trained for mass-market forecasting.
With in-house forecasting, you can tailor models to the factors that matter to you – store-specific sales patterns, regional holidays, weather conditions, promotional timing, and operational constraints. These inputs make your models smarter and your decisions sharper.
Owning the forecasting pipeline also means tighter integration with your internal systems – ERP, sales, inventory, and logistics. This enables richer feature sets, better data flow, and ultimately more accurate forecasts that reflect how your business actually runs.
Empowering teams and streamlining costs
In-house forecasting breaks down silos and fosters true cross-functional ownership. Business and IT teams collaborate to design, test, and evolve the solution together. That’s a different approach from a dynamic where IT maintains a connection to a third-party “black box” that feeds data to downstream systems.
When forecasting lives inside your walls, business teams naturally get closer to the data, and IT gains a deeper understanding of operational drivers and strategic priorities. That synergy unlocks faster iteration – model updates happen in days, not weeks – and teams can respond to shifts like promotions or local demand spikes in real time.
Instead of logging a vendor support ticket and waiting, or hoping that the generic system eventually catches a trend, your teams are in control. You can respond to real-time signals like sales surges or weather disruptions and fine-tune forecasts with context that only your team understands. This leads to faster decisions, fewer delays, and more accurate actions where they matter most.
There’s also a meaningful financial upside. A typical third-party forecasting platform can cost $200,000 to $500,000 in yearly licensing alone before factoring in integrations, training, vendor support, or data access fees. These tools often charge by user count or usage tiers, making long-term scaling expensive, with total costs easily exceeding $1 million per year.
By contrast, once in-house models are deployed, they incur only incremental infrastructure and staffing costs that are often absorbed by your existing data and IT teams. You control the roadmap, avoid ballooning vendor fees, and reinvest in capabilities that compound your competitive advantage.
“For many organizations, the ROI goes beyond improved forecasting – it delivers real money savings while reducing risk and growing internal expertise.”
Owned forecasting is easier than you think
Building or improving your forecasting capability doesn’t have to be a massive, multi-year project. In fact, it often takes less time and delivers more value than trying to force-fit a third-party tool into your business. While these tools can appear turnkey at first, the reality is often messy and complex.
Experience across our team has shown that third-party implementations are rarely as fast or seamless as they are promised to be. Time is frequently lost in the hidden complexity of configuring integrations, making vendor workflows compatible with internal processes, training users, and retrofitting features to fit your needs. Add vendor timelines and one-size-fits-all assumptions to the mix, and the process slows even further.
By contrast, an in-house forecasting solution:
- Protects your proprietary insights.
- Capitalizes on what makes your business unique.
- Strengthens operational resilience and reduces risk.
- Supports agility through tight feedback loops.
- Reduces long-term delivery cost.
Start small. Build a pilot model, focus on a key product line or region, and scale from there. The key is to own the process, so you can adapt quickly, improve continuously, and align your forecasting with your strategy.
Critical operations require resilient forecasting
The Blue Yonder outage proved that even mature third-party platforms aren’t immune to failure. When forecasting drives your supply chain, downtime isn’t just inconvenient, it’s a risk.
Bringing forecasting in-house with Dataiku, Snowflake, and modern SQL pipelines builds control, resilience, and agility into your operations.
When you create a scenario, you have the option of a Sequence of steps or Custom Python script. We will be focusing on the Sequence of steps option, but be aware that (like in many other areas of Dataiku) a scenario can be defined entirely with Python code, allowing for maximum flexibility. Of course, this flexibility comes with a tradeoff of being a bit more difficult to maintain as it would not come with a visual user interface.
Let’s select Sequence of steps and give the scenario a name: “Process Transaction Data”. Similar to when a new project is created, Dataiku will auto-generate a “Scenario ID” based on the name we’ve provided, which will be a unique identifier for this scenario within our project.
Configuring the scenario
Generally, the creation of a scenario consists of 4 parts:
- Defining the scenario steps. This includes all of the functionality that we’d like to automate as part of this scenario definition.
- Setting up the scenario triggers. These are all of the events that would trigger the scenario to start running, such as a specific schedule or the changing of a dataset.
- Configuring the scenario reporters. Reporters are used to send out notifications either before or after a scenario has run. The reporter might send out error messages or a summary of the data processed within the run.
- Enabling the scenario to run automatically. Even after configuring triggers on a scenario, it’s still necessary to enable that scenario to allow automated execution within an environment. You might, for example, define and test a scenario on your Dataiku design node but leave that scenario disabled. Once you publish your project to an automation node in a production environment, you might then enable the scenario to run on a schedule in that environment.
For this example, we’re going to get started in the “Steps” tab to define the actions we want to take in our scenario. This allows us to define any number of steps that will execute sequentially as the actions in our scenario. Steps are added to a scenario by clicking the Add Step button in the lower left-hand corner of the screen. In the pop-up list, you’ll notice that there are many step options. Listed below are a few of the most common steps that can be added to your scenario. You’ll find a more detailed description of these steps in the Dataiku documentation.
- Build / Train: Builds either a model, dataset, folder, or evaluation store. Specific to the dataset build, you can configure whether the step should build a single dataset or recursively build upstream datasets.
- Clear: Clears the underlying data out of a dataset. This is a very nice option for saving on storage space if you have a number of “intermediate” datasets that you can purge after a scenario execution.
- Run checks / Runs all defined checks on a specific dataset: If any of the checks fail, this scenario step will also fail by default.
- Execute SQL: Executes arbitrary SQL statements, allowing you to – for example – query or update a table as part of the scenario execution.
- Execute Python Code: As in so many places in Dataiku, Python code can be embedded in a scenario step for execution. This allows for great flexibility as you could perform nearly any operation using the Dataiku API from a Python step.
- Define Scenario Variables / Set Project Variables / Set DSS Global Variables: These steps are all very useful for the purpose of checking and modifying variables for use in your project execution. These variables can be used for tasks such as recording the last processed record (think incremental processing) or storing computed values.
- Run another Scenario: This can be a very useful step to trigger the run of another scenario at any specific point within your scenario execution. This scenario could be one in the same project, or even from another project that you have access to.
Creating a simple scenario
For our example, we’re going to be creating a relatively simple scenario that performs the following steps:
- Build the project’s flow up to the split recipe, which will populate the data for the transactions_known and transactions_unknown datasets which are used in the next steps to train a machine learning model.
- Re-train our model based on this newly processed data.
- Re-score the transactions_unknown dataset by building the score dataset.
Step 1: Build/Train
Let’s get started by adding a “Build/Train” step to our scenario. As you’ll see in the figure below, this is accomplished by selecting the Add Dataset to Build option and choosing our dataset. In our example project, I’m going to select the transactions_known dataset, which is the input to our prediction model.
You’ll notice that it is possible to add more than one dataset to build in this step. For example, if you had multiple flow paths that you would like to build, you could add datasets from each, and (if possible) Dataiku will build them in parallel (as compared to serial/sequential).
This is potentially much more efficient than adding the additional dataset builds as subsequent scenario steps since the scenario steps are executed sequentially (top steps would need to finish before the next step will start).
Another very important topic to understand in a build scenario step is the “Build mode” as highlighted in the figure below.
The build mode has 4 options, each producing very different results:
- Build only this dataset. As its name implies, this option will build only the selected dataset, without affecting the rest of the flow.
- Build required datasets. This is a very powerful yet tricky option, which is similar to the “smart recursion” feature in DSS flow builds. With this option, Dataiku will determine which upstream datasets it thinks are necessary to rebuild and then rebuild all recipes/datasets between the selected set and any upstream sets requiring rebuild. This is typically a great option but requires significant testing to verify that Dataiku will compute the upstream requirements in the way you hope.
- Force-rebuild dataset and dependencies. With this option, Dataiku will rebuild all upstream datasets. If you know that you do want to rebuild everything upstream from a target dataset, this is likely the best option since it takes any ambiguity out of what the scenario step is expected to perform.
- Build missing dependent datasets then this one. This is a much less common option which allows for the build of any missing data (only) prior to re-building of the selected dataset.
For our example, let’s set this build mode to “Force-rebuild” which will cause the step to create a job which will rebuild all of the upstream datasets as shown in the figure below.
The next step that we will take in our scenario is to add a step to re-train our machine-learning model with the new data we’ve just processed in the previous step.
Step 2: Retrain
Now that we’ve created a step to rebuild a portion of our project flow, let’s create a second step to re-train the model deployed to the flow on this new data. This step will be very similar to the previous “build” step. To get started, we’ll choose the same Add Step => Build / Train option as we selected previously. In this new step, rather than selecting the Add Dataset to Build button, we’re going to choose the Add Model to Build option. You’ll see that this selection will open a prompt to allow us to select our previously deployed model.
In the left-hand pane, you’ll see that, like a preparation recipe, this new step appears below the first step we created. Recall the steps in the scenario are executed sequentially from top to bottom. With the following step configuration options, it’s also possible to specify how to handle a failure at any step:
For example, we may want to execute certain steps with every scenario execution, regardless of whether a previous step had failed.
Step 3: Scoring data
With the initial build and retrain steps complete, the last step we’re going to add to our scenario is a final build step which will use our freshly retrained model to score an unknown dataset. As in the previous steps, this is done with the Add Step => Build / Train option. In this case, we’ll be adding the transactions_unknown_scored dataset as the target of the build.
Triggers
Now that we’ve set up the steps that define the actions that will be performed by our scenario, let’s jump back to the “Settings” tab where we’ll find the Triggers and Reporter settings. Triggers are set up on a scenario to define when the steps will start, which could be based on a variety of events. The most common types of scenario triggers are:
- Time-based. A very common request is to run a scenario on a time schedule such as nightly, weekly, or monthly. With a time-based trigger, this is easy to define – and it’s even possible to set up more complex schedules such as “the 2nd Wednesday of every month”.
- Trigger on SQL query change. This is a trigger type that works for SQL-based datasets and functions by running a query on that dataset and activating when the output of the query changes. A common use case of this trigger type might be “SELECT max(date) from XXX” if you wanted to trigger a scenario to run if the maximum date in a table had been updated.
- Trigger after scenario. The two previous trigger types are kicked off based on an external event, but this scenario is different in that it is started when another scenario has completed. This “other” scenario could be within the same project, or from another project that you have access to – making it a very flexible way to chain the build of project flows.
- Custom trigger. Similar to the “Custom Python Script” option for creating scenarios we defined above, the custom trigger option allows us to define triggering events using Python which allow for a great deal of flexibility.
In our demo project, we’re going to define a time-based trigger to kick off our scenario steps execution nightly at midnight as shown in the screenshot below. You’ll also notice that a trigger can easily be disabled so the configuration can be saved but not enabled for execution.
For our demonstration, we’ve added a single trigger, but it’s feasible to add multiple triggers to any scenario which could define multiple events to kick-off execution.
Enabling the trigger
Although we have configured and enabled a scheduled trigger on our scenario, there’s one additional step that must be taken to allow the trigger to execute. In the topmost section of this “Settings” tab, you’ll see an Auto-triggers toggle under the “Run” header. By enabling this toggle, the scenario triggers will be executed in the Dataiku environment that is hosting the Dataiku project.
Reporters
To this point, we’ve covered the topics of scenario steps and triggers which are the majority of what we’d typically need to set up our Dataiku flow for automated execution. The last element that we’re going to cover is focused on delivering and communicating the results of these scenario runs. One of the key components to the success of data science projects is the ability to monitor status and receive notification of failures – and that’s the role of scenario reporters.
Adding a reporter
Adding a reporter is done in a very similar manner to the trigger that we added in the previous section. In the case of the reporter, we have a number of options for the delivery channel including Mail, Slack, Microsoft Teams, and Twilio. In order to use these channels, it’s necessary to have an administrator set them up for your organization.
The technical configuration of these notification channels is beyond the scope of this article, but you can see the reporter options by selecting the Add Reporter button as shown in the screenshot below. More information on each of the reporter types can be found in the Dataiku documentation.
Generally, after adding a reporter to your scenario, there are some common options for any of the reporter types:
- Send on scenario: Specifies whether the reporter will fire before or after the scenario steps execute. “End” is the most common, but it can also be very handy to choose “Start” to notify when a scenario execution has kicked off.
- Run Condition: When toggled on, the run condition determines whether the reporter should conditionally send results. In the screenshot above, you’ll see the “outcome != ‘SUCCESS’” condition which indicates that the reporter will only send a message if the scenario encountered an error.
- Channel: In order for a reporter to function, a communication channel must be specified, which could be email, messenger, or a more custom output. This channel needs to be set up at a Dataiku instance level by an administrator.
The remaining configuration options are channel-specific, but generally specify the messaging content and formatting that will be received when the reporter executes.
Easy automation with Dataiku Scenarios
Consisting of steps, triggers, and reporters, scenarios make it easy to automate end-to-end data workflows and pipelines within Dataiku projects. As you’re looking to move your Dataiku projects from development into production, scenarios are an essential component that allows for scheduled execution and complex orchestration in your team’s environment.
About the author
With over 20 years of experience as a Lead Software Architect, Principal Data Scientist, and technical author, Ryan Moore is the SVP of AI and Analytics at Amplifi, and our resident Dataiku Neuron. He provides Data Science architecture and implementation consultation to organisations around the globe.
