Introducing ForecastFlow
A dynamic and probabilistic approach to estimating effort in Software Development
ForecastFlow is a framework that uses employee prediction markets for predicting software releases. It offers a dynamic and probabilistic approach to forecasting, rather than relying on the static estimates of story points and velocities. The system leverages the collective intelligence of the team and provides a market-based mechanism for more accurate, real-time, and evolving predictions.
1. Core Principles
Crowdsourced Intelligence: Employees from various roles (developers, QAs, product managers, etc.) contribute to predictions based on their knowledge, experience, and insight into the work at hand.
Market Dynamics: Team members participate in a prediction market where they "trade" estimates of software release dates based on their understanding of the project.
Real-Time Adjustments: Unlike static story points, prediction markets allow for dynamic adjustments as new information (e.g., progress, roadblocks, priority shifts) becomes available.
2. Market Setup
Participants:
Everyone involved in the project participates, including developers, QAs, product managers, designers, and even stakeholders with domain knowledge.
Each participant holds a virtual "currency" to trade in the prediction market.
Market Instrument:
Release Date Contracts: These are digital contracts predicting when a software release will be completed. Participants can buy or sell shares of these contracts based on their belief in the probability of the release occurring by a certain date.
Example: A contract for “Release on Dec 1st” could be bought at a price representing the collective belief (e.g., 70% chance).
As more information becomes available, participants trade these contracts, driving the price up or down.
Initial Market Setup:
A set of potential release dates is created for each software project.
Participants "bet" virtual currency on specific release dates or ranges of dates (e.g., end of Sprint 4).
Prices reflect the likelihood of those dates happening: higher prices indicate higher confidence, lower prices indicate uncertainty or doubt.
3. Trading Mechanism
Market Updates:
Participants can continuously update their estimates based on:
Changes in team velocity
Emergence of blockers or delays
New features being added
Progress reports from various teams
System or infrastructure challenges
Dependencies on other projects or external factors
As conditions evolve (e.g., scope changes or bugs), market prices will fluctuate to reflect the new expectations.
Market Incentives:
Participants who correctly predict the release date receive rewards (virtual currency or other incentives).
Mis-pricing or poor estimates result in "losses," driving participants to provide more accurate forecasts.
Prediction Accuracy Mechanism:
The system tracks individual participants' prediction accuracy over time, incentivising contributions from those with a strong track record of accurate forecasting.
Feedback loops ensure that repeated participation enhances market intelligence, making future predictions more precise.
4. Replacing Story Points and Velocity
No Need for Story Points:
Story points are static estimates that do not account for uncertainty, complexity, and real-time changes.
In the prediction market, the uncertainty of each task is reflected in the price of the contract predicting release dates. High uncertainty tasks will drive prices down, effectively acting as dynamic risk weighting.
Instead of assigning arbitrary story points, teams “trade” based on the complexity and uncertainty they foresee, which offers more fluid and nuanced feedback.
Velocity Becomes Redundant:
Velocity, a retrospective measure of productivity, becomes less relevant when the prediction market provides a real-time reflection of the probability of achieving a release by a certain date.
This system adapts to the complexities of software development—fluctuating scope, technical debt, external dependencies—better than velocity-based projections.
5. System Integration
Tooling:
Integrate the prediction market into you project management tooling (e.g., Jira, GitLab, Trello).
The market dashboard can show the current "market price" for each possible release date, helping stakeholders see the expected timeline at a glance.
Historical data from past projects can also be used to adjust the initial market setup for new projects.
Real-Time Insights:
Since prediction markets are dynamic, the market price provides real-time insight into the project’s progress, the impact of new features or delays, and the likelihood of meeting release deadlines.
Automated Signals:
Alerts can be set to notify teams when there’s a significant drop in the likelihood of an on-time release (e.g., when a contract price for the next sprint drops below 50%).
6. Psychological Safety and Anonymity
To promote openness and avoid groupthink or fear of reputational damage, the system can anonymise market participants, allowing them to trade without fear of judgment.
This ensures more honest market feedback, capturing the collective intelligence more effectively.
7. Market Closure and Evaluation
The market closes when the release date is confirmed (or missed).
The actual release date is compared against the final market predictions, and participants are rewarded based on the accuracy of their trades.
Post-release analysis provides insights into how the market responded to different factors, helping teams learn and improve for future releases.
8. Benefits Over Story Points and Velocity
Dynamic and Real-Time: Unlike static story points, which can quickly become irrelevant, prediction markets adjust to changes in real-time.
Collective Intelligence: The market aggregates the diverse perspectives of everyone involved, leading to more accurate predictions.
No Arbitrary Estimation: Story points and velocity are often subjective and detached from real progress, while prediction markets are grounded in ongoing development reality.
Incentivised Accuracy: Since participants are rewarded for correct predictions, there’s an inherent incentive to be more thoughtful and realistic in their assessments.
Conclusion
By introducing employee prediction markets to forecast software releases, teams gain a flexible, dynamic, and probabilistic approach that surpasses traditional estimation methods like story points and velocities. This system not only makes predictions more accurate but also drives engagement and incentivises team members to contribute their best insights. Over time, it can lead to better project planning, improved delivery timelines, and enhanced adaptability to changing circumstances.
