Summary

Discover how TVNZ transformed their experimentation process, scaling from one experiment per month to several continuous A/B tests with Split feature management and experimentation (acquired by Harness).

• Accelerated experimentation: TVNZ scaled their experimentation efforts, running multiple tests per day, improving their speed and efficiency along the way.
• Enhanced data-driven decisions: With Split’s real-time results and automatic statistical calculations, TVNZ gained granular insights into feature performance, so they could make smarter product decisions without hesitation.
• Improved feature rollouts: By using feature flags, TVNZ minimized risk by easily toggling features on/off and enabling targeted experiments for customer feedback.
• Streamlined data integration: Split's seamless integration with TVNZ's existing data pipelines and Google Analytics helped save engineering time and improved data tracking.

About TVNZ

Television New Zealand (TVNZ) is New Zealand’s state-owned, commercially funded broadcaster. Their media reach includes TVNZ 1, 2, and DUKE television channels, along with the TVNZ OnDemand streaming platform, connecting with millions of New Zealanders daily. As the digital landscape expanded, TVNZ sought a solution to enhance their product experimentation capabilities, scale innovation, and reduce the long-term maintenance costs of their in-house systems.

The Challenge

Slow, manual experimentation processes

TVNZ faced significant challenges with their in-house experimentation system, which required DevOps teams to manage feature flags and manual steps for A/B testing. These processes were slow, limiting their ability to iterate quickly on customer-facing features. Their goal was to move from running one experiment per month to several ongoing experiments, using a more automated and scalable system that would also provide higher statistical rigor.

The Solution

Faster, more scalable experimentation

TVNZ chose Split to build a reliable feature flag foundation, improve A/B testing, and automate their experimentation processes. With Split, the team could now easily toggle features on and off at the press of a button, reducing the risk of negative customer impacts.

• Streamlined experimentation: Split automated the A/B testing process, ensuring proper randomization and statistical calculations.
• Real-time insights: Product and engineering teams could view experiment results in real time, enabling faster decisions on feature rollouts or rollbacks.
• Dynamic targeting and experimentation: TVNZ used Split to segment users dynamically for personalized experiences, and adjust sample sizes and variations in real-time.

Nathan Wichmann, Product Manager at TVNZ OnDemand, highlights how Split allowed the team to move much faster in their experimentation and make data-driven decisions: “With Split, we’ve gone from running one experiment every few months to having so many experiments that we need to think carefully about sequencing and prioritization.”

Results

Scaling experimentation and innovation
The integration of Split into TVNZ’s development process resulted in measurable improvements:

• Faster experimentation: TVNZ transitioned from infrequent to continuous experimentation, dramatically speeding up their feedback loop.
• Increased experimentation cadence: The ease of creating and running experiments helped the team run multiple tests concurrently, achieving higher velocity in decision-making.
• Enhanced decision-making: With Split's real-time results and integration with their data pipeline, TVNZ could evaluate the impact of new features and quickly pivot when necessary.
• Feature debt reduction: TVNZ could safely revisit and re-assess old features, using Split to experiment with their removal or improvement.
• Business impact: The experiments helped the team measure and track product performance, directly linking innovation to business success.

“We use feature flags and run experiments a lot more because it’s easier now with Split. We can target a random sample size. We can increment the size of that sample really easily, and we can toggle it off immediately if there’s an issue.”— Mel De Soysa, Business Analyst, TVNZ

Future

Building a culture of experimentation
As TVNZ continues to scale their product and innovate, Split will be integral to maintaining their culture of experimentation. The team is able to test, measure, and refine product features while ensuring they align with key business metrics.

Nathan Wichmann explains, “Split has been really useful for us. We’ve been able to measure product impact and show that the ideas we’ve worked on have produced tangible business results.” TVNZ’s ongoing use of Split empowers them to continue evolving and making informed decisions for their products and customers.