Episode 40: Making campaign decisions with data
The loop of hypothesis, experiment, result and decision; understanding sample size; and common mistakes with data.
5 orders on Monday, 1 on Tuesday. Neha panicked: “Stop the ads!”
One day of data is the weather; four weeks of data is the climate. You don’t move house because of the weather.
The decision loop
Example hypothesis: “If the size chart is right below the image on the product page, conversion will rise 10%+, because fear of size is the biggest blocker (Chapter 36 data).”
A good hypothesis has: the change, the expected result and the reason.
How much data is enough?
- Small numbers vary a lot by chance. 3 vs 5 orders = maybe luck
- Look at a full cycle: weekdays/weekend, salary week, festivals (at least 1–2 weeks)
- The smaller the difference, the more data you need to prove it
- Use A/B test calculators (online) to estimate how many visitors/orders you need
Common data traps
| Trap | Example |
|---|---|
| Correlation ≠ causation | Orders rose at Diwali and a new ad also ran. Who gets the credit? |
| Cherry picking | Showing only the best days |
| Vanity focus | Views doubled, orders stayed the same |
| Survivorship | Asking only customers who stayed, not those who left |
| Attribution bias | Every platform takes credit for the sale |
Set the rules in advance
Write before the test what you’ll do for each result. Then you won’t change your mind afterwards.
| If | Then |
|---|---|
| Conversion +10% or more, for 3 weeks | Make it permanent |
| Between ±10% | No difference; keep the simpler version |
| −10% or worse | Go back to the old one |
Numbers + people = the whole story
Numbers tell you what happened. Customers tell you why.
- 5 customer calls a month (10 min)
- A list of common questions from DMs
- Words that keep coming up in reviews
When both say the same thing, the decision is solid.
StyleStreet’s real problem
Marketing in reaction mode
Neha looked at the numbers every day and changed something: the price one day, the ad another day, the caption style the next. After 3 weeks there was no way to know what had worked.
There was data, but no system.
Practical example: StyleStreet experiment log
| # | Hypothesis | Result (3 weeks) | Decision |
|---|---|---|---|
| 1 | Size chart below the image → conversion ↑ | +18% | Permanent ✅ |
| 2 | Free shipping ₹999+ → AOV ↑ | AOV ₹960 → ₹1,080 | Permanent ✅ |
| 3 | A daily story poll → DMs ↑ | +4% (noise range) | Optional; keep it simple |
| 4 | A 10% discount pop-up → conversion ↑ | +6%, but margin −₹96/order | Stopped ❌ |
Test 4 lifted sales a little, but cut profit. The decision is made on profit.
Your mission: Build an experiment system
For yourself or StyleStreet:
- Write 3 hypotheses (change + expected result + reason)
- Write each one’s main metric and duration
- Write the decision rules before the test
- Create the columns of an experiment log sheet
- Write your own examples of 3 data traps
- Write 5 customer-call questions (to understand the “why”)
Quiz
-
What should a good hypothesis contain?
- A. Just an idea
- B. The change, the expected result and the reason
- C. Just a number
- D. Nothing
Show answer
Correct answer: B. The change, the expected result and the reason
Only then does the test mean something.
-
1 order one day, 5 another. What do you do?
- A. Stop the ads immediately
- B. Look at 1–2 weeks of data for the trend
- C. Change the price
- D. Change everything
Show answer
Correct answer: B. Look at 1–2 weeks of data for the trend
Short-term data is very random.
-
Orders rose at Diwali, and a new ad was also running. Credit the ad?
- A. Yes, definitely
- B. Carefully: the season could be the reason too (correlation ≠ causation)
- C. No, never
- D. Credit Diwali
Show answer
Correct answer: B. Carefully: the season could be the reason too (correlation ≠ causation)
Happening together isn’t the same as causing.
-
A discount lifted conversion 6% but cut profit. Decision?
- A. Keep it, conversion rose
- B. Stop or change it; profit is the goal
- C. Double the discount
- D. Ignore it
Show answer
Correct answer: B. Stop or change it; profit is the goal
The business goal is profit, not just conversion.
Output for the client project
Your project file now has a Decision System:
- Hypothesis template
- Pre-set decision rules
- Experiment log (4 tests)
- Customer feedback routine
Stage 7 complete! StyleStreet now runs on data, not guesswork.
Golden rule: Don’t move house because of the weather. Write a hypothesis, set the rules first, get enough data, and decide on profit.