How can we identify the significance of a/b testing with big sample and short period?
How can we identify the significance of a/b testing with big sample and short period?
For example, we wanted to test two landing pages for approving new users' retention, gave 10000 new users to each page everyday from Monday to Sunday.
Plan A got second day retention user 3010,2890,3010,2890,3010,2950,2890 each day, with mean 2950 and standard deviation 60;
Plan B got second day retention user 2910,3090,2910,3090,2910,3090,3000 each day, with mean 3000 and standard deviation 90.
So can we say that plan B is significantly better than plan A?
If the retention rate, mean and standard deviation remain unchanged, the test period expands to 10 weeks or reduces to 2 days,does the conclusion remain unchanged?
If the retention rate, mean and standard deviation remain unchanged, the test traffic expands to 100 thousands or reduces to 100 each day,does the conclusion remain unchanged?
If the whole experiment remains unchanged, but 10000 users are divided into 10 groups, each group 1000 users, and second day retention user also divided into 10 groups, does the conclusion of the experiment remain unchanged?