I am doing a research about glass and conventional cockpits. Research question: " Is there a workload difference between cocpits? I am gonna use 30 pilots in my research. Which one is more sound statistically: 30 pilot flying same pattern on each cockpit then compare these total 60 data or 15 pilot flying same pattern on glass cockpit and 15 pilot flying same pattern on conventional cockpit then compare these total 30 data?

Both are possible - one would be independent samples and the other would be paired samples, but the big issue with the first design would be how flying one cockpit might affect performance on the second cockpit. At the very least, you would need to randomize the order.

The paired test would be statistically more efficient, but the order effect might rule that out.

Thanks for your reply Jon. The cocpits will be different but flight pattern will be same. You think this might affect pilot performance on the second cocpit! Can you suggest a way to mitigate the order effect? Or any readings that might help to me understand this effect?

I would definitely be worried about the sequence effect, but only someone who knows the domain can say for sure. The safest thing to do would be to sacrifice a little statistical power - it would probably not be much if the pilots are randomly assigned to a cockpit type. If you do go with the two-cockpit version, be sure to randomize the order.

Consider, also, that a pilot might do one test but then never do the second one. With the paired model, you lose that first set of data. In the independent samples model, you could still use the one you have.

Jon I really appreciate your help. I am not good at statistics. I am a pilot. according to me which one i fly first it wont change my performance on the second. Let me put the question other way. There is one model of car with two types of speedometer: one with analog other with digtal type. I'll ask the drivers to keep the constant speed while driving on straight and bending roads (without outside visuals). do you think will first test effect the performance of drivers on the second test?

My other problem is if i use independent samples there will be 15 pilot on each type of cocpit. will 15 be good enough to make statistacal analysis?

If you were me which aproach do you choose for a SCI paper?

With the car analogy, I might think that having driven the route once, I would be more familiar with it on the second run, and that would influence my performance. I can't say whether that is a factor for your pilot study.

In preparing the design, you could do a comparative power analysis of the two possibilities. SPSS Statistics has a power calculator (new in V27). If your measure is a mean, say, you would need to know about what the average and standard deviation are. If it is a proportion, you would need an idea of what the proportion would be. In both cases, you would plug in the sample sizes and the effect sizes or the desired power, and it would tell you the differences between the paired and independent sample designs.

This answer looks legit.

"Both are possible - one would be independent samples and the other would be paired samples, but the big issue with the first design would be how flying one cockpit might affect performance on the second cockpit. At the very leastyt1s, you would need to randomize the order.

The paired test would be statistically more efficient, but the order effect might rule that out."

Dear Billy,

Thx for your answer. Is there any way to rule out the order effect other than randomization of order? Can you offer me some readings?

Other question:​ If I flown the pilots on both cockpits (half of the pilots will fly firstly with glass then analog cockpit and the other half will firstly fly with analog then glass cockpit) and compare results with both tests (paired and unpaired: all the data will be used in paired test, first flights' data used in unpaired test) does it make sense?