Eileen, in my experience (20 years in system i CM) a lot of our customers are already adopting Agile techniques without a pilot. Where the pilot comes into play is when the existing tools are a limitation and new tools must be adopted. This is also a good time to look at the current overall workflow to take advantage of appropriate tools. As the other responders have pointed out, the goals of the transformation should be identified as early as possible so the pilot is accurate.
From my perspective, coming from a traditional IBM i centric Change Management background, the pessimistic change management tools for native i source that most companies are using now are limiting for concurrent development. I will include our own CM tool in that group. It's hard to be responsive when you can not easily manage concurrent development.
Our toolset was integrated with a true source control tool (Rational Team Concert) over 6 years ago and now integrates with the current commonly used tools, Git, Jenkins, JIRA, even TFS. So when we do a pilot we are handling the automation side of things to support the desired agile workflow out to deployment/testing of artifacts. Our metric for success is that we have a consistent and successful process with the desired outcomes.
However that is a functional pilot (as is my normal focus) and as other responders have pointed out there is also a cultural shift involved that must be 'piloted' as well.
I have successfully implemented a functional workflow just as the customer management requested only to have the development staff so resistant to their changed role and functions that it was rejected from the ground up during training or the bare minimum was done by the developers on an ongoing basis making the reporting side of things nonfunctional.
So I'll say 2 things.
First, you must have development staff involved with the design of the target agile workflow (and not just the early adopters) and there must be some positive results for the developers (better tools, increased automation which streamlines their interventions) so they buy in.
Second, you must also look at a pilot as an opportunity to refine the agile workflow, such that the first pilot might be perceived to fail so the second pilot can be successful. This means you can NOT live and die by target dates.
Our first customer for true source control is a good example of this. After over a year of using RTC for their native source they were getting frustrated with the initial branch (called a stream in RTC) design and we went back and spent a week re-implementing their branch structure and developer workflow and training developers. This was a painful decision but the outcome was positive.
Thanks Jeff Tickner
CTO North America
Arcad Software