Businesses must continually innovate at speed and at scale to improve their market share and profitability.

In today’s world, much of this innovation is driven by information technology. Consequently, IT is no longer regarded as just a back-office department like accounting, but rather as a primary driver of growth.

This shift naturally implies more accountability on the part of IT, specifically with software development. As a result, the software development industry must excel in innovation in order to keep pace with the demands and expectations of companies!

Of the 10 largest public companies, all ten owe the vast majority of their revenue streams to software-related services, be they integral to the company’s core product - as is the case with Apple, or used to operate their vast, complex network of loans and ledgers in the case of  J.P Morgan Chase.

Not one of these companies would be able to function on such a scale without its current level of IT and software infrastructure, while very few would last more than a decade without updating it.

However, software development needs to contend with several key challenges, e.g., developers might not use coding best practices, thus exposing the application to security risks, etc.

A vast majority of these issues relate to how software development is managed. As projects become increasingly complex, so does the management process. If a start-up is to one day develop a product that is as complex as Amazon’s shopping site from scratch, then a new solution is needed.

Machine learning (ML) is emerging as a key solution to several of these development challenges.

In this article, we will take a look to see how it can revolutionize software development. Let’s start with the challenges that software development faces today.

Software development challenges

Software development project failures are common. One 2017 study found that only 34% of projects are completed on time and only 42% on budget. The key challenges that impact software development include:

• Sub-optimal infrastructure;
• Fluid and/or inadequately defined requirements;
• Non-compliance with coding best practices;
• Insufficient test planning and low test coverage;
• Using outdated technologies;
• A growing complexity with many stakeholders, integration challenges, and conflicting priorities;
• Vulnerability to security risks.

As a result, businesses are increasingly turning to cloud computing, improved project management techniques, better tools and frameworks, and innovative technologies to address these challenges. Machine learning is at the forefront of much of the current innovation.

A brief introduction to Machine learning (ML)

Machine learning (ML) is an advanced computing technology that uses Artificial Intelligence (AI) to create computer systems that can learn and improve independently of much in the way of human input. It requires large amounts of data (the more data the more accurately the system can learn), and computer programs that can “train” the system from the data.

Machine learning algorithms are able to identify patterns in the data to formulate predictions and make decisions based on them. These algorithms are designed to make the system learn automatically without human intervention.

They are several types:

• Supervised ML algorithms: These can apply past learning patterns on new data and make decisions based on labeled examples.
• Unsupervised ML algorithms: Here, the data isn’t labeled, and the system will need to find and describe hidden structures from the data.
• Semi-supervised ML algorithms: These algorithms use both labeled and unlabeled data.
• Reinforcement ML algorithms: In this case, the system produces inferences, discovers errors or rewards, and adjust its inferences based on what it has learned.

How ML can make a difference in software development

There are various ways in which ML can change the way we develop software.

1. Identifying deviations from coding guidelines

Deviations from coding guidelines are common occurrences, and they can have a large impact.

The Open Web Application Security Project (OWASP) Top 10 Application Security Risks – 2017 report identified several application security risks that are a direct result of developers not following coding guidelines. Injection, XML external entities (XXE), and cross-site scripting (XSS) are examples of security vulnerabilities that arose from deviations from the best coding practices.

Modern software development projects are complex, and it’s often incredibly difficult as well as time-consuming to identify such deviations from coding guidelines manually. However, ML software can automatically analyze code and identify deviations.

Once identified, experienced code reviewers can then review these deviations in more detail and make changes. As their experience and sophistication develop over time, these ML models will only get better and better at finding such deviations, thereby reducing the need for human input even further.  

2. Getting actionable intelligence from code

Large enterprises often have many software development projects running simultaneously. Some of these projects are aimed at being transformational projects in terms of how the company operates. Just consider how many companies are now scrambling to launch some kind of blockchain solution and you will get a clear picture of what I mean by transformational projects.

With such high stakes, enterprises need to know how these projects are progressing. After all, not only will these projects spur new growth, but will also impact their core business requirements in terms of training, which tools and frameworks are important for them, and what kind of people/skills they will need hire to in the future, etc.

ML enables enterprises to learn the answers to these questions by extracting actionable intelligence from code.

An example of one such an ML platform that is already available is source{d}.  This platform lets you gather intelligence from your code on Git or other version control platforms and integrate it with your existing business intelligence (BI) tools.

3. Improving project management using ML

Project managers in charge of modern software development projects need to focus a great deal on the strategic aspects of the product. However, mundane operational activities place a heavy demand on their time and therefore act as a distraction.

ML can transform project management in various ways, e.g.:

• ML can help PMs to forecast cost and revenue based on historical data.
• It’s possible to use ML in conjunction with an organizational information repository and other external information to identify risks.
• Many PMs utilize tools like mind maps to express complex interplays between entities, relationships, and constraints. These can be very useful for complex projects. ML algorithms can use mind maps and organizational knowledge repositories to create network diagrams and work breakdown structures (WBSs).
• High-quality documentation is crucial with complex software projects, however, reviewing the availability and quality of documents is costly as this requires highly experienced reviewers. ML algorithms can inspect the documentation and identify whether key documents exist, moreover, they can give insight into the quality of such documentation.

Such overwhelming advantages is why we at DevTeam.Space use data-driven processes, which include AI and ML-powered real-time dashboards. These processes help us to proactively monitor the progress of all of our projects simultaneously, something which helps us optimize project and team management.

By empowering client-side project managers by giving them access to this data, we have experienced a dramatic reduction in roadblocks such as problems integrating post plan features, etc. Machine learning-driven approaches such as this are unquestionably going to become the mainstay of future software development.

4. ML in coding and testing

There is significant manual effort required when coding. This is both time-consuming and very costly. ML is becoming an increasingly important way to reduce this. The following examples illustrate how:

• Stack Overflow Autocomplete: This is an ML-powered tool for JavaScript coding that uses Stack Overflow data to create code. The tool creates new code by inferring the functionalities of the existing code.
• DeepCode: This AI and ML-powered tool uses data from GitHub repositories, and analyses code to find bugs and security vulnerabilities. Its ML algorithms can establish the intent of each specific element of code in addition to identifying syntax errors.

ML can make a difference in testing too, e.g.:

• It can accelerate manual testing, e.g., the sorting of log files.
• As more functionalities are added to an application, ML algorithms used for creating test cases evolve and add test cases for the new functionalities. As we know from the recent Boeing tragedy that resulted from software failure, it’s crucial to create and execute test cases for every new feature.
• ML-powered testing will also be better at finding bugs since test cases created by ML algorithms will focus on the intent of the code.

The future of ML-powered software development

By its very nature, software involves a significant degree of abstraction. As a result, software development will always involve the active participation of human beings. It is unlikely that ML and AI will not take over software development completely. However, in the near future, these technologies will take over several tasks and will serve to make them much easier for humans to accomplish.  

In this article, we have looked at how ML can find deviations from coding standards, improve coding and testing, create actionable intelligence from code, and help PMs to manage software development projects better.

In the short-term, the use of ML in software development will likely focus on finding deviations from coding standards and identifying bugs. As the ML algorithms are trained with more data, the technology will likely add significant value in managing complex software development projects in the future too.

The sky is truly the limit when it comes to machine learning in software development in the longer term. It is likely that even in an industry that is heralded as the solution to job layoffs due to factory automation, that we begin to hear grumbles from out of work code reviewers and developers sometime in the not too far off future on account of machine learning solutions.

Aran Davies

Writer for DevTeam.Space