Global AI and Data Science

 View Only

Call for Code Spot Challenge for CodeNet

By Susan Malaika posted Mon August 16, 2021 03:51 PM


The contest is over - Thank you for your contributions

Many Congratulations: Team Abiret

See Linked In Post

As preparation for competing in the challenge we ran these crowdcast sessions:


Increasingly software plays a role in every aspect of our lives, from managing Covid-19 and our finances, to interacting with other humans in education, work, and entertainment. There are 
critical shortages of software developers and experts across the world. With large numbers of services and apps in popular programming languages and over 200 billion lines of mission critical legacy COBOL code, having methods to analyze, develop, and maintain software in more automated ways is essential for progress. As a step along the way to automation, join the challenge to find programs that perform similar tasks. The  team that makes the best prediction wins. The challenge is based on an extract from
Project CodeNet which is a large-scale dataset with approximately 14 million code samples, each of which is an intended solution to one of 4000 coding problems.

The goal of this challenge is to introduce data scientists to AI for code concepts leading to improvements in the way software is produced and maintained.

Useful Links

    Background on Call for Code

    Call for Code is a global program that invites developers and problem-solvers around the world to build and contribute to sustainable, open source technology projects that address social and humanitarian issues, while ensuring the top solutions are deployed to make a demonstrable difference. Created by David Clark Cause and supported by Founding Partner IBM, Charitable Partner United Nations Human Rights Office, and Program Affiliate The Linux Foundation, Call for Code has grown beyond a single global challenge into an always-on, tech-for-good platform with multiple ways to get involved.

    The Challenge Synopsis

    Working in collaboration with Women in Data Science, we will:

    • Provide education on the codenet dataset, code representation, and NLP (Natural Language Processing)
    • Help you form or be part of a team

    Working in collaboration with your team, you will:

    • Build your models and submit your predictions
    • Join crowdcast sessions to learn about NLP,  and code representation, network with other participants, and mentors

    We will announce the winning team in January 2022 ---(or soon after  when all checks have take place!)----

    You will find a lot more information on the leaderboard

    The Prize

    One winner at the top of the final leaderboard at the end of January 2022 (or when IBM declares the contest closed) gets 5K US Dollars.

    Registering for the Contest

    How to Win

    The winning team will ultimately depend on the outcome of the of the 'final phase' of the challenge.  The ranking of a solution depends on two criteria: the accuracy of the predictions and the submission time of the solution. The accuracy is calculated as follows: number of correct predictions/ total number of predictions. The key metric used is 'meta-score', which accounts for both accuracy and submission time. A higher accuracy score will always result in a higher meta-score and therefore a higher position on the leaderboard. For submissions with identical accuracy scores, the earlier submission will always rank higher. Potential winners have to submit a description of their method and a URL to a github repo with working code to allow verification of the eligibility of the submission. The team with the highest meta-score (as long as verified as eligible) will be the winner."

    You will find a lot more information on the leaderboard

    The Phases of the Contest

    Phase Name Start in UTC & US Eastern End in UTC & US Eastern Crowdcast
    Experimentation phase

    2021-10-19 - 23:59:00 UTC

    2021-10-19 - 19:59:00 US Eastern

    2021-11-09 - 23:59:00 UTC

    2021-11-09 - 18:59:00 US Eastern

    October 27, 2021  

    Call for Code Spot Challenge for CodeNet - Setting up your system

    Dev phase

    2021-11-09 - 23:59:00 UTC

    2021-11-09 - 18:59:00 US Eastern

    2022-01-26 - 23:59:00 UTC

    2021-01-26 - 18:59:00 US Eastern

    Final phase

    2022-01-26 - 23:59:00 UTC

    2022-01-26 - 18:59:00 US Eastern

    2022-01-28 - 23:59:00 UTC

    2022-01-28 - 18:59:00 Eastern


    The Terms and Conditions

    Please do make sure you have agreed to the Participation Agreement for the Call for Code Spot Challenge for CodeNet before you start submitting to the leaderboard. (See Registering for the Contest Section)
      • No IBMers or Red Hatters can participate to win the prize - although they can experiment and build models
      • A contestant can have one account only on the leaderboard and can be in one team only after the first phase
      • Teams can be up to 4 people, with at least half the team identifying as women or non-binary
      • Teams must be registered on this leaderboard by January 27, 2022
      • No team mergers are allowed
      • IBM can restrict the number of teams competing
      • No sharing of notebooks and models privately between teams is allowed unless you make the content available to all
      • At the end of January 2022, or when IBM declares the contest closed, the leaderboard determines the winning team
      • The top teams on the final leaderboard will be asked to share their models and scripts - and provide information on the tools they used as well as any other open datasets they incorporated

      We hope that you will join us.