Innovation, project vs product culture in Data Science

Data Science carries the jobs of tomorrow. It is closely linked to the understanding of the business usecases, the behaviors and the insights that will be extracted from existing data. The stakes are both human and technological. They integrate innovation, the understanding of business needs and the need to get into production.

Science and Engineering

Data Scientists must find an unusual balance between the discovery of new knowledges (science) and its robust understanding (engineering). They must be simultaneously innovative and insightful while providing reproducible results and production quality codes. However, the research and development work methods required for exploration are often antithetical to the way in which production code is constructed.

As such, the methodologies developed and deployed by Bell Labs allowed their teams to maintain the leadership of global innovation for several decades. Innovation is a process that can be promoted. The book “The Idea Factory: Bell Labs and the Great Age of American Innovation” written by Jon Gertner retraces this adventure.

John Pierce, Bell Labs engineer and writer, defines 4 pillars on which innovation is built. Firstly, the management must be technically competent, preferably with an academical and engineering experience. Secondly, the work should not be conditioned by budget allocation in order to avoid too much financial pressure and short-term objectives. Thirdly, the work must be long-term, ie several years, in order to bring all the benefits of research and deliver reliable and scalable solutions to society. Fourthly, stopping a project should not lead to a sense of failure but to a valorization of the undertaken efforts and the acquired knowledge.

Culture project vs product culture

The majority of the teams we work with are basing their developments on Agile methodology, which is a very good thing! Building the product incrementally decreases the technical risks and share the progress of developments.

However, the execution of a strategy (with its famous “Roadmap”) defined at the beginning of the project must be continually challenged. Teams are more focused on the features to be developed than on the impact of these features. Although using Agile development, the culture remains a project culture, with its limits.

In addition, team organization is too often composed of a mixture of V-model and Scrum methods, the consequence of which is a disproportionate ratio of manager versus developers with an overflow of tasks layered on top of each others and with unclear objectives.

Finally, a brutal delivery process toward teams without functional and sometimes technical knowledge do not involve the teams in the construction of a product with complex and ambitious evolutions.

A product oriented culture, conversely, invites to experience another approach: experiment and learn about the strategy as you go. The method validate step by step (and it’s not so simple) that what we want to develop is what we need to develop.

The development and support of services use almost permanent teams as long as the service is operational. The budget may vary from year to year, but it is usually sufficient to permanently fund a sustainable development organization for the life of the service. Teams are funded to work on a particular problem or solution over a period of time; the nature of the work is defined by the stakes of the company rather than by the provision of functionalities.

In product mode, no project is subcontracted to the suppliers or awarded to teams dedicated to the operation. Instead, providers and operators may be asked to provide a stable, long-term, ideal team with key roles endued with internal talents.

The product must be managed throughout its life by the same team which does not prevent its constitution to evolve according to the service demands and imperatives.

Conclusion

Creation requirements must be maximized and aligned with the Data Lake information. Data Scientists must strike a balance between innovation and the necessary production requirements of their ideas.

Several models are possible. Some are quickly applicable and others, even if already in production in the most innovative companies, are in advance of phase. The strategy of each company determines its positioning: conservative with the risk of missing out on the “Next Big Thing” but with immediate; and/or innovative solutions with the risk of investing in a technology without future but with the opportunity to acquire a comparative advantage over the competition.