For an optimal functioning of any system, we often need a mixture of two opposing qualities.
We need to be data driven, but we need to think out of the box too. We need to listen to the users, but we also need to make things that would benefit the users, even if they do not say they want it.
In technology, we need to invest time in learning new ways doing old things in more efficient ways, but at times we just need to get job done in the most dependable way.
When I was an intern at Intel, Shekhar Borkar, who was the head of our reasearch wing, said a very interesting thing in a meeting with his team (which I imagine were more than hundred in number). He said that overall, he would like one third of his team's research work to succeed and other two thirds to fail. If all the projects were succeeding, that means the team was not taking enough risks. If all the projects were failing, that means the research team is working on unrealistic problems. The right proportion, for his research group, was that one third of the projects should succed. For production teams, perhaps more projects should succeed. I think for production teams, one in four projects should fail.
Long ago I read the book "The selfish gene" by Richard Dawkins, which had a significant impact on the way I think. One of the problems discussed in the book is the following.
Suppose, in a jungle, there is a population of jackals. Jackals, as you might know, feed on the food left over by predators like lions and tigers. Let's say there are two types of jackals: the hawkish jackals, which are combative and love to fight, and the dovish jackals, which are peace loving. If there is a piece of meat and two jackals approach it simultaneously, then what happens next depends on the tendencies of the jackals, whether they are hawkish or dovish.
If two hawkish jackals happen to approach the meat, then they fight over it. Ultimately, both of them get bruised quite badly, but one of them gets bruised more than the other and is forced to give up. The victor, though bruised, finally get the meat. If a hawkish and a dovish jackals happen to have a contest, there is no contest really. Hawkish jackal is ready to fight, and dovish jackal does not want a contest. Dovish jackal leaves the meat and goes away. If two dovish jackals meet, then they threaten each other, but no one actual hurts the other. Finally, one of the jackals gets bored and leaves the other to eat the meat.
If you are to be born in a jungle, is it better to be a hawkish jackal, or a dovish jackal? That depends on the proportion of hawks and doves in the current population. For instance, if all the jackals in the jungle are hawks, it is beneficial for you to be dove: there is no point being a hawk getting nearly fatally bruised in every encounter, just for a piece of meat. On the other hand, if all the jackals in the jungle are doves, it is beneficial for you to be a hawk: you will win every encounter. In a realistic scenario, where there is a mix of hawks and doves, the correct answer would depend the exact payoffs in various scenarios. Perhaps, when two hawks meet, the payoff for the loser is -100 (since he is very bruised), and the payoff for the victor is -90 (he is also bruised, but gets the meet). When a hawk and a dove meet, the payoff is perhaps 10 for the hawk and 0 for the dove. When two doves meet, the payoff could be 5 for the winner (he gets the meat worth 10 units, but loses 5 units in posturing) and -5 for the loser (he also loses 5 units in posturing). Using above numbers, one can derive whether being a hawk or dove is more beneficial in a given situation.
Thus, in a jackal population that evolves naturally, there will be a mix of hawks and doves.
Is your organisation completely data driven? Then perhaps you need some creative thinkers. Or is it mostly driven by creative thinkers? They perhaps can get out of touch with reality and need a data driven approach.
Are some projects of your team or company speculative enought that they fail? If not, time to do something wild. Are most projects of your company failing? Perhaps then you are out of touch with market.