Prof Zangwill offers QUICK THOUGHTS ON UNCERTAINTY IN DECISION MAKING

A guest post by:   Willard Zangwill, Ph.D., Professor, University of Chicago, Booth School of Business

Rachel Kaberon, in preparation for the Strategy Management Practices Issues Group discussion of the Chicago Booth Alumni Club, asked me to put together a page or two of thoughts about uncertainty in decision making.  Since she had helped me with software I have developed to assist in complex decision making, this was my chance to return the favor.  Hence, here are some thoughts that strongly influenced my thinking about uncertainty and how I have tried to suggest how people might better predict the future and make better decisions.

 UNCERTAINTY IS RAMPANT.

First is that uncertainty is remarkably uncertain, and our efforts to predict it are likely worse than we often assume.   Overconfidence bias is indeed strong.   What demonstrated this to me was the outstanding work of Philip Tetlock[i]. He studied how accurate were the predictions of experts and pundits in the political or economic  areas. These people were similar to the prognosticators we see on television or other experts discussing what might happen to events in the future.  Tetlock examined such predictions for years and studied tens of thousands of them, which was a huge undertaking.

What Tetlock discovered was how bad the predictions were.   They were only slightly better than chance.  Not the result one might expect, but worse.  Too many events seem to unexpectedly occur in the future.

Interestingly, the prognosticators that were most confident and sure of themselves, were wrong more than the more cautious forecasters who hedged and added conditional statements.    The confident experts tended to gain more support and attention, as their confidence convinced others, but that did not make them more right.

How could predictions be so faulty?  By and large, we tend to think we predict better than we do because if we are wrong, we give ourselves excuses.  We suggest that no one could forecast what really happened, or that events no one could have foreseen occurred. That process absolves us of blame and provides exoneration.   The net result, however, is that the future is harder to predict than most of us are likely to believe.

 WHAT HELPS

Given this conundrum that we have to predict events, but are probably not that good at it, what can be done.  Here are a couple of experiments that I have found useful to try to build upon.

As Gary Klein[ii] has noted, Research conducted in 1989 by Deborah J. Mitchell, of the Wharton School; Jay Russo, of Cornell; and Nancy Pennington, of the University of Colorado, found that prospective hindsight—imagining that an event has already occurred—increases the ability to correctly identify reasons for future outcomes by 30%.

The concept  is illustrated by the following.  Consider some upcoming event, say a presidential election.  Then think of reasons why a particular candidate might win.

Now do the following.  Assume it is now after the election. And assume it has just been announced that candidate has won by a solid margin.   Now think of reasons that triumph occurred.   You will likely think of  more reasons.  In essence, assuming an outcome and carefully imagining it, helps you think of reasons why that outcome might occur.   Perceiving those additional reasons then helps as you proceed to analyze the situation.

A much different approach in a study by Armstrong and Green[iii], was also quite helpful for forecasting the future.  In brief, they had subjects predict the outcome of past situations that were unknown to the subjects.    Since these were past situations, the actual outcome was known, although the subjects did not know those outcomes.  After the subjects made their predictions about the outcome of these situations, the accuracy of the predictions were then determined.

At this juncture, the experimenters then changed the situation.  They required that the subjects first consider several situations analogous to the one they had to predict; these were analogous situations where the subjects knew the outcomes.   Once they considered those several analogous situations, now the subjects were told to predict the situation in question.  The success rate went up substantially.  In fact, when a group of subjects were involved and they carefully compared analogous situations, the accuracy of the prediction roughly doubled.

The message seems to be this.  When we forecast an event, we tend to do that by thinking of some similar event that we know.  That similar event we know, gives us ideas about the outcome of the event we are trying to predict.  Now take this one step further.  If you consider several events roughly similar to the one you are trying to predict, it is like increasing the sample size. The accuracy of your prediction should rise.    Moreover, just examining how several situations similar to the one you are considering turned out, is illuminating, and by exposing the complexities of the situation,  provides useful insights.

 BREAKTHROUGH THINKING

Given the difficulty of predicting the future and the challenges thereof, it might help to broaden our decision-making framework and, in particular, to do more breakthrough thinking as that might provide us with an advantage.  Considering breakthrough thinking, as least for most people, good breakthrough ideas seem to occur almost randomly, as we tend to think about an issue and the exciting idea somehow jumps into our minds.    But there do seem to be procedures that help them occur more frequently and more when needed.   The key insight is to look and examine where the breakthrough idea is more likely to occur.

To illustrate, suppose you cannot find your car keys and have searched all over the house.  In frustration, you ask your spouse.   He/she replies that they are on your dresser.  Despite the mess on your dresser ( not necessarily yours, but certainly mine) you dash over to your dresser and with only a little rummaging, quickly find your keys.

As another example, when they search for oil, they do not put the exploratory well anywhere. But they first conduct detailed geological and seismologic examinations to locate where the oil find is more probable.

The concept for breakthrough ideas is the same.  Suppose you have one million possible ideas to search through in order to discover your breakthrough idea.  Finding that breakthrough idea from among the million possibilities,  is not likely to be easy.  This explains why getting breakthrough ideas is usually a challenge, as it required a quite large search to uncover it.

On the other hand, now suppose you obtain some clues as to where that exciting idea might be found that narrows your search down to ten possibilities.   You can easily search the ten and, in all likelihood, uncover the breakthrough idea.

The insight is to examine where the breakthrough idea is likely.   It is like drilling where the oil is likely, and you will more easily find it.   One of the concepts behind the software for decision making I developed takes advantage of this and seeks to suggest where the breakthroughs will be more likely, helping you to more easily discover it.

 CONCLUSIONS

The uncertainty of the future is probably far greater than most of us assume. Here I have tried to suggest some means that might help reduce that uncertainty and improve decision-making.  There are other ways as well, and they should help as you proceed to make difficult decisions for the future.


[i] Philip Tetlock, “Expert Political Judgment: How Good Is It? How Can We Know?” (Princeton)

[ii] Klein, Gary, “Performing a Project Premortem,”  Harvard Business Review, Sept.2007

[iii] Kesten C. Green and J. Scott Armstrong   “Structured analogies in forecasting”, University of Pennsylvania, 9-10-2004

Advertisements

BIG DATA: Big Deal or Just Big Business?

Technology evolves and for those of us who spend their lives adapting and endeavoring to keep up with the advancements it’s hard  not to notice a curious underlying dynamic.   Data and our ability to calculate or manipulate it for greater meaning is a little like resolving the chicken and egg paradox.  More of one begs more of the other, and yet we continue to ask which came first as if that question were important.  For many of us, our interest in  closing the uncertainty gap wishes for more data. We expect it will  help minimize the error or noise because the present picture of relationships remains a little too ambiguous. The constraint in this case is often our own experience and knowledge.

Professionally,  my own work warns against this unconscious bias.  I simply ask people to imagine three dots and ask that they line them up.  I then remind them that all three dots are coincident data points in time, and ask whether this new piece of information has changed their vision of the dots?  I then ask them to place the dots on an axis of time, and tell them that the dots now represent demand, growth or performance like ROI.  Does the way you’ve visualized the dots changed again?  I explain that the context I’ve added snapped into their own experience to create an image that creates a new puzzle as what they see fights with their expectation and they need more data to explain it.

The Economist in revisiting the Growth Matrix in 2009, put it another way. Bruce Henderson, credited with originating this framework reportedly believed  “while most people understand first-order effects, few deal well with second-and third-order effects. Unfortunately, virtually everything interesting in business lies in fourth-order effects and beyond”.

Big Data and the volume variety and velocity of its availability now has several partners,  real time processing power and plummeting data storage costs and lest we forget, simple access and manipulation tools  placing the data in an ever increasing number of users’ hands.  It is the number of people who now want to use the power of analytics that lends Big Data its influence, or at least that’s what several Chicago Booth alums who shared their thoughts last week recognized.

On May 18, 2012 Chicago was busy preparing for the arrival of NATO delegates and support.  The result was many businesses strongly encouraged their employees to work from home, leaving the monthly strategy discussion homeless.  We took advantage of the opportunity to launch our first virtual discussion combining a real-time interaction platform  (Group Systems Thinktank) and conference call (freeconferencecall.com).  Interest in the topic proved overwhelming prompting us to open up a second lunch interaction following  our usual early morning time.  The comments that follow represent a condensed version of the conversation.  Note, links to the discussion prep video and articles we encouraged participants to review in advance can be found at the bottom of this post. Also, a full transcripts are available to those who interested in seeing the automated output from ThinkTank, just drop me a note.

What’s the deal

Its a toss up whether mobility or big data has captured the imagination of business media more. The duel isn’t the point. Other driving forces and a growing need for critical thinking skills that were already in short supply.  Data reduction may be an emerging competency.  As the earlier references to Henderson point out, the questions you are trying to answer don’t get any easier just because you suddenly have access to more data. What to do with this new wealth of rich information are the bigger questions and challenges not merely for business but for consumers as well.

In the process of  generating the following list of examples, the interaction on Thinktank let participants also provide some links, raise new questions and add additional comments.

Twitter ,Telephone call records, Smart Grid, Real time Electricity meter  data , Nike + ,Scanner data,  Comments from Call Centers,  Providers’ case and disease management notes, EMR records, Geospatial (GoogleEarth, Navistar, etc), Mobile and GPS,  Gov’t DBs (big-data in an unstructured/non-uniform sense) , The quantified self, Output or processed Data from SAS, Salesforce.com, other enterprise databases, Loyalty program, Amazon purchase history, Mint.com, Moneyball (Big data in baseball), the new NSA data warehouse in Utah, QR, The internet of things, Data.gov, The London Datastore, created by the Greater London Authority (GLA–Chicago has similar initiative) offers citizens open access , Netflix movie recommendations, SAP’s HANA usage (profiled in the report on their Sapphire Conference )

Twitter for example has evolved in ways that surprised their founders and also launched a number of new businesses with very unusual purposes. As one article pointed out routing the data can be equally important as tallying it, as illustrated by Procter and Gamble’s practice of funneling social media conversation/data to the appropriate person’s screen for monitoring and response.  In other words to be meaningful, sometimes just knowing something happened is enough, it doesn’t necessarily have to be mathematically manipulated to derive value.

Persistent challenges remain in dealing with the enormous variety of formats in which data are presented — some  sources are difficult to analyze — their pages long data dictionaries  often include details about its collection.Add to that the realization that Data is not just numbers anymore. The automatic semantic annotation required to make sense of this has also entered a new era.

Facebook, LinkedIn, Google+, Pinterest and similar social media sites would fit in here, as well.  All offer a richness of information, much of it real-time, that can be monitored, mined and used to drive decisions and actions. The customer center conversations , or customer audio recordings , transcribed to text, and then subjected to text analytics is helping improve performance management practice, allows for campaign conversion performance tracking etc

The impact or convergence of the evolving technologies with all of this new data, is as overwhelming as the scanner data that was available and stored historically but few had resource capability or interest in mining it. That era has passed and with it, new questions and promises arise.https://i2.wp.com/www.connectedaction.net/wp-content/uploads/2012/04/20120421-NodeXL-Twitter-Global-Warming-Labeled-Groups-Network.png

Can we better understand and use consumers sentiment as in  do retail customers use more electricity/phone service/etc in a method that correlates to changes in the economy? weather? Or SAPs HANA data, now makes cancer DNA genome type analysis possible in minutes.

FB and Twitter,  may be drawing more attention, but those who  bring together the different streams  and mining more deeply old sources such as scanner data  are also causing quite a stir. Target found out that teenage girl was pregnant before her dad did using methods such as these.

Add the RFID  and imagine the benefit to stores knowing the quantity of each SKU that they currently have?  Can they coordinate sharing of product between their brick-and-mortar stores?

Perhaps the focus on twitter, etc., rather than scanner data comes from hoping that trending sentiment will precede and can be used to predict purchase, rather than log it as with scanner data. At cars.com, a participant shared  that vehicle search data on our website is predictive of sales.

It looks like QR codes might achieve the same benefits of RFID as smart phones are becoming ubiquitous.

The Geo-spatial data, according to McKinsey’s recent report on Big Data as the next big innovation, quantified billions in time savings from just helping consumers avert traffic.

Several other behavioral nudging based on real time feedback is now possible. Nike is exploring and furthering this automatic feedback. Check this article for more brands using the quantified self, and more information

TOOLS

SAP’s HANA, Google’s BigQuery , Splunk, Hadoop and NoSQL databases , Tableau, Tibco Spotfire, Omniture ,Pentaho (open source BI), Amazon’s Web Service suite (more of a platform), Cloud computing platforms, Data visualization, most business users, make the data preparation easier and allows them to focus more on analysis and develop insights in combination with Machine learning tools (neural networks, support vector machines, natural language processing, etc.)

Decision making , can and does BIG data make more accuracy possible?

It offers higher granularity — like Target’s ability to create coupon books customized to individual households, Or integrate GPS level stuff — e.g. texting coupons to customers right while they’re standing next to a certain store.  Or 2nd/3rd order analysis – correlating Target’s sales with weather data; creating ‘real-time’ personalized coupons; identifying ‘trend-setters’ among the customer base to influence ‘trend-followers’ coupons

The downside? Detecting or separating out spammers from these data sets or paid to  express a certain sentiment. Totally! Like those girls paid to say great things about clothes on Facebook — not maybe necessarily analyzing big data, but using the platform.

Greater real-time evidence  can reduce risk and  insure assumptions in product/service development and marketing are on or off target.

Hard to appreciate the analytic without the qualitative context or understanding;  but maybe some strange new ideas can come out of the data, like the “diapers next to beer” epiphany. Data needs some drivers to make it meaningful, as in cause and effect. In part because qualitative data is harder to analyze.

Is quant vs. qual or the social science methods to data collection really that different from the scientific data analysis approach?  Both approaches seek to explain cause and effect, or the relationship between a stimulus to produce a predicted response. The problem is  too many people will extend a model beyond it’s capacity. Claiming “the data said so”  lets people off the hook and avoids responsibility for  decision-making.

Suggested tips include  avoid extending a model beyond its capacity, or  understand and differentiate descriptive and predictive Stats. Likewise, be wary of  finding trends that don’t exist (e.g. “data mining” or “straws that look like needles”) and confusing correlation with causality.

Perhaps cross-validation from trained analysts can help avoid  these . Danger of expecting tools to automatically extract value from large datasets.  Need to ensure good analysis, disciplined hypothesis generation, etc.

The data, even when analyzed, does not represent the decision.  This is true with small and big data.

FINAL TAKEAWAYS

 

  • Big data is here to stay – need to figure out how to use it effectively.
  • I liked the point that lots of data is around and that people just don’t know what to do with it.  The best BIG DATA process or engine in the world still won’t create the insights that are needed.
  • Corporate culture is a huge factor–the problem is not availability of data, but commitment and focus of corporate leaders to shape a culture that moves the organization in that direction.
  • Big data is here. It’s a tool and like any other, it’s the latest and greatest on the block, with a bit too much hype. But it has a definite value in providing and stronger qualitative base to identifying trends and activities.
  • My realization  is that, once again, the technology is interesting, but it is the corporate culture and will that will matter. The culture and vision lead; the strategy and models follow.
  • sharing the questions with a wider audience confirms concerns and clearly lots of assumptions that need to be played out. There is a large dark side that we still don’t understand; but the positives and opportunities for real time decision.
  • Big Data! The piles get higher and  higher and wider and wider…to what purpose? That implies the need to “mine” the data, reduce it and subject it to analysis before it can be made useful.
  • Big data will revolutionize business but it is not strategy, potential for a lot of false positives .
  • The wise use of big data offers a huge opportunity for developing differentiating strategies and for finding new product/service needs.
  • “Big data” is the current term for things that have existed for a long time.  All types and sizes of organizations can benefit from big data if they recognize the importance of the human component (not just the data and software) and have specific objectives in mind before starting.
  • Much of the expertise about analytics developed over many decades still applies, and there are new dimensions to integrate and understand because of the availability of the technologies and data.
  • Everyone on the line has experience with Big data, so I don’t think it’s so scary.  Most people have business perspectives, wanting to teach the Analysts that their conclusions need to be driven by business needs.   My comment is that as leaders, and those trained with some behavioral awareness through business school, it is _OUR_ responsibility to try and massage the analysts towards an enthusiasm for our world view…;-)
  • The human side of utilizing the technology and expertise is just as challenging as ever. (Cognitive biases, communication skills, influencing skills) Garbage in – garbage out is a big risk without proper attention and skill in applying the technology and in communicating.  The data, even when analyzed, does not represent the decision.  This is true with small and big data.

In closing, let me return to my observations about the limitations of developing strategy rooted in an expectations of the experience curve relationship.  The frame with which you approach the problem often has far more bearing than the data, your analysis or the tools.  Or at least, in June we  plan to look at some of the assumptions around growth as the ultimate strategy.

Please throw your responses, or continue to post links for others as Big Data continues to be quite newsworthy as its impact and influence continues to unfold.

Articles and links:

We suggest an  Optional  short 5 min. video tutorial , EMC produced to understand what Big Data IS?    http://youtu.be/eEpxN0htRK
The following are required advance reading.

1. IBM’s Institute for Business Value, in collaboration with MIT Sloan Management Review  2010 research findings
Analytics: The new path to value: How the smartest organizations are embedding analytics to transform insights into action

  http://public.dhe.ibm.com/common/ssi/ecm/en/gbe03371usen/GBE03371USEN.PDF

2. Strata keynote  short 7 min. video by Google’s Digital Marketing Evangelist Avinash Kaushik- a bit irreverent and a little over the top – a bit irreverent, bordering on over the top – but not boring  effort to help us understand the problems and the approach undertaken by Google.

Big Data Imperative 
March 2012
http://www.youtube.com/watch?v=CrSX97elHDA

3. Tom Davenport’s Culture of Analytics

April 5, 2011
http://smartdatacollective.com/clifffigallo/34719/tom-davenport-s-culture-analytics 

4. Inside P&G’s digital revolution
McKinsey Quarterly November 2011
http://www.mckinseyquarterly.com/Retail_Consumer_Goods/Strategy_Analysis/Inside_PGs_digital_revolution_2893
overseeing the large-scale application of digital technology and advanced analytics across every aspect of P&G’s operations and activities—from the way the consumer goods giant creates molecules in its R&D labs to how it maintains relationships with retailers, manufactures products, builds brands, and interacts with customers. The prize: better innovation, higher productivity, lower costs, and the promise of faster growth…

One more optional overview

The age of big data
NYTimes, Feb 2, 2012
http://www.nytimes.com/2012/02/12/sunday-review/big-datas-impact-in-the-world.html

 

Is it Too Late for a Web Strategy?

Old spice man

If you don't know this man, then you're missing out on one of the more popular twists in popular culture and marketing of 2010. 

This is the Old Spice campaign's man of mystery.  Intentionally I did not insert the web video, nor am I interested in chasing down the viewer stats, though sales report isn't great.  It's here because the ad reference exemplifes multi-channel linked marketing strategy and came up  in last Friday's monthly Chicago Booth Alumni Club's Discussion around  Strategic Management Practices.

Wearing my research hat, and doubling as a typical consumer, the first place I turned to find the reference was to type the key word phrase "old spice man" into my google search bar located at the top of my web browser. My search was not to purchase, engage in conversation or to gain proximity to someone with product experience –that would need  some different key words.  The campaign as well as my search process shows the evolution of the internet and the effect of its influence in our lives.  The shifting trends exhibited below in this wonderful chart  was the focus by Chris Anderson and Michael Wolff in the provocatively titled September 2010 article in Wired The Web is Dead, long live the Internet

Internet traffic trends 2010

CISCO compiled data using the Cooperative Association for Internet Data Analysis (CAIDA). The chart suggests that Video and Peer to Peer traffic is increasing while the use of the world wide web is declining.  This data is somewhat misleading and the chart's suggestions that mobile apps, and other specialized channel options, will displace the web browser  is not so clear-cut.

Is this graph a credible and reliable translation of the geek speak from  CAIDA?  A more recent  analysis than what appeared in Wired, expresses the following:

" Continuing its growth in traffic, connectivity, and complexity, the current Internet is full of applications with rapidly changing characteristics."

Overall, CAIDA has found that traffic on the internet continues to grow,  which is not adequately represented by the two- dimensional graph CISCO and WIRED depicted. Growth does accurately reflect the transition and growing emergence of traffic off the world wide web and into  alternative internet based transmission paths (e.g. mobile based and other applications that allow real time streaming).  

This same transition mimics strategies used by effective  marketers who link the brand messages and campaigns across  multiple media platforms.  Key words provide the bridge. The more consistent their use across the growing number of media platforms,  the more certain an organization's promotion efforts will  intersect key consumer touch points on or offline.   Ideally, consumers pick up these same key words  and carry them across other natural communication channels, further enhancing the brand's reputation and in theory  increasing sales.

If your business is selling Search Engine Optimization (SEO) this emphasis on key words appears  great for business. It's not however where a capable marketing strategy should invest the majority of its budget.  Not merely because there is some danger to pursuing this strategy (see the The dirty little secrets of search in last week's New York Times); but the greater, more complex objective is reputation management and not key word optimization.  

 Historically, brand owners/creators controlled media messaging and placement.  To successfully sell, you "paid" for the privilege of being placed in front of consumers walking through the yellow pages or by a billboard, listening over radio/TV  or their eyeballs scanning newspaper or specialty publications. Product packaging, placement and promotion  are often  budgeted separately and only occasionally linked for a "special" promotion (e.g. cause marketing or a contest).  The rise of the world wide web, added the category of "owned" media to the marketing mix and budgets had to cover the cost of website development, content writers and traffic analysis, including SEO.  With Social Media, a third area– "earned" warrants increasing budget and management attention to monitor the customer-created channels and chatter of your brand enthusiasts  as well as brand detractors. (see complete description in Branding in the Digital Age by David Edelman). 

 The Edelman article's case study of a TV manufacturer across one touch point within the wider consumer decision journey proves far more  instructive than my earlier reference to the Old Spice ad and its multi-channel focus. 

"A costly disruption of the journey across the category made clear that the company’s new marketing strategy had to deliver an integrated experience from consider to buy and beyond . In fact, because the problem was common to the entire category, addressing it might create competitive advantage."    

Unlike Old Spice, the manufacturer opted to shift the marketing emphasis away from paid media.  Focusing on owned and earned media seems to enhance the effectiveness of their key words and multi-channel linkages, and engage traffic where it mattered most at the buy, and enjoy, advocacy, bond  touch points. This is not a prescription for all brands, but the case is instructive in identifying the disconnects and deficiencies in common web based strategies, or even of marketing extravaganzas disconnected from the ongoing conversations that are circling your business, product and/or brand.

Whether or not you belief in Chris Anderson's prognosis about the death  of the Web or buy into David Edelman's Consumer Decision Journey research, few organizations appear to have fully leveraged these changes.  Increasingly, an ability to execute and efficiently allocate resources to address the demands presented by the growing number of communication channels  will  distinguish successful companies from their competitors.  The changes create more opportunities for strategy to take a more commanding role in managing and driving the combined efforts, either internally or with the help of outside specialty firms.

Additonal Discussion Take Aways

  • Social networks are informative, free sources of intelligence that naturally build out and generate mutual trust and benefits to buyers and sellers. 
  • The role of the marketer is merely to influence and no longer the producer/director of the brand experience.
  • The responsibility for marketing  is changing and increasingly is upending internal role limitations  and requiring participation from unlikely sources e.g. corporate governance, communication standards and guidelines.  Employees share roles with customers and the more acquainted with internal policies, strategies and planning the more they can aide and assist in  wider message consistency. 
  • Authenticity has become ever more important.
  • Fluidity and increasing knowledge of terminology around the digital communications space is a valuable skills set…not just for marketers and IT folks. 
  • As reputation management rises and people do business more and more with the people that they know,  is there anything really being created of value, and are other marketing and sales efforts as necessary?
  • How do these lessons translate or enhance B2B sales? 
  • It's not the web vs. the internet differentiation that matters, as much as recognizing how one innovation(social media)  has brought into focus an array of  deficiencies and gaps within an organization (marketing departments) as well as an industry (e.g. advertising) The challenge is how to best integrate the old with the new. 
  • In the end, the prescription to know your customer before creating your strategy remains the first and foremost lesson. Knowing what your customer wants will always be helpful but successful business requires more.
  • True differentiation in products being marketed remain beneficial but the emphasis should be toward innovation in developing products. 
  • Important to remember the shape of the adoption curves with new technology and Chris Anderson's point that new doesn't replace old. New merely creates more table space to accommodate more preferences.  The challenge is the frequency we change, resort and revisit our marketing activities and resource priorities. 
  • Both  articles confirm the importance of social media and keeping up with changing technologies.  They also call attention to the  the challenges organizations  face in trying to bring them together  to create successful communities around their products and/or brands.

 

Any added thoughts, perspectives or cases are welcome.

Added citations

Edelman makes some of the same points in this article:

Four ways to get more value from digital marketing

By David C. Edelman, McKinsey Quarterly, March 2010

https://www.mckinseyquarterly.com/Four_ways_to_get_more_value_from_digital_marketing_2556

 

Trust Agents, Using the web to build Influence    by Chris Brogan and Julien Smith

NOW Revolution, 7 shifts to make your business faster   by Jay Baer and Amber Naslund