Newspapers: Open Your Archives

One of these days, a newspaper currently charging a premium for access to its article archives will do something bold: It will open the archives to the public -- free of charge but with keyword-based advertising at the margins.

I predict that the result will pleasantly surprise the bean-counters. There'll be a huge increase in traffic at first, once people realize they can read their local history without paying a fee. Eventually, though not instantly, the revenues will greatly exceed what the paper had been earning under the old system. Meanwhile, the expenses to run it will drop.

And, perhaps most important, the newspaper will have boosted its long-term place in the community. It will be seen, more than ever, as the authoritative place to go for some kinds of news and information -- because it will have become an information bedrock in this too-transient culture.

I've believed this move is necessary for newspapers for a long time. Almost three years ago, in an article for Library Journal called "Yesterday's Headlines," Rich Wiggins looked at the growing "archive digital divide," in which some people put things on the Web in a permanent way and others put them behind walls. He asked me what I thought. I replied, in part, "I have a feeling that the newspaper industry would be better served by opening up the archives and Googling them (and selling related ads based on keywords entered) than charging for individual searches."

I recognize the institutional and financial hurdles that will make it difficult to pull off in many companies today, even if they like the idea in a general sense. But I also believe it's almost inevitable.

Some major newspapers already do it. I would love to know how much the San Francisco Chronicle, which offers its archives at no charge (and loads search results with too many ads), get in revenue compared with the San Jose Mercury News, which charges for older articles.

For me, again, the heart of this issue is more about a newspaper's community role than revenue, even though I believe the right thing from the community standpoint is also the best from a financial one.

In an important recent posting on the PressThink blog, the Guardian's Simon Waldman explained "The Importance of Being Permanent" -- the notion that Web postings should have permanent, accessible Web addreses, or URLs. He wrote, in part:

Permanence means understanding that when you put something on the Web it should be there for ever: ideally in the same place for perpetuity. It means that if I link to it now, someone else can follow that link in two days, two weeks or two years' time. (I'm not going to lay out the business models in this piece, but I'm not excluding the possibility of pay-to-view; it's the position that counts, not the price.)

Waldman pointed out that the mainstream news industry, with a very few exceptions, doesn't see things this way. The consequence of the industry's tendency to push its older journalism behind a wall, where the initial links disappear and people now have to pay for access, is a loss of identity and diminution of authority.

Jay Rosen believes journalists should demand it, since their work is being hidden from most of the world as a result of such policies (he has more on this in a posting today. As he notes, last week's Harvard conference on Blogging, Journalism & Credibility produced a surprising consensus in favor of opening archives. Maybe that's not so unexpected after all, given that almost all of people in the room were professional journalists and bloggers: "content" types. (It's easy for me, as a proud former member of what newspaper publishers call "cost centers," to advise companies to give up a certain source of revenue for a speculative one.)

There are two immediate questions for publishers. First, how much money are they making today from pay-per-view archives? The Poynter Institute's Bill Mitchell said over the weekend (on a mailing list), "The range of news orgs generating significant revenue from archives with current biz models may be more limited than I had assumed."

That's good news. It means that the financial risk of changing the model is small in many, if not most cases.

Second, and related to that, what are the benefits of shifting to an open archive? I'm utterly in Waldman's camp that it would heighten a newspaper's authority, helping keep it at the center -- or not far from there -- of the community's civic agenda.

Every newspaper of any quality has published hundreds of articles that readers can find nowhere else and which bloggers, among others, would surely cite and point to as a vital part of the permanent record of a community. These include investigative pieces, certain features and other stories. If available upon publication at a permanent URL, they quickly rise in search engine rankings, where others will find them later.

I'm convinced that increasingly sophisticated Web advertising, especially keyword-based text ads, will create a revenue stream of some size for such stories. This will be especially the case when they've moved high enough in search-engine rankings to be found without searching the newspaper's site, but that's not crucial.

A locally targeted ad based on a keyword will bring new kinds of advertisers to the newspaper: small businesses that couldn't afford to buy space in the print edition. This is new money. It may not replace what papers are losing to the online competition, but it's worth something.

Many articles won't have that broad appeal. But they will have special meaning to smaller numbers of people who will want to point to them from their personal sites. They will reinforce people's sense that the newspaper is a medium of record in their lives. That's also worth something.

If I was a publisher with a pay-per-view archive, here's what I'd do:

1) Re-publish every article in the archives with a unique URL, outside the pay-wall. It would be helpful if the articles published since the newspaper went online could have the same URLs, but don't worry if that's too expensive; if the stories are important enough, they'll be found and pointed to. It'll just take a little longer.

2) Leave every new article on the Web at the URL it had upon publication. That's easier.

3) Encourage the readers to use the archives, with house advertisements, website notices e-mail to local librarians and other ways to get out the word.

4) Let local bloggers know that you welcome their links, and that you've made the change in part because they need it, too.

5) If a local blogger points to your article, use Trackback or other such technology to point back. (But be careful of link spamming.)

I don't know if there are ways to share revenues with people who point to news articles, or if that might raise some difficult ethical questions. I do know that the archives should be open in a way that encourages cooperation with bloggers and other Web publishers, even when you're competing in other ways.

Someday soon, some paper is going to try this. It'll be a great experiment. I believe it'll be a successful one, too.

Comments

[Jay Rosen]

Dan: Very compelling case. You put it masterfully. I too believe someone will try it.

Here are some further elaborations, twists, angles:

* Once you take the plunge, the editorial staff has to figure out how to do journalism so as to continually improve the (future) value of the open archive, without distorting what you do to gather and present the news.

* A partnership with public libraries in producing an open archive of past editions might be possible. Consider free public access in the context of the gift economy.

* At the conference Bill Mitchell came up with a phrase, "locking up the content" to describe the current policy. Exactly. And now it should be unlocked.

* Bill and others assured me that most journalists working in Newsroom USA know that their work will never make it into Google and other search engines. I don't think they realize it.

Click my name for more on this.

[spriggs]

as pravda insists on charging for its archives, i'd taken to reformatting the content i felt strongly enough about to rant about on my blog. i wouldn't have violated their copyright if i knew the url i'd pointed at would still be availablefor freetwo weeks after it was posted...

[Karen M.]

I like to blame Microsoft... on principle. Wasn't it Bill Gates who predicted that eventually it would be the information or content that would generate income on the internet? (If not him, who was it...)

I'm fortunate in having Lexis/Nexis access available through a university library, but not everyone does, and it's not that convenient anyway.

This has been an issue in medical and scientific peer-reviewed journals, too. However, it appears that the trend is toward more openness.

[Eric]

I worked for the newspaper in Grants Pass, Oregon. And the publisher was so fearful of the Internet he would only allow a single page to be put up, and only then with links to advertisers.

Interestingly they've carried to this day an ironc quote near the top:

"The theory of a free press is that truth will emerge from free discussion, not that it will be presented perfectly and instantly in any one account."

-- Walter Lippmann

Now they have an archive search through the library for a very reasonable $20 an hour! What a deal!
Free discussion? Not in Grants Pass! You have to pay to enter this marketplace of ideas there.

[Thalia]

In fact Lexis-Nexis does pay the NYT a not insubstantial amount of money for the right to have copies of the archival materials. And the contract, as far as I recall, does specify that the NYT cannot make this content available to anyone else for less. So unless you have a revenue model that replaces the income from Lexis-Nexis, I venture to guess that the NYT, and all other newspapers, will keep their archives locked.

[Frankke Potential]

Er, um, I don't mean to embarrass you, but I must point out that the NY Times is currently evaluating going in the opposite direction, charging for their daily content and raising prices for accessing their archives.

Also, in your position, you should know that Nexis recently agreed to higher licensing fees to the leading periodicals in a series of deals well publicized in the Library Science trade journals. (And there has always been an element of exclusivity to these arrangements).

[Jozef Imrich]

Mr Waldman, the Director of Digital Publishing at The Guardian, rightly pointed out that "permanence" is one of the most important characteristics of the Net, even eclipsing such factors as "immediacy" and "interactivity."

As someone who used to do a lot of digging and data mining for a wide variety of clients, nothing is more annoying than not being able to access information in a timely fashion. Often users will settle for a less reliable information in order to meet deadlines. Lexus and Nexus and other providers of specialised research databases caught in the web of exclusivity have built so many iron curtains within their own databases that often even librarians and researchers give up on using them if time is of an essence.

I often find useful information and links on free search engines such as Google or Soogle. Many of these search providers archive amazing amount of data that can be accessed if a researcher is able to fine-tune keywords with +, -, site:, source and other useful search techniques... However, even the best techniques have limitations. When a document or a file is behind the wall, coined subscribers only, and moved to another URL address it is lost to a general public.

In Australia hardly any surburban newspapers are available on line. In the country areas the situation is not better.

Most of the 1,400+ daily newspapers in the United States continue to offer their content on their Web sites for free. Cyberjounalist compiled a list of the more than 35 newspapers that charge for access.
Newspapers that charge for their sites

Most newspapers allow free access for a week or so.

How can anyone expect working people to be involved in discussions of public interest in public life or be engaged as citizens when information that is older than a week dissappears from the radar. No wonder we are all developing the short-term political memory syndrome...

[Karl]

If newspapers not only opened their archives, but published a web based API (RESTful my preference) independent developers could add value to those archives.

[Danny Sullivan]

"It will open the archives to the public -- free of charge but with keyword-based advertising at the margins."

Which is great as long as ad revenue is flowing. When it is not, nice to have some alternative streams of income -- not to mention nice to have some disconnect between being beholden to advertisers entirely.

"Every newspaper of any quality has published hundreds of articles that readers can find nowhere else and which bloggers, among others, would surely cite and point to as a vital part of the permanent record of a community."

Hey -- we don't need bloggers. Search engine could easily full text index these articles and point at them. Google, for example, already does for news search. And the permanent URLs could be retained. The problem is that the search engines generally don't want to link to content behind password-protected walls, though Google Scholar is an exception. Want a first step in getting archives more accessible? Put some pressure on search engines to start indexing behind the barriers. It's not a technical problem to do so -- they just don't want to.

"If a local blogger points to your article, use Trackback or other such technology to point back. (But be careful of link spamming."

C'mon, Dan -- hundreds of articles per day, and you want to monitor that? And you trackback to bloggers but not to many non-blogging sites that might link to you? Why the special treatment here? Plenty of non-blogs will help generate traffic to the search engines without expecting some type of trackback reward.

[Tom Guarriello]

Amen to this suggestion, Dan. For me, the NY Times has the most infuriating pay-for-content model. I view, and link to their archives often. They say they charge "as little as $1.05 per article." Here are the various ways in which you can purchase:

Purchase Single Article -- $2.95

Purchase as part of Article 4-Pack -- $7.95
($1.99/article - a 33% savings!)
Pack expires after 30 days

Purchase as part of Article 10-Pack -- $15.95
($1.60/ article - a 46% savings!)
Pack expires after 90 days

Purchase as part of Article 25-Pack -- $25.95
($1.05/ article - a 65% savings!)
Pack expires after 6 months

Now, I don't like paying for these articles (I'm a paper-version subscriber, BTW), but, it's the way of the world, right?

But tell me this: what is their rationale for expiring the bloody purchase that I've made? I buy 25, use 18 and lose the last 7. Why? Nothing but pure "got the customer by the shorts" greed. And for this, some day, they will pay. Customers fight back--when they can--after being treated this way.

[ethan fremen]

The big reason NYTimes doesn't do this is because they cannot. As of Tasini vs. NYTimes, they would have to get permission from every author in their archive; a huge and extremely expensive undertaking. They could, in theory, buy more rights from authors and make articles available from now on, but they can't trrivially do this for back articles.

[Joe Kulhavy]

Thank you for your on-spot essay.

Several years ago, I sent an email to our local paper (Austin-American Statesman, owned by Cox Enterprises) pointing out that from my perspective as a researcher, the San Angelo Standard Times has a much higher Web profile, notwithstanding that this regional West Texas paper has a much smaller readership and budget than the Austin paper. The reason? The San Angelo Standard Times' archive is deep, free, and readily accessible through major search engines like Google.

To his credit, the Statesman's webmaster did respond to my email, although his response completely missed the point of my email. He stated that because the Statesman relies on advertising, it must subsidize access to its older items.

What he failed to see was that I was not just complaining that the Statesman (and the Times, and the Dallas Morning News, and the Wall Street Journal, etc., etc.) were charging for archived content. I was objectively announcing the doom of the pay-for-articles business model.

I wasn't expressing an opinion; I was just stating a fact. The San Angelo Standard Times (a thin regional daily from rural West-central Texas) is a more significant and high profile Internet paper than many national dailies, precisely because the Standard-Times archive is freely searchable and deep.

[Daniel Conover]

The Tasini precedent has been cited as justification for keeping archives behind some kind of wall, but I suspect that's a really an excuse for locking down the archives at some papers. the general attitude seems to be "hey, we're getting ripped off by people who don't buy our paper but get our news for free online, so why shouldn't we charge for the archives and get some of that back?" There's a real push by some executives (NYT and elsewhere) to charge for practically everything.

It's just an old business model that hasn't caught up to current reality, much less thought ahead to the new information environment. It will be washed away within a couple of years.

As for Tasini, i believe the issue is really freelance work, not every author. Plus, since the decision, most papers have instituted new freelance contracts that cover this contingency. It should be zero problem to post new archival material, and one would think that creative minds could find solutions to opening the archives... if that was actually a goal.

[Anspar Jonte]

Obviously, newspapers have a point in selling archives, or the practice wouldn't bother Dan so much. Sure, it's a different type of person who needs archived news, but there's still a need. And they can still make a buck charging for it.

Sure, newspapers are better off providing free archives, just like record companies are better off leaving file traders alone. Clamp down on access and you force a few people to pay. Open yourself up and you make more money in the long run.

[Anspar Jonte]

er, more importantly, it's offensive that I can subscribe to the print edition of a newspaper, and be charged extra fees for access to electronic archives. The newspaper does not differentiate me from non-subscribers. It seems I am buying newsprint and not the news content.

[Joe Clark]

New York Times Link Generator, people.

[Joe I.]

Funny thing is the NY Times online will start, from press reports, a subscription system soon like the WSJ and Salon. The site will no longer be free so I think the tide is going the other way.

[Carmi]

Wonderfully put. Beyond the near-term business models, one needs to look at how the media consumers of tomorrow - today's kids - will relate to whatever product the old-world media are putting out.

Today's kids don't subscribe to newspapers. As a result, media organizations - especially, but not limited to, print - need to figure out what model will hook them in to become loyal consumers just like their parents were.

The fate of online archives factors directly into this issue because it will influence how these new new-media consumers will interface with the organization. And if the model doesn't work for them, the old dinosaurs will ultimately suffer.

Carmi Levy
http://writteninc.blogspot.com

[Howard Melman]

In the early 80s, if you wanted today's news, you paid $0.25 for the paper (or whatever) and if you wanted old stories, you went to the library and used microfilm and got the info for free, I did this all the time as a student. I realized in the early 90s that the model was being turned upside-down, now papers gave away current news online and charged for online archives. When the Wall Street Journal changed their online version to be subscription-based everyone watched to see if it would work. Up until then, I thought they had the best online news, making the best use of hyperlinks and the web, but I was unwilling to subscribe for it. Turns out they do ok and are profitable in this model, but few have copied it. We've also moved classified ads online and I understand this was responsible for half a paper's revenue and now that is going to other places like Craigslist or eBay. Now if you want to take away revenue from archives (which I doubt is very much), just what is the business model for an online paper supposed to be?

[Steve Cisler]

I'm surprised those posting don't realize the tax-supported services already offered by many, many public libraries. They negotiative with the database providers who have contracted with the newspapers to provide full text access to subscribers. Most of you can get a free library card and get access to the archives (S.J. Merc, NY Times, and many others) at no cost to you. For instance InfoTrac provide this:

Full text articles (1996 - present) of 126 US and foreign newspapers including, Christian Science Monitor, La Opinión, New York Times, San Francisco Chronicle, Times London and the San Jose Mercury News (last 4 years only)...."

For those people in rural areas (and small libraries) the state libraries make the contracts with the database providers and this gives those with Internet access pretty good coverage. However, people in places like Boston, S.F., San Jose, Dallas, have more choices.

[jimmypermo]

'There'll be a huge increase in traffic at first, once people realize they can read their local history without paying a fee.'

Remember that the infrastructure cost on web archives goes up exponentially as the traffic increases. Many newspaper use a partner to host this content, such as Proquest. Can you imagine what such a 3rd party will start charging when they realize that the server traffic is going to experience a huge increase? That would have to be a hellava lot of Google ad click throughs.

[Henry]

In all the discussion over the past few years regarding the amount publications earn by charging for access to their archives I don't remember seeing any amounts mentioned. I'm surprised that this number is proving so difficult to determine. One would assume that they would not do this without a clear financial gain in doing so.

Does anybody have any guess what the revenue is? Can someone find out?

Reporters, report... There must be a way to get this information.

[Richard Silverstein]

When I saw your article & its title all I could say was "YES!" You are so right & media owners are so wrong in walling off their content. I'm espeically PO'ed at my newspaper of record the NY Times which does this. I've had some e mail conversation on this subject with their website staff & they claim that they want to move more in this direction. But I think they need an external push from readers.

I'm delighted by the idea of paying small fees to bloggers for links to online material. It would vastly increase the amt. of linking & thus vastly increase the level of cross-viewing by blog readers who explore these media links. THe majority of my media links are to NY Times. What are the odds that they'll ever see me or other bloggers as both a resource & revenue generator? Kinda doubt it...

My hyperlink below links to one of my posts on this subject.

[anne]

The New York Times just made it harder to email an article even from today's paper! Just, today! What a tragedy. I love and buy the paper, but use the website to mail articles.

[Ted]

Have you all missed a Trackback from above?
Luke Rosenberger
writes:
Dan Gillmor and Cory Doctorow argue that
newspapers should open up their archives on
the web. What they don't seem to be
grasping is that those backfiles are
already a valuable asset for the newspaper
publishers when they sell electronic rights
to the likes of EBSCO, Gale, LexisNexis,
ProQuest, and others -- cash in the bank as
opposed to the ad-supported model that
Gillmor proposes.

Outsell Inc., a leading research and advisory firm that focuses exclusively on the Information Content Industry, estimates more than $50 billion is spent each year on paid content.

Offering up valuable archives for the good of community, history and bloggers vs. $50 billion? Got any other windmills to tilt at?? Good luck!