Why OpenCalais?

Over the last few months you’ve probably seen a number of announcements about how OpenCalais has been chosen by one organization or another to support its business.

In a number of recent meetings I’ve been asked the (very fair) question, Why OpenCalais and not one of the other entity extraction services out there?

Given that the question seems to be coming up more often as the number of extraction services increases, I thought I’d get my best understanding of why many major players we’ve announced (and an equal number we haven’t) have chosen to go with OpenCalais. And – at the end – I’ll mention a few reasons why others haven’t chosen OpenCalais.

So, in no particular order, why do organizations choose Calais?

Thomson Reuters

OpenCalais is provided by Thomson Reuters – the largest professional information organization in the world.

If you’re interested in kicking around some semantic technologies in your spare time this doesn’t really matter. If you’re incorporating those technologies deep within your business – or, as is true with many users – actually building a new business on top of them, this becomes pretty important. Basically – you need to know that the service is going to be there for you.

Facts & Events

With the increase in structured content assets like Wikipedia / DBpedia, it’s become pretty easy to knock out a basic entity extraction tool. And – while we like entity extraction as much as anyone else – it’s really just the tiniest starting point in what you can and will need to do.

OpenCalais extracts a wide range of facts and events from unstructured content and lets you know what’s happening in your content – not just tags for things.

• Facts are things like “John Doe is CEO of XYZ Corporation.”
• Events are things like “XYZ Corporation today announced that it would acquire ACME Corporation.”

OpenCalais is the only service that does this in a production-strength manner.

Reliability

OpenCalais stays up. It’s hosted in mirrored data centers thousands of miles apart from each another. It’s monitored 7*24. It basically doesn’t go down – even during system upgrades and maintenance. We stopped adding 9s after we got beyond 99.99% uptime.

Accuracy

We’ve been building the tools underneath OpenCalais for over a decade. They’ve been used by hundreds of organizations and many many thousands of end users. One of the things we’ve learned is that accuracy matters. While no NLP system is perfect, we’re convinced ours is the best and we have some ideas in the pipeline to increase accuracy even more.

Integration

We basically focus on providing great semantic plumbing. But we know that not everyone wants to be a plumber. We’ve worked to integrate (or motivate others to integrate) OpenCalais with a wide range of tools including Drupal, WordPress, WordPress Multiuser, Oracle, Lucene, Coldfusion, Flash, Firefox, Prolog, Lisp, Django, Java, PHP, Python, Alfresco, Perl, .NET, Ruby, TopBraid and a few others.

From content management systems to language-specific libraries – there are lots of ways to get started quickly.

Linked Data

We’re serious about Linked Data. We’re also worried about the proliferation of incorrect links and the effects of link rot. So, rather than just pointing to Linked Data assets out on the cloud and risking that they’ll go stale, we host our own Linked Data cloud, which is kept up to date with both Thomson Reuters contributed content as well as regularly validated links to other sources such as DBpedia, Freebase and others.

SocialTags

Pure semantic extraction is great – but sometimes you need more. If you’re writing about Porsches and Ferraris you’d probably like to have categorization concepts like “sports cars” and “automobiles” returned to you with your semantic metadata. OpenCalais does this via our ever-improving SocialTags concept tagging capability. It’s good now, and it’s going to get a lot better soon.

Focus

OpenCalais is here to provide great semantic plumbing. We’re not trying to sell ads. We’re not trying to provide the prettiest decorations for blogs. We build the plumbing – you architect the solutions.

Now, in a spirit of transparency, here’s why some people don’t choose OpenCalais:

Languages

We’re great in English and okay in French and Spanish (we extract entities but neither facts nor events in these two languages). We intend to implement more languages in the future – but for the time being we’re concentrating our efforts on improved functionality and accuracy in English.

Complexity

OpenCalais isn’t a simple tagging tool. What it returns to the calling application is a reasonably complex RDF construct. It takes a little time to get up to speed on RDF and how to use it in your applications. We think it’s worth it because it’s the most flexible and powerful format we know of.

Performance in Knowledge Domain ‘x’

Where ‘x’ is fashion or square dancing or rugby. OpenCalais is optimized for performance in the general world of business – that’s where we excel.

We have extended OpenCalais to take steps in other areas (such as sports, media, etc.) – but if you need deep semantic extraction capabilities related to protein binding – there are better places to look.

English, Official Blog, OpenCalais Strengths, OpenCalais Value

We are happy to report that we have resolved the bug that we identified in the initial 4.3 release, and that the new and improved OpenCalais 4.3 is up and running on all servers.
 
As a quick reminder, here are the new features and expanded capabilitites of the OpenCalais service.  As always, please let us know if you run into any issues or have any questions.
 
New in OpenCalais 4.3
 
Improved ‘Social Tags’: We are expanding on our popular Social Tags categorization technique by adding more generalized, aggregate tags.
 
For example, if a blogger is comparing the racing performance of sports cars like the Ferrari 308 GTB and Porsche 959, OpenCalais 4.3 will suggest auto racing and motorsport as Social Tags, in addition to the more obvious sports cars.
 
NEW! ‘News Names’: We are instituting a process of name normalization that represents a first step toward our more robust vision for person disambiguation. Whenever a partial or extended name appears in content, OpenCalais 4.3 will return the names it finds as usual, but will now also suggest the most commonly used form of that same name.
 
For example, for articles containing Barack Obama, Obama or Barack Hussein Obama, OpenCalais will suggest not only the partial or extended name it found, but also the more frequently used Barack Obama.
 
New Entities, Facts and Events in English, including:
 

• New Natural and Manmade Disaster attributes that reveal these disasters’ effects
• Supporting data for upcoming events that will enable OpenCalais to recognize new Movies, Music Albums, etc., as well as anticipated Medical Treatments
• More Political Events and new items such as Diplomatic Relations, Political Endorsements, Poll Results and Voting Results
• Enhanced Person Career extraction that includes political party affiliations where those are included in the text.

 
The 4.3 release also features improved Simple Format and Microformat outputs, as well as several extraction bug fixes. For technical details, please see the full release notes here.

English, Official Blog

We released OpenCalais 4.3 a day or so ago - and we’ve run into a few issues as we’ve rolled it out into production. We think we have a handle on the fixes needed – but to play it safe we’re going to roll back to Release 4.2 for at least the weekend.
 
Sorry for any inconvenience – but we’d rather play it safe and take a day or two to get things totally in shape.
 
Tom

English, Official Blog

Big News!

To kick-start 2010, we have updated virtually every tool we have in the OpenCalais arsenal. Check them out and let us know your thoughts.

• Version 4.3 of the OpenCalais service features enhanced Social Tags for categorization; improved disambiguation; new entities, facts and events in English; and an entirely new feature called News Names.
• By popular demand, the improved Tagaroo plugin for WordPress gives users more control over the tagging process.
• Our URL submission tool, SemanticProxy, has been normalized to ensure that its results are identical with those you would get by programmatically submitting content to OpenCalais via the API. Additionally, we are now supporting JSONP. You can now specify a callback function when requesting results in JSON.
• Our Gnosis plugin for Firefox now includes categorization of processed Web pages in the sidebar and can sort entities by alphabetical order or according to frequency or relevance.
• And finally, we have published the OpenCalais Schema in OWL, enabling OWL enthusiasts to apply ontology based tools on OpenCalais metadata.

Below please find further details on OpenCalais 4.3, Tagaroo and SemanticProxy.

New in OpenCalais 4.3

Improved ‘Social Tags’: We are expanding on our popular Social Tags categorization technique by adding more generalized, aggregate tags.

For example, if a blogger is comparing the racing performance of sports cars like the Ferrari 308 GTB and Porsche 959, OpenCalais 4.3 will suggest auto racing and motorsport as Social Tags, in addition to the more obvious sports cars.

NEW! ‘News Names’: We are instituting a process of name normalization that represents a first step toward our more robust vision for person disambiguation. Whenever a partial or extended name appears in content, OpenCalais 4.3 will return the names it finds as usual, but will now also suggest the most commonly used form of that same name.

For example, for articles containing Barack Obama, Obama or Barack Hussein Obama, OpenCalais will suggest not only the partial or extended name it found, but also the more frequently used Barack Obama.

New Entities, Facts and Events in English, including:

• New Natural and Manmade Disaster attributes that reveal these disasters’ effects
• Supporting data for upcoming events that will enable OpenCalais to recognize new Movies, Music Albums, etc., as well as anticipated Medical Treatments
• More Political Events and new items such as Diplomatic Relations, Political Endorsements, Poll Results and Voting Results
• Enhanced Person Career extraction that includes political party affiliations where those are included in the text.

The 4.3 release also features improved Simple Format and Microformat outputs, as well as several extraction bug fixes. For technical details, please see the full release notes here.

Tip Top Tagaroo!

By popular demand, we have also made two key improvements to our Tagaroo plugin for WordPress.*

Tagaroo no longer suggests tags while you type (which many users had found disruptive). Instead, you simply click a button when you are ready to see tag suggestions, and then select the tags you want.

We also added the ability for you to select / highlight a portion of text in your post and get tag suggestions for that text alone, in the event that you don’t want to tag the whole post.

*Note: You must have a hosted site – or your own server – where you have access to install WordPress plugins. Blogs hosted on WordPress.com won’t work with Tagaroo.

SemanticProxy Support for JSONP

Finally, we have also made two important changes to our URL submission tool, SemanticProxy.com.

First, we normalized the service to ensure that the results it returns are identical to those you would get by programmatically submitting content to OpenCalais via the API. This entailed discontinuing the HMTL cleaning that SemanticProxy did on its own, which had occasionally caused the service to return false content-type errors.

Second, we added the ability for you to specify a callback function when requesting results in JSON format. Callbacks are useful for web service requests in client-side JavaScript and provide a relatively simple way to invoke web service requests across domains.

To use it, specify the callback function name followed by the JSONP output format. For example: http://service.semanticproxy.com/processurl/licenseid/jsonp:processresul...

The JSONP output format wraps resulting JSON output text in parentheses and the provided callback function name. For the request above, wrapped results would be: processresults({"doc":{"info":...json output...”} );

As JSON is native to JavaScript, you can access the elements inside the returned output, similar to passing a JSON object reference to the processresults function. In other words, if your JavaScript has a processresults function which takes a JSON object as input and manipulates it, this function will be invoked automatically.

Please see the attached demo html to see this in action.* 

*Note that the demo will not work in IE. Also, we have restricted the volume for this demo key.  If it does not resolve, please plugin your own OpenCalais API key.

Jump in!

Let us know what you think about these updates. We appreciate your feedback and look forward to any suggestions you may have.

English, Official Blog

Michael Fagan undertook a review of various entity and term extraction tools this past weekend.

While the use case (content types, goals, etc.) of the test are not clear, we are heartened to see folks beginning to look at the growing variety of tools and services in the space.

You can read the full posting - as well as a commentary from our Tom Tague - here: http://faganm.com/blog/2010/01/02/1009/ 

Tom's commentary is also excerpted below: 

Michael:

Tom Tague from OpenCalais here.

Wow - you covered a lot of territory here and I could probably spend at least as much space responding to some of the points you've raised - but I’ll try to stay focused on just a few key items.

First of course is the use case - which you don’t reveal. And of course it’s difficult to evaluate tools without the intended use well understood. Are you “tagging” news? Analyzing a large corpus of documents for network effects?

For example - if your use case is simply for entity extraction then the volume of entities is rarely the goal - but rather a mixture of recall and precision. You can have perfect recall and low precision - or the reverse. The goal is to find the appropriate balance. If your use case requires named (e.g. typed) entities, then tools such as Yahoo should not be in the mix - they are term and not entity extraction engines.

Facts & Events: I was also a little surprised that you stopped at entity/term extraction. Most real-world use cases want to understand what’s happening in the text and are heavily dependent on facts and events - an area in which OpenCalais shines. Facts and events reveal the relationships between entities, and make up the core elements of “aboutness,” which are key values / benefits that many use cases for semantic technology seek to derive.


Semantic Links: It appears you missed our connection to the Linked Data Cloud on your “Semantic Links” section.  For a growing number of entities that we return, we also return links to a rapidly growing set of Calais-provided, Thomson Reuters information assets that follow the semantic Linked Data standard.  These dynamically generated pages also provide relevant ‘sameAs’ links to key resources in the Linking Open Data cloud.

You can see these by entering the text of a news article into our demonstration tool at http://viewer.opencalais.com, copy and paste in a news story that features a number of company names, hit submit, and view the extracted entities, facts and events in the left hand rail.  Then expand on the companies, and click on one to find the Calais asset. (For instance, see the Bank of America asset here: http://d.opencalais.com/er/company/ralg-tr1r/e80e12df-622c-3c3e-86dc-a3ffdcc39e25.html (Traditionally, we have also included sameAs links to DBPedia and Freebase, and those will be back.  Right now we are adapting to a new format.)

RDF: While the general developer population may lack familiarity with RDF at this time, as you note you do, developers that work with large textual content sets are moving to learn it now.  While a variety of alternate representation ideas are available - RDF is the W3C standard and provides the right transport layer for rich knowledge representation. The text/simple format you chose is designed to support simple tagging / entity extraction use cases and leaves much of the richness extracted behind.

Length: OpenCalais supports entry of text documents up to 100K in length. We’ve found this supports the vast majority of our users well while conserving systems resources.

Usage: We welcome the use of OpenCalais for commercial or non-commercial purposes and allows users to submit up to 50,000 documents per day at no charge. After that we need to discuss some sort of value exchange.

We could probably name several other tools - but you absolutely should include Zemanta in any entity extraction test case.

Again - thanks for putting in the time to compare tools. I’d encourage you to come back and revisit the subject in the future.

Regards,

Tom Tague, OpenCalais Initiative Lead
(@TomTague)

English, Official Blog

Big news!

OpenCalais has been nominated by ReadWriteWeb as among the top Web products for 2009.

Please add your voice to the mix by voting in ReadWriteWeb's Top 10 Web Products competition, here: http://www.readwriteweb.com/archives/poll_your_favorite_web_products_of_2009.php

You can select up to 10 nominees to support. We humbly request your consideration of OpenCalais.

Many thanks,
-Krista

English, Official Blog

Based on some forum chatter, I realized that we haven't shared our latest publisher case studies - including The New Republic and Associated Content - here in the blog.  Rather, we posted them out on Slideshare and linked to them in the newsroom.

So I think it's worthwhile to also link to that PPT here - which we developed for the U.C. Berkeley Media Technology Summit at Google - and to detail below some of the ways that publishers are using the RDF that the OpenCalais service returns.

Please feel free to post any questions below and we will be happy to jump in.

Best,
-Krista

English, Official Blog

Want to learn more about our connection to the Linked Data Cloud?

Check out our simple, high-level whitepaper, and let us know if you have questions.

@OpenCalais on Twitter

English, Official Blog

The OpenCalais Partner page has launched, and - once again - we'd like to thank the innovative publishers and creative entrepreneurs who have embraced OpenCalais and made us part of their sites and services.

Not on the list, but think you should be? 

Email us at Partners (at) OpenCalais (dot) com with two sentences on your app, site or service, and send along a quality image of your logo. 

We'll be glad to add you to the list and include you in our next blog post or press release.

English, Official Blog

Add 10 to the list of innovative sites and services that use OpenCalais to reduce costs, deliver compelling content experiences and mine the social web for insight. See our press release for more details on each.

We are thrilled to recognize the following new sites and services that are changing the way we engage with news and the social Web. They join a growing number of others in media, publishing, blogging, and news aggregation who use OpenCalais.

The newest publishers, joining CBS Interactive / CNET, Huffington Post, DailyMe and others in using OpenCalais, include:

The New Republic – http://www.tnr.com – The new website uses OpenPublish, an OpenCalais-enabled Drupal-powered Content Management System (CMS) to increase editorial productivity, improve search engine optimization, and drive reader engagement, including faceted search, recommended reading sidebars and – coming soon – automatically generated topic hubs.

Al Jazeera English’s new blogging network – http://blogs.aljazeera.net/ – features Al Jazeera correspondents from around the world. All posts in the new blog network are semantically tagged using OpenCalais for optimal search and navigation. It also uses a Creative Commons license and allows users to sign-in to comment using Facebook Connect, Twitter or OpenID.

Slate Magazine’s News Dots Network –http://slatest.slate.com/features/news_dots/default.htm – News Dots visualizes the most recent topics in the news as a concise network of related topics. Like a human social network, the news tends to cluster around popular topics, and most stories are more closely related than one might think. Behind the scenes, News Dots scans all the articles from major publications—about 500 a day—and submits them to OpenCalais to identify the relevant people, places, companies, topics, etc.

I *heart* Sea – http://iheartsea.com/ – is a hyperlocal news aggregation site that collects some of the best blogs in Seattle, especially those serving the Capitol Hill area. I *heart* Sea uses OpenCalais to automatically tag the keywords of the blog posts in aggregates, to make it easier to find related information.

Innovative new media monitoring and intelligence tools using OpenCalais include:

Tattler (app) – http://tattlerapp.com/ – is an open source topic monitoring tool for today's Web. Tattler finds and aggregates content from the Web on topics users ask it to monitor. Using OpenCalais and other Semantic Web technologies, Tattler mines news, websites, blogs, multimedia sites, and other social media like Twitter, to find mentions of the issues most relevant to users’ selected topics, making it easy for users to filter, organize, share, and take action on content gathered from the real-time Web.

Interceder – http://www.interceder.net – is a social media monitoring tool that makes it easy to track trending topics and search through the latest content from major news Web sites, blogs, Twitter and YouTube.  Interceder uses the Daylife API, OpenCalais, Freebase, and Yahoo! Pipes to retrieve the latest news from major news websites.

AskJot – http://www.askjot.com – Ask Jot is a tool for analyzing web pages for keywords, and displaying them as links to search results from various services around the Web. Developed by John Wright of Wright Labs and formerly known as Semantalyzr, Ask Jot uses OpenCalais,  The New York Times article search API, DBPedia, the Yahoo! Answers API, the flickr API and many more.

New services using OpenCalais to deliver intelligent content experiences include:

Feedly – http://www.feedly.com –This Firefox plug-in brings to life user-selected inputs from Google Reader, friendfeed, Twitter, RSS feeds and more in an easy-to-read and engaging magazine-style format. Feedly uses OpenCalais and other semantic technologies for clustering, linking and organizing the content experience in an intuitive fashion that is nicely integrated into the browsing experience.

OpenPublish –http://www.opensourceopenminds.com/openpublish– Based on the popular open source publishing platform Drupal, OpenPublish is a next-generation CMS that has been tailored to the needs of today's online publishers (magazines, newspapers, journals, trade publications, broadcast and wire services). Developed by Phase2 Technology, it uses semantic metatagging from OpenCalais to streamline content operations, automatically create topic hubs and recommend related articles and archived ‘more from this author’ stories.

DocumentCloud – http://www.documentcloud.org – Founded by reporters from The New York Times and ProPublica, and funded by the Knight Foundation, DocumentCloud is a unique online resource that will offer public access to news reporters’ original source materials, including documents, media files and more. OpenCalais processes materials available through DocumentCloud to make it easy for users to explore connections between newsmakers, corporations, transactions and even quotations across documents and across the full collection of source information.

English, Official Blog, media, news aggregation, publishing

Good news for fans of OpenCalais and investigative journalism alike.

The DocumentCloud initiative – winner of this year’s largest grant from the John S. and James L. Knight Foundation – has lined up some two dozen partners, everyone from Thomson Reuters, The Wall Street Journal and The New York Times, to the ACLU National Security Project, The National Security Archive, the Center for Investigative Reporting and many more.

DocumentCloud is a unique online resource – found at http://www.documentcloud.org - that will provide public access to news reporters’ original source materials.  It will debut in a beta version by the end of this year.

Here are some of the stories that resulted from the news. Please help us share them with friends to let everyone know that this powerful tool for citizen journalism is on the way.

Nieman Journalism Lab: DocumentCloud adds impressive list of investigative journalism outfits.
http://www.niemanlab.org/2009/09/documentcloud-adds-impressive-list-of-investigative-journalism-outfits/

Journalism.co.uk: Thomson Reuters partners KNC winner DocumentCloud

http://www.journalism.co.uk/2/articles/535928.php

The Knight Foundation: More news from DocumentCloud

http://www.knightblog.org/more-news-from-documentcloud/

NYConvergence: Newspapers, mags, non-profits partner with DocumentCloud

http://www.journalism.co.uk/2/articles/535928.php

Editor’s Weblog: OpenCalais joins DocumentCloud; set to host a wealth of primary sources. http://www.editorsweblog.org/multimedia/2009/09/opencalais_joins_documentcloud_set_to_ho.php

The New York Observer: The Atlantic, The New Yorker, Mother Jones, WNYC, More Join Data Archive Experiment DocumentCloud.
http://www.observer.com/2009/media/atlantic-new-yorker-mother-jones-wnyc-more-join-data-archive-experiment-documentcloud

English, Official Blog

Here’s what’s up. Over the last several months you may have read about partnerships between OpenCalais and organizations like CNET/CBSi, The Huffington Post, Magus, Associated Content, and a variety of others. And we, of course, think those are great.

But, we realized we’re not doing a good enough job of sharing some of the basic ways that people are getting value out of OpenCalais. We have the great opportunity to talk to a wide range of individuals and organizations every week and we think about this stuff – a lot – ourselves. So – the purpose of this blog entry (and more to come) is to just simply start sharing our thoughts and what you can do with OpenCalais. We can’t share everything (NDA’s and all that) – but we can share a lot.

This post is going to cover two things: 1) What OpenCalais does at a ridiculously high level and 2) a few starter ideas for what you can do with it. Follow-on posts will cover additional ideas and we’ll try to keep them coming at a good clip.

What Does Calais Do?

Several things actually.

• It analyzes text you send it and extracts entities (people, organizations, geographies, etc.). In many cases, it links those entities to the world of Linked Data.
• It extracts facts – like the fact that John Doe is the CEO of Acme Corporation or such.
• It extracts events – like mergers, earning announcements, natural disasters and a bunch of others.
• It attaches a topic to the text as a whole, much like a newspaper would (Sports, Finance, Health, etc.).
• It creates SocialTags – our attempt to “tag” the article a way a human would to file it away somewhere.

There is a whole lot more going on in the background – but those are the basics.

Why is OpenCalais Unique?

• First, there are a lot of entity extraction tools out there. Some are good – some not so great.  Entity extraction is fine (if a little mundane) – but the real power of understanding text comes from understanding Facts and Events. Calais is the only tool that does that well.  It may be the only tool that does that at all.
• Second, it is high performance. We process millions and millions of transactions every day for a diverse set of users. On average they take less than 0.75 seconds to complete.
• Third, it’s free for up to 50,000 submissions per day for commercial or non-commercial purposes. If you need more or want an SLA we have commercial options and have deals in place with clients who process millions of transactions per day.
• Fourth, you can count on it being here. OpenCalais is provided by Thomson Reuters – the world’s largest content company. We’re not going anywhere – so you can feel confident building a business or solution on top of OpenCalais technology.

How Can I use OpenCalais?

We’re not going to try and structure this as use cases for different groups (publishers, bloggers, museums, etc.) – rather we’re just going to talk about the general types of things you can do with OpenCalais. You’ll have to do the translation to your situation on your own. We’re also not going to put out the complete technical recipe – just the general concept.  And – no smoke and mirrors – this is all stuff you can do today.

The first round…

Triage — A simple use that saves time and money. If you’re faced with a large influx of content (say press releases) and you have a staff reviewing them for material that might be relevant to you – OpenCalais can probably help. Send the document to OpenCalais, get back the metadata and apply a few business rules. If you only care about mergers and acquisitions, then filter for that type of event and throw the rest away. We have real-world cases where this has reduced the volume of material to be processed by 60% and improved the accuracy of results by 10%.

Workflow — Triage with a twist. Use the metadata returned by OpenCalais to route documents to the right person and/or system based on the facts and events inside the document.

Content Enhancement — There’s a whole world of Linked Data out there and OpenCalais can be your entry point. For example – take in press releases, and extract the companies mentioned in them. Use OpenCalais’ Linked Data entry points to get the SIC codes and the link to DBPedia. Access DBPedia and enhance your content with other information about the company like locations, people, products. Access Geonames to figure out what region the company is located in. Take that enhanced content and do cool things (like triage and workflow and presentation) with it.

Alerting — Give users the ability to be alerted when certain types of content becomes available. Unlike simple keyword alerting with OpenCalais + Linked Data you can construct alerts like, “Tell me when there is M&A activity for a company in the Steel industry.”

Media Monitoring — Whether you’re a media monitoring company or do it for your own company – it just got easier. Take in a content feed (social media, press releases, news), use OpenCalais to categorize and organize it, – put the results in a database and set some trigger levels.

Content Harmonization — Are you managing content from diverse sources? OpenCalais can take content from multiple sources (different news feeds, different museum collections, etc.) and apply a consistent set of metadata tags to all of it. Pop it in your content management system and you can treat it all as one harmonized content asset.

Automated News Portals — Want to create a general purpose news portal? Or maybe one that deals only with baseball news? Great. Subscribe to and/or acquire some content sources, and feed them through OpenCalais. Then use the metadata to throw away what you don’t care about and to organize the rest by topic, geography, person – whatever. A great example of an off-the-shelf solution that does this is OpenPublish.

Finer-Grained / Higher-Value Syndication — Do you have content consumers via RSS or other syndication methods? Give them a better experience by allowing them to create their own channels based on OpenCalais metadata. Create channels based on region, types of events, companies, etc. – or any combination of those and other items.

SEO — Something we get asked about all the time – we know people are experimenting – but they’re not being very public about their experimentation. Here’s a simple idea though: make your content more search friendly. Two routes: One easy, one a little harder.

Route 1: Translate events into human readable text and get it on your page. Have a complicated article about an LBO of company x by people y? OpenCalais will identify an M&A event. Take that event and turn it into a tag like “Acquisitions” – something people might actually search for. Don’t just use it as a metatag – incorporate into the page via navigation or whatever so Google pays attention.

Route 2: Use linked data to enhance your content. If you’re talking about a company or geography use OpenCalais Linked Data to enhance the page with additional information from Dbpedia, Geonames, CIA world fact book or a bunch of other sources.

New Presentation Metaphors — With consistent metadata extraction across your content you can implement new navigation and search tools. Two examples. The Powerhouse Museum (here’s an example) tags everything in their collection and shows them as search terms. Some interesting insights emerge. Second example: Slate Magazine processes the day’s news and creates a network diagram of what’s connected to what (here it is). Pretty interesting. Our recommendation here: unless your audience is researchers, start simple and expand – it’s easy to overwhelm the average user with novel graphics, etc.

Looking Ahead

That’s just a start. We’ll keep publishing this entry with new ideas as we get the time. There are about 15 additional ones on the list today that we haven’t covered here – and more will be coming in the next few weeks.

And remember – these are building blocks. Mix and match them to create something really cool, and let us know what you’re doing.

English, Official Blog

English, Official Blog

Hi all – just a little snippet of news today.

We’ve been busy working to keep up with the increased demand for Opencalais services. A lot of engineering going on in the background around scalability, performance and reliability.

The good news is that it is all paying off. Over the last three months we’ve achieved 100% uptime while improving performance and a delivering a number of functional improvements in the service - all while seeing growing utilization.

One result of our recent work is that the system is more efficient and scalable. Given our goal of connecting the world’s content – scalability is more than a little important.

And we’d like to share some of those improvements with the OpenCalais community. While we can’t offer everyone unlimited usage – we can turn the dial up a bit. So – effective immediately we’ve increased the daily transaction allowance for OpenCalais to 50,000 transactions per day – a 25% increase.

Of course, OpenCalais continues to be offered at no charge for commercial or non-commercial use. You don’t need to serve ads, embed links you don’t want or anything else. All we ask is that you share your stories of how you’re using OpenCalais with the world.

We’ve signed quite a number of agreements for OpenCalais Professional recently (same service, more transactions and an SLA). Based on what we’re hearing from our users we’re also working on some new pricing models for startups and (enormously) high volume media monitoring applications – more to come soon. If you’re interested in our Professional solutions please drop us a note at professional@opencalais.com and we’ll get back to you right away.
 

English, Official Blog

The Y Combinator's 'Future of Journalism' challenge presents a great opportunity for OpenCalais developers.  The seed-stage venture group has turned the conventional approach to journalism on its head, calling for sustainable online business ideas that can also support journalistic endeavors.

They ask: What would a content site look like if you started from how to make money—as print media once did—instead of taking a particular form of journalism as a given and treating how to make money from it as an afterthought?

Read more about it here, and remember how OpenCalais can help:

OpenCalais uses natural language processing (NLP) to “read” an article, extracting the ‘who, what, when, where and how’ from the story. Breaking content down into its basic elements makes it easier to manipulate – automating the creation of topic hubs and microsites – and improves its search relevance. With OpenCalais, publishers can:

• Automate: Automatically tag the entities, facts and events in content to increase its value.
• Enhance: Enrich content with open data from Wikipedia, Shopping.com, Geonames and more.
• Engage: Optimize the user experience, increase engagement and drive repeat visits.
• Extend: Increase reach to new search engines, aggregators, ‘related stories’ apps and more.
• Connect: Compete in tomorrow’s media ecosystem of enriched and interconnected content

English, Official Blog

We noted in our OpenCalais 4.1 press release that version 4.2 would follow swiftly, and here it is!  The upgrade, which began rolling out yesterday, is automatically available to all OpenCalais users - with no changes required on your part..

As Tom noted in his original 4.1 blog post, the 4.2 improvements include:

New Granularity in News Categories (4.2)

We’ve added a number of additional news categories. Our news topic categorization capabilities have been expanded and now map quite closely to 17 top levels of the IPTC Newscodes.

OpenCalais entidad extracción en español (4.2)

OpenCalais now supports entity extraction in Spanish. The detailed list of supported entity types are covered in the release notes.

Linked Data Breadth, Depth and Access (4.2)

We’ve significantly upgraded the Linked Data URIs for Company and the content is refreshed more frequently.

If you’re investigating Linked Data – particularly around Companies – you’ll find the changes especially useful. With one call to OpenCalais and a few HTTP fetches you can build a complete picture of a company, its industry, its competitors, its location and many other items.

We’re also exposing our Linked Data endpoints in a new format: JSON. In addition to HTML and RDF you can now retrieve companies, geographies and any other Linked Data URI as JSON by appending .json to the URI or calling us with an appropriate caller type.

English, Official Blog

The latest version of Tagaroo is available here.

Below are some of the new and important features of this new release

• Social tags are suggested as tags and appear at the top of the tags list in bold.
• Document categories are suggested as tags and appear at the top of the tags list.
• Many new entities and events are also suggested as tags. We have changed the descriptive text (what you see as a suggestion) for events, which unifies the suggested tags, so that, for example, 'Trial', 'Conviction' and 'Arrest' appear as 'Judicial Event' instead of as three separate tags.
• Entity tags are sorted by their relevance scores. You can control the 'Relevance Score' threshold from Tagaroo's settings, so that entity tags with a lesser relevance score do not appear as suggestions.

Check out the newly released Tagaroo and see how it can enrich your Wordpress blog.

English, Official Blog

OpenCalais Release 4.1 Available Today

The Gist:

• Introducing Social Tags – a knowledgebase-driven tagging solution
• Entity extraction now supported in English, French and Spanish
• Significant improvements to Linked Data depth and breadth
• Introducing the “Recession Pack” of topical fact and event extraction

Over the last several months we’ve been hard at work in the boiler room doing a fair amount of engineering work on OpenCalais. While not exciting – we’ve made significant improvements to the system’s reliability and scalability.

Now it’s time to release some user-visible enhancements. OpenCalais 4.1 is released today and Release 4.2 will follow in just two to three weeks. Unlike our last several releases, these will be rolled out as simple updates to the existing web service. If you’re not interested in any of the new features – no changes are required to your application.

Here’s what’s coming in Release 4.1 and 4.2:

Folksonomies, Ontologies, Vocabularies and Stuff (4.1)

OpenCalais is a great semantic data extraction engine. If you write an article about the relative merits of Porsche and BMW at the test track in Leipzig, we’ll diligently identify Porsche and BMW as companies and Leipzig as a geography. We’ll create Linked Data URIs to represent these things and open up access to the Linked Data ecosystem so you can enhance your article with other content assets.

But… sometimes you just want a great description. The kind of tags a human would put on the article. Like “Car racing” or “Automobiles”. The kind of tag that would, for example, be very searchable and therefore …. SEO’able (that is definitely is not a word).

In 4.1 we’re introducing OpenCalais Social Tags. Social Tags is our attempt to emulate how a human might tag the document. Social Tags does some fairly sophisticated analysis of your entire document and maps it to a knowledgebase based on Wikipedia and other assets. From that process we generate Social Tags.

We’d suggest you experiment with using them for content tagging and navigation – we’d also really like to see some experimentation around using the Social Tags as keywords for ad placement and meta-tags for HTML pages. Sounds like an opportunity for SEO and improved ad placement to us.

Because it’s a new approach Social Tags is going to require ongoing refining and tuning. You can expect some strange results in its first few months out in the real world. When you see them – we’d love to hear about them in our Forum.

New Granularity in News Categories (4.2)

We’ve added a number of additional news categories. Our news topic categorization capabilities have been expanded and now map quite closely to 17 top levels of the IPTC.

OpenCalais entidad extracción en español (4.2)

Beginning with release 4.2 OpenCalais will support entity extraction in Spanish. The detailed list of supported entity types are covered in the release notes.

Linked Data Breadth, Depth and Access (4.2)

We’ve significantly upgraded the Linked Data URIs for Company. The content is refreshed more frequently, company competitors are cross-linked to the appropriate OpenCalais Company URI and links to new information sources such as CrunchBase are now included. If you’re investigating Linked Data – particularly around Companies – you’ll find the changes especially useful. With one call to Opencalais and a few HTTP fetches you can build a complete picture of a company, its industry, its competitors, its location and many other items.

We’re also exposing our Linked Data endpoints in a new format: JSON. In addition to HTML and RDF you can now retrieve companies, geographies and any other Linked Data URI as JSON by appending .json to the URI or calling us with an appropriate caller type.

Opt-In Publishing of Document Metadata URIs (4.1)

By default OpenCalais stores all document level metadata as a Linked Data URI and makes is accessible via a (secret) identifier. This is useful if you want to share document metadata with someone else by providing them with a list of URIs rather than a massive file. Beginning with Release 4.1 this will shift from our default behavior to an opt-in function. No reason to make many millions of document-level URIs accessible if you don’t plan to share them later.

New and Enhanced Events and Facts (4.1)

Given the current…. environment many of our new events and facts focus on company performance and actions. We’ve added a wide range of event types including company accounting changes, labor issues, layoffs, earnings restatements, delayed filings and quite a few others. Go wild. We’re calling it the OpenCalais Recession Pack – and we hope it proves quite useless in the near future.

Bug Fixes

Yes, it’s true. We have had some bugs. Release 4.1 improves accuracy and reliability of a wide range of extractions and addresses a few specific processing errors we’ve discovered.

Wrap

Release 4.1 is a small but important milestone for OpenCalais. Over the past 18 months we’ve dramatically expanded the range and improved the quality of our entity and event extraction capabilities. We’ve achieved our goal of providing the highest-quality entity extraction toolkit available and done it at a scale that’s surprised even us. In addition to entity extraction we’ve invested in the heavy lifting necessary for sophisticated fact and event extraction – which sets the stage for a whole new class of significantly more sophisticated semantically enabled applications.

With the release of Social Tags we are taking this capability into a whole new arena. While our current Social Tags database is based on Wikipedia and other public assets, we don’t plan to stop there. Social Tags is a general-purpose solution that can apply to any knowledge domain. As we move forward we’ll be investigating opportunities to provide tagging in areas ranging from economics to environment to politics. Social Tags is just the first step down an exciting path.

Documentation for the new release has been updated and is located here. The release notes can be found here.

English, Official Blog

We’ll be releasing version 4.1 of OpenCalais on Monday 15 June (and Release 4.2 about three weeks later!). This release has a number of new features – but for the moment we want to focus on making certain it is backwards compatible with your existing applications.

Beginning now, version 4.1 is exposed for compatibility testing at the API access point of http://beta.opencalais.com. Just point your application to this address, make sure everything works and then point back to the default http://api.opencalais.com.

This beta will be available until the morning (EST) of June 15. At that point we’ll be rolling out the upgrade of OpenCalais across all of our server locations, and the beta servers will be retired.

What new in 4.1?  Well – that’s a secret until we release on June 15. At that time we’ll publish another blog post, updated documentation and provide release notes. A few hints: a new language, social content tagging and semantic metadata uniquely tuned to current events.

English, Official Blog

On Thursday, we announced that CBS Interactive / CNET is our first major media commercial partner for OpenCalais. You can see the press release here.

CNET is using OpenCalais for semantic analysis of its tech product reviews, award-winning news, and blog postings on consumer electronics and technology, where it will streamline CNET’s content operations, drive audience engagement, and further extend CNET’s reach across the Web.

In addition, CNET is joining us in publishing core data assets for public, programmatic use on the Linked Data cloud. CNET will allow its original content -- such as tech product reviews on laptops, TVs, smart phones, and digital cameras; news articles and blog posts from its CNET News editorial staff; and parts of its core technology product catalog - to be available for public use.

You can read more about CNET’s pioneering work in the ReadWriteWeb write-up.

You can also read how Huffington Post and DailyMe are using OpenCalais to create local microsites and offer greater personlalization, respectively.  DailyMe's Neil Budde offers more detail in this Now Possible story as well.

Let us know how you are using OpenCalais in time for SemTech 2009 (June 14 - 18) so that we can showcase your work!

English, Official Blog

Click here to find an excellent, professional report from PriceWaterhouseCoopers (the PWC Quarterly Technology Forecast) on the impact of Semantic Web and Linked Data – primarily with an enterprise vs. web focus.

It's well worth a read and sharing with others.

English, Official Blog

Interesting news yesterday from Google. Basically they’ve announced that their new Rich Snippets feature will harvest semantic metadata from web pages using Microformats or RDFa.

As is to be expected – Google is a little uncommunicative about whether they’ll harvest this from every web page that has it and exactly how they’ll use this information in displaying search results (or advertising) – it’s still an important development. It’s important because – if they do something interesting with the content – it will provide the incentive for several million websites to begin embedding semantic metadata – which we think is a good thing.

Some time ago OpenCalais released Marmoset – a tool designed to provide automated microformat generation for the Yahoo crawler. Marmoset sits quietly and waits for Yahoo to come along. When the crawler arrives Marmoset tags the page using OpenCalais and inserts microformats at the end of the page for a subset of the detected entities. It’s not a perfect solution – the microformats should really be implemented in-line with the content – but it does serve the purpose of feeding Yahoo the metadata at no additional coding cost to the content publisher.

Today we’re releasing an updated Marmoset that does the same thing for Google’s Rich Snippets. The tool and documentation are located here.

In the future we’ll look at more elegant solutions that embed RDFa directly in-line with the content. For the time being, Marmoset is a no-cost mechanism for jumping on the Google Rich Snippets platform.

Tom

English, Official Blog

Interesting news yesterday from Google. Basically they’ve announced that their new Rich Snippets feature will harvest semantic metadata from web pages using Microformats or RDFa.

As is to be expected – Google is a little uncommunicative about whether they’ll harvest this from every web page that has it and exactly how they’ll use this information in displaying search results (or advertising) – it’s still an important development. It’s important because – if they do something interesting with the content – it will provide the incentive for several million websites to begin embedding semantic metadata – which we think is a good thing.

Some time ago OpenCalais released Marmoset – a tool designed to provide automated microformat generation for the Yahoo crawler. Marmoset sits quietly and waits for Yahoo to come along. When the crawler arrives Marmoset tags the page using OpenCalais and inserts microformats at the end of the page for a subset of the detected entities. It’s not a perfect solution – the microformats should really be implemented in-line with the content – but it does serve the purpose of feeding Yahoo the metadata at no additional coding cost to the content publisher.

Today we’re releasing an updated Marmoset that does the same thing for Google’s Rich Snippets. The tool and documentation are located here.

In the future we’ll look at more elegant solutions that embed RDFa directly in-line with the content. For the time being, Marmoset is a no-cost mechanism for jumping on the Google Rich Snippets platform.

Tom

English, Official Blog

Just a quick note to say we hope to see you at one of two key industry conferences this spring.  Here's what we're up to in May and June:

1.) The Web 3.0 conference takes place in New York City at the New Yorker Hotel on May 19 & 20, 2009.

We will be offering a media industry and publisher-centric presentation on day one of the show.  We look forward to catching up with everyone on the east coast, and hearing about the semantic and Linked Data projects folks have been working on over the past year.

As the Web 3.0 conference producers note, the time is ripe for innovation:  "In turbulent economic times, it is critically important to understand what opportunities exist to make our businesses run better."

We'll look to offer some hands-on ideas for how OpenCalais can help you do just that.

2.) The 5th Annual Semantic Technology Conference - aka "SemTech 2009" - takes place at the Fairmont Hotel in downtown San Jose, June 14 -18, 2009.

We will help to kick-off this conference by sharing key insights, inspirations and ideas - as well as areas that demand greater innovation & collaboration - gleaned during OpenCalais' first year out. 

We'll also offer our first ever 'OpenCalais user track' at SemTech, which will start on Tuesday, June 16th at 2 p.m.  Here is the line-up as it stands today; the final version will be appearing on the SemTech site soon.

Let us know if you plan to be at either show - we'd love to meet face-to-face.

English, Official Blog

Developers - we've posted information about the Resolved Entity Analyzer in the Showcase. It analyzes entities and outputs in an xml file containing simplified data gathered from OpenCalais and Linked Data.

Check it out here.

English, Linked Data, Official Blog, RDF output, disambiguation

As you may have noticed we've had some intermittent stability and response time issues over the last few days. We believe we have the problem solved - or at least quarantined.

We have one particularily high volume user who submits a very wide range of content types to us. Due to some errors in the way they were using the API and some errors in the way we were handling errors (whew..) - we were seeing system utilizations that were off the chart.

We've moved that user to their own little quarantine island until we get things worked out with them. As soon as we did this - the remainder of our servers paused for a moment, took a deep breath, and then went back to almost idle - where they belong.

So - things are looking good. We'll continue to keep a close eye on the system and make sure things stay settled down.

We learned a few things about how to debug these types of errors and will be faster in the future.

Thanks all for your patience.

Tom

English, Official Blog

Last chance for testing. Effective early in the morning hours of 3/17 all calls going to Calais will be redirected to version 4 of the service. If you haven't tested your application with version 4 this would be a really really good time to do so.

Details are located here.

While we will be sympathetic if your tool, website, application breaks when we cut over - we won't be rolling back to Release 3.

English, Official Blog

English, Official Blog

We're going to keep this very simple and fast:

Write a web-accessible mashup using the new NYT newswire API @ http://bit.ly/11amDC and Calais. Do something cool and do it fast.

(Update: Use of the other new NYT News API's is also eligible)

Entries are due Friday 3/6. Send entries to team@opencalais.com with "NYT Mashup" in the title so we don't lose them.

Depending on the number of entries we may 1) puzzle over them that weekend and announce a winner), or 2) come up with some community voting mechanism (anybody know a good off-the-shelf option?).

U.S. Residents only (sorry - but the Kindle is not much use elsewhere anyway).

Will have to add a couple basic big company rules later - but they won't get in the way. Start building!

Tom

English, Official Blog

In preparation for the new 4.0 release of OpenCalais on March 15th, the gang at Phase2 Technology has done some massive upgrades on the Drupal Calais modules.

While many of these upgrades were under the hood, there are also some pretty great user accessible features to make you smile.

You can read all about it on Frank Febbraro's agile approach blog.

English, Official Blog