The story of NewsGator Go!

As has been blogged about, and formally announced, NewsGator Go! for Windows Mobile was released this morning.  If you’re using a Windows Mobile device, and you haven’t tried it yet, go download it now!  I’ll wait.

So before I get to the story, a little background about why we think mobile applications are important.  Skip down a little if you’re tired of hearing me talk about this. :-)

Many other mobile applications today are designed around consuming content.  But most of them act like they are the sole source of content, and you’re going to read it all there.  But that’s not how most of us actually work.  The phone or PDA is a companion device to our laptop or desktop, typically.

I remember back a while, when I got my first Windows Mobile phone that could actually synchronize over-the-air to Exchange Server.  No docking, no cables…just up-to-date email, synchronized with Exchange.  A second view of my same mailbox.

It changed my life.

I now have my email everywhere we go, and I’m notified within seconds when someone sends me a new message.  I can respond to it, file it, delete it, whatever I want – all while I’m standing in line at Starbucks. And when I get back to my desk, everything I did is already reflected in Outlook.  Sometimes I think I should get away from my email, and I’m a little too reachable…but that’s another story. ;-)

We’re doing the same thing for the rest of your content – be it blogs, news, or internal corporate data from your CRM system.  The content you want, on the device you have in your hand, wherever you are.  Laptop, desktop, internet cafe, phone, PDA – it doesn’t matter, your stuff is there waiting for you.

So onto the story.  Back in March, we acquired SmartRead and SmartFeed for Windows Mobile, and hired Kevin Cawley in the process.  We started morphing what was then called SmartRead into the application we wanted it to be.

It was a long process – we spent a bunch of time with the application, and identified exactly where we wanted to be in terms of user experience.  And then we went to work.

There were some MAJOR changes to what was originally SmartRead.  We changed the user experience dramatically – one of the requirements was smooth, intuitive, one-handed operation on a phone.  I think we hit that mark.  We also didn’t want the phone to “lock up” and be unresponsive while content was being downloaded – check.  Memory usage was a big problem – especially in devices like the Treo 700w, where Palm only left a tiny bit of memory for applications to use.  The list went on and on – but we weren’t going to ship before it was ready.

We even made some changes to the NewsGator Online API to more efficiently support the mobile applications.  These changes have made a dramatic difference in memory and bandwidth requirements on the client – and the sync process is MUCH faster than it used to be.

In the end, Kevin’s estimate is that the released application is about 10% original SmartRead code, and 90% new.  After six months of engineering, it’s quite a different product than it was!

NewsGator Desktop Sync for IE7 and Vista

Back at the MIX06 conference in Las Vegas, I demonstrated synchronization live and on-stage between Vista, IE7, and the NewsGator platform (including FeedDemon, NetNewsWire, and an early build of our windows mobile product).

Today, I’m proud to be able to point you to the first public beta of NewsGator Desktop Sync.  This will fully synchronize your NewsGator subscriptions (including our clients such as FeedDemon, NetNewsWire, etc, when running in sync mode) with Vista and/or IE7 and their included RSS platforms.

As I’ve said before, Vista and IE7 are going to have a big impact on syndication as we know it…and as promised, NewsGator is extending that experience beyond the desktop.  If you’re running IE7 RC1 or later, or Windows Vista RC1 or later, download it and give it a try!

Why I can’t try Picasa Web Albums

Almost a year ago, I reviewed a few photo hosting sites.  After that, but quite a while ago, I got an invitation to try out Picasa Web Albums, and I just now got around to clicking on the link to try it out.  But I didn’t get far…here is an excerpt from their Terms of Service (emphasis mine):

Google claims no ownership or control over any Content submitted, posted or displayed by you on or through Picasa Web Albums. You or a third party licensor, as appropriate, retain all patent, trademark and copyright to any Content you submit, post or display on or through Picasa Web Albums and you are responsible for protecting those rights, as appropriate. By submitting, posting or displaying Content on or through Picasa Web Albums, you grant Google a worldwide, non-exclusive, royalty-free license to reproduce, adapt, distribute and publish such Content through Picasa Web Albums, including RSS or other content feeds offered through Picasa Web Albums,  and other Google services.  In addition, by submitting, posting or displaying Content which is intended to be available to the general public, you grant Google a worldwide, non-exclusive, royalty-free license to reproduce, adapt, distribute and publish such Content for the purpose of displaying, distributing and promoting Google services. Google will discontinue this licensed use within a commercially reasonable period after such Content is removed from Picasa Web Albums. Google reserves the right to refuse to accept, post, display or transmit any Content in its sole discretion.

Hmm…this is a problem for me.

I have some images of people with whom I have a model release, and some of those releases cover self-promotional use only.  So I can (and do) put these images on my web sites (available to the general public) for the purpose of promoting myself and my photography, but I can’t allow someone else to use it to promote themselves or their service.  I own the copyright to these images – but I don’t have the usage rights required to do what Google is talking about.  If one of those images wound up on a billboard in Times Square, promoting some Google service, I’d be potentially liable for a lot of usage fees.

As another (possibly more common) example, I have some images from the San Diego Wild Animal Park.  On the back of my entry ticket, it said none of my photos taken there could be used for commercial purposes without express written permission from the park – which of course I don’t have.  So I again don’t have the usage rights to grant Google the license they are looking for for these images – promoting Google services is clearly commercial use.

One more example – do you have a photo that happens to have a recognizable face in it?  You know, like a passer-by?  You will, in general, need a model release from that person before you can use that image for (non-editorial) commercial purposes.  Or before you can grant Google the license to use it in this manner.

Images of the Eiffel Tower, that are taken at night when the lights are on, apparently require permission from someone before they can be sold.  I can’t remember where I read that…but if you search on, say, istockphoto, you’ll notice there are no Eiffel Tower pics for sale with the lights on at night.  Again – you can take the pictures, and you can put them on your web site, but no commercial use.  So you can’t upload those either, because you can’t grant the license necessary.

IMHO, there is a difference between a service that lets me share and distribute my content to others (like many other photo sharing sites), and a service that helps themselves to a pretty much unlimited license to my content (eek).  I did a search before posting, and was surprised to only notice one other blog that had mentioned this…

In Google’s defense, this probably isn’t the spirit of the agreement…but when I read the words, this is my interpretation – so I’m going to have to pass on trying out the service for now.

RSS/Atom security risks and FeedDemon

The blogosphere has been busy talking about the SPI Dynamics presentation at the Black Hat security event.  Some of the articles following the presentation have mentioned FeedDemon as a vulnerable application…this isn’t actually true.

Nick Bradbury goes into detail on his blog.

Bottom line – despite what’s being reported in the press, current versions of FeedDemon are not vulnerable to any of the serious exploits they reported.

NewsGator is hiring!

We have quite a few open positions…take a look, and if you’re interested, or you know someone else who might be interested, let me know or contact our HR folks directly from the linked pages below!  This is all going to sound a bit like a sales pitch – but we have lots of positions, and I’m excited about them all. :-)

All positions are in Denver, Colorado.  Local candidates are preferred, but we’d be willing to relocate under the right circumstances.  The web site is a few days out of date…but here are the positions we’re looking for at the moment:

Software engineer – NewsGator Online.  As I mentioned a while ago, we have a TON of cool stuff coming in our online system…and we’d like YOU to come help us build it.

Web UI designer – Private Label.  The activity in this business is exploding, and we need an awesome UI guy to help drive the state of the art here.

Systems Engineer – production operations.  Due to the incredible growth of our online platform (both our online application and API usage), we’re expanding our data center operations team significantly.

Network Engineer – production operations.  This one isn’t on the web site yet – but we need a kick-ass network engineer who is experienced not only in high-traffic highly available networks, but who also has experience with network architecture and design for multiple data centers.

Database Administrator – production operations.  If you’ve forgotten more about SQL Server 2005 than most SQL people have learned yet, and you’re interested in working on a highly available distributed database, we’re your shop.

Delphi software developer (contract).  FeedDemon and TopStyle are both written in Delphi, and we need an engineer to help out on both fronts with these applications.  FeedDemon is the most popular RSS desktop application for Windows – come help shape its future!

Software presales engineer – if you like to work with customers, and you can help drive sales for NewsGator Enterprise Server, this is your spot.  I can’t over-emphasize the potential here – this group is growing quickly.

The other positions on the web site (as of right now) have been filled.  BUT – if you’re an awesome software engineer, interested in working with cutting-edge technologies in a fast-moving industry, send us your resume!

NewsGator platform roadmap – Part II (a look forward)

A few days ago, I wrote the first part of this roadmap, which basically summarized what we’ve been up to over the last 16 months.  If you haven’t read it yet, go do it now – you’ll need it for background.

So now onto the good stuff…what’s coming?  What are we thinking?  Where are we going?

NewsGator Online

As I said 16 months ago, the proposed feature list is long and distinguished – and it still is.  There is so much to do here…some of the short-term planned additions range from more interactive feed discovery mechanisms (based on the larger community of users and their subscriptions), to completely different user interface paradigms (where a user could potentially select from different options, each catering to a different kind of user).

A larger initiative is around the whole paradigm. Techies aside, users don’t want to think about feeds, and subscriptions, and searching for content.  They want to be exposed to content that is relevant and interesting to them, without thinking about all this “RSS” and “feeds” techno-babble.  Disagree?  Go find 5 people you don’t know at happy hour, and ask them about RSS – and then come back and let us compare notes.

There are exceptions of course – we all want to subscribe to our friends’ feeds, or other specific feeds…but in general, I believe the above holds true.

Given all that, we’re really rethinking the way we present information to the user, and the way users discover new information.  We’re designing ways for people to participate in a larger community if they wish, and get more value out of the content they consume, at the point they discover it.  While we all have our own set of feeds, and we all participate to some extent in the larger ecosystem, there is a lot of potential in linking people with similar interests to each other.  Some users will continue to use our system as they always have – and others will use it in completely different ways.  We’re testing a couple of approaches on this right now – I think it’s truly a game-changer.

We’ll also continue to add support for more applications and devices – when we chat about this in the office, we refer to this as “NewsGator everywhere”.  This is pretty straightforward – wherever you are, whatever device or application you’re using, content from NewsGator (either pre-selected or selected by you) will be available to you.  A very recent example – NewsGator has a plug-in for the new Yahoo! Messenger beta.  I encourage you to try this out, and you’ll get an idea where we’re going with the NewsGator everywhere concept.

NewsGator Inbox, FeedDemon, NetNewsWire

As I mentioned before, the enthusiasm around these products has continued to grow – people obviously see the value in a rich, synchronized, offline-capable user experience for consuming content.  Moving forward, online integration will get tighter, and more complete – ranging from the low hanging fruit like FeedDemon “News Bins” becoming Clippings (and thus synchronize with the entire platform), to more involved features like analytics-related features (recommendations, interest-based surfacing, etc.) and community-related features.

These products will also continue to evolve and innovate in their own right; Nick and Brent, for example, are both pretty open about what they’re working on, and what’s coming – but the magnitude of the things we’re looking at now, from an overall user experience perspective, is pretty amazing to watch.  It’s fun to watch them continue to innovate as they always have, and cool to watch them start to take advantage of the resources available to them now.  As we come up with new ways to consume information, we can roll it out on desktop and online clients together in ways that haven’t been done before.

And just for a tidbit of news thrown in here, NetNewsWire Lite should be publicly available later today.  How’s that for a super-short-term roadmap item!  ;-)

And because all of these products are written using the online synchronization system (via the API), they work with our private label platform, and they can all be reconfigured to talk to an instance of NewsGator Enterprise Server.  More on that below.

NewsGator Mobile

As I mentioned, we released a beta of NewsGator Mobile for Windows mobile a few weeks ago.  In the next month or two, that will go gold, and we’ll also release a beta of NewsGator Mobile for java-enabled phones.  If you’re using a palm-based Treo, a Blackberry, or any of a number of other devices, you’ll get a similar rich experience as Windows mobile users.

Moving forward, these kinds of applications is where it’s at.  Truly delivering information at your fingertips – but this time, it’s not the information that MSN or Yahoo! has selected for you – it’s information you’ve subscribed to yourself, or information you’ve otherwise indicated an interest in.

What we’re focused on here is user experience.  Folks don’t spend hours reading on their mobile phones – the key is to provide the right information, at the right time, in a way that’s usable.  And this includes audio and video content as well – on certain devices, this can be a great experience, and is becoming more and more important, and we’re going to bring that to market.  One of the mobile applications we acquired along with SmartRead was a mobile podcasting client; expect to see that technology rolled into our mobile applications.

NewsGator core platform

This is the heart of our entire product line (with the exception of NewsGator Enterprise Server).  Moving forward, we’re investing a lot in the platform.  We’re building out more support for deep analytics (which we can use to deliver different kinds of user experience), and building out a much deeper metadata engine (which means if a client retrieves content from our system, they’ll get much richer data than they otherwise would).  We’ll have other ways to “slice” our data to get what you need, without having to subscribe to hundreds of feeds.

The API has been very successful, and we process millions of API calls per day from client applications, web services, and private label clients.  This traffic actually makes up a large percentage of our overall system traffic – which I think is a testament to the popularity and utility of the API.  Moving forward here, we’re obviously very committed to the API story, and we’ll continue to enhance it as we add platform capabilities.

NewsGator Enterprise Server

There’s so much activity going on here, I’m not sure I can even do it justice.  NGES 1.3 tightly (and optionally) integrates out of the box with Active Directory, Exchange Server, and Sharepoint.  Coming soon is integration with additional LDAP directory stores, and more out-of-the-box tools for portal integration (it’s possible now with the NGES API, but our customers are asking for turn-key examples).  More collaboration features are on their way – the “group clippings” feature in 1.3 was just the tip of the iceberg.  A continued focus on surfacing relevant data – both human and automated mechanisms are in place now, and we’re providing additional tools for both.  And as I mentioned before, it’s available in appliance or installable shrink-wrap form today…although we’ve found that shrink-wrap is usually preferred by our customers.

And in the next service release, the last tweaks will be in to allow the clients (FeedDemon, NetNewsWire, Inbox, mobile clients) to synchronize directly with NGES.  This is great stuff – our best-of-breed clients on every platform working with NGES, giving customers the choice they want.

We also will be adding a hosted version of NGES.  Right now, we have our existing NewsGator Online system for individuals and small groups, and we have NGES for larger groups (generally 100 to 100K users).  But for a small business group, say 5-100 users, we don’t really have a solution specifically targeted at them.  So we’re building a hosted business product, which will have similar capabilities to NGES, but delivered as a hosted solution.  We’ll see one version of this very shortly in Japan, and we’ll have a different hosted product for the US and European market later this year.

NewsGator Private Label Platform

This business has been fun.  We’ve been surprised at the cool stuff our customers come up with that they’d like from our private label platform, and they’re really helping guide this business to be exactly what the market needs.  One example – analytic information is clearly important – that is, surfacing information about a customer’s content that they didn’t necessarily know.  For example, if CNN publishes an article on their site, we have certain information about that article (readership, link relationships, ratings, etc.) that even CNN doesn’t have – and we can leverage that information into a cool and unique user experience for them.  It turns out that that kind of information is very valuable not just to CNN, but to other brands that want to foster a closer relationship with their user communities.

As we add more metadata and more relevance-related information, which will grow as a result of the way we’ll be presenting end-user information in our online systems, the information gets even more valuable for our private label clients…and content contributed by users can get a wider and wider distribution (when appropriate).

And as J.B. Holston (our CEO, for those that don’t know him) mentioned in a recent article, we’re working with selected PR and advertising agencies on using this data to provide a unique and compelling advertising paradigm.  Think of this as conversational advertising – in your user’s eyes, it’s not just about the product – it’s also about the company being an expert, and participating in the relevant conversation.  The private label products allow them to do just that…watch for some large and cool announcements about this shortly.

NewsGator platform roadmap – Part I (a look back)

About 16 months ago, I wrote NewsGator platform roadmap – where are we going?  It’s a little hard to believe it’s been that long…it seems like yesterday.  In any case, though, it’s about time to talk about this again, as we’ve come a long way in these last 16 months, and we’ve got a lot of exciting things planned. And it’s worth pointing out that everything I said we were going to do last time, we’ve done…so the roadmap track record is 100%. :-)

I’ve been working on this post for a while, and finally hit on the problem that’s kept me from finishing it sooner.  There has been so much going on these past 16 months, in so many different business lines, that most folks aren’t aware of all of it – and this makes talking about the roadmap difficult. So in this first part of the roadmap, I’m going to take a look back – and try to give you a baseline with which I can talk in part II about where we’re going.  Hopefully this won’t sound like a commercial – that’s not the intent; I just want to give you the background against which the forward-looking post will make more sense.

So let’s get started!

NewsGator Inbox, FeedDemon, NetNewsWire

What can I say about these products in the last year?  NewsGator Inbox got its shiny new name, and a completely revamped synchronization system where just about everything is synchronized.  FeedDemon and NetNewsWire joined the NewsGator family, and both have since had new releases (2.0 and 2.1 respectively) that add synchronization and other online-supported features.  It’s been a busy time for all of these apps…and frankly, I’ve been floored with the excitement level surrounding these products – it hasn’t let up even a little.

The synchronization is now tight and fast.  We’re on our third-generation sync platform, and it behaves exactly the way you think it would.  Read a post in FeedDemon, and it shows up read in NetNewsWire.  And mobile.  And the web.  And…well, you get it.  We’ve had a few hiccups with this system (including a fairly significant outage), but we think we’re past the worst of it.  And even with the occasional problem, the net gain for the user from this system is worth it, at least according to most of the feedback we get.

I’ve said many times that part of our product strategy is to have an awesome user experience on every platform.  People have asked why we didn’t just build a cross-platform client application in java or something, that would run on every platform.  Well, we’ve taken a different strategy.  Rather than take a lowest-common-denominator approach with a single application, we’ve instead chosen to have best-of-breed on every platform.  FeedDemon (for Windows), NetNewsWire (for Mac OS X), and Inbox (for Outlook) all provide this kick-ass experience in their respective environments, and each is the market leader in its segment.

NewsGator Mobile

Mobile is a big part of our product strategy – both for the consumer and enterprise markets.  I think RSS and mobile is a match made in heaven.  BUT – and this is a big but – I think a mobile RSS reader is useless unless it is synchronized with the other devices you use.  No one really wants to read all of their feeds on a mobile device; they want to read some of them there, and they want to be sure they don’t have to download something onto their phone that they just read in the office.

We’ve had our mobile HTML reader for some time, and it’s hugely popular.  It’s such a simple application, and yet so useful.  But HTML applications alone don’t really cover all of the use cases we want to hit.  What about rich functionality, like clipping, emailing, forwarding, and other per-article functions?  What about when you’re on a plane, with no wireless service, and you’d like to read your feeds?

Enter our new mobile applications.  The first is NewsGator Mobile for Windows Mobile devices – this is in beta right now, and delivers a rich and synchronized experience to Windows-based mobile devices.  It’s a great user experience – if you haven’t tried it, I’d encourage you to download the beta and give it a shot.

NewsGator core platform

At the heart of our online system is what we affectionately call the NewsGator core platform.  It needs a cool code name, I suppose…but even with a boring name, it’s evolved into an incredibly powerful beast. :-)  And growing very fast.

This platform provides content retrieval and archival, index/search, subscription management, synchronization (including incremental), analytics, and other base-level services.  These services are exposed for our own web applications (such as NewsGator Online), but are also exposed via web services API’s (SOAP and REST both).  These API’s are the same ones that our own client products use, including Inbox, FeedDemon, NetNewsWire, and our mobile products (more on those below).  Our private label platform also uses these API’s.

I talked about the synchronization system above…but one more thing to point out here.  It’s fast.  If you’re running FeedDemon, with several hundred feeds, it used to take a while to refresh all of those feeds, especially on a slow connection.  But with the latest version, which uses the new sync system, only the incremental updates have to be sent to the client…which means you can refresh hundreds of feeds in just a few seconds.

NewsGator Online

This is our online application, the one you see when you sign in at http://www.newsgator.com.  It’s growing at a huge rate – as we’ve added features, and added sync with client products into the core platform, users have been signing up en masse…it’s exciting to watch.

Most of what I have to say here is where we’re going – so I’ll save that for part II.

NewsGator Enterprise Server

My roadmap post 16 months ago was the first time we uttered its name publicly…but now NGES is shipping (actually in its fourth release, 1.3), and it’s getting some serious adoption in medium and large companies alike.  Enterprise RSS usage is skyrocketing – the use cases are clear, and and value proposition is compelling.

NGES is usually sold as a shrink-wrap product, to be installed on the customer’s servers.  It’s also available as an appliance – although interestingly, we don’t get a lot of demand for that.  A few smaller customers find the appliance idea appealing…but in the larger installations, and these are six-figure deals we’re talking about, we’ve found nearly all of our customers want to install on their own equipment.

NewsGator Private Label Platform

This was a sleeper – it came out of nowhere.  And now it’s a significant part of our business.

Publishers today, and other site operators, are very committed to doing what they need to do to drive additional traffic to their sites, and to increase customer loyalty.  The industry has been telling them “build RSS feeds, that’s the answer.”  This is great – but it’s a double edged sword…what happens is many folks consume their content in RSS-enabled consumer portal sites like My Yahoo! and others.

Additionally, there is a conversation happening, in the blogosphere or elsewhere, around the content produced by that publisher – but it’s not easy for them to find and surface this information in a way that’s useful for their readers.

These problems and others are what the NewsGator Private Label platform was built for.  Publishers (or anyone else, for that matter) can add anything from a complete RSS reading experience for their users, to exposing conversational information (or “hot topics” and the like).  They can either do this in a very custom manner, using the NewsGator platform API, or they can choose a turnkey solution and get up and running in literally days.  Examples of real customers of this platform include My USA Today, and My Newsweek – both are powered by the NewsGator private label platform, and they deliver completely different user experiences, designed by the companies themselves.

 

So now we should all be on the same page…in my next post, later this week, I’ll lay out the future for all of these product lines.

Grill the VC’s about syndication!

As some of you may know, I’m moderating a panel at the Syndicate conference next week.  Here are the session details:

A Look at the Horizon: A VC’s Perspective on Syndication
05/16/2006, 3:45 PM – 4:40 PM
Moderator:
Greg Reinacker, Founder & CTO, NewsGator Technologies, Inc.

Speakers:
Seth J. Levine, Principal, Mobius Venture Capital.
Richard Levandov, General Partner, Masthead Venture Partners.
Richard Fishman, Managing General Partner, RSS Investors.

Venture Capitalists have identified syndication technologies as a broad new area that’s ripe for investment. In this session, hear from the VCs that have invested the how’s, why’s and where’s of their interest.

What’s on your mind?  What questions would you like to hear the panel address?  Post a comment here, or post on your own blog and trackback me, and I’ll bring your questions to the conference and we’ll discuss them!  Here’s your chance to really get inside their heads, and ask the hard questions!

Anatomy of an outage

The last 12 hours will not go down in the books as NewsGator’s happiest hours…as posted here, there was an outage and NewsGator Online was down for several hours.  On a normal day, I’d be thrilled that “NewsGator” was the #1 search on Technorati; but not today. :-(

But while this was painful for us, it was painful for our users too.  So I’m going to walk through what actually happened, in quite some detail, and why it took so long to come back online.  And there are still a lot of questions here – our platform/operations folks are on the phone with EMC and Microsoft both as I type this post, trying to get to the root cause of all this.  But I’m getting ahead of myself.

Warning – lots of detail ahead.  Feel free to skip below if you don’t want to read all the details.

Monday 5:00pm

A query starts running against the database that contains user information, kicked off by an automated agent job. This query is run periodically, and is used for some reporting data.  It normally takes a minute or so to run…but this time, for some reason (I don’t have the details), it was going to block on something and take 300+ minutes.

Monday 6:00pm

SQL transaction log files are growing much more quickly than normal, due to the errant query.  It’s not yet noticed, however, as it’s still below the relatively high log monitoring thresholds we have set up.

Monday 7:00pm 

I’m sitting in the office with Darryl, who runs our platform team, and owns all of the non-GUI portions of the online platform.  We’re noticing large I/O bottlenecks on one file group in one of our databases, and some quick calculations indicate we are at basically 100% of the hardware capacity.  This has been consistent for the past week or so.

We use an EMC fiber-channel SAN for database storage.  The database file group that was running into problems is striped across 20 disk spindles.  We had a plan in place to increase the number of spindles to increase I/O capacity, which was going to happen later this week…but given the data at hand, we decided to schedule it for Monday night.  The immediate plan was to add 10 more disks to the stripe for that file group, bringing the total up to 30, and increasing I/O capacity for the group by 50%.  (For techies out there, we’re measuring I/O operations per second; physical disks have a maximum number of these operations they can handle per second, and the SAN caching increases the effective rate in some cases…but generally, the only way to really increase I/O capacity is to add drives).

So…we allocate the extra drives, and begin the re-striping process.  This happens transparently to the applications (i.e. SQL Server), as the SAN has the ability to re-stripe on the fly.  We’ve expanded storage a number of times, exactly like this, with no problems.  Total I/O capacity is impacted a bit during the process, but everything stays up and running.

Or at least it’s supposed to.

Monday 10:00pm

System starts throwing intermittent errors when applications try to connect to the database in question.  Monitor alerts are sent.  It’s pretty critical, as this database contains all user-related data (such as authentication info) for all system users.

Monday 10:30pm

Darryl discovers the errant query (the one that started at 5pm), and terminates it.  At this point the SQL transaction logs are well over 300GB.  SQL starts to recover, but intermittent errors still occur.

Tuesday 2:00am

SQL Server shuts down – cause is as of yet unknown, although there are some suspicious log entries prior to the failure.  It automatically restarts, and begins the recovery process to bring the database online, including replaying this huge 300GB log.  Other cluster members don’t take over, because the database requires recovery – as opposed to having a server problem.

However, while it’s trying to recover, a huge amount of traffic is still coming in from the web sites (mostly API and non-US sites at this time), which causes a large number of SQL transactions.  Some of these transactions deadlock with the recovery process, which causes SQL to terminate again and restart recovery.

At this point, every request coming into the system that hits that particular database fails.

Tuesday 2:30am

It’s clear by now that the only way to bring the database back online is to eliminate all database traffic.  “Maintenance” pages go up, all sites come down, and the database restarts its recovery process.

Tuesday 10:15am

SQL Server completes the recovery process and brings this database online.  Back-end services are turned back on in priority order, and sites are brought back online by 10:30am.

So there you have it. I’m not a database expert myself, so I may have some of these things not quite right, but this is more or less what happened.  The root cause, and which details along the way are actually relevant, is as of yet unclear.

The big question is why did SQL Server fail completely?  One possible cause would be if the SAN expansion caused some disk errors, and left the database in an inconsistent state.  So as you can imagine, we have tickets open with both EMC and Microsoft, and we will get to the bottom of this.  There are also other issues – such as the intermittent failures – that are cause for concern.

Frankly, this was a pretty frustrating experience.  We have a lot of redundant systems – pretty much any piece of hardware in our data center could fail, and we can absorb it without a significant outage.  For example, if an entire SQL box would have lost power, fallen on the floor, and broken into pieces, no problem, we’d have an approximately 10 second outage.  But this case, where the database gets into an inconsistent state, wasn’t helped by the redundant systems.

Moving forward, we’re doing a number of things. Specifically, we still need to get to the root cause of this particular problem, but in general, we’re working a lot on our back-end systems to build additional redundancy, scalability, performance, and resiliency.  We’re going to partition data in different ways, so a failure like this will only affect a portion of the user base, a portion of the content, or a portion of the functionality.  Our back-end cache servers will expand to be able to serve data while underlying data store is offline.  And lots more, which I won’t go into details about just now, since this post is getting a little long.

But for now – yep, we screwed up.  To our customers who were affected – you have my personal apology, and my assurance that we’re doing everything we can to ensure this doesn’t happen again.

NewsGator, IE7, Microsoft, and my first ‘fanboy’ accusation

I got an email today, which was sent to several aliases at newsgator.com.  I was typing up a response, when I noticed the same text of the note on a comment to the recent TalkCrunch podcast (which featured executives from four different RSS-related companies, including myself for NewsGator)…and given that the comment was made in public, I thought I would respond publicly as well.  Here’s the comment in its entirety:

John Pywtorak
April 5th, 2006 at 11:24 am

“Greg Reinacker, Founder and CTO, NewsGator” sounded like a Microsoft fanboy.

To suggest hat IE7 with RSS will be revolutionary and change the web is marketing speak and completely wrong. What about Firefox and other RSS readers? What about iTunes? I think it will have a longer and greater mpact than IE7.

From his discussion I can only come to the conclusion that your service is IE specific and that is the browser you will provide biased support for. I thought of using your service, but now I will not.

This is the first time I’m aware I’ve been called a fanboy. :-)  Here is my response:

John,

Sorry you feel that way.  Let me try to clarify what I was trying to say, and if you disagree, then we can agree to disagree.

I believe we’re in the middle (er, beginning) of a paradigm shift on the internet.  And that shift isn’t about RSS; it’s not about Atom.  It’s about subscribing to content.  Users will come to expect that when they visit a web page, there is a consistent way to subscribe to the content on that page.  If they can’t subscribe to some particular site’s content, they’ll consider it a bug.  They will likely visit their favorite search engines less often – and they will “cruise” less – because the content they care about is being delivered to them.

Microsoft is going to ship IE7 soon, and Windows Vista (including IE7) sometime after that.  Now here’s the thing.  I don’t think IE7 is the best experience for reading this content the user has subscribed to.  But – and this is a big but – it’s going to be the first exposure to RSS and subscriptions that millions of people see.

Tidbit – Microsoft said at one point they plan to ship 475 million units of Vista in the first 24 months.  That’s a lot.  If I add up the total number of people in the world that have ever heard of RSS, I bet it’s a lot less than 475 million.

Anyway, back to my point.  People aren’t going to look for subscribe buttons on a site any more; they’re going to look for “subscribe” buttons in the browser or client they’re using.  That browser might be Firefox, it might be Safari, it might be IE7.  But out of those three, IE is probably going to have the most market share.  That’s not fanboy talk – that’s real life.

So back to NewsGator.  We’ve announced that we will have synchronization support for the RSS platform that’s contained in IE7 and Vista – I demonstrated it personally on stage at MIX06.  This means people can use the cool auto-discovery and subscribe tools in the browser, and they can use as much or as little of the IE7 reading experience as they wish.  We’ll synchronize all of that information to either NewsGator Online, or their internal NewsGator Enterprise Server installation.  Once there, they can use a _different_ Vista/IE7 machine to read it, or use their mobile device, or use a rich desktop client like FeedDemon or NetNewsWire, or use a web client from a shared computer, or…you get the point.  We take the RSS experience and extend it off of the user’s single machine, while at the same time leveraging the strengths of IE7.

Does this mean we only support IE?  Does this mean we don’t love Firefox, Safari, and other browsers?  No!  Our system supports whatever browser you want to use.  Where possible, we integrate with the subscription and auto-discovery mechanisms in all of these browsers to varying degrees.  We’re not “biased” in any way – other than we devote resources to the platforms that our customers ask for.

So anyway, John, and everyone else reading this – I’m happy to continue this discussion with you if you’d like.  Just leave a comment below, write on your own blog, or you can even call me – 303-552-3900 – and we can chat.