The new StackExchange Leaderboards are now live! Any site that has turned on the setting to be shown in the directory is now being tracked and ranked.

On the main page we show the top 4 sites in each of four categories.  Our goal with these categories was to select a mix of high traffic, small-but-active, and fast-growing sites.

The full list page shows the full list of sites, and lets you sort by any of the columns.  It also adds a few extra columns that didn’t quite make the main page but still have some interesting data.

We have a lot of flexibility in what stats we track so we’ll probably be tweaking them down the road.  If you have suggestions for information you’d like to see go ahead and post them on meta.stackexchange.com.  We also have lots of ideas for expanding it into a full directory of StackExchange sites, but we figured we’d get the leader boards out for now and see where things go.

There are a lot of sites still missing!  If you’re an admin and you don’t see your site, make sure you’ve filled out the StackExchange Directory settings under Admin | Content.  We recalculate the stats every night, so you should show up some time tomorrow!

We’re doing a small release on Monday morning to fix a handful of things.  As with Beta 7, admins will receive a notice letting them know when they’ve been upgraded.

StackExchange Directory Opt-In

We’re working on developing a new section on StackExchange.com that will eventually be a directory of StackExchange sites complete with interesting statistics and a leaderboard of the top sites in several categories.

Because we don’t want to expose information without the account owner’s permission, we’re adding some new settings under Admin > Content that allow admins to decide whether and how their site will appear.

Usage Information

A new page under admin > usage will show administrators their daily and monthly pageview totals for billing purposes.  For more information see the pricing information on our website.

Bug Fixes

We’re sneaking in a few bug fixes that didn’t quite make it into Beta 7 or were introduced in Beta 7:

• Gmail (and other email clients) will again correctly show HTML emails
• URL entities (such as ‘+’) will no longer be decoded in CSS
• Improved support for Polish keyboards (thanks Tomasz)
• Hide user’s preference tab when viewing another user as a moderator since it was showing the wrong information

Beta 7 is finally ready to go.  We’ll upgrade 25% of sites Monday morning then the rest on Tuesday if all goes well.  If you’re an administrator on a site you’ll get a notification message on the site letting you know when you’ve been upgraded.

Database Export

The main feature introduced in Beta 7 is the ability to export your database to XML.  Administrators can create an export at most once every 24 hours by going to the account tab under the admin section.  We’ll queue up the export and then send you an email when it’s ready to download it.

Exports should ordinarily take around 1-2 minutes, but we expect there might be a large volume of requests when the feature is first released, so please be patient early in the week.

The exports are similar in format to the Stack Overflow data dumps, a zip archive of XML files corresponding to database tables.  Unlike the Stack Overflow dumps, however, they include information that is normally private or only available to administrators, so please be careful what you do with them.  The only data we don’t include are the password hashes and some session and authentication information.

For any administrators who want to create anonymized data dumps like the Stack Overflow data dumps, it should be pretty straightforward to compare your data to the Stack Overflow dump and see how they are anonymizing it.

New Customization Options

We’ve added a few more things that administrators can customize.

• Admins can now customize the home page Title and Description for SEO
• Admins can now customize the email from and reply-to addresses
• Admins can now make the “banner low” ad spot show up for users with less than 200 reputation and/or for questions with less than 2 answers.

Badge Changes

We updated to the latest badge behavior from Stack Overflow.

• Fixed Citizen Patrol, Yearling, Popular Question, Notable Question, Famous Question, Disciplined, Peer Pressure, and Fanatic badges
• Disabled Tumbleweed and Generalist badges
• Added Pundit, Reversal, Mortarboard, Epic, and Legendary badges
• All badges will be awarded retrospectively

User Throttling / Reputation Changes

• Administrators, moderators, and users with more than 20 reputation are no longer limited in the number of questions and answers they can post
• Unregistered users can again create tags in bootstrap mode
• Unregistered users can now vote while in bootstrap mode (note: this applies to unregistered users, not anonymous users)
• Raised new user link limit in posts to 5
• Users will no longer see the captcha before being told they can’t post
• New users can now post unlimited links and images while in bootstrap mode
• Users are again considered new when they have less than 10 reputation (was incorrectly set to 19 in Beta 6)

CSS / HTML Customization Fixes

• Sidebar modules now have appropriate HTML ids so they can be styled
• The HTML structure of the topbar has been modified slightly to make it easier to customize
• Removed inline styles from “wide” sidebar elements to make them easier to customize

Other Fixes

• The Google MediaBot crawler should no longer be blocked
• Deleting a former mod/admin no longer crashes
• We now attempt to convert non-ASCII characters in tags instead of stripping them.  This is a partial solution, not full unicode support in tags.
• We pulled in the latest version of the markdown editor from Stack Overflow, which should fix some minor markdown bugs
• System messages will now correctly appear to all users immediately (they were being cached incorrectly across our web servers)
• The “register” link will no longer appear for administrators viewing an unregistered users’ profile
• Removed reference to opensearch.xml, since it isn’t implemented for StackExchange

Testing the Changes

We have not yet added a way to preview changes on your site before they go live, but as of today all new sites are being created on Beta 7 so you can create a new account and test against that.

In particular, if you have made modifications to the markdown editor or extensive changes to the top bar, you should check that your modifications still work.  We made an effort to make sure that the changes would be backwards compatible, but we can never be completely sure.

The End of Beta is Near

With the release of Beta 7, we’ve finished all major feature development planned for the beta.  We’ll do at least one more release with additional bug fixes and to add the account usage data and billing information, and then we will end the beta and convert all existing beta accounts to their free, 45-day trial.

Feedback

As always, questions and comments are welcome on meta.stackexchange.com.

We experienced a brief outage this morning between around 10:44 and 11:05 EST.  More information can be found on the Fog Creek Network Status Blog.

Everything should be back up and running normally now.

We’ll be rolling out fixes for a few bugs introduced in Beta 6 tomorrow:

• Unregistered users will no longer see “That email address is in use” when attempting to register with a password
• Changing email address will now correctly update the email you log in with
• Can now easily add an email/password login to an account that registered with OpenID and vice-versa
• Modifications to OpenID CSS on login page now work as expected
• Publishing primary domain will no longer fail with a warning about the confirmation token

As with Beta 6, we’ll roll out to a random 25% of sites on Tuesday and the rest of the sites on Wednesday.  There is still no notification letting you know you’ve been upgraded, but we’re working on it.

Update

As of 9:21 AM EST on Wednesday, all sites have been upgraded to the latest version.

Beta 6 is finally ready to be deployed again after the deploy and rollback last week.  We’ll deploy it to the first 25% of sites early on Monday (2/1), and if all goes well we’ll roll out to the rest on Tuesday (2/2).

What’s in Beta 6?

See the original post What’s new in Beta 6 for most of the release notes.

After the first aborted deploy, we made a lot of performance improvements.  We also removed the ill-conceived Bootstrap Mode banner, and made some minor tweaks to sign-up.

Note that from now on we’ll be posting release information here on the blog, so if you’d like to know when there’s an update you should subscribe.

What took so long?

After last week’s outage when deploying, we wanted to make absolutely certain that Beta 6 performed well enough.  We spent a lot of time optimizing the performance and creating tests to measure the performance under load to make sure that everything would go smoothly.

What’s next?

Beta 7 is already in the works.  Our newest developer, Emmett Nicholas, was able to keep plugging away at database export while the rest of the team was figuring out how to re-deploy Beta 6.  We hope to have that and a few other fixes out within a few weeks.

Feedback

Questions? Comments? Bring them over to meta.stackexchange.com.  Now that the crisis is past the team should be able to spend more time answering your questions.

On January 20th, we experienced a partial outage between 12:31 PM and 2:55 PM EST shortly after upgrading all sites to Beta 6.

The root cause of the problem turned out to be changes made in Beta 6 which resulted in a higher CPU load on our servers.  When we reached our peak hours (12:00 - 2:00 EST) the CPU on one of the web servers reached 100% and requests routed to that server started timing out.

We ultimately restored service by rolling all sites back to Beta 5.

What went wrong?

1) Poor regression testing

We should have caught the performance regression before we pushed the beta out to everyone.  Our existing tests didn’t catch the regression because we were measuring total page load time and not measuring the actual processing time on the server.  Beta 6 included changes that improved total page load time*, which masked the regressions in processing time.

Fix: We have created regression tests to monitor this

* Combining CSS into a single request and turning on content compression both improved page load time while not appreciably affecting processing time

2) We should have rolled back sooner

We found out about the problem almost immediately thanks to an alert watching for timeouts, but wasted valuable time trying to fix the problem and do an emergency deploy.  We rolled back late only as a last resort.  If we had rolled back immediately the outage would have been much shorter.

Fix: We will ensure that we are always backwards compatible to the last stable release, and do a rollback at the first sign of problems

3) We rolled out to 100% of sites too quickly

When we rolled out Beta 6, we went directly from a few testing sites to 100% of existing sites.  If we had instead rolled out to a smaller percentage of sites, we might have noticed the increased CPU load without crippling the servers.

Fix: Future rollouts will be done progressively over a day or a few days (depending on the magnitude of the changes), rolling out to increasing percentages of sites until we reach 100%

4) Insufficient extra capacity in our web servers

Since we launched the beta, we have grown much faster than expected.  We were getting close to needing a new server and this outage convinced us to add it now rather than in a few weeks or a month so that we have additional head room.  Having that additional capacity might have kept the sites functioning until we rolled back.

Fix: We are adding an additional web server

Other Issues

While attempting to fix the load problems, we reset the load balancer instance.  This is something normally done only during off-peak maintenance hours.  This time, resetting the instance resulted in the load balancer rejecting all redirects which made it impossible to log in (among other things).  Resetting the load balancer again fixed the problem.  We do not know what caused this.

Fix: We have set up a monitor to alert us immediately if this happens again

Conclusion

The StackExchange team would like to apologize for any problems that this caused, and thank you for your continued patience and support during the beta.

If you have any questions, please post them at meta.stackexchange.com.

Welcome to the new StackExchange blog!

Check back or subscribe here to find out about upcoming releases and features, and any other updates we might have about StackExchange.

If you have any questions or comments, please leave it at meta.stackexchange.com.