site title

Topic: cc-wiki-dump

Stack Exchange Creative Commons data now hosted by the Internet Archive

01-23-14 by David Fullerton. 21 comments

We’ve been publishing an anonymized dump of all user-contributed Stack Exchange content since 2009. Unfortunately, at the end of last year our former host, ClearBits, permanently shut down. So we set out to look for a new home for our data dumps, and today we’re happy to announce that the Internet Archive has agreed to host them:

The Stack Exchange Data Dump at the Internet Archive

We’ve been big fans of the Internet Archive for a long time, and we’re really happy to be working with them on this.

Wait, what’s this data dump?

All community-contributed content on Stack Exchange is licensed under the Creative Commons BY-SA 3.0 license. As part of our commitment to that, we release a quarterly dump of all user-contributed data (after carefully sanitizing it to protect user private data, of course).

Each site can be downloaded individually, and includes an archive with Posts, Users, Votes, Comments, Badges, PostHistory, and PostLinks (new). You’re free (and encouraged!) to share, remix, analyze and build on top of this data any way you want, as long as you follow the attribution requirements.

What are the attribution requirements?

In keeping with the spirit of sharing and proper attribution, and as the “attribution” part of the CC BY-SA license, we require that you do the following when you use the data:

  1. Visually indicate that the content is from the Stack Exchange network

  2. Link back to the original source question or answer

  3. Display the author names for each question and answer you show

  4. Link back to the author’s user page

Those links should be ordinary hyperlinks directly to the Stack Exchange site, without “nofollow” or any obfuscation or redirection tricks plainly visible on the page (we’re looking at you, content farms).

I’m too lazy to download this giant zip file. Can’t I just play with it online?

You’re in luck! We also make the data available through the Stack Exchange Data Explorer (an open-source project maintained by community member Tim Stone) which lets you run SQL queries directly against a copy of the data. It’s updated weekly, and includes some data that’s not in the data dumps in order to keep the size of the downloads reasonable.

If you want to access the data programmatically, we also have a pretty expansive JSON API that returns similar data to the dumps (but updated in real time with the websites). If you need help, we have a whole site for people developing apps on top of the API.

So take our data for a spin! We love seeing what people create with it, from apps to research papers or even machine learning contests. Making this data easily accessible is just our way of giving back to the community that has made Stack Exchange so successful.

Stack Exchange Data Explorer 2.0

01-15-12 by Sam Saffron. 4 comments

It has been a year and a half since we launched Data Explorer. In the past few months Tim Stone (on a community grant) and I have pushed a major round of changes. Thanks Tim!

Recap on last years changes

Since we publicly launched data explorer, the most notable change contributed back from the community was support for query plans, big thanks to Justin for submitting the patch.

We also added quite a few bug fixes/features, mostly around merging users. Some features were added to defend against an onslaught of public queries. A few features were added to support non Stack Exchange data dumps, most notably a system for white listing. Our very own Rebecca Chernoff ported Data Explorer to ASP.NET MVC3 amongst many other fixes.

The current round of changes offers some very cool new functionality, which is worth listing:

Query revisions

When we created Data Explorer there was no way to track a query’s “lineage”. This was particularly problematic because we had no way of updating featured queries or shared queries. Even I complained about this on meta.

The new pipeline works just like Gist, you can track the history of your query as you are editing (attributing the various editors along the way):

You can link to a specific revision, or simply share a “query set” by using the permalink. By sharing a “query set” you can later on fix up any issues the query has, without needing to update the link. The new pipeline allows you to “fork” any query created by other users and tracks attribution along the way.


We added some basic graphing facilities, supporting 2 types of line graphs:

The first type is a simple graph, where the first column represents the X-axis and the other columns the data points. For example: a graph of questions and answers per month.

The second type is a bit trickier, it unpivots the second column in the result set. For example: a graph of questions per tag for top 10 tags.

Huge open source upgrade

Data Explorer consumes a fair amount of open source libraries. In the past year and a half many have evolved. We took the time to upgrade them all.

The excellent Code Mirror was updated to the 2.0 version, the new version no longer uses messy iframes. Marjin wrote a great post explaining the changes, a fantastic read for any JavaScript developers.

SlickGrid, which in my opinion is the best grid control built on jQuery, was upgraded to version infinity.

100% more Dapper

Dapper our open source micro ORM is the only ORM Data Explorer uses. We took the time to port the entire solution to Dapper. I even added a few CRUD helpers so you are not stuck hard coding INSERT and UPDATE statements everywhere.

Data Explorer is a good open source example of how we code web sites at Stack Overflow. It is built on our stack using many of our helpers. Dapper and related helpers are used for data access. It uses the same homebrew migration system we use in production and an interesting asset packaging system I wrote (for the record, Ben wrote a much more awesome one that we use in production, lobby him to get it blogged). It also uses MiniProfiler for profiling. MiniProfiler is even enabled in production, so go have a play.

Lots of smaller less notable fixes

  • We now have a concept of “user preferences”, so we can remember which tab you selected, etc.
  • We remember the page you were at and try to redirect you there after you log on.
  • We attribute the query properly to the creator / editor from the query show page.
  • You can page through your queries on your user page.
  • Support for arbitrary hyperlinks
  • Revamped object browser, you can collapse table definitions
  • Lots of other stuff I forgot :)

You too can run Data Explorer

At Stack Exchange we run 3 different instances of Data Explorer. We have the public Data Explorer and a couple of private instances we use to explore other data sets. The first private instance is used for raw site database access. The other is used to browse through our haproxy logs.

There is nothing forcing you to point Data Explorer at a Stack Exchange data dump, the vast majority of the features work fine pointed at an arbitrary database.

Hope you enjoy this round of changes.

If there are any bugs or feature requests please post them to Meta Stack Overflow. Data Explorer is open source, patches welcome.

Creative Commons Data Dump Sep ’11

09-19-11 by Jeff Atwood. 8 comments

While we will always continue to produce Stack Exchange creative commons data dumps, we are moving to a quarterly schedule for all future dumps. We won’t be blogging each and every one, as it gets a bit monotonous. Please subscribe to our Clear Bits creator feed to be automatically notified when new Stack Exchange creative commons data dumps become available!

The latest version of the Stack Exchange Creative Commons Data Dump is now available. This reflects all public data in …

… up to September 2011.

Download from ClearBits

This month’s Stack Exchange data dump, as always, is hosted at ClearBits! You can subscribe via RSS to be notified every time a new dump is available.

If you’d prefer not to download the torrent and would rather play with the most recent public data in your web browser right now, check out our open source Stack Exchange Data Explorer.

Have fun remixing and reusing; all we ask is for proper attribution.

Creative Commons Data Dump Jan ’11

01-15-11 by Jeff Atwood. 8 comments

IMPORTANT: This torrent was originally uploaded incomplete. Our apologies. If you downloaded it before ~ 8 pm Pacific on January 16th, 2011, you should re-download it now. The correct size is > 3 GB; anything smaller is incorrect.

The latest version of the Stack Exchange Creative Commons Data Dump is now available. This reflects all public data in …

… up to Jan 2011.

Download from ClearBits

This month’s Stack Exchange data dump, as always, is hosted at ClearBits! You can subscribe via RSS to be notified every time a new dump is available.

Please read, this is not the usual yadda yadda! Three things:

  1. Because the dumps are quite a bit of work for us, we’re moving to a tri-monthly schedule instead of monthly. Meaning, you can expect dumps every three months instead of every month. If you have an urgent need for more timely data than this, contact us directly, or use the Stack Exchange Data Explorer, which will continue to be updated monthly.
  2. The attribution rules have changed to forbid JavaScript generated attribution links.
  3. As of November 2010, we enhanced the format of the data dump to include more requested fields, full revision history, and many other pending meta requests tagged [data-dump]. That’s why the dump is so much larger, but we did break it out in individual files per site within the torrent, so you can download just the files you need.

If you’d prefer not to download the torrent and would rather play with this month’s data dump in your web browser right now, check out our open source Stack Exchange Data Explorer. Please note that it may take a few days for the SEDE to be updated with the latest dump.

Have fun remixing and reusing; all we ask is for proper attribution.

Re-Launching Stack Exchange Data Explorer

12-13-10 by Jeff Atwood. 22 comments

Since we launched the Stack Exchange Data Explorer in June, we’ve been actively maintaining it and making small improvements to it. But there is one big change — as of today, the site has permanently moved from to

If you’re wondering what the heck this thing is, do read the introductory blog post, but in summary:

Stack Exchange Data Explorer is a web tool for sharing, querying, and analyzing the Creative Commons data from every website in the Stack Exchange network. It’s also useful as for learning SQL and sharing SQL queries as a ‘reference database’.

We are redirecting all old links to the new path, so everything should work as before. Why did we make this change?

Mostly because we decided to move off the Windows Azure platform. While Microsoft generously offered us free Azure hosting in exchange for odata support and a small “runs on Azure” logo in the footer, it ultimately did not offer the level of control that we needed. I’ll let Sam Saffron, the principal developer of SEDE, explain:

Teething issues

When we first started working with Azure, tooling was very rough. Tooling for Visual Studio and .NET 4.0 support only appeared a month after we started development. Remote access to Azure instances was only granted a few weeks ago together with the ability to run non-user processes.

There are still plenty of teething issues left, for example: on the SQL Azure side we can’t run cross database queries, add full-text indexes or backup our dbs using the BACKUP command. I am sure these will eventually be worked out. There’s also the 30 minute deploy cycle. Found a typo on the website? Correcting it is going to take 30 minutes, minimum.

Due to many of these teething issues, debugging problems with our Azure instances quickly became a nightmare. I spent days trying to work out why we were having uptime issues, which since have been mostly sorted.

It is important to note that these issues are by no means specific to Azure; similar teething issues affect other Platform-As-A-Service providers such as Google App Engine and Heroku. When you are using a PAAS you are giving up a lot of control to the service provider. The service provider chooses which applications you can run and imposes a series of restrictions.

The life cycle of a data dump

Whenever there is a new data dump, I would log on to my Rackspace instance, download the data dump, decompress it, rename a bunch of folders, run my database importer, and wait an hour for it to load. If there were any new sites, I would open up a SQL window and hack that into the DB. This process was time consuming and fairly tricky to automate. It could be automated, but it would require lots of work from our side.

Now that we migrated to servers we control, the process is almost simple — all we do is select a bunch of data from export views (containing public data) and insert them into a fresh DB. We are not stuck coordinating work between 4 machines across 3 different geographical locations.

Did I mention we are control freaks?

At Stack Overflow we take pride in our servers. We spend weeks tweaking our hardware and software to ensure we get the best performance and in turn you, the end user, get the most awesome experience.

It was disorienting moving to a platform where we had no idea what kind of hardware was running our app. Giving up control of basic tools and processes we use to tune our environment was extremely painful.

We thank Microsoft for letting us try out Azure; based on our experience, we’ve given them a bunch of hopefully constructive feedback. In the long run, we think a self-hosted solution will be much simpler for us to maintain, tune and automate.

There’s also few other bits (nibbles?) of data news:

  • We won’t be producing a data dump for the month of December 2010, but you can definitely expect one just after the new year. We apologize for the delay.
  • SEDE will continue to be updated monthly as a matter of policy to keep it in sync with the monthly data dumps.

Remember, SEDE is fully open source, so if you want to help us hack on it, please do!

And as usual, if you have any bugs or feedback for us, leave it in in the [data-explorer] tag on meta, too.