Go Back   Steam Users' Forums > Steam Discussions > Suggestions / Ideas

Reply
 
Thread Tools Display Modes
Old 06-19-2011, 10:35 PM   #1
Cheeseness
 
Join Date: Mar 2011
Reputation: 88
Posts: 700
Why Isn't There A Steam Survey FAQ?

So with the Steam system profiler now apparently working on my platform of choice, I've been looking for information on the hardware/software survey.

Specifically, I think it would be nice to see an up to date, concise FAQ which covers general methodology questions and specifics on how things work from the user end (what data is gathered, how data is gathered, how statistics are derived, how participation is handled, how users opt in, links to any relevant privacy policies, etc.). Most of that stuff might be obvious or at least assumable, but that's really no substitute for a proper factual resource.

It may be that this stuff is covered in the survey prompt, but given that I haven't actually seen one since the random sampling started, I don't really feel that that's good enough.

I'm happy to keep this post updated if anybody has any solid information, but longer term, it'd be really nice to see an official FAQ on the support site or linked to from the survey page.


Gathered/Assumed Facts

So far, I've been unable to find much in the way of solid information, and I thought I'd compile what I have found together for anybody else seeking information. Please note that at the time of writing, I have not been able to find any "official" confirmation for many of the statements below (they're based mostly on information provided by volunteer moderators and forum members). Confirmed information is bolded.
  • The survey is conducted monthly and results are published for the previous month
  • From July 2010 to early in 2013, the survey results included data for installed software on Windows users' machines with a notice that Mac data would be coming soon (only data from the initial July 2010 results was published)
  • Though historical data does not seem to be publicly available, the survey has been running (in one form or another) since February 2004
  • There seems to be no way to opt out of the software part of the survey
  • Users are prompted to participate by a Steam pop-up window
  • If you opt out of one, you are not opted out of future surveys
  • Survey results are taken from a random sample of the Steam community - supposedly 1/12th with the assumption that every Steam user should be prompted once a year (previously every Steam user was prompted once a month?)
  • The survey is anonymous - at least to the extent that support staff are unable to assist with technical queries relating to the collection of data
  • For a period of months before March 2012, a bug prevented Steam users who hadn't already participated in previous surveys from being prompted to participate in future ones



Support Articles
  • At the time of writing, there were no relevant results on the Support site when searching for "survey".


Survey Results Notices
  • Accompanying the March 2012 results
    Quote:
    Why do many of the Steam Hardware Survey numbers seem to undergo a significant change in April 2012?

    There was a bug introduced into Steam's survey code several months ago that caused a bias toward older systems. Specifically, only systems that had run the survey prior to the introduction of the bug would be asked to run the survey again. This caused brand new systems to never run the survey. In March 2012, we caught the bug, causing the survey to be run on a large number of new computers, thus giving us a more accurate survey and causing some of the numbers to vary more than they normally would month-to-month. Some of the most interesting changes revealed by this correction were the increased OS share of Windows 7 (as Vista fell below XP), the rise of Intel as a graphics provider and the overall diversification of Steam worldwide (as seen in the increase of non-English language usage, particularly Russian).

Forum Threads


Client Update notes

Broken, but seems to be a duplicate of a Steam Blog post?
New Software Survey (16 July 2010)


Steam Blog posts

Edit: added some horrible colouring to separate stuff out a bit
Edit: added confirmation that opting out is not meant to opt users out of future participation, that surveys are random, and that the survey is anonymous (thanks to Support Tech Seth)
Edit: added a copy of the notice regarding the impact of the survey prompting bug fixed in March 2012

Last edited by Cheeseness: 10-24-2013 at 12:36 AM. Reason: Updated to mention that software data is not included anymore
Cheeseness is offline  
Reply With Quote
Old 06-22-2011, 06:57 PM   #2
Cheeseness
 
Join Date: Mar 2011
Reputation: 88
Posts: 700
An update: I submitted a support ticked asking for some solid information on how the survey works, and whether or not it was normal that I haven't seen a prompt in years.

The response I got back was that this forum is the best place to get answers from the Valve team, so I guess anybody who wants to have more information available about how the survey works should feel free to bump this thread.
Cheeseness is offline   Reply With Quote
Old 07-01-2011, 11:42 PM   #3
Cheeseness
 
Join Date: Mar 2011
Reputation: 88
Posts: 700
Updated with details from my support ticket that confirmed a few points regarding opting out and survey timeframes.

One interesting piece of information was the suggestion that anonymity is preserved to an extent that my question of "is there a technical problem preventing me from participating?" couldn't be answered.

Whilst anonymity is good, if it prevents underlying problems with sampling/data collection from being visible, that potentially undermines the value and validity of the results.
Cheeseness is offline   Reply With Quote
Old 04-01-2012, 06:50 PM   #4
Cheeseness
 
Join Date: Mar 2011
Reputation: 88
Posts: 700
Updated to include details of the two client updates since November 2011 that have contained updates related to the Hardware & Software Survey.

Also worth mentioning is the recent addition of a notice regarding the impact of the survey bug fixed in the 2nd March 2012 Steam client update:
Quote:
Why do many of the Steam Hardware Survey numbers seem to undergo a significant change in April 2012?

There was a bug introduced into Steam's survey code several months ago that caused a bias toward older systems. Specifically, only systems that had run the survey prior to the introduction of the bug would be asked to run the survey again. This caused brand new systems to never run the survey. In March 2012, we caught the bug, causing the survey to be run on a large number of new computers, thus giving us a more accurate survey and causing some of the numbers to vary more than they normally would month-to-month. Some of the most interesting changes revealed by this correction were the increased OS share of Windows 7 (as Vista fell below XP), the rise of Intel as a graphics provider and the overall diversification of Steam worldwide (as seen in the increase of non-English language usage, particularly Russian).
It's interesting to note that the most recent (March 2012) survey results now contain "Unknown" and "Unknown 64 bit" operating systems which hadn't appeared in previous survey results. Whatever it is that they represent appears to have been previously hidden by the bug.
Cheeseness is offline   Reply With Quote
Old 05-03-2013, 02:24 PM   #5
Cheeseness
 
Join Date: Mar 2011
Reputation: 88
Posts: 700
It seems I'd forgotten to mention, but Linux results began appearing in January this year.

In March, the threshold for inclusion in the "Other" category was reduced to reveal additional Mac OS and Linux versions.

We are currently seeing two "N/A" operating systems in the survey results.
Cheeseness is offline   Reply With Quote
Old 10-22-2013, 02:37 AM   #6
Cheeseness
 
Join Date: Mar 2011
Reputation: 88
Posts: 700
A few months back, the "Other" threshold was increased again. The results from the software portion of the July 2010 have now been removed, but I'm not certain when.
Cheeseness is offline   Reply With Quote
Reply

Go Back   Steam Users' Forums > Steam Discussions > Suggestions / Ideas


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



All times are GMT -7. The time now is 04:10 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Site Content Copyright Valve Corporation 1998-2014, All Rights Reserved.