Guidelines tab

 

Surveys | Data research | Diary & Probe studies | Controlled Experiments

 

Surveys

Any study where you collect data by asking people questions is a survey. This can be conducted using paper questionnaires, email, web-based survey forms, or occasionally by telephone or in public spaces. The people who participate in a survey are generally called 'respondents'.

A study in which you analyse data about people and their activities that has been collected without contacting them individually, or specifically asking them to respond to questions, is described as data research. See that section for further details.

A study in which you ask people to keep records of their daily life, or their usage of some technology, is described as a diary study. These are described in more detail in the section about diary and probe studies.

Practicalities - Surveys

Questionnaire design

Questionnaires generally include a combination of closed questions (predetermined responses, either yes/no or multi choice), Likert scales to indicate strength of agreement with a statement, and open questions (free text, which must be coded for analysis).

It is easy to make serious errors when you first attempt to design a questionnaire. There are many textbooks and online guides - make use of them. If possible, ask an expert to review a prototype of your questionnaire, and try it out in advance with several pilot respondents. Typical traps include biased questions, ambiguous questions, poor 'guard' logic, inconsistent response formats, failure to anticipate some valid answers, or reasons for not giving an answer.

Recruitment

Who do you want to respond to your survey, and is this sample expected to be representative of a larger group? Most surveys are initiated from some database or email list. You should ensure that using it in this way is consistent with the terms of use, including any Data Protection Act considerations. You need to check this with the owner of the list.

It is possible to recruit directly by telephone or pedestrian samples. These approaches are stressful and time consuming, and should only be attempted with expert guidance and preparation.

Incentives and compensation

Not everyone who you ask to complete a survey will do so. It is reasonably common to encourage survey responses by offering a gift or other incentive to randomly selected respondents. This often requires that you collect contact details, which raises issues of anonymity as below.

Anonymity

Most surveys are anonymous - they do not record either the name of the respondent, or the name of any institution that the respondent represents. This can be inconvenient if you realise that you need more data after collecting responses (either clarification, compensation for errors in the survey design, or investigating subsequent research questions). Nevertheless, we advise to make surveys anonymous whenever possible.

If it is essential to contact respondents subsequently, it may be acceptable to request an email address, but this should be optional. If email addresses are collected, your data will then be subject to the terms of the Data Protection Act

Many surveys incorporate demographic data (age, gender, education etc). This should be minimised - you should not collect any demographic data unless it is related to a specific research question. Demographic data may include personal details that would bring your research within the terms of the Data Protection Act, in which case precautions noted below must be taken.

Tools

There are a range of tools for administering online surveys. You should check whether you can extract all your data, and whether they have any limit on number of responses. The most popular at the time of writing, SurveyMonkey, does have a limit. SurveyBob has no limit, but displays advertising on some pages.

Data Retention

If survey responses do not include any personal data, then the data may be retained. If they do contain personal data, then they fall within the terms of the Data Protection Act. Personal data should be kept secure (see data security below). Data that would allow a respondent to be identified should be kept in a separate place throughout the research project, with an anonymised code used during analysis work and at publication time. It is good practice to destroy any personal data after a stated period of time.

Informed consent

In general, voluntary completion of a questionnaire or interview can be taken as consent for this data to be used in research. Nobody should ever be compelled to participate in a research survey (for example, students should not be required to participate in research as a condition of course grading). You may wish to assure rspondents that no personal data is collected, or if it is collected, that it will not be published, and will be destroyed

Advice on Survey Validity

Sampling, Response Rate and Selection

You will want to make claims, in presenting your research, that your results are applicable to a larger number of people beyond those that responded - that you had a representative sample. How can you justify that your recruitment database was genuinely representative? Only a subset of those in the database will have responded (often between 5 and 50%). Can you be sure that those who didn't respond would have given the same answers (what are the reasons they didn't respond - might these be related to any of the questions)?

Coding and Analysis

Closed questions can be used as a basis for statistical comparisons, either investigating differences between groups within your sample, or correlations between responses to questions. Survey responses are not generally particularly sensitive measures, so the statistical techniques available might not be straightforward.

Single value statistics have little research relevance unless they can be related to an external comparison or prior hypothesis. (30% of respondents said they liked your product - but what would they have said about a different product?)

Where your survey included open questions, how will you draw conclusions about patterns or trends across their answers? This will involve creating a set of coding categories, assigning each answer to one or more categories, and dealing with those that fall outside the coding scheme, are ambiguous and so on. You should probably get a second person to re-code the same data, and make a statistical inter-rater reliability analysis.

It may be the case that you did not have prior research hypotheses relating to some open questions. In this case, it can be valuable to follow a rigorous process by which codes and potential theoretical concerns are derived from the collected data (for example, Grounded Theory methods). However, these are time-consuming. It is unwise to collect large amounts of verbal data without having a firm plan in advance of how it will be analysed.

 

 

Data Research

If you are directly collecting data on human behaviour, those providing the data should be aware that you are doing so. In the case of surveys, this is already clear. However, data research often makes use of data that was originally collected by other people, in which case it is important to be sure that your own use of the data is consistent with the terms under which it was originally collected. This is discussed below: "Data obtained via third parties"

If data is collected as a side effect of other actions, for example by people using a piece of research software, the situation is not so straightforward.

Network monitoring

Monitoring of network traffic is subject to legal constraint in the UK, and research making direct use of network traffic may be subject to the Telecommunications (Interception of Communications) Regulations 2000. Whatever the legal status of the means by which the data has been collected, it will still be subject to the constraints described by GDPR legislation.

If network traffic is being monitored on an official university network, then the project may be subject to further constraints as defined in terms of use for the UK Joint Academic Network (JANET).

Data obtained via third parties

Data that has been provided by a third party (e.g. market research, student records, customer data, system usage data) is likely to be subject to the terms of the GDPR. You should ensure, at the time you receive the data, that the person supplying it is doing so within the terms allowed by the Act.

It is increasingly common to carry out research using data that has been 'scraped' automatically from websites. However, this should only be done with permission from the administrators of the site concerned. Note that most social networking sites (Facebook, Twitter etc) have terms of use that explicitly forbid this kind of research, unless prior permission has been obtained.

Data Security

All data on human behaviour should be kept secure, and should be distributed only to known individuals as consistent with the terms on which the data was obtained (i.e., data like this should never be placed on websites, or distributed to mailing lists).

Data that is not anonymised, or could be used to identify specific classes of people, requires special precautions. You should consult departmental system administrators to ensure that storage locations on departmental machines and servers are appropriately secure against access by anyone other than the researcher needing to use this data.

Data that is not anonymised should not be stored outside the department, on personal laptops, or carried on removable media, without encryption. (There are several options by which data files can be securely encrypted before copying to such devices). Remember that many applications may create working files or other disk traces from which data could be recovered. In the case of very sensitive information, physical destruction of disk drives and other media at the end of the project may be appropriate.

Anonymity

Anonymisation is a valuable tool that allows data to be shared, whilst preserving privacy. An introduction to the principles involved can be found here.  If at all possible, you should arrange that data is provided to you in an anonymised format. Anonymised data should not include names, addresses, email addresses, or date of birth.

It should not be possible to combine anonymised data with other data in a way that would make the subjects identifiable, for example by combining office numbers with a university directory, or student registration codes with exam entries.

There are sophisticated statistical inference attack techniques that can be used to 'de-anonymise' data. If your data is going to be placed in a public archive, it would be wise to consult an expert in these techniques, to ensure that privacy cannot be compromised.

Identifying specific classes of people

As a general matter of research ethics, if the data you are studying may lead to research outcomes affecting an identifiable class of people, an explicit ethical review should be requested. This is more likely to arise in social science or clinical research, but take care if your own research questions are likely to involve technology use by groups that are sensitive in any way. Examples include:

• genealogy of war criminals
• politicians and sex offenders
• estimating genetic potential for disease in an identifiable class of living people

 

 

Diary Studies

A diary study is one in which participants keep records of their activities, at regular intervals, over a period of time. These may be a record of their experiences using technology, or as a means for the researcher to gain understanding of ordinary life situations in which technology might be usefully applied.

An experience sampling study, like a diary study, collects data about ordinary experience, but with the intention of capturing information about situations that are completely unremarkable, and that participants might not think to record in a diary. This is done by prompting the participant at (semi)random intervals to make a note what they are doing at that precise moment.

A cultural probe study is one in which participants are asked to reflect on their own situation, providing data that is more open to interpretation by technology designers, such as postcards, photographs or more imaginative media.

A technology probe study involves placing a new piece of technology into some context of use, but without predefined conceptions of how it ought to be used. As with cultural probes, researchers take a more interpretive approach to the data that is collected, in contrast to an evaluative approach.

Practicalities

Participants in studies of these kinds usually need to be thoroughly briefed in advance, in the course of which they will have given consent to participate. Each contribution they provide is also made voluntarily. As with all studies, it should be made clear to participants that they are free to withdraw from the study at any time.

All data should be stored and processed anonymously, for example using serial numbers or initials of the participants rather than their full names. If it is expected that publication of the research will involve direct quotations from diaries, participants should give permission for this to be done.

It is increasingly common to carry out diary and experience sampling studies using mobile devices, for example sending participants an SMS message at random intervals, asking them to record their location or activity at the moment they receive it. In a study like this, it can be very convenient to use recording capabilities of the device to record audio, video or photographs. If it is necessary for a research publication to include an image captured by participants, those images should not include any identifiable faces - if they do, then permission should be obtained before publication, from the person whose face appears.

 

Controlled Experiments

This page is intended for use by students and researchers whose research involves recruiting people from outside your own research team to take part in experiments.

A controlled experiment is an experimental setup designed to test hypotheses.

A controlled experiment has one or more conditions (independent variables) and measures (dependent variables).

A randomised controlled trial is an experiment in which participants are assigned at random to different conditions, in order to test in an objective way which of several alternatives is superior.

A pilot study is a trial run of an experimental procedure, not expected to produce valid research data.

Controlled experiments may or may not require human participants. This page is only about controlled experiments involving human participants.

Introduction - Controlled experiments

Controlled experiments are difficult to design and analyse. Students in experimental psychology take practical classes in experiment design before they attempt to conduct their own original research. However, all experiments with human participants conducted by students in Technology and Physical Sciences have the character of original research, from a psychology perspective. It is therefore a common experience for technology researchers to find that their first experiment produces meaningless or null results, often after a great deal of effort. This is wasteful of time and resources, both for the researchers and participants, so should be avoided. For this reason controlled experiments should only be carried out by researchers who are trained in experimental design and analysis, or under direct supervision of researchers with suitable training. If you have little experience you should consult senior researchers.

Some important considerations include design for:

• Reliability (Would you get the same measurement again?)
• Validity (Are you measuring what you claim to be measuring?)
• Internal validity is the relationship between your measurement and what you think it tells you about the experimental task.
• External validity is the relationship between what you measure in the lab, and the phenomenon in the outside world.

A well-known reference book is:

Krik, R.E. Experimental Design: Procedures for Behavioral Sciences.

Practicalities - Controlled experiments

Preparation

Experimental design

It is easy to make serious errors when you first attempt to design a controlled experiment. There are many textbooks and online guides - make use of them. Ask an expert to review your experimental design , and try it out in advance with several pilot studies.

There are a number of critical factors that could cause the experimental results to be invalid, and it is important to anticipate these and avoid them. One way to do so is to plan, in advance, how you propose to write up the results of the experiment. Think about the conclusions that you would draw if the result of the experiment is consistent with your hypothesis. How would you present your results in a way that convinces the reader that conclusion is justified? What would the results of data analysis have to be to support this kind of presentation? What experimental method will produce data that can be analysed in this way? What is the best way to express an hypothesis compatible with that method? If you can explain your reasoning in this way, before you start the experiment, you will have a much better chance of avoiding the invalid and/or inconclusive results that are so often obtained by inexperienced experiment designers.

Pilot studies

It is very hard to get an experimental procedure right the first time. Every experiment should therefore include at least one pilot session, with a participant whose results you expect to discard from the final data analysis. For this reason, it is common to use a pilot subject whose results you would not expect to be valuable - for example, because they are aware of the experimental hypothesis, have specialist expertise, or similar. Family members and (fellow) students can be useful.

Where an experimental paradigm is unconventional, or there is substantial uncertainty about either the measures or the hypotheses, you should consider a pilot study involving several participants, in which each of the experimental conditions is used, and a preliminary data analysis can be conducted.

Recruitment

In order for research to have good external validity, the recruited participants should be representative of the population about which you want to make research conclusions. However, in practice, undergraduate and graduate students are often recruited because this is easier. If you plan to do this, it is a good idea to think in advance how you will justify it to reviewers or assessors of your work.

Where children are involved in research, recruitment is likely to be via schools or parents. Some experiments with children, or with vulnerable adults, may also require that members of the research team undergo a "CRB check".

Where participants have been recruited on the basis of a medical condition, it is likely that your research will require approval via the NHS Research Ethics Service.

It is increasingly common to recruit experimental participants via platforms such as Amazon Mechanical Turk or CrowdFlower. There are many distinct ethical implications of experiments conducted using these tools that are rather different to those arising in the conduct of experiments in a laboratory.

Conducting the experiment

Treatment of participants

In most experiments, participants are asked to carry out an experimental task while being observed, or while their responses are being measured. It is of paramount importance that participants are treated with dignity and respect. Remember that you are in a position of power from the participants' perspective. You need to inform yourself about participants' rights and then disclose these rights to the participants. Among those rights:

• The right to stop participating in the experiment, possibly without giving a reason.
• The right to obtain further information about the purpose and the outcomes of the experiment.
• The right to have their data anonymised.

This list is not exhaustive.

It is often the case that people being asked to use new technologies while under observation find the experience stressful. It is very important to reassure participants that your objective is to identify possible faults in the technology, and not to test the participants' own ability or intelligence. If they have trouble completing an experimental task, you should reassure them further, emphasising that they have had this experience because the technology is inadequate, and that it is not a reflection on their own ability. Experimenters should never offer any comment with regard to participants' intelligence, aptitude, or other factors that might give people the impression that a scientific judgment of their ability has been performed. This is especially the case if standard psychometric tests are being employed as one of the experimental measures. An experimental situation in technology or physical sciences is not a proper psychometric assessment, and psychometric test results should not be directly communicated to participants.

Informed consent

It is very important for participants to understand that their participation in the experiment is completely voluntary. In order to ensure that they understand this, experimenters should prepare a 'consent form', stating the nature of the experiment and the rights of the participant. Before the start of the experiment, participants should be asked to read this form, and sign it to indicate that they have read and understood their rights.

You may wish to assure participants that no personal data is collected, or if it is collected, that it will not be published, and will be destroyed. These things can be mentioned in a consent form.

If a participant appears to be experiencing any stress (for example due to task difficulty, or perhaps through factors unrelated to the experiment), it is important to remind them that they are free to withdraw at any time.

If a participant is experiencing physical pain (e.g. because of extensive use of the mouse for the task) then abort the experiment immediately and consult a senior colleague or the appropriate university ethics committee for advice on whether to proceed with the experimental procedure.

In the case of children (in the UK, under the age of 18), consent must be given by a parent. The experimenter may also be subject to CRB check.

Participant briefing

For the purposes of experimental control, every participant should be given the same instructions before they commence the experimental task. Briefing instructions are normally written out in full, in order to ensure that this is done. The instructions can either be read from a script by the experimenter, or given to the participant to read, after which they are asked if they have understood everything, and are ready to start.

If an experimenter script is used, it is a good idea for this to include all instructions and actions that the experimenter must carry out throughout the experimental session. This script should be tested during the experimental pilot, and helps gain maximum value from the pilot as a 'debugging' session for the main experimental procedure.

Debriefing

At the end of an experimental session, participants should normally be debriefed. Debriefing involves a short interview, often semi-structured, with some prepared questions that you ask every participant, and follow-up questions in the event that interesting points are raised.

This provides a valuable data collection opportunity, especially as participants' subjective experience of the experiment could be of value in interpreting either their individual performance, or behaviour observed more broadly across the sample group. It may be useful to discuss your experimental hypothesis with participants, because they might well be able to warn you of potential problems with task validity, from their perception of the task.

Whether or not you expect to gain useful information for research purposes, debriefing also provides an opportunity for the participant to reflect on the experience they have had. It is a good idea to complete the debriefing interview by asking whether there is anything else the participant would like to tell you.

Incentives and compensation

It is recommended to compensate participants for their time, although compensation need not be financial. People may be very willing to participate in experiments from which they gain interesting feedback, or experiments that are intrinsically enjoyable (for example games). A token gift (chocolates, a book or report, software, or a memento such as copies of a scan) may be sufficient reward. Nevertheless, many departments in Cambridge routinely recruit experimental participants, and payment may be expected after a formal experiment. If the participant has incurred direct costs such as travel these should be reimbursed.

If a participant chooses to withdraw, or not to complete the experiment, they should still be compensated. Experiments in which incentive payments are varied according to task performance are considered to be unethical. A standard procedure where incentive is a central hypothesis (for example experiments in economic judgment) is to offer participants variable payment at the outset, but then to pay all participants the same (usually maximum) amount at the close of the session.

The university has issued rules on procedure to be used, and how much compensation should be given to participants. Finance division policy on payments to research volunteers is described in this document:

http://www.admin.cam.ac.uk/cam-only/offices/finance/procedures/expenses/expenses/volunteers.html

Data Retention

If the data collected does not include any personal data, then the data may be retained. If they do contain personal data, then they fall within the terms of the Data Protection Act. Personal data should be kept secure. Data that would allow a participant to be identified should be kept in a separate place throughout the research project, with an anonymised code used during analysis work and at publication time. It is good practice to destroy any personal data after a stated period of time. In most cases, experimental data is used only by the person conducting the experiment. If this is not the case, see the page on sharing research data.

Significant ethical issues

This page is intended to address relatively routine research in the schools of Technology and Physical Sciences. If your experiment involves any of the following activities, then more serious questions must be addressed, and you will need to consult the relevant university ethics committee:

• Experiments involving animals are subject to the animals scientific procedures act
• Medical and other invasive experiments on human participants must be reviewed by the NHS research ethics service.
• Psychological manipulation of human participants (deception, emotional manipulation, etc.).

This list is not exhaustive. When in doubt consult senior colleagues and relevant university ethics committees.

References

Some popular books are:

• Kirk, R.E. Experimental Design: Procedures for Behavioral Sciences.
• Robson, C. Experiment, Design and Statistics in Psychology