Trustworthiness of Think-Aloud Protocols in the Study of Translation Processes

Defeng Li, The Chinese University of Hong Kong

Presented by: Sitti Fatimah Saleng (120221521886) State University of Malang

 

v  Introduction

The study of translation has been increasing since the mid 1980’s. It has yielded many interesting insights to the mental translation process. This article intends to serve as a critical inquiry into the validity and reliability of think-aloud protocols adopted in the study of the translation process based on theories about empirical research methodology. However, there are problems with the research designs, so the trustworthiness of finding is open to debate. In order to enhance the reliability and validity of such studies, research into the method of using think-aloud protocols as a means of data collection in examining the translation process is of prime importance.

v  Safeguards for the trustworthiness of empirical research

Empirical research can be divided into two kinds – rationalistic and naturalistic, which are known today as quantitative and qualitative. While some researchers believe that they can be used in combination, one complementing the other (Mertens 1998), others contend that they are epistemologically different and hence mutually exclusive. This research concern with qualitative research design only.

According to Guba (1981), the criteria for assessing the trustworthiness of naturalistic inquiries, determine the trustworthiness of an empirical study:

1)      Truth value. How can one establish confidence in the ‘truth’ of the findings of a particular inquiry for the subjects/respondents with which, and the context in which, the inquiry was carried out?

2)      Applicability. How can one determine the degree to which the findings of a particular inquiry may have applicability to other contexts or with other subjects/respondents?

3)      Consistency. How can one determine whether the findings of an inquiry would be consistently repeated if the inquiry were replicated with the same/similar subjects/respondents in the same/similar context?

4)      Neutrality. How can one establish the degree to which the findings of an inquiry are a function solely of subjects/respondents and conditions of the inquiry and not of the biases, motivations, interests, perspectives of the inquirer?

Regarding with the four concern above, safeguards for the trustworthiness of empirical research, especially naturalistic qualitative research, are recommended, including the following:

1)      Voluntary participation and guarantee of anonymity. In order to encourage subjects to be truthful in their responses and to minimize the chances of subjects intentionally supplying data they believe is being sought, the investigator needs to brief the subjects about the purpose and the method of the study, especially what the subjects are required to do in the experiment, what will happen to the data, and how their anonymity will be ensured during and after the study. The subjects’ participation should be voluntary.

2)      Purposeful sampling. The researcher usually should avoid generalizations of findings on the grounds that virtually all social and behavioral phenomena are context-bound. Thus it is not possible to come up with ‘truth’ statements that have general applicability. One must be content with statements that are descriptive or interpretative of a given context.

3)      Triangulation To cross-check data and interpretations, it is recommended that the researcher relies on triangulation, whereby a variety of data sources, different investigators and different methods are utilized. Whenever possible, the research team should be divided so that the perceptions of several investigators can be compared. Also, different data-collection methods – e.g. questionnaires, interviews, analysis of documents – should be brought to bear.

4)      Prolonged engagement. Since an inquiry can be affected by a bewildering array of interlocking factor patterns and therefore pose formidable problems of interpretation, the researcher should have prolonged engagement at a site in order to overcome, as far as possible, distortions produced by the researcher’s presence and to give the researcher as well as the respondents opportunities to test their own biases and perceptions.

5)      (Near-) Natural situation Related to prolonged engagement, it is strongly recommended that the research context be kept as close to the subjects’ ‘normal’ environment as possible. Any external interference or change caused by the study should be kept to a minimum. Experts on think-aloud protocols also recommend this.

6)      Peer debriefing, stepwise replication and intercoder reliability. Researchers should regularly detach themselves from the site and seek out and interact with other professionals engaged in similar work. They should expose their thinking to this jury of peers and answer whatever questions arise. In data interpretation and analysis, stepwise replication is recommended. That is, the investigators should be divided into two groups which will deal separately with data sources, which should also be divided, so that their interpretations and results can be compared. Related to this is intercoder reliability, which experts on think-aloud protocols regard as an essential check.

7)      Member checks. The investigator should check with the subjects about the authenticity of the data and the interpretations through member checks. It is the investigation for checking by the people who were the source of those materials.

8)      Thick description. To enable judgments about how well the research context fits with other contexts, thick descriptive data, i.e. a rich and extensive set of details concerning methodology and context, should be included in the research report.

 

Those mentioned above are some of the most important considerations. They can be roughly summarized in Table 1.

Stage of the research	Safeguards
Data collection	v  Voluntary participation v  Anonymity assurance Purposeful sampling v  Triangulation Prolonged engagement (Near-) Natural situation
Data analysis	v Peer debriefing, stepwise replication and intercoder reliability v Member checks
Reporting	v Thick description
Application	v  Refrain from generalizing

 

v  Design of the study

To examine the research design of TAP studies on translation processes. The sample consisted of 15 published articles, book chapters and monographs, all written in English, on translation processes using TAPs as the research design. The sample was then examined with particular attention given to the design of the reported studies. The safeguards listed in Table 1 were used as the guide in identifying and recording the measures taken by each study to insure its trustworthiness. Also included as parameters were two further safeguards required for TAP research, namely training for research subjects to familiarize them with TAPs and a background check to determine their suitability for such research.

 

v  Result and discussion 

Parameters		Articles, book chapters and monographs
		Number		Percentage
		Yes	No/Not reported	Yes	No/Not reported
Data collection	v  Voluntary participation v  Anonymity assurance v  Purposeful sampling v  Triangulation v  Prolonged engagement v  (Near-) Natural situation v  Training for subject v  Background check	3 0 6 9 1 7 4 0	12 15 9 6 14 8 11 15	20 0 40 60 6.7 46.7 26.7 0	80 100 60 40 93.3 53.3 73.3 100
Data analysis	v Peer debriefing, stepwise replication and intercoder reliability v Member checks	2 0	13 15	13.3 0	86.7 100
Reporting	v  Thick description	8	7	53.3	46.7
Application	v  Refrain from generalizing	11	4	73.3	26.7

 

For the 15 research reports surveyed, it was found that the following three safeguards were used the most frequently: refraining from generalizing findings (73.3%), triangulation of data-collection methods (60%), and thick description in reports (53.3%). Three of the listed safeguards, namely assurance of anonymity and background checks (under data collection) and member checks (under data analysis) were not reported as being used in any of the 15 surveyed studies. In addition, over 70% of the studies did not use or did not report using one or more of the following: prolonged engagement (93.3%), peer debriefing/intercoder reliability (86.7%), voluntary participation (80%), and training for subjects (73.3%).

 

v  Conclusion

There is no doubt that the pioneering work on translation processes using think-aloud protocols has produced many interesting findings which otherwise would have remained mostly unknown. However, the research designs of the studies conducted to date leaves much to be desired, which in turn has seriously undermined the trustworthiness of many of the findings. As such, many of the findings emanating from these projects constitute little more than working hypotheses about translation processes, which are to be confirmed or refuted by future research relying on enhanced rigor and trust- worthiness (Ivanova 2000: 48).