Trustworthiness of Think-Aloud Protocols in the Study of Translation Processes

Defeng Li, The Chinese University of Hong Kong

Presented by: Sitti Fatimah Saleng (120221521886) State University of Malang

 

v  Introduction

The study of translation has been increasing since the mid 1980’s. It has yielded many interesting insights to the mental translation process. This article intends to serve as a critical inquiry into the validity and reliability of think-aloud protocols adopted in the study of the translation process based on theories about empirical research methodology. However, there are problems with the research designs, so the trustworthiness of finding is open to debate. In order to enhance the reliability and validity of such studies, research into the method of using think-aloud protocols as a means of data collection in examining the translation process is of prime importance.

v  Safeguards for the trustworthiness of empirical research

Empirical research can be divided into two kinds – rationalistic and naturalistic, which are known today as quantitative and qualitative. While some researchers believe that they can be used in combination, one complementing the other (Mertens 1998), others contend that they are epistemologically different and hence mutually exclusive. This research concern with qualitative research design only.

According to Guba (1981), the criteria for assessing the trustworthiness of naturalistic inquiries, determine the trustworthiness of an empirical study:

1)      Truth value. How can one establish confidence in the ‘truth’ of the findings of a particular inquiry for the subjects/respondents with which, and the context in which, the inquiry was carried out?

2)      Applicability. How can one determine the degree to which the findings of a particular inquiry may have applicability to other contexts or with other subjects/respondents?

3)      Consistency. How can one determine whether the findings of an inquiry would be consistently repeated if the inquiry were replicated with the same/similar subjects/respondents in the same/similar context?

4)      Neutrality. How can one establish the degree to which the findings of an inquiry are a function solely of subjects/respondents and conditions of the inquiry and not of the biases, motivations, interests, perspectives of the inquirer?

Regarding with the four concern above, safeguards for the trustworthiness of empirical research, especially naturalistic qualitative research, are recommended, including the following:

1)      Voluntary participation and guarantee of anonymity. In order to encourage subjects to be truthful in their responses and to minimize the chances of subjects intentionally supplying data they believe is being sought, the investigator needs to brief the subjects about the purpose and the method of the study, especially what the subjects are required to do in the experiment, what will happen to the data, and how their anonymity will be ensured during and after the study. The subjects’ participation should be voluntary.

2)      Purposeful sampling. The researcher usually should avoid generalizations of findings on the grounds that virtually all social and behavioral phenomena are context-bound. Thus it is not possible to come up with ‘truth’ statements that have general applicability. One must be content with statements that are descriptive or interpretative of a given context.

3)      Triangulation To cross-check data and interpretations, it is recommended that the researcher relies on triangulation, whereby a variety of data sources, different investigators and different methods are utilized. Whenever possible, the research team should be divided so that the perceptions of several investigators can be compared. Also, different data-collection methods – e.g. questionnaires, interviews, analysis of documents – should be brought to bear.

4)      Prolonged engagement. Since an inquiry can be affected by a bewildering array of interlocking factor patterns and therefore pose formidable problems of interpretation, the researcher should have prolonged engagement at a site in order to overcome, as far as possible, distortions produced by the researcher’s presence and to give the researcher as well as the respondents opportunities to test their own biases and perceptions.

5)      (Near-) Natural situation Related to prolonged engagement, it is strongly recommended that the research context be kept as close to the subjects’ ‘normal’ environment as possible. Any external interference or change caused by the study should be kept to a minimum. Experts on think-aloud protocols also recommend this.

6)      Peer debriefing, stepwise replication and intercoder reliability. Researchers should regularly detach themselves from the site and seek out and interact with other professionals engaged in similar work. They should expose their thinking to this jury of peers and answer whatever questions arise. In data interpretation and analysis, stepwise replication is recommended. That is, the investigators should be divided into two groups which will deal separately with data sources, which should also be divided, so that their interpretations and results can be compared. Related to this is intercoder reliability, which experts on think-aloud protocols regard as an essential check.

7)      Member checks. The investigator should check with the subjects about the authenticity of the data and the interpretations through member checks. It is the investigation for checking by the people who were the source of those materials.

8)      Thick description. To enable judgments about how well the research context fits with other contexts, thick descriptive data, i.e. a rich and extensive set of details concerning methodology and context, should be included in the research report.

 

Those mentioned above are some of the most important considerations. They can be roughly summarized in Table 1.

Stage of the research	Safeguards
Data collection	v  Voluntary participation v  Anonymity assurance Purposeful sampling v  Triangulation Prolonged engagement (Near-) Natural situation
Data analysis	v Peer debriefing, stepwise replication and intercoder reliability v Member checks
Reporting	v Thick description
Application	v  Refrain from generalizing

 

v  Design of the study

To examine the research design of TAP studies on translation processes. The sample consisted of 15 published articles, book chapters and monographs, all written in English, on translation processes using TAPs as the research design. The sample was then examined with particular attention given to the design of the reported studies. The safeguards listed in Table 1 were used as the guide in identifying and recording the measures taken by each study to insure its trustworthiness. Also included as parameters were two further safeguards required for TAP research, namely training for research subjects to familiarize them with TAPs and a background check to determine their suitability for such research.

 

v  Result and discussion 

Parameters		Articles, book chapters and monographs
		Number		Percentage
		Yes	No/Not reported	Yes	No/Not reported
Data collection	v  Voluntary participation v  Anonymity assurance v  Purposeful sampling v  Triangulation v  Prolonged engagement v  (Near-) Natural situation v  Training for subject v  Background check	3 0 6 9 1 7 4 0	12 15 9 6 14 8 11 15	20 0 40 60 6.7 46.7 26.7 0	80 100 60 40 93.3 53.3 73.3 100
Data analysis	v Peer debriefing, stepwise replication and intercoder reliability v Member checks	2 0	13 15	13.3 0	86.7 100
Reporting	v  Thick description	8	7	53.3	46.7
Application	v  Refrain from generalizing	11	4	73.3	26.7

 

For the 15 research reports surveyed, it was found that the following three safeguards were used the most frequently: refraining from generalizing findings (73.3%), triangulation of data-collection methods (60%), and thick description in reports (53.3%). Three of the listed safeguards, namely assurance of anonymity and background checks (under data collection) and member checks (under data analysis) were not reported as being used in any of the 15 surveyed studies. In addition, over 70% of the studies did not use or did not report using one or more of the following: prolonged engagement (93.3%), peer debriefing/intercoder reliability (86.7%), voluntary participation (80%), and training for subjects (73.3%).

 

v  Conclusion

There is no doubt that the pioneering work on translation processes using think-aloud protocols has produced many interesting findings which otherwise would have remained mostly unknown. However, the research designs of the studies conducted to date leaves much to be desired, which in turn has seriously undermined the trustworthiness of many of the findings. As such, many of the findings emanating from these projects constitute little more than working hypotheses about translation processes, which are to be confirmed or refuted by future research relying on enhanced rigor and trust- worthiness (Ivanova 2000: 48).

A Critical Review of Qualitative Interviews in Applied Linguistics

By Steve Mann

University of Warwick

Presented by Sitti Fatimah Saleng (120221521886), State University of Malang 

 

v  INTRODUCTION

This article aims to address the following question: What can applied linguistics learn from related disciplines with regard to the qualitative interview? The article assesses the contributions of qualitative sociology, anthropology, and, in particular, the interactional perspectives of ethomethodology/CA, and discursive psychology.

 

v  THE QUALITATIVE INTERVIEW IN THE SOCIAL SCIENCES

The intention is to open up areas of debate and interest related to the use of qualitative interviews in the wider social sciences. There is no shortage of writing about qualitative interviews in the social sciences and there is inevitably quite a variety. In the variety of resources available above there is plenty of advice and also plenty of metaphors used to describe the interview process.

 

·         Diciplinary perspectives

Theorize the interview in the fields of qualitative sociology, anthropology, discursive, psychology, and ethnomethodology is necessarily selective and the intention is to draw out some important contributions and dilemmas, using them as a series of critical and discursive perspectives with which to review qualitative interviews in applied linguistics.

When the interview is theorized as ‘active,’ the interviewer’s contribution to the co-construction of interview content, and the local accomplishment of the interview, is explicitly acknowledged and thus becomes a topic for analysis.

Fundamentally, ‘no matter how formalized, restricted, or standardized’ the nature of the interview, there is ‘interaction between the interview participants’ (1995: 18) and there will inevitably be spontaneous and ‘improvized’ elements. In this view, all interviews are already sites of social interaction, where ideas, facts, views, details, and stories are collaboratively produced by interviewee and interviewer.

 However, the extent to which that interaction is (i) acknowledged as requiring analysis, and subsequently, (ii) included or excluded from the research report, depends on whether the researcher has theorized the interview as active or not.

 

·         Discursive dilemmas

1.       Co-construction.

One main outcome of the literature that has problematized the qualitative interview is that it is now well established that interview talk is inevitably a co-construction between the interviewer and interviewee.

2.       A greater focus on the interviewer

The social science literature has focused primarily on distinctive features of the interviewee (e.g. attention to age, race, gender, and issues of power). The importance of co-construction is that it inevitably requires more attention to be paid to what the interviewer is bringing to the process.

3.       Interactional context

The interview ‘produces situated understandings grounded in specific interactional episodes’ (Denzin and Lincoln 1994: 353). Although the interviewer may be interested in getting a perspective on the lived experience and context which the interviewee inhabits, the interview enacts its own context. According to Baker (2004), there has been a great deal of discussion in qualitative sociology around the idea of giving ‘voice’ to interviewees, especially where issues of discrimination, under-representation and asymmetries of power are concerned. However, there is also a worry that these voices can become decontexualized, taking attention away from the interactional context and the role and contribution of the interviewer.

4.       The ‘what’ and the ‘how’

Another key issue is that the researcher needs to be more conscious of the interview process rather than simply ‘mine’ the products. Donnelly (2003) sees the challenge for qualitative researchers as being to shift from a ‘what’ perspective to a ‘how’ perspective and ‘to articulate as fully as possible the processes associated with the data analysis of interviews’.

 

v  A CRITICAL AND DISCURSIVE VIEW OF QUALITATIVE INTERVIEW USE IN APPLIED LINGUISTICS

In discursive psychology, as we have seen, Potter and Hepburn’s (2005) view is that there are prevailing problems associated with the use of qualitative interviews that have not been resolved. In qualitative sociology, there have been ‘varying degrees of enthusiasm’ (Silverman 2007: 572) for the position that interviews are ‘accounts’ and ‘representations’ and are ‘collaboratively produced’. The articles try to emerge and acknowledged response to problems associated with the use of qualitative interviews In order to see such a call is justified, He intend to look at several articles that use interviews published in relevant applied linguistics journals (e.g. Applied Linguistics, TESOL Quarterly, etc.). He will then return to the four discursive dilemmas modified from Potter and Hepburn (2005) in relation to applied linguistics. In doing so, He will draw attention to common problems and also highlight useful contributions, including articles in this collection.

 

·         A Selection Of Papers From Applied Linguistics

The studies below highlighted the use interviews as the primary method for generating data and are a representative sample of strong research articles that use one of a variety of qualitative interviews (i.e. semi-structured, life-history, ethnographic).

1.      Semi-structured

Varghese and Johnson (2007) use ‘semi-structured’ interviews to explore evangelical pre-service teachers’, but somehow they just concern with the discourse of the interviewees and ignore the contribution of interviewer in building up the interview. Then, the readers’ views are limited since there is no access to the interactional development.

2.      Life-history

Study on Sri Lankan teachers by Hynes (2005) state that the contribution of interviewer and methodological issues in interview co-construction are commented explicitly as trustworthiness, because the process of co-construction cannot be accessed in its transcripts or analysis.

3.      Ethnographic

Palfreyman (2005) presents interview data to illustrate the processes of ‘Othering’ (a group’s construction of a shared Us–Them representation of another group). His study provides more details data about the interview schedule provided with loosely-structured talk. Unfortunately, there is no explanation how the interviewer elicits and explores significant issues from the interviewees.

                       

The selected articles above use interviews as the primary method for generating data. However, there are a growing number of articles that draw on interviews to support other data collection instruments (e.g. questionnaires). Regarding to the issues raised above, the interview must focus on the role of the interviewer, transparent with regard to transcriptions, and balance the ‘what’ with the ‘how’, accounting for interactional context.

Borg (2009: 363), in a study of teachers’ attitudes to research, uses qualitative interviews as follow up data to questionnaires. The ‘interviews were transcribed in full’ but there are no transcripts included, either in the article, the appendices, or in any on-line supplementary resource. Representation aside, the analysis concentrates on coding and classification are not enough to provide an exhaustive qualitative analysis.

Shi (2010) is a good example of an article where there seems to be an appreciation of the interview as a co-constructed event, but the presentation of the data does not allow the reader any view of this co-construction. Shi does mention the interviewer contribution but, beyond a few phases like ‘when prompted to identify’(a glimpse of an interviewer at work), the interviewer presence is largely absent.

 

·         Discursive Dilemmas In Applied Linguistics

There are four discursive dilemmas in applied linguistics proposed by Roulston (2010) and Talmy (2010). They are co-construction, greater focus on the interviewer, interactional context and the “what” and “how” interview is conducted.

1.      Co-construction

Co-construction dilemma deals with is the need of representing talks with the transcription. To strengthen the research, interviewee is asked to validate the transcription and to clarify the transcript. Talmy (2010) states that qualitative applied linguistics research conceptualizes interview data as objective or subjective reports from interviewees, focusing on content.

2.      A greater focus on the interviewer

During the interview, the focus should be given to both interviewer and interviewee because the relationship built between interviewer and interviewees tend to have important implications towards the process of interview.

3.      Interactional context

Each interview has its own interactional context. There are two kinds of context, research context related to the issues and interactional context in which the talk raises. Interviewee contributions are always produced in negotiation with the interviewer (Rapley 2001: 317) and extracts from transcripts ‘should always be presented in the context in which they occurred. Richards (2003) includes a number of longer transcripts and analysis so that the reader can access the account of the tension between legitimate talk in interviews and something more like gossip. The account is valuable because we get a perspective on the interactional context and we get a consideration of how the interviewers’ participation is ‘significantly implicated in what the respondents end up saying’ (Wooffitt and Widdicombe 2006: 56).

4.      The ‘what’ and the ‘how’

The ignorance of the keys of co-construction, interviewer identity, and interactional context seems to clarify the absence of interviewer role in the interview. According to Sealey, the role of interviewer should be described in his corpus-based study.

v  

PARAMETERS OF SENSITIVITY

It is common for our students to provide ‘a summary of participant’s observations, richly interspersed with quotes, presented as analysis’ (Pavlenko 2007:163). This is particularly worrying when a study claims to be constructivist in nature and yet pays no attention to issues of co-construction in the shaping of these quotes. The following table presents an indicative list of the parameters of sensitivity that a researcher needs to build up.

Parameters of sensitivity	Comments
Agenda led « Conversational	Important balance in semi-structured interviews
Naturally occurring « Manipulated voices	A degree of structure and an important analytic perspective on how the data will be treated.
Direct « Indirect	Sensitivity around the issue of directness covers a number of issues such as hypothetical questions, indirect probes such as asking about other people’s views, offering anecdotes, texts.
Conventional « Active	Considerations of the balance between rapport and empathy and more ‘active’ or confronting stances This might be compared with interview moves associated with ‘active listening’.
Empathy « Disclosure	The balance between trying to view things from the interviewee’s perspective and ‘contribution’ or ‘disclosure’
Reports « Accounts	Recognizing the co-production and situated nature of what is presented. It also covers a monologic / dialogic parameter.
Rapport « Overrapport	Spelling out the difficulties of both ‘underrapport’ and ‘overrapport’.
Context free « Context shaped	The use of context free quotes and longer sections which show interactional context.
How « What	Might also be called process and product. This is a useful overall parameter—to make sure there is at least some reflexive element