Proceedings of the 2nd Workshop on Data Science with Human in the Loop: Language Advances, pages 105–111June 11, 2021. ©2021 Association for Computational Linguistics

105

A Computational Model for Interactive Transcription

William Lane, Mat Bettinson, Steven BirdNorthern Institute, Charles Darwin University

Abstract

Transcribing low resource languages can bechallenging in the absence of a comprehen-sive lexicon and proficient transcribers. Ac-cordingly, we seek a way to enable interac-tive transcription, whereby the machine ampli-fies human efforts. This paper presents a com-putational model for interactive transcription,supporting multiple modes of interactivity andincreasing the likelihood of finding tasks thatstimulate local participation. The approachalso supports other applications which are use-ful in low resource contexts, including spokendocument retrieval and language learning.

1 Introduction

Understanding the “transcription challenge” is aprerequisite to designing effective solutions, mini-mizing bottlenecks (Himmelmann, 2018). We mustface realities such as the lack of a good lexicon, theshort supply of transcribers, and the difficulty ofengaging people in arduous work. Sparse tran-scription is an approach to transcribing speech inthese low-resource situations, an approach which iswell suited to places where there is limited capac-ity for transcription. Sparse transcription admitsmulti-user workflows built around shared data, forhuman-in-the-loop transcriptional practices, or “in-teractive transcription” (Bird, 2020b; Le Ferrandet al., 2020).

Sparse transcription is ‘sparse’ because we donot produce contiguous transcriptions up front. In-stead, we transcribe what we can, and lean on com-putational support to amplify those efforts acrossthe corpus. This is not suggested as an alternativeto contiguous transcription, but as a more efficientway to produce it, especially in those situationswhere linguists and speakers are “learning to tran-scribe” (Bird, 2020b, page 716). Sparse transcrip-tion relies on word spotting. Wordforms that occurfrequently in the transcribed portion of a corpusare used to spot forms in the untranscribed portion.

These are presented for manual verification, speed-ing up the contiguous transcription work while in-dexing the entire corpus.

Sparse transcription accepts the realities of earlytranscription: we lack a good lexicon; we need togrow the lexicon as we go; and we do not have aready workforce of transcribers. Moreover, in thecontext of language documentation, transcriptionis iterative and interactive. Linguists and speakersleverage complementary skills to accomplish thetask (Crowley, 2007; Austin, 2007; Rice, 2009).

Sparse transcription leverages the kind of workspeakers are motivated to do. For example, when itcomes to recordings, speakers tend to engage withthe content more than the particular form of expres-sion (Maddieson, 2001, page 215). Identifying keywords and clarifying their meanings is often moreengaging than puzzling over the transcription ofunclear passages (Bird, 2020b). An indexed corpuscan be searched to identify additional high-valuerecordings for transcription.

We report on a computational model for inter-active transcription in low-resource situations. Wediscuss the kinds of interactivity which the sparsetranscription model enables, and propose an ex-tension which provides real-time word discoveryin a sparse transcription system. For concretenesswe also present a user interface which providesreal-time suggestions as the user enters words.

We work with speakers of Kunwinjku (ISO gup),a polysynthetic Indigenous language of northernAustralia. Members of this community have ex-pressed interest using technology to support theirown language goals. Through this work we hopeto support language learning and corpus indexing,and produce locally meaningful results that helpto decolonize the practice of language technology(Bird, 2020a).

This paper is organized as follows. Section 2gives an overview of the sparse transcription model.Section 3 describes a particular use case of sparse

106

transcription: interactive transcription. In Section 4we describe the system architecture and the de-sign decisions which enable an interactive human-computer workflow. Section 5 describes the userinterface and shows screenshots of the implemen-tation. We conclude with a summary in Section 6.

2 The Sparse Transcription Model

Following Bird (2020b), we understand transcrip-tion to be the task of identifying meaningful unitsin connected speech. These units belong to a grow-ing inventory (the glossary, or lexicon); their or-thographic representation is generally not settled.We add each new meaningful unit to the glossaryas it is encountered, initializing the entry with aform and a gloss. Thus, a transcriptional token is apairing of a locus in the speech stream with a glos-sary entry. We are agnostic about the size of thisunit; it could be a morpheme, word, or multi-wordexpression.

Transcription begins with a lexicon. There isalways a word list, since this is what is used for es-tablishing the distinct identity of a language. Theremay also be some historical transcriptions, andthese words can be included in the initial lexicon.From this point on, transcription involves growingthe lexicon.

The speech stream is broken up into ‘breathgroups’ which we use as manageable chunks fortranscription. In the course of transcription, it is anatural thing for a non-speaker linguist to attemptto repeat any new word and have a speaker say itcorrectly and give a meaning. Thus, the processis interactive in the interpersonal sense. We hearand confirm the word in context, and record it inthe lexicon with a lexical identifier and a pointer towhere it occurs in the media. In the background,a sparse transcription system uses this confirmedglossary entry to spot more instances.

Word spotting is an automatic task which discov-ers putative tokens of glossary entries. Glossaryentries are already stored with pointers to occur-rences in particular breath groups. Discoveringnew instances through word spotting then becomesa retrieval task, where each breath group is seenas a mini-document. Breath groups which are de-termined to contain the exemplar lexical entry arequeued for speaker confirmation. Confirmed spot-tings are updated with pointers to their respectivebreath groups.

Word spotting proceeds iteratively and interac-

tively, continually expanding the lexicon while tran-scribing more speech. As we focus on completingthe contiguous transcription of a particular text,we grow the lexicon and the system attempts todiscover other instances across the wider corpus.As the system calls our attention to untranscribedregions, which may be difficult to complete fora variety of reasons, we effectively marshall thewhole corpus to help us.

A sparse transcription system is a form of com-puter supported collaborative work, in that it al-leviates productivity bottlenecks via automationand asynchronous workflows (Greif, 1988; Hanke,2017). The sparse transcription model—organizedaround a growing glossary of entries with pointersto instances in speech—can underlie a variety ofspecial-purpose apps which support various tasksin the transcription workflow. For example, Le Fer-rand et al. (2020) demonstrate the use of a wordconfirmation app based on word-spotted data forthe purpose of confirming automatically-generatedhypotheses.

We have prototyped a system which implementsthe core functionalities described in this section,and which includes a user interface which supportsinteractive transcription. Figure 2 gives a schematicview of the sparse transcription model.1

3 Learning to Transcribe

A linguist, learning to transcribe, is capable of lis-tening to audio and quickly transcribing the lex-emes they recognize. As lexemes are recorded,they are added to the transcriber’s personal glossary.Entries in this glossary may be morphs, words, orother longer units such as multi-word expressions.The record-keeping of the glossary helps managethe linguist’s uncertainty in an accountable way, asthey give the task their best first-pass. As is thestandard behavior in sparse transcription, a glos-sary is updated with links from glossary entries tothe segment of audio in which they were found.

Speakers of the language can access a view ofthe linguist’s glossary entries, and confirm entrytokens for admission to the global glossary. Thedesign decision to maintain personal glossaries forindividual users and postpone adjudication with ashared, canonical glossary is an extension of theconcept defined in the sparse transcription model.

1The system prototype and a reference implementation ofthe sparse transcription model can both be found at https://cdu-tell.gitlab.io/tech-resources/.

107

Figure 1: Word spotting in the sparse transcription model begins when the user confirms the existence of a glossaryentry in the audio. A token is created for that instance of the glossary entry, and can be used to spot similar instancesin other breath groups across the corpus.

Figure 2: The Sparse Transcription Model: Audio issegmented into breath groups, each one a mini spo-ken document where words may be spotted (with givenprobability); interpretations span one or more breathgroups (Bird, 2020b).

Multiple transcribers can contribute to the sharedglossary, initializing their own project with the cur-rent state of the global lexicon.

Confirmed glossary entries can be used to spotsimilar entries across the whole corpus, maximiz-ing the efforts of the learner, and providing morepointers from a glossary entry to breath groupswhere it occurs. Over time, this process leads tomore contiguous transcriptions as the transcriberrevisits and revises their lexicon in the course oftheir transcription work.

However, there is an opportunity here to getmore immediate feedback from the system. Asparsely transcribed breath group (whether sys-tem or human transcribed) provides signal aboutthe breath group as a whole. Combined with thefact that the human is currently engaged in enter-ing their hypotheses, we can provide system sug-gestions conditioned on sparsely transcribed datawhich are updated interactively as the user types.Anchored at the locus of a known lexeme, andconditioned on additional available signal i.e., apredicted phone sequence, the system posits sug-gestions for untranscribed regions. We can refer tothis as ‘local word discovery’ (Fig. 3).

Working together with the system, a linguist’shypotheses can be queued for confirmation in thesame way that word spotting queues hypothesesfor speaker confirmation. Simultaneously, the tran-

scriber leverages a model to get immediate feed-back on the connections between what they hearand what a model encodes about the language, po-tentially aiding language learning (Hermes and En-gman, 2017).

Up to this point, we have established the inter-active nature of transcription on three levels. First,it is interpersonally interactive, as a linguist workswith speakers to associate forms with meanings.Second, sparse transcription is interactive in thesense that it attempts to amplify the effort of tran-scribers by propagating lexical entries across thewhole corpus via word spotting.

Finally, the implementation of local word dis-covery is interactive in the context of the “learn-ing to transcribe” use case. It occupies a distinctniche with a smaller feedback loop than word spot-ting: transcription hints are polled from the modeland filtered with every keystroke (Figs. 6-8). Itis improved by word spotting because contiguoustranscriptions reduce uncertainty in the input to thelocal word discovery model. It allows a linguistto prepare and prioritize work for the interperson-ally interactive task of confirming entries with aspeaker.

Figure 3: Sparsely transcribed input can be leveragedfor local word discovery methods which are comple-mentary to word spotting.

4 System Architecture

The interactive transcription use case calls for avariety of computational agents. Some agents ser-

108

Figure 4: The system architecture

vice computationally-expensive batch tasks, whileothers are coupled with user events down to thelevel of keystrokes.

Agents are implemented as containerized ser-vices, some corresponding to long-running tasks,e.g. media processing, while others are integral tothe user interface, e.g. phone alignment. The im-plementation supports RESTful endpoints, and areal-time websocket-based API.

The API layer responds to events in the client,and endpoints support the methods in the datamodel. There are three main kinds of operation;simple CRUD operations like uploading media,data model operations such as adding a token to aglossary, and real-time queries such as word discov-ery. Data validation is distributed across the clientand the server, for performance reasons and to mit-igate the effects of network dropouts. The clientreplicates a subset of the server data model, storingthis in the browser’s database and synchronizing itwith the server opportunistically.

We utilise a continuous web socket session to re-lay user input to the server, fetching and displayingresults in real time. Commonly seen in web search,this is a form of distributed user interface wherecomputational resources are distributed across plat-forms and architectures (Elmqvist, 2011). This isachieved via asynchronous programming with ob-servable streams, via implementations of the Reac-tive X pattern for JavaScript (rxjs) on the client andPython (rxpy) on the server. Input events from thebrowser are filtered, debounced and piped througha websocket transport to a session handler on theback end. Similarly, components of the client sub-

scribe to session event streams coming from theback end, such as aligning user input to a phonestream, and presenting a series of word comple-tions.

The system makes use of several agents whoseimplementation may vary across contexts or evolveover time. We have implemented the followingagents:

Audio pre-processing. When a user adds an au-dio file to a transcription project, the audio is pre-processed and we store metadata and alternativerepresentations which are useful for downstreamtasks. For example, the pipeline includes voiceactivity detection (VAD), which identifies breathgroups. Next, we calculate peaks–acoustic am-plitude values–which we use to visualize speechactivity over time. Finally, the audio is resampledand sent to the phone recognition agent, and theresults are displayed beneath the waveform as extrainformation to support transcription.

Phone recognition. Allosaurus is a universalphone recognizer trained on over 2,000 languages(Li et al., 2020). The model can be used as-is toprovide phones from a universal set, or it can befine-tuned with language specific phonemic tran-scriptions. The model currently we currently de-ploy is fine-tuned on 68 minutes of Kunwinjkuspeech across 5 speakers. We calculated a 25.6%phone error rate on 10 minutes of speech from ahold-out speaker.

Word spotting. Word spotting traditionally is au-dio exemplar matching against spans of raw audio

109

(Myers et al., 1980). It has been shown to be fea-sible in low resource scenarios using neural ap-proaches (Menon et al., 2018b,a). Le Ferrand et al.(2020) describes several plausible speech represen-tations suited for low-resource word spotting.

Local word discovery. This is distinct fromword spotting, which locates more tokens of exist-ing glossary entries. Local word discovery attemptsto fill in untranscribed regions between existing to-kens. This agent provides transcription hints via asmaller feedback loop, the third kind of interactiv-ity discussed in Section 3. The system retrieves thepotentially large set of suggested words, and filtersit down interactively as the transcriber types. Themodel is free to favor recall, because the raw sug-gestions do not need to be immediately revealed.

We implement local word discovery using a fi-nite state analyzer for Kunwinjku (Lane and Bird,2019), modified to recognize possible word-formsgiven a stream of phones and the offsets of knownlexemes. We use PanPhon to estimate articula-tory distances between lexemes and phone subse-quences to obtain rough alignments (Mortensenet al., 2016).

5 User Interface

The user interface (Fig. 5) is inspired by minimal-ist design, motivated by the need for an inclusiveagenda in language work (cf. Hatton, 2013). Inthe left column is a waveform which has been au-tomatically segmented into breath groups. Belowthe waveform is a map of waveform peaks, to fa-cilitate navigation across long audio files. Usefulcontext is also displayed, including the transcriptof the preceding breath group, followed by the se-quence of phones produced from the audio, withuser transcriptions aligned roughly to the phone se-quence. Below this is the input box, scoped to thecurrent breath group, where users enter lexemes,with occasional suggestions offered by the localword discovery module, and which filter interac-tively per keystroke (Figs. 6-8).

In the right column, there is a running transcriptof the audio file, with the text of the transcript forthe current breath group shown in bold.

The user interface is designed to be navigable en-tirely through the keyboard, to support ergonomictranscription (cf. Luz et al., 2008).

6 Conclusion

Transcription is especially challenging when welack a good lexicon and trained transcribers. Con-sequently, we seek to bring all available resourcesto bear, including the knowledge of speakers, lin-guists, and a system, all of whom are “learning totranscribe.”

We presented a use case for interactive transcrip-tion and showed how this can be supported withinthe sparse transcription model. In designing andimplementing a sparse transcription system for aspecific use case, we elaborated on some conceptspresented in (Bird, 2020b). We examined variouskinds of interactivity in low-resource language tran-scription, and we proposed local word discovery asa grammatically-informed approach to word spot-ting. This allows individual users to manage theirlocal lexicon independently of the task of curating acanonical lexicon, enabling multi-user workflows.

Finally, we reported on the architecture and im-plementation of an interactive transcription system.It enables a transcriber to take care of much of thearduous transcription task up front, and to allocatemore meaningful work for speakers. The productof interaction with the system is an expanded lex-icon, which can be used to index the corpus forinformation retrieval, thus supporting the commu-nity goal of access to knowledge locked up in manyhours of recorded audio. Additionally, we antici-pate that support for growing personal lexicons willbe a valuable resource for the language learningthat takes place alongside transcription. In short,the system is designed to produce the content thatlanguage communities care about, in a way thatleverages the kind of language work that people arewilling to do.

Operationalizing the sparse transcription modelmakes it possible to streamline field-based tran-scriptional practices, and is expected to lead tofurther implementations of special purpose inter-faces that support transcription of low-resource lan-guages.

Acknowledgments

We are grateful for the support of the WarddekenRangers of West Arnhem. This work was coveredby a research permit from the Northern Land Coun-cil, and was sponsored by the Australian govern-ment through a PhD scholarship, and grants fromthe Australian Research Council and the Indige-nous Language and Arts Program.

110

Figure 5: The transcription user interface connects to the data model, which facilitates word spotting and localword discovery agents.

Figure 6: Local word discovery predicts possible words in the audio, conditioned on known lexemes and a flexibleinterpretation of the surrounding sounds.

Figure 7: As the user continues typing, the list of suggestions is filtered down to those which remain compatible.

Figure 8: Thus, the user is guided to grammatically valid transcriptions which can be added to their lexicon.

ReferencesPeter Austin. 2007. Training for language documen-

tation: Experiences at the School of Oriental andAfrican Studies. In Victoria Rau and Margaret Flo-rey, editors, Documenting and Revitalizing Austrone-sian Languages, number 1 in Language Documenta-tion and Conservation Special Issue, pages 25–41.University of Hawai‘i Press.

Steven Bird. 2020a. Decolonising speech and lan-guage technology. In Proceedings of the 28th Inter-national Conference on Computational Linguistics,page 3504–19, Barcelona, Spain.

Steven Bird. 2020b. Sparse transcription. Computa-tional Linguistics, 46:713–744.

Terry Crowley. 2007. Field Linguistics: A Beginner’sGuide. Oxford University Press.

Niklas Elmqvist. 2011. Distributed user interfaces:State of the art. In Distributed User Interfaces,pages 1–12. Springer.

Irene Greif. 1988. Computer-Supported CooperativeWork: A Book of Readings. Morgan Kaufmann.

Florian Hanke. 2017. Computer-Supported Coopera-tive Language Documentation. Ph.D. thesis, Univer-sity of Melbourne.

John Hatton. 2013. SayMore: Language documenta-tion productivity. Presentation at International Con-ference Language Documentation and Conservation.

Mary Hermes and Mel Engman. 2017. Resounding theclarion call: Indigenous language learners and doc-umentation. Language Documentation and Descrip-tion, 14:59–87.

111

Nikolaus P Himmelmann. 2018. Meeting the transcrip-tion challenge. In Reflections on Language Doc-umentation 20 Years after Himmelmann 1998, vol-ume 15 of Language Documentation and Conserva-tion Special Publication, pages 33–40. University ofHawai’i Press.

William Lane and Steven Bird. 2019. Towards a ro-bust morphological analyzer for Kunwinjku. In Pro-ceedings of the 17th Annual Workshop of the Aus-tralasian Language Technology Association, pages1–9.

Éric Le Ferrand, Steven Bird, and Laurent Besacier.2020. Enabling interactive transcription in an In-digenous community. In Proceedings of the 28thInternational Conference on Computational Linguis-tics, pages 3422–28. International Committee onComputational Linguistics.

Xinjian Li, Siddharth Dalmia, Juncheng Li, MatthewLee, Patrick Littell, Jiali Yao, Antonios Anasta-sopoulos, David R Mortensen, Graham Neubig,Alan Black, and Florian Metze. 2020. Universalphone recognition with a multilingual allophone sys-tem. In Proceedings of the International Conferenceon Acoustics, Speech and Signal Processing, pages8249–53. IEEE.

Saturnino Luz, Masood Masoodian, Bill Rogers, andChris Deering. 2008. Interface design strategiesfor computer-assisted speech transcription. In Pro-ceedings of the 20th Australasian Conference onComputer-Human Interaction: Designing for Habi-tus and Habitat, pages 203–10.

Ian Maddieson. 2001. Phonetic fieldwork. In PaulNewman and Martha Ratcliff, editors, LinguisticFieldwork, pages 211–229. Cambridge UniversityPress.

Raghav Menon, Herman Kamper, John Quinn, andThomas Niesler. 2018a. Fast ASR-free and al-most zero-resource keyword spotting using DTWand CNNs for humanitarian monitoring. In Inter-speech, pages 3475–79.

Raghav Menon, Herman Kamper, Emre Yilmaz, JohnQuinn, and Thomas Niesler. 2018b. ASR-freeCNN-DTW keyword spotting using multilingualbottleneck features for almost zero-resource lan-guages. In Proceedings of the 6th InternationalWorkshop on Spoken Language Technologies forUnder-Resourced Languages, pages 182–186.

David R. Mortensen, Patrick Littell, Akash Bharadwaj,Kartik Goyal, Chris Dyer, and Lori Levin. 2016.Panphon: A resource for mapping IPA segmentsto articulatory feature vectors. In Proceedings ofthe 26th International Conference on ComputationalLinguistics, pages 3475–84. Association for Compu-tational Linguistics.

Cory Myers, Lawrence Rabiner, and Andrew Rosen-berg. 1980. An investigation of the use of dy-namic time warping for word spotting and connected

speech recognition. In Proceedings of the Interna-tional Conference on Acoustics, Speech, and SignalProcessing, volume 5, pages 173–177. IEEE.

Keren Rice. 2009. Must there be two solitudes? Lan-guage activists and linguists working together. InJon Reyhner and Louise Lockhard, editors, Indige-nous language revitalization: Encouragement, guid-ance, and lessons learned, pages 37–59. NorthernArizona University.