Tokens

Tokens represent the boundaries between words & symbols of a title or passage's text. They are used both to lookup Dictionary definitions in the Reader, and to map text in a language to its corresponding Audio.

The following operations are supported for token modification:

Selection

Clicking on a token in the text will select it. Once a token is selected, it will be highlighted in green and the selection overlay will be displayed:

The selection overlay displays:

• the token's language
• its index within the containing passage
• the containing passage's index within the chapter
• the character offset & width within the passage or title's text
• the token's part of speech, if available
• time offset and duration for the token's audio, if available

Multi-select

Multiple tokens may be selected by holding shift while clicking another token within the same passage (tokens may not be selected across multiple passages). All selected tokens are highlighted in green:

When multiple tokens are selected, the selection overlay displays:

• the tokens' language
• the selected tokens' indices within their containing passage
• the character offset and width of the selection
• the number of tokens in the selection
• time offset and duration of the selection's corresponding audio segment, if available

Creation

To create a token, simply click on any character in a passage or title's text which is not part of an existing token. A new token will be created containing the clicked character.

Deletion

To delete the selected token or tokens, click the button labeled "Delete Token(s)" from the selection overlay action menu:

Alternatively, the Delete hotkey performs the same operation.

Shift forward/backward

Shifting a token forward increases its character offset, while shifting it backward decreases the offset.

To shift the selected token or tokens, click one of the buttons labeled either "Shift Forward" or "Shift Backward" from the selection overlay action menu:

Alternatively, the left and right arrow hotkeys perform the same operations.

Note that tokens may not be shifted such that they overlap other tokens.

Resize

Resizing a token changes the number of characters that it represents.

To resize a token, click one of the buttons labeled either "Expand Token" or "Contract Token" from the selection overlay action menu:

Alternatively, the up and down arrow hotkeys perform the same operations.

Resizing is only available for singular selections, and tokens may not be resized to overlap other tokens.

Change part of speech

A token's part of speech is indicated by a Universal POS tag (upos).

To change a token's part of speech (upos), click the button labeled with its current part of speech on the selection overlay:

A dialog will be displayed allowing selection of the desired new upos:

Once the desired part of speech is selected, press the button labeled "confirm" to save.

Split passage at token

To split a passage at a particular token, press the button labeled "Split Passage" from the selection overlay action menu:

The selected token, and all subsequent tokens from the selected passage, will be moved to a new passage immediately below the original.