Swiping Action for Displaying a Translation of a Textual Image

14 Pages·2014·1.27 MB·English

US 20140081620A1
(19) United States
(12) Patent Application Publication (10) Pub. N0.: US 2014/0081620 A1
Solntseva (43) Pub. Date: Mar. 20, 2014
(54) SWIPING ACTION FOR DISPLAYINGA (52) US. Cl.
TRANSLATION OF A TEXTUAL IMAGE CPC .................................... .. G06F 17/28 (2013.01)
USPC ............................................................ .. 704/3
(71) Applicant: ABBYY SOFTWARE LTD., Nicosia
(CY) (57) ABSTRACT
_ Disclosed is a method that involves acquiring an image With
(72) Inventor: Ekatenna solntseva, MOSCOW (RU) text, displaying all or a portion of the image on an electronic
_ _ _ device. In response to detecting a sWiping action or gesture,
(73) Asslgnee? ABBYY software Ltd” Nlcosla (CY) displaying a result of translation on a screen of the device. A
?rst screen or display becomes a second one. Original text in
(21) APP1~ NOJ 13/694,900 a ?rst language or source language may be easily and quickly
compared to translated text shoWn on a second screen through
(22) Filed? seP- 18, 2012 a sWiping gesture. Electronic dictionaries and machine trans
lation may be used. These services may be independently
Publication Classi?cation stored and operated from different locations including on the
device performing the translation, on a server or across a
(51) Int. Cl. network (LAN, WAN, etc.). Optional manual correction of
G06F 17/28 (2006.01) the translated text is also possible.
/104 102
f /
( Speisekarte
1. Kos-Teller: Suflaki und Kalamari mit Zaziki und
Reis
2. Muffins mit Mandeln und Schokoladen Raspeln (
3. Kése-Platte: Chaloumi, Schafskése, Kése
Tiropitakia, Tirokeftede
4. Krautsalat
5. Eis mit Kokosraspeln
108
Patent Application Publication Mar. 20, 2014 Sheet 1 0f 6 US 2014/0081620 A1
104 102
( Speisekarte f
1. Kos-Teller: Suflaki und Kalamari mit Zaziki und
Reis
2. Muffins mit Mandeln und Schokoladen Raspeln {
3. Kase-Platte: Chaloumi, Schafsk'aise, Kase
Tiropitakia, Tirokeftede
4. Krautsalat
5. Eis mit Kokosraspeln
k
Fig. 1A
106
// \
( Menu /
1. Kos-Teller: suvlaki and calamari with tzatziki \
and rice
2. Muffins with almonds and chocolate shavings
3. Cheese-teller: haloumi, feta, tiropitakia, cheese
bails
4. Cabbage salad
5.lce with coconut shavings
Patent Application Publication Mar. 20, 2014 Sheet 2 0f 6 US 2014/0081620 A1
)P/
102
/
( Muffins mit Mandeln und Schokoladen Raspeln \
Mandeln, Mehl, Backpuiver, Vanillezucker und Salz
vermischen. In einer anderen Schiissel die Eier,
Zucker und den Amaretto leicht schaum
schlagen, danach Stuck fiir Sttick die B
dazugeben. Danach die Eimasse zur Mehimas
dazumischen und die Schokoraspeln hinzufijige .
Zuietzt die Masse in die Muf?nformen geben und
bei ca. 180 Grad fur 20-25 Min. backen.
108
Fig. 1C
106 102
Muffins with almonds and chocolate shavings ( \
Asallmt.o nIdns ,a nfoltouhre,r bbakoiwnlg bpeoawtd etrh,e veagniglsl,a ssuuggaarr aa n‘d "i ' '
amaretto frothy, then add the butter piec
piece. Then to mix the egg mixture to flour :
add the grated chocolate mass. Finally te*
mixture into the muffin tins and at about 18‘
degrees for 20-25 minutes baking.
A
/ / J
/ Fig. 10 L
108
Patent Application Publication Mar. 20, 2014 Sheet 3 0f 6 US 2014/0081620 A1
200 \
201
Text or image with text to be
translated
' a 202
Identifying a text or a rectangle of the
image containing a text
+ 203
Performing optical character
recognition (OCR) of text image
+ 204
Machine or dictionary translation J/
$ 205
Displaying or rendering the
translation on other screen
l 206
Swiping between screens }/
s translation satisfied.
YES
Word-by-word adjustment
4/ 207
208
L->( Translated screen orimage )/
Fig. 2
Patent Application Publication Mar. 20, 2014 Sheet 4 of 6 US 2014/0081620 A1
207
301
Touching a screen with translated
text
1
302
Identifying a word or word
combination to change
i 303
Displaying variant(s) oftranslation or
wordforms
l
Receiving selection of best or
preferred variant
l
Inserting or substituting translated
word or word combination in text
Fig. 3
Patent Application Publication Mar. 20, 2014 Sheet 5 0f 6 US 2014/0081620 A1
f Menu /
1. Kos-Teller: suvlaki and calamari with tzatziki
and rice ' v
balls
4. Cabbage salat
5. Ice with coconut
k
k
Fig. 4A
/106 102
f / \
r Menu / X
1. Kos-Teller: suvlaki and calamari with tzatziki
and rice v
2. Muffins with almonds and chocolate@
3. Chesse-teller: haloumi, feta, tiropitakia, cheese
balls
4. Cabbage salat
5. Ice with coconut
\ j
k J
Fig. 4B
Patent Application Publication Mar. 20, 2014 Sheet 6 0f 6 US 2014/0081620 A1
/ 500
506 508
( (
|NpUT OUTNPUT
DEVICHS) DEV|CE(S)
516
502 + i \
kn
Processor <—> ASPPOLFITCWAATRIEON 514 \-/
i A
512
\f‘ NETWORK(S) MASS STORAGE 510
r‘\/
Fig. 5
US 2014/0081620 A1 Mar. 20, 2014
SWIPING ACTION FOR DISPLAYING A German text to be translated into another language through a
TRANSLATION OF A TEXTUAL IMAGE machine translation system, etc.
[0014] FIG. 1D shows an accurate English translation of
BACKGROUND OF THE INVENTION the text shown in FIG. 1C switched by a swiping gesture. FIG.
2 shows a ?owchart of operations performed, for example by
[0001] 1. Field
[0002] Embodiments of the invention generally relate to the software, in accordance with an embodiment of the present
?eld of automated translation using electronic dictionaries disclosure concerned with translation.
and various applications in such areas as machine translation. [0015] FIG. 3 shows a ?owchart of operations performed,
[0003] 2. RelatedArt for example by software, in accordance with an embodiment
[0004] There is a lot of electronic devices with display of the present disclosure concerned with the method of cor
screens capable of displaying text, including devices with a rection of translated text.
touch screen, for example many mobile devices, such as [0016] FIG. 4A shows an example of selecting a particular
smartphones, mobile phones, laptops, tablet computers, net word to be corrected and opening an entry with other possible
books, personal digital assistants (PDAs), e-book readers, variants of translation of selected word in output language.
photo and video cameras. [0017] FIG. 4B shows an example of correcting translated
[0005] These devices have enough resources forusing elec text by inserting another more appropriate or desired variant
tronic dictionaries, which may be installed locally. Machine of translation.
translation system may be available over the Internet, WiFi, [0018] FIG. 5 shows an exemplary hardware for imple
Bluetooth, and through these and other types of networks and menting the technology of the present disclosure.
protocols.
[0006] Many of the aforesaid devices involve advanced DETAILED DESCRIPTION
displays and associated logic capable of supporting non-key
board type input methods. For example the devices may sup [0019] In the following description, for purposes of expla
port pen-based input, voice-based input or touch-based input. nation, numerous speci?c details are set forth in order to
[0007] Also, most of above-identi?ed electronic devices provide a thorough understanding of the invention. It will be
apparent, however, to one skilled in the art that the invention
may involve the use of a camera and other components to
facilitate receiving, capturing and using photographs and can be practiced without these speci?c details. In other
images with text. instances, structures and devices are shown only in block
diagram form in order to avoid obscuring the invention.
SUMMARY [0020] Reference in this speci?cation to “one embodiment”
or “an embodiment” means that a particular feature, structure,
[0008] In one embodiment, the invention provides a
or characteristic described in connection with the embodi
method including receiving an image with text (photos, PDF,
ment is included in at least one embodiment of the invention.
Word ?les, pictures, etc.) with text to be translated on the
The instances of the phrase “in one embodiment” in various
screen of electronic device, automated translation of the text,
places in the speci?cation are not necessarily all referring to
displaying the result of translation on the screen of electronic
the same embodiment, nor are separate or alternative embodi
device, providing a user an opportunity to swipe or refresh
ments mutually exclusive of other embodiments. Moreover,
screens and compare the original text with the translated one.
various features are described which may be exhibited by
Some or all of the screens of the electronic device is swapped
some embodiments and not by others. Similarly, various
or refreshed upon the device receiving a swiping action.
requirements are described which may be requirements for
[0009] Electronic dictionaries may comprise a software
some embodiments but not other embodiments.
program, software logic which operate with dictionary data.
[0021] Advantageously, the present invention discloses an
The software program may include a shell, which provides a
electronic device that allows a user to perform machine or
graphical user interface, morphology models to display
dictionary translation of an image or photograph containing
in?ected forms, a context search that uses an index, a teaching
text, display the translated variant of the image on another
module, etc. The dictionaries that the user needs to use may be
screen of an electronic or mobile device, and to switch
independently stored in different locations including the
between screens by performing a swiping gesture. The terms
computer device or a server in a local area network (LAN) or
“screen” and “another screen” imply different views or rep
a wide area network (WAN) such as the Internet.
resentations on a physical screen or parts of a same physical
[0010] An electronic device may also connect to a machine
screen and refer to a representation of an original image and
translation system. The machine translation system may be
a translated one. The electronic device preferably includes or
located on a remote server, access to which is available via a
accesses additional optical character recognition (OCR) soft
network connection or protocol to the server, e.g. over the
ware, ?rmware, algorithms or functionality to support trans
Internet.
lation of text. Such OCR software can identify the regions on
the image where the text is located and then converts the
BRIEF DESCRIPTION OF THE DRAWINGS
image in the regions into a text format. OCR software may be
[0011] FIG. 1A shows an example ofa user interface dis installed at any remote server and run via the Internet, Blue
played on a touch screen of an electronic device with German tooth, etc. so that an OCR functionality need not be limited to
text to be translated with dictionary into another language. steps or algorithms performed on the device that displays,
[0012] FIG. 1B shows an accurate English translation of the captures or accesses an image of text.
text shown in FIG. 1A on the screen switched by a swiping [0022] The method(s) allow a user to correct results of an
gesture. OCR function, an OCR result, and to re-initiate a translation
[0013] FIG. 1C shows another example of a user interface of the original text, or portions of the original text such as
displayed on a touch screen of an electronic device with based upon a correction made by a user. For example, a user
US 2014/0081620 A1 Mar. 20, 2014
may change a Word recognized as “mike” to “make” to enable set of ingredients in a food recipe. The choice of dictionary
a correct translation of the Word in the image into a second or translation alloWs one to save memory and resources of an
target language. electronic or mobile device. An example of a text image
[0023] Additionally, the disclosed methods also alloW a translated under a dictionary mode is shoWn in FIG. 1A and
user to compare the result of a machine and/or dictionary FIG. 1B. An image With German text appears in FIG. 1A and
translation of the text (in a second or target language) With the an accurate translation of the text appears in FIG. 1B.
text in a ?rst or original (source) language. The methods may
[0029] In alternative embodiment, a pulling of a portion of
include the alternating use of a forWard and backward sWip
the visible language doWn (off the screen) and then raising it
ing motion or action to a touch sensitive screen or in relation
back upWard (Without lifting the ?nger from the surface of the
to a touch sensitive screen. The forward and backWard sWip
device display) translates only those lines or portions of text
ing action may be, for example, respectively, a left sWiping
that Were temporarily moved off the screen. Thereby a portion
motion and a right sWiping motion, or an upWard and doWn
of the text in the image displayed or rendered (by being in a
Ward sWiping motion, or by a counter-clockWise and clock
state of encoded text) to be translated may be selected. For
Wise rotation of a device etc.
example, if three lines of text from a menu Were pulled off the
[0024] The methods also alloW a user to look through alter
screen, only those three lines are translated. Similarly, in
natives of translation and to optionally and manually make
another exemplary embodiment, When tWo top lines are
Word-by-Word correction of the text by insertion or replace
sWiped upWard and off the screen, these tWo lines are trans
ment of a ?rst translation With another variant of translation.
lated When the screen is brought up and doWn through a
The opportunity to manually correct the translated text (in the
combination of a sWiping motion upWard folloWed directly
target language) is useful in the case When a user is not by a sWiping motion doWnWard.
satis?ed With the translation proposed by system translation
or When a Word is not translated due to text that Was not [0030] In yet another alternative implementation, With ref
accurately recogniZed. erence to FIG. 1A, a translation is not performed until a
[0025] A manual correction of the target language com sWiping motion 108 is performed on the touch screen, touch
prises, for example, touching a user interface element asso pad, or through some other input device. If a suf?ciently long
sWiping motion is made, the image (or encoded text) shoWn
ciated With a certain Word or Word combination in the trans
lated text in an output language, displaying a list of alternative on the display or screen is moved off the side of the display,
variants of translation for the source Word or Word combina and the display is refreshed With a translation of the text. In
tion (e.g., in a balloon or in a pop-up or overlay WindoW), another alternative embodiment, a translation may be made
alloWing a user to look through, ?nd or enter other possible through a particular or separate softWare application, or a
variants of translation of the Word or Word combination, modi?cation may be made to the operating system such that
receiving an input from a user that indicates a variant of a translation through a sWiping motion may be accomplished
through any softWare application. For example, a sWiping
translation, and performing an insertion or substitution of an
appropriate variant of translation in the text instead of the motion, When used in the context of a SMS texting program,
variant originally or ?rst proposed by the translation system. translates text from a ?rst or source language to a second or
target language. The sWiping motion may be performed on a
[0026] In alternative embodiment, a subsequent translation
from a target language back to a ?rst (source) language may message-by-message basis, on the basis of an entire screen of
be made to alloW a user to verify that the translation to the text, or on the basis of some or all of a history or thread of
target language carries the correct meaning or sense. Such conversation (based on a con?guration setting of the SMS
texting program).
subsequent translation may be made by using a distinctive
and different motion or sWiping action in relation to the [0031] In another alternative embodiment, the translation
touch-sensitive screen. Only a portion of the translated text of the source text into tWo target languages may be per
may be re-translated to a source language. Such portion may formed4either sequentially or at the same time, depending
be indicated by a different sWiping action (e.g., tWo-?nger on a con?guration setting. In this scenario, in the system, tWo
action, three-?nger action), or by selecting a portion of text, target languages shouldbe preliminary speci?ed. A ?rst sWip
and then subsequently making a sWiping motion in a back ing gesture in a forWard direction initiates a translation into
Ward orientation or motion. the ?rst target language, and folloWing after the ?rst sWiping
[0027] A system user can preliminarily select or “pre-set” a gesture, a second sWiping gesture in a forWard direction ini
mode of translation (dictionary translation mode, machine tiates a second translation into the second target language. So,
translation mode, etc.), for example depending on the kind of if user speaks or is familiar With tWo or more foreign lan
text or accessibility of an Internet or cellular telephone con guages, for example Spanish and German languages, a user
nection. In one exemplary embodiment, tWo modes of trans can translate the source text into tWo languages based on this
lation are available: a dictionary translation and a machine procedure. The desired target languages should be speci?ed
translation. Dictionary translation performs a Word-by-Word in the settings of the system or softWare application. In the
or phrase-by-phrase translation based on translation of Words example, the ?rst sWiping of the screen in a forWard direction
and Word combinations available in one or more dictionaries. yields the result of the ?rst translation into Spanish, and the
A machine translation performs a sentence-by-sentence or second sWiping in the forWard direction shoWs the result of
phrase-by-phrase translation of the source text. In other translation of the original (source) text into German. The
embodiments, other modes of translation may be available sWiping action makes translation betWeen languages much
including various services of machine and automated trans easier than previously available. Instead of a user searching in
lation. a menu for a setting for a target language, a sWiping motion
[0028] Word-by-Word dictionary translation may be alloWs a user to translate a text betWeen tWo or more lan
selected in a case When text is represented as a string of guages With a single gesture. Also, the opportunity to go back
grammatically unconnected Words, for example a menu or a a previous screen by using a sWiping gesture in a backWard
US 2014/0081620 A1 Mar. 20, 2014
direction remains for a user. This process is available for shoWn upon detection or performing of a sWiping action 108
translation of the source text into more than tWo target lan such as With a ?nger. A picture element 110 (e. g., picture of a
guages. muf?n) is not modi?ed and is persisted from a ?rst language
[0032] Advantageously, the method of translation With the or source display to a second language or target language or
help of translation and other types of dictionaries is most display.
convenient in the circumstance Where a device is Without [0040] Referring noW to FIG. 2, there is shoWn a ?owchart
Internet access or When the Internet connection costs too 200 of operations performed by a translation application in
much. Dictionaries may be installed locally on the electronic accordance With one embodiment of the invention.
device. [0041] The process 201 is responsible for receiving and
[0033] On the contrary, machine translation is more reason displaying a text or textual image on a screen of electronic
able to use in a case When a text to be translated includes device. For example, it may be a photograph, portable docu
sentences or Words With linguistic and grammatical links ment format (PDF) ?le, or Word processor ?le. After receiv
betWeen them. An example of a text image translated under ing the image With the text to be translated, the disclosed
the direction of a machine translator is shoWn in FIG. 1C and method is activated. In one embodiment the translation can
FIG. 1D. start by detecting a pressing of a button of a camera or camera
[0034] In one embodiment, a method of translating a sen application to initiate and capture a photograph or image With
tence from one source language into another output or target text or by detecting a ?rst sWiping motion (108) in relation to
language includes analyZing the source sentence using lin an existing or original image.
guistic descriptions of the source language, constructing a [0042] At ?rst, the softWare application identi?es a text or
language-independent semantic structure to represent the at least one rectangle in an image that includes a text 202.
meaning of the source sentence, and generating an output [0043] An optical character recognition (OCR) softWare
sentence to represent the meaning of the source sentence in 203 is applied to the identi?ed region in the image With text.
the output language using linguistic descriptions of the output The result of the OCR is a Word, Word combination, sentence
language. or longer passage of text represented by a string of characters.
[0035] FIG. 1A shoWs an example of an electronic device At the OCR step 203, morphology dictionaries may also be
102 comprising a display screen 104. The content presented used as higher OCR accuracy and error correction is often
on the display screen or touch screen 104 may be outputted or achieved by comparing the recognition results With similar
rendered to the display by any application (eg an image Word forms in one or more morphology dictionaries. At least
vieWer, a pdf-vieWer, e-book reader, a Web BroWser, e-mail, one of a machine translation or dictionary translation is per
Word, Notepad, a textual message) that sends an image of text formed 204. The output language of translation may be pre
to the display screen 104. liminary selected or con?gured through a selection of a target
[0036] The electronic device 102 may comprise a general language in a softWare application.
purpose computer embodied in different con?gurations such [0044] Also, the dictionary or set of dictionaries may be
as a smartphone, cell phone, digital camera, desktop personal preliminarily selected by the user or may be programmati
computer (PC), laptop computer, or any other gadget and cally or automatically made available or selected based on
having touch screen as a display screen. FIG. 5 of the draW one or more Words that are successfully recogniZed or trans
ings shoWs exemplary hardWare for implementing the user lated. For example, if an initial OCR operation encounters or
electronic device 102, in accordance With one embodiment. recogniZes a Russian Word, the softWare application may
[0037] The electronic device 102 may be provisioned With assign the source language as Russian, and the target lan
softWare that includes a dictionary application to implement guage as another language other than Russian. Dictionary
techniques disclosed herein. The application may include one softWare may use, by default, one or more dictionaries or a
or more local dictionaries. Additionally, the application may user may specify one or more desired dictionaries. A default
be able to access one or more remote dictionaries located on dictionary on a given subject may be selected if the dictionary
a remote server via a netWork connection to a server (not softWare determines that the text belongs to a specialiZed
shoWn). subject (e.g., medicine, laW, automobiles, computers).
[0038] The electronic device 102 may be also provisioned [0045] Additionally, the electronic dictionary may include
With softWare that performs machine translation or a machine a morphology module, so that the query Word or Word com
translation system may be located on a remote server and run bination need not be in a base, or “dictionary” formithe
via netWork connection to the server, for example a Model morphology module identi?es the base form of an in?ected
Based MT system, such as one disclosed in Us. Pat. Nos. form. If more than one base form is possible, the morphology
8,195,447 and 8,214,199. In a preferred implementation, the module identi?es possible alternatives. Also, in some cases,
system provides syntactically coherent output. Syntactic and the morphology module may determine a grammatical form
morphological descriptions of the input and output languages of the source Word, for example, a number for nouns or form
are used for this purpose. The teachings of these patents are of verbs, to select a proper form for the translation in an output
language.
incorporated by reference to the extent necessary to enable
the instant invention. [0046] Also the morphology module identi?es relevant
[0039] Referring to FIG. 1C, a touch screen 104 of an Word forms of variants of translation, for example, in accor
electronic device 102 is shoWn With German text. This text dance With the form of the translated Word in the original
represents a set of grammatically agreeing Words and inter sentence such as tense forms of verbs or a noun in a plural
connected sentences, so it is reasonable to translate this text form.
With an implementation of machine translation. FIG. 1D [0047] The result of translation is displayed on another
shoWs the result of machine translation in English Where the screen of the electronic device at step 205 or may be queued
translation shoWn is a true and accurate translation of the or rendered onto a virtual screen that is physically not visible
German text shoWn in FIG. 1C. The translation in English is until after a sWiping motion is detected by the device. A