Funders: 
This work was funded by the European Union&#8217;s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie Actions (grant agreement number 657440) and the Norwegian Association of the Blind and Partially Sighted.&#160; 
 
Film makeup: 
Makeup artist Rachael Parrey.

<ol>
	<li>Ramachandran, V. S., Hubbard, E. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Synaesthesia+-+A+Window+Into+Perception,+Thought+and+Language.">Synaesthesia - A Window Into Perception, Thought and Language.</a> Journal of Consciousness Studies. 8 (12), 3-34 (2001).</li><li>Bremner, A. J., Caparos, S., Davidoff, J., de Fockert, J., Linnell, K. J., Spence, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bouba&#8217;&#8217;and+&#8216;&#8216;Kiki&#8217;&#8217;+in+Namibia?+A+remote+culture+make+similar+sound-shape+matches,+but+different+taste-shape+matches+to+Westerners.">Bouba&#8217;&#8217;and &#8216;&#8216;Kiki&#8217;&#8217; in Namibia? A remote culture make similar sound-shape matches, but different taste-shape matches to Westerners.</a> Cognition. 126 (2), 165-172 (2013).</li><li>Davis, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+fitness+of+names+to+drawings.+A+cross-cultural+study+in+Tanganyika.">The fitness of names to drawings. A cross-cultural study in Tanganyika.</a> British Journal of Psychology. 52 (3), 259-268 (1961).</li><li>Maurer, D., Pathman, T., Mondloch, C. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+shape+of+boubas:+sound-shape+correspondences+in+toddlers+and+adults.">The shape of boubas: sound-shape correspondences in toddlers and adults.</a> Developmental Science. 9 (3), 316-322 (2006).</li><li>Ozturk, O., Krehm, M., Vouloumanos, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Sound-symbolism+in+infancy:+Evidence+for+sound-Shape+cross-modal+correspondences+in+4-months-olds.">Sound-symbolism in infancy: Evidence for sound-Shape cross-modal correspondences in 4-months-olds.</a> Journal of Experimental Child Psychology. 114 (2), 173-186 (2013).</li><li>Bar, M., Neta, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Humans+Prefer+Curved+Visual+Objects.">Humans Prefer Curved Visual Objects.</a> Psychological Science. 17 (8), 645-648 (2006).</li><li>Bertamini, M., Palumbo, L., Gheorghes, T. N., Galatsidas, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Do+observers+like+curvature+or+do+they+dislike+angularity?.">Do observers like curvature or do they dislike angularity?.</a> British Journal of Psychology. 107 (1), 154-178 (2016).</li><li>Quinn, P. C., Brown, C. R., Streppa, M. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Perceptual+Organization+of+Complex+Visual+Configurations+by+Young+Infants.">Perceptual Organization of Complex Visual Configurations by Young Infants.</a> Infant Behavior and Development. 20 (1), 35-46 (1997).</li><li>Silvia, P. J., Barona, C. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Do+people+prefer+curved+objects?+Angularity,+expertise,+and+aesthetic+preference.">Do people prefer curved objects? Angularity, expertise, and aesthetic preference.</a> Empirical Studies of the Arts. 27 (1), 25-42 (2009).</li><li>Nielsen, A., Rendall, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Sound+of+Round:+Evaluating+the+Sound-Symbolic+Role+of+Consonants+in+the+Classic+Takete-Maluma+Phenomenon.">The Sound of Round: Evaluating the Sound-Symbolic Role of Consonants in the Classic Takete-Maluma Phenomenon.</a> Canadian Journal of Experimental Psychology. 65 (2), 115-124 (2011).</li><li>D&#8217;Onofrio, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Phonetic+Detail+and+Dimensionality+in+Sound-shape+Correspondences:+Refining+the+Bouba-Kiki+Paradigm.">Phonetic Detail and Dimensionality in Sound-shape Correspondences: Refining the Bouba-Kiki Paradigm.</a> Language and Speech. 57 (3), 367-393 (2014).</li><li>Westbury, C., Hollis, G., Sidhu, D. M., Pexman, P. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Weighing+up+the+evidence+for+sound+symbolism:+Distributional+properties+predict+cue+strength.">Weighing up the evidence for sound symbolism: Distributional properties predict cue strength.</a> Journal of Memory and Language. 99, 122-150 (2018).</li><li>Oberman, L. M., Ramachandran, V. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Preliminary+evidence+for+deficits+in+multisensory+integration+in+autism+spectrum+disorders:+The+mirror+neuron+hypothesis.">Preliminary evidence for deficits in multisensory integration in autism spectrum disorders: The mirror neuron hypothesis.</a> Social Neuroscience. 3 (3-4), 348-355 (2008).</li><li>K&#246;hler, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Gestalt psychology. , (1929).</li><li>Monaghan, P., Mattock, K., Walker, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Role+of+Sound+Symbolism+in+Language+Learning.">The Role of Sound Symbolism in Language Learning.</a> Journal of Experimental Psychology: Learning, Memory, and Cognition. 38 (5), 1152-1164 (2012).</li><li>Fort, M., Martin, A., Peperkamp, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Consonants+are+More+Important+than+Vowels+in+the+Bouba-kiki+Effect.">Consonants are More Important than Vowels in the Bouba-kiki Effect.</a> Language and Speech. 58 (2), 247-266 (2015).</li><li>Graven, T., Desebrock, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bouba+or+kiki+with+and+without+vision:+Shape-audio+regularities+and+mental+images.">Bouba or kiki with and without vision: Shape-audio regularities and mental images.</a> Acta Psychologica. 188, 200-212 (2018).</li><li>Fryer, L., Freeman, J., Pring, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Touching+words+is+not+enough:+How+visual+experience+influences+haptic-auditory+associations+in+the+&#8220;Bouba-Kiki&#8221;+effect.">Touching words is not enough: How visual experience influences haptic-auditory associations in the &#8220;Bouba-Kiki&#8221; effect.</a> Cognition. 132 (2), 164-173 (2014).</li><li>Fontana, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Association+of+Haptic+Trajectories+to+Takete+and+Maluma.">Association of Haptic Trajectories to Takete and Maluma.</a> Lecture Notes in Computer Science. 7989, 60-68 (2013).</li><li>Graven, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+blind+individuals+discriminate+braille+characters:+An+identification+and+comparison+of+three+discrimination+strategies.">How blind individuals discriminate braille characters: An identification and comparison of three discrimination strategies.</a> British Journal of Visual Impairment. 33 (2), 80-95 (2015).</li><li>Graven, T., Bello, F., Kajimoto, H., Visell, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=How+attention+is+allocated+when+using+haptic+touch:+Shape+feature+distinction+and+discrimination+strategy.">How attention is allocated when using haptic touch: Shape feature distinction and discrimination strategy.</a> EuroHaptics 2016, Part 1. , 380-393 (2016).</li><li>Heller, M. A., Calcaterra, J. A., Burson, L. L., Tyler, L. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Tactual+picture+identification+by+blind+and+sighted+people:+Effects+of+providing+categorical+information.">Tactual picture identification by blind and sighted people: Effects of providing categorical information.</a> Perception &amp; Psychophysics. 58 (2), 310-323 (1996).</li><li>Heller, M. A., Gentaz, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Psychology of Touch and Blindness. , (2014).</li><li>Pathak, K., Pring, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Tactual+picture+recognition+in+congenitally+blind+and+sighted+children.">Tactual picture recognition in congenitally blind and sighted children.</a> Applied Cognitive Psychology. 3 (4), 337-350 (1989).</li></ol>

The authors have nothing to disclose.

The presented protocol is appropriate for investigating the effect of visual imagery and learning shape-audio regularities, when they occur and how robust they are; in specific individuals and groups of individuals.&#160;
This protocol would be appropriate for testing not only the accuracy but also the exploration time: From when the experimenter removes their left hand from the participant&#8217;s fists (in the blind and blindfold-group) / the carton plate or foam board (in the vision-group) to when the participant says &#8216;yes.&#8217; In the statistical analyses, the longest exploration time per trial per experimental group would be divided into four and the number of &#8216;fast,&#8217; &#8216;medium&#8217; and &#8216;slow&#8217; participants in each experimental group counted: &#8216;Fast&#8217; using &#8804;25% of this exploration time, &#8216;medium&#8217; using 26-75%, and &#8216;slow&#8217; using &#8805;75%. Across trials, the participants would be counted as in Experiment 2: That is, as &#8216;fast&#8217; if counted as &#8216;fast&#8217; in &#8805;66.6% of all trials, as &#8216;medium&#8217; if counted as &#8216;medium&#8217; in &#8805;66.6% of all trials, and so on. (For &#8216;inconclusive&#8217;, i.e. the participant cannot be counted as &#8216;fast,&#8217; &#8216;medium&#8217; or &#8216;slow&#8217;, cf. Experiment 3.) It would be possible to test the exploration time for correct and incorrect answers separately, depending on the sample size and/or the number of correct and incorrect answers. Whether testing accuracy, exploration time or both, there are critical protocol steps that must be followed: The three experiments should be conducted in fixed order, and their trials presented in fixed order, for all participants. The participants should not be made aware that the test materials in all three experiments include bouba and kiki. Further, the auditory information from the MP3 player should be played only once and not repeated back orally to the participants (all three experiments). In Experiment 1 and Experiment 2, the tactile/visual bouba and kiki should be presented in the same position and orientation. For the vision-group (in all three experiments), the test materials should be set up behind a carton plate/foam board. Finally, in the blindfold-group, the participants should not remove their blindfold until Experiment 3 has been completed. These steps are to ensure that the amount and type of information about the tactile/visual and the auditory bouba and kiki is kept under control across all experimental groups. In addition, it is crucial in Experiment 2 to present the auditory bouba/kiki as soon as the participant starts exploring the tactile/visual bouba/kiki. This will ensure that multimodal (congruous or incongruous) information is presented. Moreover, this protocol would also be appropriate for testing other groups of individuals. To integrate these, such participant groups could be added as separate experimental groups: For example, a color perception-group and a shape perception-group;22 a sudden vision loss-group and a progressive vision loss-group;20,21 and/or specific groups of individuals with a hearing impairment and/or an Autism Spectrum Disorder13 could be added.&#160;
With this protocol, it would be possible to establish a group norm (for accuracy and/or exploration time) for specific groups on each instant and long term effect, including each repeated within-group measure, and/or task; then to test individuals against their group norm: Whether any particular individual or the entire group is in need of further training in noticing the shape-audio regularities. A group norm for each instant and long term effect, including each repeated within-group measure, and/or task would also make it possible to determine what kind of training is needed to notice the shape-audio regularities: For example, more trials with the tactile/visual-auditory bouba/kiki-effect (Experiment 1) and/or more trials with congruous and incongruous tactile/visual shape and audio (Experiment 2). Indeed, it would be possible to increase both the number of trials and the number of repeated within-group measures in each task as well as altering the time between them, thereby allowing more individualized training and testing.&#160;
However, this protocol would need modifying if testing what type of materials (e.g., bouba or kiki, filled or outlined tactile/visual shape, and/or auditory sound or word) best ensures the learning of shape-audio regularities. A possible way of modifying this protocol could involve presenting the different materials in blocks: For example, filled tactile/visual shapes and auditory sound and/or outlined tactile/visual shapes and auditory word (Experiment 1). These blocks could be presented in random order to all the participants in one experimental group; or alternatively, each one of the four blocks to separate experimental groups (Experiment 1). This protocol would also need modifying if testing even further the robustness of the visual imagery and/or learned shape-audio regularities within and across tasks. The modification could involve adding repeated within-group measures with different tactile/visual shape and/or audio than those included in the training; for example, with the auditory sound for &#8216;maa-boo-maa&#8217; and &#8216;tuh-kee-tee&#8217; after training with tactile/visual bouba and kiki shapes and auditory words (Experiment 1)1,3,4,10,11,12,13,14,15,16. Another option could involve presenting a third tactile/visual shape which is not mainly curved/angular together with the tactile/visual bouba and kiki shapes in Experiment 110,11,12,13,14,15,16. An auditory nonsense word and a sound could be created for this tactile/visual shape to produce a third shape-word pair14,15,16 and a third shape-sound pair17 (Experiment 1). The shape-sound pair could then be included in Experiment 2 as catch-trials; or alternatively as regular congruous trials and with corresponding incongruous trials. Finally, the sound (from Experiment 2 and Experiment 1) and/or the nonsense word (from Experiment 1) could be presented in Experiment 3. A third option could involve not conducting Experiment 2 and Experiment 3 at all, but rather including them as tasks in the repeated within-group measures in Experiment 1, in random order.
The presented protocol is unique in that it keeps under control both the visual imagery and the sensory information during training and testing. It is flexible: Large and small samples and specific individuals can be tested against themselves (cf. the within-group testing) and/or other groups of individuals (cf. the between-group testing). It is possible to test both accuracy and exploration time, change the number of trials and repeated within-group measures in each experiment and the time between them, and change or even collapse the order of trials and/or experiments.

95% of the world&#8217;s population shows an auditory-visual bouba/kiki-effect, associating the visually rounded shape with the spoken word, &#8216;bouba;&#8217; and the visually angular shape with the spoken word &#8216;kiki;&#8217; and they do this even when they have not had any experience with either the shape or the word1. The bouba/kiki-effect precedes language learning; it occurs across languages1,2,3,4,5, with both bouba and kiki (cf. people&#8217;s preference for visual curves over visual angles6,7,8,9), and it depends on the combination of vowels and consonants (e.g., it does not occur with &#8216;bibi&#8217; and &#8216;kuku&#8217;.)1,5,10,11,12. This shape-word association occurs with other shape and word pairs as well, as long as they are mainly curved, as bouba, and mainly angular, as kiki11,13; and/or they have the same combination of vowels and consonants as &#8216;bouba&#8217; and &#8216;kiki,&#8217; for example: &#8216;Baluma&#8217; and &#8216;takete,&#8217; &#8216;maluma&#8217; and &#8216;takete,&#8217; &#8216;uloomo&#8217; and &#8216;takete,&#8217; &#8216;maa-boo-maa&#8217; and &#8216;tuh-kee-tee&#8217;1,3,4,10,11,12,14. In fact, people associate the rounded visual shapes with the spoken words containing continuant consonants (such as /l/ and /m/) and open back vowels (such as /&#593;:/, and /&#596;:/), and the visual angular shapes with the spoken words containing plosive consonants (such as /k/ and /t/) and close front vowels (such as (/&#949;/ and /I/)11,12,15,16. They are influenced more by the consonants than the vowels, especially by the voiceless ones (e.g., /k/ and /t/)11,12,15,16. Indeed, it seems the features of the global shapes &#8211; their curves and angles17 &#8211; and the sound of the spoken words &#8211; their melody17 &#8211; are the most crucial, as opposed to the global shapes and words themselves.
One study has investigated the bouba/kiki-effect with tactile bouba and kiki instead of the visual bouba/kiki shapes and found that participants who were visually impaired showed a significantly less robust bouba/kiki-effect (~64%) than the fully sighted (~90%)18. This study argued that the significantly less robust bouba/kiki-effect amongst the participants who were blind and partially sighted was caused by a lack of visual imagery: The fully sighted participants had noticed regularities in their environment that are not easily accessed when vision is impaired18. It is not clear from this study, however, whether visual imagery is necessary for the bouba/kiki-effect to occur: Only six of the 42 participants with a visual impairment were congenitally totally blind18, thus had no visual imagery at all. Furthermore, the other participants, none of them blindfolded, may have observed the experimenter&#8217;s lip movements when announcing the bouba/kiki word &#8211; rounded lip movements when announcing &#8216;bouba&#8217; and angular lip movements when announcing &#8216;kiki&#8217;1,18 &#8211; showing a tactile-visual-auditory bouba/kiki-effect instead of drawing upon any visual imagery. Additionally, any effect of noticing tactile-auditory regularities was not investigated, for example, by comparing the bouba/kiki-effect on trial one and trial four instead of calculating the effect across all four trials. Moreover, the tactile bouba and kiki were rather different across trials: In trials one and two they were 3D and 2D shapes (curved v angular); in trials three and four, identical in shape (circle) and dissimilar in texture (smooth v rough; smooth v spiky)18. In a related vein, one study has investigated a kinesthetic-auditory bouba/kiki-effect in blindfolded (fully sighted) individuals and found that, after a two-minute training period of holding a robotic stylus programmed to draw trajectories of the bouba and kiki shapes, 82% showed the bouba/kiki-effect19. It is not clear from this study, however, whether the bouba/kiki-effect occurred because of the training period: The study did not include a pre-test, nor a control group.&#160;
To this end, we have investigated the role of visual imagery in the bouba/kiki-effect, whether training in noticing tactile/visual-auditory bouba and kiki regularities affected the bouba/kiki-effect and the recognition of individual tactile/visual bouba and kiki shapes, and finally what mental images these shape-audio regularities produced17. To generate bouba/kiki shape-audio regularities &#8211; not merely, for example, kiki-shape/kiki-word particularities1 &#8211; this study included two types of tactile/visual bouba/kiki shapes (filled; outlined) and two types of auditory bouba/kiki [audio (word; non-word sound)]17. Three groups of individuals participated in three experiments: Blind (N = 12), blindfold (N = 12), and vision (N = 12). The experiments were conducted in fixed order across participants and with no break between them, to keep under control the participants&#8217; amount and type of experience with bouba and kiki. In Experiment 1 (pre-test-post-test design with three repeated within-group measures) the participants were asked to pick out the tactile/visual shape they associated with the auditory bouba/kiki; in Experiment 2 (within-subject design), to name one tactile/visual shape and some audio (sometimes congruous; sometimes incongruous) as &#8216;bouba&#8217; or &#8216;kiki;&#8217; and in Experiment 3, (post-test only design) to draw the tactile/visual shape they associated with the auditory bouba/kiki17. Overall, this study suggested that the effect of visual imagery is robust across tasks, but not long term within each task, whereas the effect of learning shape-audio regularities is robust long term within each task, but not across tasks17. This article presents the protocol from this study17. The presented protocol is appropriate for investigating the effect of visual imagery and learning shape-audio regularities, when they occur and how robust they are; in specific individuals and groups of individuals. This protocol is unique in that it keeps under control both the visual imagery and the sensory information during training and testing.&#160; &#160;

<table><tbody><tr><td>Bouba/kiki filled</td><td></td><td></td><td>30 mm horizontal, 23 mm Vertical and 0.5 mm thickness m vertical, and 0.5 mm line height, and 0.5 mm embossed embossed height</td></tr><tr><td>Bouba/kiki outlined</td><td></td><td></td><td>30 mm horizontal, 23 mm Vertical and 0.5 mm thickness m vertical, and 0.5 mm line height, and 0.5 mm embossed embossed height</td></tr><tr><td>Bouba/kiki sound (sine wave)</td><td></td><td></td><td></td></tr><tr><td>Bouba/kiki word (human voice)</td><td></td><td></td><td></td></tr><tr><td>Carton plate/foam board for covering bouba/kiki (white)</td><td>Ryman</td><td>230461860</td><td>297 mm horizontal x 210 vertical 0 mm vertical</td></tr><tr><td>Copy paper [white (80 g/m2 )]</td><td>Ryman</td><td>250030000</td><td>210 mm horizontal x 148 mm vertical</td></tr><tr><td>Foam board for bouba/kiki picture cards [white (5 mm thick)]</td><td>Ryman</td><td>230461860</td><td>50 mm horizontal x 50 mm vertical</td></tr><tr><td>MP3-player (smartphone with the VLC app for iOS installed)</td><td>Carphone Warehouse</td><td></td><td></td></tr><tr><td>Plastic embossing film</td><td>RNIB Shop</td><td>ZM04</td><td>210 mm horizontal x 148 mm vertical</td></tr><tr><td>Rollerball pen [black (1.0 mm tip)]</td><td>Ryman</td><td>827134001</td><td></td></tr><tr><td>Rubber mat</td><td>RNIB Shop</td><td>LC177</td><td>230 mm horizontal x 150 mm vertical</td></tr><tr><td>Saddleback style</td><td>RNIB Shop</td><td>B511</td><td></td></tr><tr><td>Swell paper</td><td>Zychem Ltd</td><td>Zytex2 paper</td><td>50 mm horizontal x 50 mm vertical</td></tr></tbody></table>

investigating the effect of visual imagery and learning shape-audio regularities on bouba and kiki

This article presents a protocol for investigating the role of visual imagery in the bouba/kiki-effect, whether training in noticing the bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, and finally what mental images these regularities produce. To generate bouba/kiki shape-audio regularities, there were two types of shapes (filled; outlined) and two types of audio (word; non-word sound). Three groups of individuals participated in three experiments: Blind, blindfold, and vision. The experiments were conducted in fixed order across participants, with no break between them. In Experiment 1 (pre-test-post-test design with three repeated within-group measures) the participants were asked to pick out the shape they associated with the auditory bouba/kiki; in Experiment 2 (within-subject design), to name one shape and some audio (sometimes congruous; sometimes incongruous) as &#8216;bouba&#8217; or &#8216;kiki;&#8217; and in Experiment 3 (post-test only design), to draw the shape they associated with the auditory bouba/kiki. The results suggest that the blindfold-group draw upon visual imagery to solve new problems, but not long term; that training in noticing bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, but differently in each experimental group; and that all experimental groups create mental images of the most characteristic shape feature of bouba (curve) and kiki (angle). In fact, the effect of visual imagery is robust across tasks, but not long term; the effect of learning shape-audio regularities is robust long term, but not across tasks. The presented protocol is appropriate for investigating the effect of visual imagery and learning shape-audio regularities, when they occur and how robust they are; in specific individuals and groups of individuals. This protocol is unique in that it keeps under control both the visual imagery and the sensory information during training and testing.

The bouba/kiki-effect
Six of the 12 participants who were congenitally blind (50%), nine of the 12 who were blindfolded (75%), and 10 of the 12 who were fully sighted (~83%) showed an instant tactile/visual-auditory bouba/kiki-effect: That is, both the blindfold and vision-group were significantly above the chance level (of 50%): &#967;2(1. N = 12) = 3.00, p = 0.08 and &#967;2(1. N = 12) = 5.33, p = 0.02 (Experiment 1, Trial 1)17. No significant difference was found by Fisher&#8217;s exact test between the three experimental groups (Experiment 1, pre-test)17. (Cf. Table 1.)
When it comes to the long term tactile/visual-auditory bouba/kiki-effect: On the first repeated within-group measure, nine of the participants who were congenitally blind showed the bouba/kiki-effect along with nine who were fully sighted and seven who were blindfolded: The blind and vision-group significantly above the chance level (of 50%): &#967;2(1. N = 12) = 3.00, p = 0.08 and &#967;2(1. N = 12) = 3.00, p = 0.08 (Experiment 1, Trial 4)17. On the second repeated within-group measure, 11 participants in the blind and vision-group showed the tactile/visual-auditory bouba/kiki-effect: Both experimental groups were again significantly above the chance level (of 50%): &#967;2(1. N = 12) = 8.33, p = 0.00 and &#967;2(1. N = 12) = 8.33, p = 0.00; and seven in the blindfold-group (Experiment 1, Trial 5)17. Finally, on the third repeated within-group measure and post-test, nine of the 12 participants who were congenitally blind (75%), six of the 12 who were blindfolded (50%), and all of the 12 participants who were fully sighted (100%) showed the tactile/visual-auditory bouba/kiki-effect; both the blind and vision-group were again significantly above the chance level (of 50%): &#967;2(1. N = 12) = 3.00, p = 0.08 and &#967;2(1. N = 12) = 12.00, p = 0.00 (Experiment 1, Trial 8)17. Fisher&#8217;s exact test found a significant difference between the blindfold and vision-group [p = 0.01. (Experiment 1, post-test)]17. (Cf. Table 1.)
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td rowspan="2">Participant group</td>
			<td>Instant bouba/kiki- effect</td>
			<td colspan="3">Long term bouba/kiki-effect</td>
		</tr>
		<tr>
			<td>Pre-test</td>
			<td>Repeated measure 1</td>
			<td>Repeated measure 2</td>
			<td>Repeated measure 3/post-test</td>
		</tr>
		<tr>
			<td>Blind</td>
			<td>50.0%</td>
			<td>75.0%</td>
			<td>91.7%</td>
			<td>75.0%</td>
		</tr>
		<tr>
			<td>Blindfold</td>
			<td>75.0%</td>
			<td>58.3%</td>
			<td>58.3%</td>
			<td>50.0%</td>
		</tr>
		<tr>
			<td>Vision</td>
			<td>83.3%</td>
			<td>75.0%</td>
			<td>91.7%</td>
			<td>100.0%</td>
		</tr>
	</tbody>
</table>
Table 1: The instant and long term bouba/kiki-effect
The recognition of bouba and kiki shapes
Eleven of the 12 participants who were congenitally blind (~92%), nine of the 12 who were blindfolded (75%), and all of the 12 who were fully sighted (100%) instantly recognized the congruous tactile/visual and auditory bouba/kiki; all three experimental groups were significantly above the chance level (of 50%): &#967;2(1. N = 12) = 8.33, p = 0.00, &#967;2(1. N = 12) = 3.00, p = 0.08, and &#967;2(1. N = 12) = 12.00, p = 0.00 (Experiment 2, Trial 1)17. (Cf. Table 2.)
Long term, 11 participants in the blind-group recognized the tactile bouba/kiki shapes together with congruous audio and 10 participants with incongruous audio: Both types of congruousness recognized significantly above the chance level (of 50%): &#967;2(1. N = 12) = 8.33, p = 0.00 and &#967;2(1. N = 12) = 5.33, p = 0.02 (Experiment 2, Trial 1-8)17. Nine participants in the blindfold-group recognized the tactile shapes together with congruous audio and eight participants together with incongruous audio; in other words, the congruous shape and audio were significantly above the chance level (of 50%): &#967;2(1. N = 12) = 3.00, p = 0.08 (Experiment 2, Trial 1-8)17. All 12 participants in the vision-group recognized the visual bouba/kiki shapes together with congruous audio and six participants with incongruous audio: The congruous shape and audio recognized significantly above the chance level (of 50%): &#967;2(1. N = 12) = 12.00, p = 0.00 (Experiment 2, Trial 1-8)17. (Cf. Table 2.)
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td rowspan="2">Participant group</td>
			<td>Instant recognition of bouba/kiki shape</td>
			<td colspan="2">Long term recognition of bouba and kiki shapes</td>
		</tr>
		<tr>
			<td>Congruous shape and audio</td>
			<td>Congruous shape and audio</td>
			<td>Incongruous shape and audio</td>
		</tr>
		<tr>
			<td>Blind</td>
			<td>91.7%</td>
			<td>91.7%</td>
			<td>83.3%</td>
		</tr>
		<tr>
			<td>Blindfold</td>
			<td>75.0%</td>
			<td>75.0%</td>
			<td>66.7%</td>
		</tr>
		<tr>
			<td>Vision</td>
			<td>100.0%</td>
			<td>100%</td>
			<td>50.0%</td>
		</tr>
	</tbody>
</table>
Table 2: The instant and long term recognition of bouba and kiki shapes
The mental images of bouba and kiki
Eight of the 12 participants who were congenitally blind [~73% (with one &#8216;inconclusive&#8217; participant drawing removed)], eight of the 12 who were blindfolded [~89% (with three &#8216;inconclusive&#8217; participant drawings removed)], and eight of the 12 who were fully sighted [80% (with two &#8216;inconclusive&#8217; participant drawings removed)] instantly drew a mental image: A tactile/visual shape that corresponded to the presented auditory bouba/kiki (Experiment 3, Trial 1)17. Both the blindfold and vision-group were significantly above the chance level (of 50%): &#967;2(1. N = 9) = 5.44, p = 0.02 and &#967;2(1. N = 10) = 3.60, p = 0.06 (Experiment 3, Trial 1)17. (Cf. Table 3.)
Regarding the long-term mental images of bouba and kiki: 11 participants in the blind-group, eight in the blindfold-group, and 12 in the vision-group drew tactile/visual bouba/kiki shapes that corresponded to the presented auditory bouba/kiki (Experiment 3, Trial 1-4)17. Both the blind and vision-group were significantly above the chance level (of 50%): &#967;2(1. N = 12) = 8.33, p = 0.00 and &#967;2(1. N = 12) = 12.00, p = 0.00 (Experiment 3, Trial 1-4)17. Fisher&#8217;s exact test found a significant difference between the blindfold-group and the vision-group [p = 0.09 (Experiment 3, Trial 1-4)]17. (Cf. Table 3.)
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>Participant group</td>
			<td>Instant mental image of bouba/kiki</td>
			<td>Long term mental images of bouba and kiki</td>
		</tr>
		<tr>
			<td>Blind</td>
			<td>72.7%</td>
			<td>91.7%</td>
		</tr>
		<tr>
			<td>Blindfold</td>
			<td>88.9%</td>
			<td>66.7%</td>
		</tr>
		<tr>
			<td>Vision</td>
			<td>80.0%</td>
			<td>100.0%</td>
		</tr>
	</tbody>
</table>
Table 3: The instant and long term mental images of bouba and kiki
Furthermore. ~83% of all participant drawings included the most characteristic shape feature of the global bouba and kiki shapes: Curve and angle, respectively (Experiment 3, Trial 1-4)17. The participant drawings differed in the quantity of curves/angles (e.g., one and five angles for the kiki word: cf. Figure 2, Trials 1, 6, and 9), and in the direction of the curves/angles [i.e. horizontal, vertical or diagonal: cf. Figure 2, Trials 2, 5, and 8 (bouba sound)], but typically did not include the global bouba/kiki shape [Experiment 3, Trial 1-4 (cf. Figure 1; Figure 2)]17. Finally, the experimental group was recognized in ~43% of the scores&#8217; answers: Five participants in the blind-group, three in the blindfold-group, and five in the vision-group; no experimental group was significantly above the chance level [of 33.3% (Experiment 3, Trial 1-4)]17.
The effect of visual imagery and learning
The presented protocol succeeded in investigating the role of visual imagery in the bouba/kiki-effect, whether training in noticing tactile/visual-auditory bouba and kiki regularities affected the bouba/kiki-effect and the recognition of individual tactile/visual bouba and kiki shapes, and finally what mental images these shape-audio regularities produced17. By including one experimental group with no visual experience (i.e. the blind-group) and two experimental groups with visual experience (i.e. the blindfold and vision-group), it was possible to test the effect of visual imagery; and by including one experimental group with no visual experience and extensive tactile experience (i.e. the blind-group) and one experimental group with extensive visual experience and no tactile experience (i.e. the blindfold-group), it was possible to test the effect of the training in noticing tactile-auditory regularities. The results clearly suggest that the blindfold-group drew upon visual imagery to solve new tactile-auditory problems, but not long term (Experiment 1&#8211;3); that training in noticing bouba/kiki shape-audio regularities affected the tactile/visual-auditory bouba/kiki-effect (Experiment 1) and the recognition of individual tactile/visual bouba and kiki shapes [i.e. together with congruous audio (Experiment 2)], but differently in each experimental group (Experiment 1-2), and that all experimental groups created mental images of the most characteristic tactile/visual-auditory shape feature of bouba (curve) and kiki [(angle) Experiment 3. Cf. Table 1; Table 2; Table 3]17. Moreover, by including repeated within-group measures of instant and long term effect (Experiment 1-3), it was possible to test when the effect of visual imagery and learning shape-audio regularities actually occurred; and by including three different tasks (i.e. the tactile/visual-auditory bouba/kiki-effect in Experiment 1, recognizing individual tactile/visual bouba/kiki shapes together with congruous and incongruous audio in Experiment 2, and drawing mental images of the bouba/kiki audio in Experiment 3), to test how robust these effects were. The results clearly suggest that the effect of visual imagery was not robust long term within each task, especially when using haptic touch [cf. the blindfold-group (Experiment 1-3)]; whereas the effect of learning shape-audio regularities was, but not across tasks [cf. the blind and vision-group (Experiment 1-3). Cf. Table 1; Table 2; Table 3.]17

Watch this Scientific Journal Video about Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki at JoVE.com

Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki

The aim of the presented protocol is to investigate the role of visual imagery in the bouba/kiki-effect, whether training in noticing the bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, and finally what mental images these regularities produce.

This article presents a protocol for investigating the role of visual imagery in the bouba/kiki-effect, whether training in noticing the bouba/kiki ...

investigating-effect-visual-imagery-learning-shape-audio-regularities

Universidad de Cartagena UDEC

Research

JoVE Journal

Behavior

10.0K Views.  University of Oxford. The aim of the presented protocol is to investigate the role of visual imagery in the bouba/kiki-effect, whether training in noticing the bouba/kiki shape-audio regularities affects the bouba/kiki-effect and the recognition of individual bouba and kiki shapes, and finally what mental images these regularities produce.

Video: Investigating the Effect of Visual Imagery and Learning Shape-Audio Regularities on Bouba and Kiki

Automated Visual Cognitive Tasks for Recording Neural Activity Using a Floor Projection Maze

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have developed visual cognitive tasks for rats using the Floor Projection Maze1,2&#160;that are optimized for visual abilities of rats permitting stronger comparisons of experimental findings with other species.
In order to investigate neural correlates of learning and memory, we have integrated electrophysiological recordings into fully automated cognitive tasks on the Floor Projection Maze1,2. Behavioral software interfaced with an animal tracking system allows monitoring of the animal's behavior with precise control of image presentation and reward contingencies for better trained animals. Integration with an in vivo electrophysiological recording system enables examination of behavioral correlates of neural activity at selected epochs of a given cognitive task.
We describe protocols for a model system that combines automated visual presentation of information to rodents and intracranial reward with electrophysiological approaches. Our model system offers a sophisticated set of tools as a framework for other cognitive tasks to better isolate and identify specific mechanisms contributing to particular cognitive processes.

We describe protocols for training rats for chronic electrophysiological recordings in fully automated cognitive tasks on a Floor Projection Maze.

Neuropsychological tasks used in primates to investigate mechanisms of learning and memory are typically visually guided cognitive tasks. We have ...

Experience is Instrumental in Tuning a Link Between Language and Cognition: Evidence from 6-  to 7- Month-Old Infants' Object Categorization

At birth, infants not only prefer listening to human vocalizations, but also have begun to link these vocalizations to cognition: For infants as young as three months of age, listening to human language supports object categorization, a core cognitive capacity. This precocious link is initially broad: At 3 and 4 months, vocalizations of both humans and nonhuman primates support categorization. But by 6 months, infants have narrowed the link: Only human vocalizations support object categorization. Here we ask what guides infants as they tune their initially broad link to a more precise one, engaged only by the vocalizations of our species. Across three studies, we use a novel exposure paradigm to examine the effects of experience. We document that merely exposing infants to nonhuman primate vocalizations enables infants to preserve the early-established link between this signal and categorization. In contrast, exposing infants to backward speech &#8212; a signal that fails to support categorization at any age &#8212; offers no such advantage. Our findings reveal the power of early experience as infants specify which signals, from an initially broad set, they will continue to link to cognition.

At 3-4 months, listening to human and nonhuman primate vocalizations boosts infants&#39; cognition; by 6 months, only human vocalizations exert this cognitive advantage. We describe an exposure manipulation that reveals the powerful shaping role of experience as infants specify which sounds to link to cognition and which to tune out.

At birth, infants not only prefer listening to human vocalizations, but also have begun to link these vocalizations to cognition: For infants as young as ...

Creating Objects and Object Categories for Studying Perception and Perceptual Learning

In order to quantitatively study object perception, be it perception by biological systems or by machines, one needs to create objects and object categories with precisely definable, preferably naturalistic, properties1. Furthermore, for studies on perceptual learning, it is useful to create novel objects and object categories (or object classes) with such properties2.
Many innovative and useful methods currently exist for creating novel objects and object categories3-6 (also see refs. 7,8). However, generally speaking, the existing methods have three broad types of shortcomings.
First, shape variations are generally imposed by the experimenter5,9,10, and may therefore be different from the variability in natural categories, and optimized for a particular recognition algorithm. It would be desirable to have the variations arise independently of the externally imposed constraints.
Second, the existing methods have difficulty capturing the shape complexity of natural objects11-13. If the goal is to study natural object perception, it is desirable for objects and object categories to be naturalistic, so as to avoid possible confounds and special cases.
Third, it is generally hard to quantitatively measure the available information in the stimuli created by conventional methods. It would be desirable to create objects and object categories where the available information can be precisely measured and, where necessary, systematically manipulated (or 'tuned'). This allows one to formulate the underlying object recognition tasks in quantitative terms.
Here we describe a set of algorithms, or methods, that meet all three of the above criteria. Virtual morphogenesis (VM) creates novel, naturalistic virtual 3-D objects called 'digital embryos' by simulating the biological process of embryogenesis14. Virtual phylogenesis (VP) creates novel, naturalistic object categories by simulating the evolutionary process of natural selection9,12,13. Objects and object categories created by these simulations can be further manipulated by various morphing methods to generate systematic variations of shape characteristics15,16. The VP and morphing methods can also be applied, in principle, to novel virtual objects other than digital embryos, or to virtual versions of real-world objects9,13. Virtual objects created in this fashion can be rendered as visual images using a conventional graphical toolkit, with desired manipulations of surface texture, illumination, size, viewpoint and background. The virtual objects can also be 'printed' as haptic objects using a conventional 3-D prototyper.
We also describe some implementations of these computational algorithms to help illustrate the potential utility of the algorithms. It is important to distinguish the algorithms from their implementations. The implementations are demonstrations offered solely as a 'proof of principle' of the underlying algorithms. It is important to note that, in general, an implementation of a computational algorithm often has limitations that the algorithm itself does not have.
Together, these methods represent a set of powerful and flexible tools for studying object recognition and perceptual learning by biological and computational systems alike. With appropriate extensions, these methods may also prove useful in the study of morphogenesis and phylogenesis.

We describe a novel methodology for creating naturalistic 3-D objects and object categories with precisely defined feature variations. We use simulations of the biological processes of morphogenesis and phylogenesis to create novel, naturalistic virtual 3-D objects and object categories that can then be rendered as visual images or haptic objects.

In order to quantitatively study object perception, be it perception by biological systems or by machines, one needs to create objects and object ...

Neuroscience

Portable Intermodal Preferential Looking (IPL): Investigating Language Comprehension in Typically Developing Toddlers and Young Children with Autism

One of the defining characteristics of autism spectrum disorder (ASD) is difficulty with language and communication.1 Children with ASD's onset of speaking is usually delayed, and many children with ASD consistently produce language less frequently and of lower lexical and grammatical complexity than their typically developing (TD) peers.6,8,12,23 However, children with ASD also exhibit a significant social deficit, and researchers and clinicians continue to debate the extent to which the deficits in social interaction account for or contribute to the deficits in language production.5,14,19,25
Standardized assessments of language in children with ASD usually do include a comprehension component; however, many such comprehension tasks assess just one aspect of language (e.g., vocabulary),5 or include a significant motor component (e.g., pointing, act-out), and/or require children to deliberately choose between a number of alternatives. These last two behaviors are known to also be challenging to children with ASD.7,12,13,16
We present a method which can assess the language comprehension of young typically developing children (9-36 months) and children with autism.2,4,9,11,22 This method, Portable Intermodal Preferential Looking (P-IPL), projects side-by-side video images from a laptop onto a portable screen. The video images are paired first with a 'baseline' (nondirecting) audio, and then presented again paired with a 'test' linguistic audio that matches only one of the video images. Children's eye movements while watching the video are filmed and later coded. Children who understand the linguistic audio will look more quickly to, and longer at, the video that matches the linguistic audio.2,4,11,18,22,26
This paradigm includes a number of components that have recently been miniaturized (projector, camcorder, digitizer) to enable portability and easy setup in children's homes. This is a crucial point for assessing young children with ASD, who are frequently uncomfortable in new (e.g., laboratory) settings. Videos can be created to assess a wide range of specific components of linguistic knowledge, such as Subject-Verb-Object word order, wh-questions, and tense/aspect suffixes on verbs; videos can also assess principles of word learning such as a noun bias, a shape bias, and syntactic bootstrapping.10,14,17,21,24 Videos include characters and speech that are visually and acoustically salient and well tolerated by children with ASD.

Portable Intermodal Preferential Looking IPL: Investigating Language Comprehension in Typically Developing Toddlers and Young Children with Autism

A reliable home-based way to assess the language comprehension of very young typically developing children, as well as those with autism, is described. The method analyzes children's eye gaze while viewing side-by-side images but hearing an audio that matches only one image. Stimuli are designed with young participants in mind.

One  of the defining characteristics of autism spectrum disorder (ASD) is difficulty  with language and communication.1  Children with ASD's onset of ...

Medicine

Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

Statistical learning, a fundamental skill to extract regularities in the environment, is often considered a core supporting mechanism of the first language development. While many studies of statistical learning are conducted within a single domain or modality, recent evidence suggests that this skill may differ based on the context in which the stimuli are presented. In addition, few studies investigate learning as it unfolds in real-time, rather focusing on the outcome of learning. In this protocol, we describe an approach for identifying the cognitive and neural basis of statistical learning, within an individual, across domains (linguistic vs. non-linguistic) and sensory modalities (visual and auditory). The tasks are designed to cast as little cognitive demand as possible on participants, making it ideal for young school-aged children and special populations. The web-based nature of the behavioral tasks offers a unique opportunity for us to reach more representative populations nationwide, to estimate effect sizes with greater precision, and to contribute to open and reproducible research. The neural measures provided by the functional magnetic resonance imaging (fMRI) task can inform researchers about the neural mechanisms engaged during statistical learning, and how these may differ across individuals on the basis of domain or modality. Finally, both tasks allow for the measurement of real-time learning, as changes in reaction time to a target stimulus is tracked across the exposure period. The main limitation of using this protocol relates to the hour-long duration of the experiment. Children might need to complete all four statistical learning tasks in multiple sittings. Therefore, the web-based platform is designed with this limitation in mind so that tasks may be disseminated individually. This methodology will allow users to investigate how the process of statistical learning unfolds across and within domains and modalities in children from different developmental backgrounds.

Presented here is a protocol introducing a set of child-friendly statistical learning tasks geared towards examining children&#8217;s learning of temporal statistical patterns across domains and sensory modalities. The developed tasks collect behavioral data using the web-based platform and task-based functional magnetic resonance imaging (fMRI) data for examining neural engagement during statistical learning.

Statistical learning, a fundamental skill to extract regularities in the environment, is often considered a core supporting mechanism of the first ...

Habituation: Studying Infants Before They Can Talk

Source: Laboratories of Nicholaus Noles, Judith Danovitch, and Cara Cashon&#8212;University of Louisville
Infants are one of the purest sources of information about human thinking and learning, because they&#8217;ve had very few life experiences. Thus, researchers are interested in gathering data from infants, but as participants in experimental research, they are a challenging group to study. Unlike older children and adults, young infants are unable to reliably speak, understand speech, or even move and control their own bodies. Eating, sleeping, and looking around are the only activities babies can perform reliably. Given these limitations, researchers have developed clever techniques for exploring infants&#8217; thoughts. One of the most popular methods makes use of a characteristic of attention called habituation.
Like adults, infants prefer to pay attention to new and interesting things. If they are left in the same environment, over time they become accustomed to their surroundings and pay less attention to them. This process is called habituation. However, the moment something new happens, infants are waiting and ready to pay attention again. This reengagement of attention following habituation is referred to as dishabituation. Scientists can use these characteristic changes in attention as a tool for studying the thinking and learning of young infants. This method involves initially presenting stimuli to infants until they are habituated, and then presenting them with different kinds of stimuli to see if they dishabituate, i.e., notice a change. By carefully choosing the stimuli shown to the infants, researchers can learn a great deal about how infants think and learn.
This experiment demonstrates how habituation can be used to study infant shape discrimination.

Source: Laboratories of Nicholaus Noles, Judith Danovitch, and Cara Cashon&#8212;University of Louisville
Infants are one of the purest sources of ...

Education

JoVE Science Education

Psychology

Developmental Psychology

The McGurk Effect

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University
Spoken language, a singular human achievement, relies heavily on specialized perceptual mechanisms. One important feature of language perception mechanisms is that they simultaneously rely on auditory and visual information. This makes sense, because until modern times, a person could expect that most language would be heard in face-to-face interactions. And because producing specific speech sounds requires precise articulation, the mouth can supply good visual information about what someone is saying. In fact, with an up-close and unobstructed view of someone&#39;s face, the mouth can often supply better visual signals than speech supplies auditory signals. The result is that the human brain favors visual input, and uses it to disambiguate inherent ambiguity in spoken language.
This reliance on visual input to interpret sound was described by Harry McGurk and John Macdonald in a paper in 1976 called Hearing lips and seeing voices.1 In that paper, they described an illusion that arises through a mismatch between a sound recording and a video recording. That illusion has become known as the McGurk effect. This video will demonstrate how to produce and interpret the McGurk effect.

The McGurk Effect: Audiovisual Illusion and Speech Perception

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University
Spoken language, a singular human achievement, relies heavily on specialized ...

Sensation and Perception

Decoding Auditory Imagery with Multivoxel Pattern Analysis

Source: Laboratories of Jonas T. Kaplan and Sarah I. Gimbel&#8212;University of Southern California
Imagine the sound of a bell ringing. What is happening in the brain when we conjure up a sound like this in the "mind's ear?" There is growing evidence that the brain uses the same mechanisms for imagination that it uses for perception.1 For example, when imagining visual images, the visual cortex becomes activated, and when imagining sounds, the auditory cortex is engaged. However, to what extent are these activations of sensory cortices specific to the content of our imaginations? 
One technique that can help to answer this question is multivoxel pattern analysis (MPVA), in which functional brain images are analyzed using machine-learning techniques.2-3 In an MPVA experiment, we train a machine-learning algorithm to distinguish among the various patterns of activity evoked by different stimuli. For example, we might ask if imagining the sound of a bell produces different patterns of activity in auditory cortex compared with imagining the sound of a chainsaw, or the sound of a violin. If our classifier learns to tell apart the brain activity patterns produced by these three stimuli, then we can conclude that the auditory cortex is activated in a distinct way by each stimulus. One way to think of this kind of experiment is that instead of asking a question simply about the activity of a brain region, we ask a question about the information content of that region. 
In this experiment, based on Meyer et al., 2010,4 we will cue participants to imagine several sounds by presenting them with silent videos that are likely to evoke auditory imagery. Since we are interested in measuring the subtle patterns evoked by imagination in auditory cortex, it is preferable if the stimuli are presented in complete silence, without interference from the loud noises made by the fMRI scanner. To achieve this, we will use a special kind of functional MRI sequence known as sparse temporal sampling. In this approach, a single fMRI volume is acquired 4-5 s after each stimulus, timed to capture the peak of the hemodynamic response.

Source: Laboratories of Jonas T. Kaplan and Sarah I. Gimbel&#8212;University of Southern California
Imagine the sound of a bell ringing. What is happening ...

Neuropsychology

Children's Reliance on Artist Intentions When Identifying Pictures

Source: Laboratories of Judith Danovitch and Nicholaus Noles&#8212;University of Louisville
Children are not the best artists. Sometimes it&#8217;s easy to pick out the characteristic triangular head, whiskers, and tail of a cat, but children often describe elaborate scenarios that they depict as a beautifully unrecognizable mess. Thus, given children&#8217;s questionable artistic talent, how do they know what their drawings, and the drawings of others, represent? One way children identify pictures is by relying on resemblance. If it looks like a cat, then it&#8217;s a cat. However, some pictures do not clearly resemble any real object. In this situation, children must use other means to figure out what the picture represents, including their understanding of what the person who created the picture intended it to represent.
By their first birthday, children are sensitive to the intentions of other people. They know that people&#8217;s actions are driven by their goals, and they can infer a person&#8217;s intentions even if the goal-directed action is not successful (e.g., they understand a person struggling to turn a lid intends to open a jar, even if they never see them succeed in opening it). By about age 3, children can use this understanding of intention to guide their interpretation of drawings and other pictorial representations. They apply this understanding both to identifying their own drawings, and to interpreting drawings created by another person.
This experiment demonstrates how to measure children&#8217;s use of intention to interpret otherwise ambiguous pictures based on the methods developed by Bloom and Markson.1

Children&#039;s Reliance on Artist Intentions When Identifying Pictures

Source: Laboratories of Judith Danovitch and Nicholaus Noles&#8212;University of Louisville
Children are not the best artists. Sometimes it&#8217;s easy ...

Visual Statistical Learning

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University

The visual environment contains massive amounts of information involving the relations between objects in space and time; certain objects are more likely to appear in the vicinity of other objects. Learning these regularities can support a wide array of visual processing, including object recognition. Unsurprisingly, then, humans appear to learn these regularities automatically, quickly, and without conscious awareness. The name for this type of implicit learning is visual statistical learning. In the laboratory, it is studied with an incidental-encoding paradigm: participants observe a stream of nonsense objects and complete a cover-task, a task unrelated to the underlying statistical structure in the stream. But statistical structure is present, and subsequent to a short exposure period&#8212;as short as 10 min in some experiments&#8212;a familiarity test reveals the extent of learning by the participants.
This video will demonstrate standard methods for inducing and testing visual statistical learning.

Incidental Encoding: Testing Visual Statistical Learning

Source: Laboratory of Jonathan Flombaum&#8212;Johns Hopkins University

The visual environment contains massive amounts of information involving the ...