About Me

Virtual reality (VR), inherently reliant on spatial interaction, poses significant accessibility barriers for individuals who are blind or have low vision (BLV). Traditional audio descriptions (AD) typically provide a verbal explanation of visual elements in 2D or flat video media, facilitating access for BLV audiences but failing to convey the complex spatial information essential in VR. This shortfall is especially pronounced in musical performances, where understanding the spatial arrangement of the stage setup and movements of performers is crucial. To overcome these limitations, we have developed two AD approaches—Spatial AD for a dance performance and View-dependent AD for an instrumental performance—within VR-based 360° environments. Spatial AD employs spatial audio technology to align descriptions with corresponding visuals, dynamically adjusting to follow the visuals, such as the movements of performers in the dance performance. Meanwhile, View-dependent AD adapts descriptions based on the orientation of the VR headset, activating when particular visuals enter the central view of the camera, ensuring that the description aligns with the user's attention directed to a particular location within the VR environment. These methods are designed as enhancements to traditional AD, aiming to improve spatial orientation and immersive experiences for BLV audiences. This demonstration showcases the potential of these AD approaches to improve interaction and engagement, furthering the development of inclusive virtual environments.

Read Article

DOI

Our research focuses on making musical performance experience in virtual reality (VR) settings non-visually accessible for Blind and Low Vision (BLV) individuals by designing a conceptual framework for omnidirectional audio descriptions (AD). We address BLV users' prevalent challenges in accessing effective AD during VR musical performances. Employing a two-phased interview methodology, we initially collected qualitative data about BLV AD users' experiences, followed by gathering insights from BLV professionals who specialize in AD. This approach ensures that the developed solutions are both user-centric and practically feasible. The study devises strategies for three design concepts of omnidirectional AD (Spatial AD, View-dependent AD, and Explorative AD) tailored to different types of musical performances, which vary in their visual and auditory components. Each design concept offers unique benefits; collectively, they enhance accessibility and enjoyment for BLV audiences by addressing specific user needs. Key insights highlight the crucial role of flexibility and user control in AD implementation. Based on these insights, we propose a comprehensive conceptual framework to enhance musical experiences for BLV users within VR environments.

Read Article

DOI

High-quality closed captioning of both speech and non-speech elements (e.g., music, sound effects, manner of speaking, and speaker identification) is essential for the accessibility of video content, especially for d/Deaf and hard-of-hearing individuals. While many regions have regulations mandating captioning for television and movies, a regulatory gap remains for the vast amount of web-based video content, including the staggering 500+ hours uploaded to YouTube every minute. Advances in automatic speech recognition have bolstered the presence of captions on YouTube. However, the technology has notable limitations, including the omission of many non-speech elements, which are often crucial for understanding content narratives. This paper examines the contemporary and historical state of non-speech information (NSI) captioning on YouTube through the creation and exploratory analysis of a dataset of over 715k videos. We identify factors that influence NSI caption practices and suggest avenues for future research to enhance the accessibility of online video content.

Read Article

DOI

Blind and low-vision (BLV) individuals often face challenges in their attendance and appreciation of musical performances (e.g., concerts, musicals, opera) due to limited mobility and accessibility of visual information. However, the emergence of Virtual Reality (VR) based musical performance as a common medium of music access opens up opportunities to mitigate the challenges and enhance the musical experiences by investigating non-visual VR accessibility. This study aims to 1) gain an in-depth understanding of the experiences of BLV individuals, including their preferences, challenges, and needs in listening to and accessing various modes (audio, video, and on-site experiences) of music and musical performances and 2) explore the opportunities that VR can create for making the immersive musical experiences accessible for BLV people. Using a mixed-methods approach, we conducted an online survey and a semi-structured interview study with 102 and 25 BLV participants, respectively. Our findings suggest design opportunities for making the VR space non-visually accessible for BLV individuals, enabling them to participate equally in the VR world and to further access immersive musical performances created by VR technology. Our research contributes to the growing body of knowledge on accessibility in virtual environments, particularly in the context of music listening and appreciation for BLV individuals.

Read Article

DOI

Safety-critical sounds at job sites play an essential role in construction safety, but hearing capability is often declined due to the use of hearing protection and the complicated nature of construction noise. Thus, preserving or augmenting the auditory situational awareness of construction workers has become a critical need. To enable further advances in this area, it is necessary to synthesize the state-of-the-art auditory signal processing techniques and their implications for auditory situational awareness (ASA) and to identify future research needs. This paper presents a critical review of recent publications on acoustic signal processing techniques and suggests research gaps that merit further research for fully embracing construction workers’ ASA of hazardous situations in construction. The results from the content analysis show that research on ASA in the context of construction safety is still in its early stage, with inadequate AI-based sound sensing methods available. Little research has been undertaken to augment individual construction workers in recognizing important signals that may be blocked or mixed with complex ambient noise. Further research on auditory situational awareness technology is needed to support detecting and separating important acoustic safety cues from complex ambient sounds. More work is also needed to incorporate context information into sound-based hazard detection and to investigate human factors affecting the collaboration between workers and AI assistants in sensing the safety cues of hazards.

Read Article

DOI

Collisions between workers and operating vehicles are the leading source of fatal incidents in the construction industry. One of the most prevalent factors causing contact hazards is the decline in construction workers' auditory situational awareness due to the hearing loss and the complicated nature of construction noises. Thus, a computational technique that can augment the audible sense of a worker can significantly improve safety performance. Since construction machines often generate distinct sound patterns while operating at the construction sites, audio signal processing could be an innovative solution to achieve the goal. Unfortunately, the current body of knowledge regarding automated surveillance in construction still lacks such advanced methods. This paper presents a newly developed auditory surveillance framework using convolutional neural networks (CNNs) that can detect collision hazards by processing acoustic signals in construction sites. The study specifically has two primary contributions: (1) a new labeled dataset of normal and abnormal sound events relating to collision hazards in the construction site, and (2) a novel audio-based machine learning model for automated detection of collision hazards. The model was trained with different network architectures, and its performance was evaluated using various measures, including accuracy, recall, precision, and combined F-measure. The research is expected to help increase the auditory situational awareness of construction workers and consequently enhance construction safety.

Read Article

DOI