Project Description

l-RIM: Learning based Resilient Immersive Media-Compression, Delivery, and Interaction

Augmented Reality/Virtual Reality (AR/VR) has been recognized as a transformative service for the Next Generation (NextG) network systems, while wireless supported AR/VR will offer great flexibility and enhanced immersive experience to users and unleash a plethora of new applications. The primary goal of this project is to explore innovative technologies in NextG and Artificial intelligence (AI) to provide a unified media compression, communication, and computing framework to enable resilient wireless AR/VR.

May 1, 2022 ~ Apr. 30, 2025

Project Team

• Shiwen Mao

• Zhu Li

• Ticao Zhang

• Anthony Chen

• Chris Henry

• Biren Kathariya

• Yangfan Sun

Related Publications (journal & magazine)

• M. Xu, D. Niyato, B. Wright, H. Zhang, J. Kang, Z. Xiong, S. Mao, and Z. Han, “EPViSA: Efficient auction design for real-time physical-virtual synchronization in the metaver,” IEEE Journal on Selected Areas in Communications, Special Issue on Human-Centric Communication and Networking for Metaverse over 5G and Beyond Networks, to appear.

• M.Xu, D. Niyato, J. Chen, H. Zhang, J. Kang, Z. Xiong, S. Mao, and Z. Han, “Generative AI-empowered simulation for autonomous driving in vehicular mixed reality metaverses,” IEEE Journal of Selected Topics in Signal Processing, Special Issue on Signal Processing for XR Communications and Systems, to appear. DOI: 10.1109/JSTSP.2023.3293650.

• Q.Luo, H. Zhang, B. Di, M. Xu, A. Chen, S. Mao, D. Niyato, and Z. Han, “An overview of 3GPP standardization for extended reality (XR) in 5G and beyond,” ACM GetMobile, vol.27, no.3, pp.10-17, Sept. 2023.

• Y. Sun, L. Li, Z. Li, S. Wang, S. Liu, and G. Li, “Learning a compact spatial-angular representation for light field,” IEEE Transactions on Multimedia, to appear. DOI: 10.1109/TMM.2022.3219671.

• Y. Sun, Z. Li, S. Wang, and W. Gao, “Learning-based depth-guided light field factorization for compressive light field display,” Optical Express, to appear.

• A. Chen, S. Mao, Z. Li, M. Xu, H. Zhang, D. Niyato, and Z. Han, “An introduction to point cloud compression standards,” ACM GetMobile, vol.27, no.1, pp.11-17, Mar. 2023.

• H. Zhang, S. Mao, D. Niyato, and Z. Han, “Location-dependent augmented reality services in wireless edge-enabled metaverse systems,” IEEE Open Journal of the Communications Society, vol.4, pp.171-183, Jan. 2023. DOI: 10.1109/OJCOMS.2023.3234254.

• Y. Sun, J. Chen, Z. Wang, M. Peng, and S. Mao, “Enabling mobile virtual reality with Open 5G, fog computing and reinforcement learning,” IEEE Network, vol.36, no.6, pp.142-149, Nov./Dec. 2022. DOI: 10.1109/MNET.010.2100481.

• J. Dai, G. Yue, S. Mao, and D. Liu, “Sidelink-aided multi-quality tiled 360° virtual reality video multicast,” IEEE Internet of Things Journal, vol.9, no.6, pp.4584-4597, Mar. 2022. DOI: 10.1109/JIOT.2021.3105100.

Related Publications (conference)

• P. Chen, B. Chen, M. Wang, S. Wang, and Z. Li, "Visual Data Compression for Metaverse: Technology, Standard, and Challenges," in Proc. IEEE MetaCom 2023, Kyoto, Japan, June 2023.

• C. Henry, M. S. Asif, and Z. Li, "Privacy preserving face recognition with lensless camera," in Proc. IEEE Int'l Conf on Audio, Speech and Signal Processing (ICASSP'23), Rhode Island, Greece, June 2023.

• C. Henry, M. S. Asif, and Z. Li, "Light-weight fisher vector transfer leanrning for video deduplication," in Proc. IEEE Int'l Conf on Audio, Speech and Signal Processing (ICASSP'23), Rhode Island, Greece, June 2023.

• R.Puttagunta, Z. Li, S. Battacharyya, and G. York, "Appearance label balanced triplet loss for multi-modal aerial view object classification," in Proc. IEEE CVPR Workshop on Perception Beyond Visual Spectrum (PBVS'23), Vancouver, Canada, June 2023.

• M. Xu, D. Niyato, H. Zhang, J. Kang, Z. Xiong, S. Mao, and Z. Han, “Generative AI-empowered effective physical-virtual synchronization in the vehicular metaverse,” in Proc. International Conference on Metaverse Computing, Networking and Applications (MetaCom 2023), Kyoto, Japan, June 2023.

• Z. Liu, F. Li, Z. Li, and B. Luo, “LoneNeuron: a highly-effective feature-domain neural trojan using invisible and polymorphic watermarks,” in Proc. ACM Conference on Computer and Communications Security (CSS'22), Los Angeles, CA, Nov. 2022, pp.2129-2143.

• B. Kathariya, Z. Li, H. Wang, G. Van Der Auwera, “Multi-stage locally and long-range correlated feature fusion for learned in-loop filter in VVC,” in Proc. IEEE International Conference on Visual Communications and Image Processing (VCIP'22), Suzhou, China, Dec. 2022, pp.1-5

• B. Kathariya, Z. Li, H. Wang, and M. Coban, “Multi-stage spatial and frequency feature fusion using transformer in CNN-based in-loop filter for VCC,” in Proc. IEEE Picture Coding Symposium (PCS'22), San Jose, CA, Dec. 2022, pp.373-377.

Related Publications (others)

• H. Zhang, Z. Han, D. Niyato, and S. Mao, “Wireless technologies for metaverse,” Chapter 4 in Metaverse Com-munication and Computing Networks: Applications, Technologies, and Approaches, D.T. Hoang, D.N. Nguyen, C.T. Nguyen, E. Hossain, and D. Niyato (editors), Hoboken, NJ: John Wiley & Sons, 2023.

• Dr. Zhu Li's AI based dynamic point cloud geometry compression technique, in collaboration with Qualcomm’s video team, has been adopted in MPEG AI based PCC test model.

We acknowledge the generous support from our sponsor

• National Science Foundation

This material is based upon work supported by the National Science Foundation under grant CNS-2148382, and is supported in part by funds from OUSD R&E, NIST, and industry partners as specified in the Resilient & Intelligent NextG Systems (RINGS) program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the foundation.