AI for Digital Humanities

Digital Humanities is a term that is spreading all over the world to denote the emerging discipline at the junction of digital and information technologies and the humanities.

The term was created to denote new centres of interdisciplinary study and new tools to provide innovative, intrinsically digital ways of creating knowledge that involve collaborative, transdisciplinary and computationally engaged research, teaching and publishing. It brings digital tools and methods to the study of the humanities with the recognition that the printed word is no longer the main medium for knowledge production and distribution (A. Burinick, Digital Humanities 2012).

In addition to natural humanistic, historical and artistic skills, there is an increasing need for cutting-edge technological skills in the fields of information technology and especially new systems and solutions which, after the digitalisation of assets, facilitate their exploitation and the intelligent creation of new knowledge.

The "AI for Digital Humanities" project is oriented towards promising research activities in the fields of media analysis, computer vision and pattern recognition, video and text understanding and natural language, artificial intelligence and machine learning. These themes are now becoming pervasive in many areas and can make an extraordinary contribution to new experiments in the museum, art-historical, tourist and cultural fields.

The project aims to address several challenges in order to:

a)     Bringing emerging technologies into the everyday life of humanities studies and into the fruition of culture and art, data and digital libraries for experts, scholars and tourists; in this context it will provide expertise support to accompany museum and cultural sites towards a path of digitalisation, use and exploitation of technologies, also in collaboration with the subjects in charge of digitalisation activities.

b)     Disseminating knowledge on digital technologies and on the potential for a positive economic and social exploitation of cultural heritage and for fostering an eco-system of knowledge and social innovation; cultural exchange and joint training will be fostered, also by means of specialised schools or Master courses and dissemination at city, national and international level. 

c)     To produce new scientific research of international relevance on different innovative technologies in the field of information technology, with specific reference to artificial intelligence for knowledge extraction, to automatic analysis of visual and multimedia data that can be exploited in the field of digital humanities and cultural heritage.

This is the outline of a project for innovation, dissemination and joint research that is stable over time, in which experts and researchers in media technologies can be brought together with experts in the humanities sectors, not only academics, both public and private, in a close project collaboration with the city and with public bodies such as museums, galleries or cultural heritage management systems. A striking example in our city are the renovated museum structures of the Estense Galleries, which, also due to their multimodal nature (the Library, the Picture Gallery, the architectural pole, the different locations in Modena, Sassuolo and Ferrara) lend themselves well to advanced experimentation.

In this context, the present research project is crucial if framed in a wider innovative ecosystem constituted by the actors involved in the activities of digitalisation and dissemination of Estense Gallery's heritage and in particular constituted by: Estense Galleries, AImageLab, humanities researchers at Estense Gallery's disposal and other public and private subjects involved by the Galleries in the digitalisation activities. 

Planned activities 

There is no doubt that cultural heritage, in its cycle of study, management and use, needs information technology. The use of the Internet for knowledge, cataloguing and comparison has now been cleared by humanistic culture. The use of information systems for the management of assets or BIM (Building Information Modeling) for the management and conservation of assets, including architectural ones, is slowly spreading also in Italy, and the most modern models of fruition cannot avoid using information technologies for virtual, immersive and augmented guided tours and for a greater involvement of the spectator. 

However, the project would like to go further and be a reference point for tomorrow's technologies applied to Digital Humanities.  

First, it wants to address the problem of digitalised data from art historical libraries, illuminated manuscripts, digitalised 3D assets or even digital native assets (such as photography or video art). The analysis of art-historical documents, digitalised 2D and 3D tangible cultural assets or native assets from multimedia collections no longer only needs data management, archiving and research systems, which are their basic foundation. Libraries, Galleries and Museums increasingly need the most advanced tools of Artificial Intelligence to represent and learn about a dynamic and interconnected heritage of culture and knowledge, to obtain natural tools for access and study, to learn styles of knowledge and language and patterns of interpretation in an automatic or semi-automatic way and to bring researchers, BBCC experts, students, stakeholders and tourists closer in an effective and modern interaction to digital or digitalised culture. 

Many basic tools in multimedia analysis have already been in use for a long time: many are used in cultural information systems for multi-digitalisation and multimedia annotation of dematerialised assets, to manage defined document standards or de-facto standards, for preservation and conservation of assets, to support research, 3D reconstruction, content analysis.

Some AI and artificial vision tools are now employed by large IT companies dealing with Digital Humanity, first of all Google with Google Art Project and Amazon with Amazon Digital Book project. Several experiments in Italy have been studied and carried out in the prototype phase, in particular at UNIMORE. Results of projects such as RERUM NOVARUM for illuminated manuscripts, or SACHER for 3D architectural models are some examples. 

For this reason we intend to propose a multi-year research programme in this field, combining the competences of Computer Engineering with the Humanities of UNIMORE and marrying the cultural needs of the Modenese territory and in particular of the Estense Galleries.

Below we intend to propose the technological part of the project and the possible contribution of the Enzo Ferrari Engineering Department and AImageLab researchers, which can be summarised in some activities: 

1. Study of the problems of digitalisation and multi-digitalisation, from a multimedia, multimodal and cross-media perspective (with specific support for the Modenese heritage in the Estense Galleries).
2. Study of archiving and preservation formats of digital assets, as well as of interrogation and access.
3. Study of the problems of archiving and access systems, including innovative ones, for non-traditional digitalised documentary assets, such as paper documents with complex layouts (such as Muratorian Codices), large digital documents (Geographical Maps) and Renaissance and Baroque musical documents.
4. Study of problems related to reuse-by-design, for natural access, reuse of documents and their parts, and economic and social exploitation of cultural heritage, both for historical and artistic purposes and for wider cultural and commercial purposes, from tourism, education to publishing and advertising.
5. Analysis and linkage of activities with Estense Gallery's Digital Humanities System, with the aim of sharing the results of activities 1-4 and identifying a set of deliverables that can be interfaced with Estense Gallery's Digital Humanities System software and hardware infrastructure.
6. Design of AI systems for automatic learning from expert skills in support of advanced search systems, for automatic annotation of digitalised data to facilitate widespread and effective access, in connection with widespread cultural knowledge on the internet and international archives.
7. Development of Machine Learning (and in particular Deep Learning DL) and computer vision models for the automatic recognition of visual content and its automatic description in natural language. This activity is known as Image or video captioning and is giving the first results in research on generic visual data and will have to be developed specifically for cultural, artistic and historical data. 
8. Development of AI and Pattern Recognition models, alongside Deep Learning for the recognition and analysis of interaction between people and things, between tourists and experts and cultural heritage to provide automatic, adaptable and customisable tools (not to categories of users but to individuals). 
9. Study and research of innovative AI tools for the understanding of pictorial, figurative and iconographic heritage, both in the collections in the Estense Galleries' picture galleries and in the manuscript and less known collections. 
10. Launch of new AI and Deep Larning research for understanding and reconstructing iconographic and figurative, 2D and 3D images, especially in reference to human beings, individuals and communities, fashionable costumes and painted actions in Renaissance and Baroque heritage collections, useful for understanding, dating and attributing works.
11. Dissemination of knowledge and potential of Digital Humanities on the territory with specific actions of internationalisation of research and dissemination of the knowledge produced to students and teachers, also in coordination with the Estense Galleries Digital Humanities System. 

The possibility of a wide-ranging interdisciplinary study is also envisaged, with links to Italian and international research centres, such as Penn University (Prof. Yia Li), the University of Tokyo (Prof. Katsuki Ikeuchi), the University of TU WIEN (Prof. Andreas Rauber), the University of Amsterdam (Prof. Theo Gevers, Prof. Cees Snoek), the Facebook Lab AI Research, and Amazon Research (Prof. Gerard Medioni). 

The project will have a strong social impact and prospects for dissemination first at city level and then as a pilot example at national level. The social impact stems from the desire to:

• Contribute to a modern, attentive and forward-looking management of cultural heritage in its digitised or digital-native form;
• Create new forms of use of cultural heritage through adaptive and innovative models of research and access to heritage;
• Create a new form of knowledge, of historical and cultural associations and thus a new form of culture;
• Create new tools for educating young people and bringing humanistic culture closer to scientific culture, also with a view to the economic and social exploitation of common cultural assets;
• Create new cultural tools not only for experts or scholars but also for citizens and tourists, enhancing the artistic and historical heritage of the area. 

The project was set up with the aim of carrying out extremely innovative research in the field of artificial intelligence and the analysis of visual and multimedia data, with direct scientific impact first and foremost, but with clear repercussions on the local area, both in social and economic terms, favouring the creation of new innovative companies and providing opportunities for networking among local bodies. 

Publications

1	Cojocaru, Iulian; Cascianelli, Silvia; Baraldi, Lorenzo; Corsini, Massimiliano; Cucchiara, Rita "Watch Your Strokes: Improving Handwritten Text Recognition with Deformable Convolutions" Proceedings of the 25th International Conference on Pattern Recognition, Milan, Italy, pp. 6096 -6103 , 10-15 January 2021, 2021 | DOI: 10.1109/ICPR48806.2021.9412392 Conference
2	Cornia, Marcella; Stefanini, Matteo; Baraldi, Lorenzo; Corsini, Massimiliano; Cucchiara, Rita "Explaining Digital Humanities by Aligning Images and Textual Descriptions" PATTERN RECOGNITION LETTERS, vol. 129, pp. 166 -172 , 2020 | DOI: 10.1016/j.patrec.2019.11.018 Journal
3	Cornia, Marcella; Baraldi, Lorenzo; Cucchiara, Rita "SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability" International Conference on Robotics and Automation, Paris, France, pp. 1128 -1134 , May, 31 - June, 4, 2020 Conference
4	Cornia, Marcella; Stefanini, Matteo; Baraldi, Lorenzo; Cucchiara, Rita "Meshed-Memory Transformer for Image Captioning" 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, pp. 10575 -10584 , June 14-19, 2020 | DOI: 10.1109/CVPR42600.2020.01059 Conference
5	Cornia, Marcella; Baraldi, Lorenzo; Tavakoli, Hamed R.; Cucchiara, Rita "A Unified Cycle-Consistent Neural Model for Text and Image Retrieval" MULTIMEDIA TOOLS AND APPLICATIONS, vol. 79, pp. 25697 -25721 , 2020 | DOI: 10.1007/s11042-020-09251-4 Journal
6	Carraggi, Angelo; Cornia, Marcella; Baraldi, Lorenzo; Cucchiara, Rita "Visual-Semantic Alignment Across Domains Using a Semi-Supervised Approach" Computer Vision – ECCV 2018 Workshops, vol. 11134, Munich, Germany, pp. 625 -640 , 8-14 September 2018, 2019 | DOI: 10.1007/978-3-030-11024-6_47 Conference
7	Tomei, Matteo; Baraldi, Lorenzo; Cornia, Marcella; Cucchiara, Rita "What was Monet seeing while painting? Translating artworks to photo-realistic images" Computer Vision – ECCV 2018 Workshops, Munich, Germany, 8-14 September 2018, 2019 | DOI: 10.1007/978-3-030-11012-3_46 Conference
8	Landi, Federico; Baraldi, Lorenzo; Corsini, Massimiliano; Cucchiara, Rita "Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters" Proceedings of 30th British Machine Vision Conference, Cardiff, UK, pp. 1 -12 , 9th-12th September 2019, 2019 Conference
9	Baraldi, Lorenzo; Cornia, Marcella; Grana, Costantino; Cucchiara, Rita "Aligning Text and Document Illustrations: towards Visually Explainable Digital Humanities" Proceedings of the 24th International Conference on Pattern Recognition, Beijing, China, pp. 1097 -1102 , August 20th-24th, 2018, 2018 | DOI: 10.1109/ICPR.2018.8545064 Conference