AI Style Transfer and Texture Mapping Generation

Project description

Allow a user to upload an input image along with a style reference painting, which will be used to generate a stylized version of the input image. This stylized image will then be transformed into a texture map that accurately mimics the surface characteristics of a real painting. The final output can be used with Canon's textured layer printers to produce a physical print that replicates the look and feel of an actual painted artwork.

Context

Canon currently creates texture maps of paintings manually by hand-drawing and outlining every detail, which is a time-consuming process. To streamline this, they aimed to develop a more automated solution using Al. As a starting point, they decided to generate synthetic paintings along with corresponding texture maps that accurately mimic real brushstrokes and surface textures. These Al-generated assets could then be used for layered printing and exhibition displays, offering a more efficient and scalable approach for showcasing textured artwork at expos and similar events.

Results

For style transfer, we found that GPT-4.1 produced the best results, so we used its API for this task. For texture mapping, we used a Pix2Pix model to generate texture maps based on the stylized images. To train the model, we used high-resolution scans of real paintings by various artists, primarily Van Gogh, Monet, Dunbar, and Weykamp. The model performs best on paintings with heavy brushstrokes. It has difficulty producing realistic texture maps for flatter painting styles.

To support testing and demonstration, we also set up a Streamlit environment that runs the full pipeline, from input image to stylized output and texture map generation. This setup allows for interactive evaluation and feedback.

Considering the current setup, the system stands at TRL 6. It has been validated in a relevant environment with an integrated prototype that demonstrates the complete workflow. However, further refinement and robustness testing are still needed before it can be considered ready for operational use.

About the project group

All team members came from a software background. We held two scheduled meetings per week to recap progress and plan our next steps. Throughout the project, we kept our stakeholder informed as much as possible and had the opportunity to experiment with a variety of techniques. Our group collaborated effectively, maintaining a strong and productive working dynamic.