Virtual fitting room of clothes in online stores

Today, the e-commerce industry faces the challenge of accepting a large number of returns. The re-shipping process is costly and time-consuming. Therefore, a tool was needed to reduce the likelihood of returning tried-on garments, to improve the conversion rate, and to reduce costs.

The biggest challenge was to build a product that replicated the realistic appearance of specific people and the clothes they try on (different sizes, prints, cuts, details).  The more accurately such a model was made, the more often it would be used by app users.

Today, the e-commerce industry faces the challenge of accepting a large number of returns. The re-shipping process is costly and time-consuming. Therefore, a tool was needed to reduce the likelihood of returning tried-on garments, to improve the conversion rate, and to reduce costs.

The biggest challenge was to build a product that replicated the realistic appearance of specific people and the clothes they try on (different sizes, prints, cuts, details).  The more accurately such a model was made, the more often it would be used by app users.

The entire process consisted of the following stages of model development: 

1. We defined with the client the goal of the project, both business and technical.

The goal was to develop a solution based on multiple collaborative models, assuming that the available data was a silhouette photo of the person trying on and a photo of the garment being tried on.

2. We collected and extracted the relevant data.

At this stage, we extracted the person’s and clothing’s relevant features. None of the features allowed for biometric identification, making the solution secure and not subject to AI Act restrictions.

3. We used the data to train each model separately. The data resulting from one model became the basis for the performance of subsequent models.

Using the collected data in the form of properly processed images, we created a realistic visualization of the user’s avatar in the clothes being tried on.

4. We implemented the model and tested it under various conditions.

In order for the user to best evaluate the clothes he or she was trying on, the solution presented a visualization in various avatar poses.

In the process, we applied the following technologies: 

• DensePose – The strategy was to estimate the dense position of a person by dividing the surface. For each pixel, we determined what part of the surface it belonged to, where it was in the 2D parameterization, and which part it corresponded to. On the right, the surface division and “point-to-part correspondence” are shown. The interpretation of this model was to “remove” the outer layer from the 3D object and stretch it to the appropriate places in the flat image.
• Segmentation (Computer Vision) – We divided an image into multiple segments or objects in the same class. This grouping task was based on certain criteria, such as color and texture. In this case, we were interested in extracting body parts and clothing items, so system users could correctly try on clothes.
• Coordinate Completion Model – We used a model that completes the voids created by the DensePose model. As a result, we have the first version of the user in the new clothing from every possible angle, allowing it to be shown in any pose.
• GAN (Generative Adversarial Networks) uses deep learning methods such as neural networks for generative modeling. This makes it possible to learn regularities or patterns in the input data so that the model can be used to generate or present new examples that could have been drawn from the original dataset.

GAN models are characterized by the ability to task them with specific inputs and outputs, i.e., saying if we give you “A” and “B” at the beginning, then we want you to do “C.” This is impossible to achieve with the help of StableDiffiusion, which is so popular today. So our A’s and B’s were the user’s photo, the result of the CCM, the segmentation, the set pose, and the clothing, and the result of the activity was a dressed-up user in the clothing of his choice.

We were able to obtain high-resolution visualizations of the garment being tried on, including reproductions of prints and other decorative elements.