Technical Architecture
Client-Side Rendering
All AR processing happens in the browser using JavaScript/TypeScript, eliminating server-side computational overhead and ensuring low-latency interactions.
Facial Mesh Tracking
Utilizes MediaPipe FaceMesh (or similar ML models) to detect 468+ facial andmarks in real-time, enabling precise positioning of accessories on facial features like ears, nose bridge, cheekbones, and neck.
Canvas-Based Overlay System
A 2D rendering context draws product images with dynamic transformations (scale, rotation, translation, opacity) synchronized to facial movements at 30-60 FPS.
Product Configuration Engine
JSON-based product definitions store dimensional metadata (default scale, anchor points, offset coordinates, rotation angles) allowing non-technical users to add new products without code changes.
Responsive Calibration
Adaptive algorithms adjust product sizing based on detected face width, ensuring consistent visual appearance across different devices, camera resolutions, and user distances.
Supported Product Categories
Earrings (studs, hoops, danglers)
Nosepins and nose rings
Necklaces and pendants
Eyewear (sunglasses,prescription glasses)
The module integrates seamlessly into React/Next.js applications as reusable components, providing hooks for product selection, camera controls, and AR state management.
How it works
Product Selection & Discovery
Users browse and select products from the product catalog page with filtering, search, and sorting capabilities.
Camera & Canvas Setup
A video element captures the user's camera feed in real-time.
A canvas layer overlays the video to render product visualizations.
Facial Landmark Detection
MediaPipe (or equivalent ML model) runs in a WebWorker to detect key facial points including ears, nose bridge, cheekbones, and neck area.
Detected landmarks are continuously passed back to the main thread for processing.
Real-Time Overlay Rendering
Product definitions containing size, positioning offsets, and scaling multipliers determine precise placement.
On each animation frame, the module calculates scale, rotation, and position transforms based on facial movement.
Transparent PNG product images are dynamically drawn on the canvas,maintaining accurate positioning during real-time face and head movements.
Interactive User Controls
Category/subcategory selection interface (nosepins, earrings, necklaces,eyewear).
Try-on section featuring snapshot capture with retake and download functionality.
Full-screen mode toggle for enhanced viewing and closer product examination.
Key Features
- Real-time, photorealistic AR overlays
- 2D product overlays on canvas
- Multiple product types: earrings, nosepins, necklaces, sunglasses, prescription glasses
- Adaptive smoothing & responsive scaling across desktop, tablet, and mobile
- Constant facial detection using mediapipe-facemesh for realtime overlays
- Consolidated alert/feedback for detection issues
- Take, retake and downnload the snapshot for side views tryon.
- Full screen mode
- Tailwind-powered, customizable UI components
- Shadcn UI library
- Modular component for each prodcuct tryon.
Use Cases
E-commerce platforms wanting “try before you buy” for jewelry and eyewear
Fashion & accessories websites offering virtual tryon rooms
Social shopping features where users share Araugmented selfies
In-store kiosks or events for contactless product demos
Tech Used
Frontend
Technologies
Framework
Next.js (version 15.3), React (version 19.0)
Language
TypeScript
Styling
Tailwind CSS, Shadcn UI library
Real-time Facial
Detection
MediaPipe (v0.4.1)
Request a Demo