Ask a robot to fold a towel and you quickly learn how little it knows about the world. Large language models learned from the internet’s text. Robots, smart glasses and embodied AI systems have no such library to draw on. They need to learn how people actually use their hands, and they need to see it from the person’s own point of view.
That gap is now one of the most practical bottlenecks in physical AI. The models are improving fast. The data to train them is not keeping up.
Our team recently delivered an egocentric video data collection program for a global technology organization building activity-recognition and robotics models. We collected an estimated 30,000 hours of first-person video, and more than 24,000 hours were annotated and approved for training. Here is what that work taught us, and what any business building real-world AI should know before it starts.
Why third-person video is not enough
Most video datasets are shot from the outside: a fixed camera watching someone work. That is useful for surveillance or sports analysis. It is far less useful for a machine that has to do the work itself.
A robot or a wearable assistant needs the operator’s view. It needs to see where the hands go, how fingers wrap around a handle, and what happens in the second before an object is set down. That is egocentric, or first-person, video. It captures hand-object interaction and the order in which tasks unfold, which is exactly what embodied models need to learn.
The business challenge: real, varied and safe, all at once
The client’s brief sounded simple. In practice, it had three goals that pull against each other.
- Models trained on staged, scripted footage learn staged, scripted behavior. The client wanted natural activity from daily life and work.
- A model that has only seen tidy kitchens will fail in a warehouse. Coverage had to span many settings, people and tasks.
- Filming people in homes and workplaces raises real questions about consent, location and ownership of the footage.
Pushing hard on one of these usually costs you another. More realism means less control. More diversity means more variation in quality. Tighter compliance adds friction for contributors. The job was to design a program that held all three at scale.
Let people film their real lives
We did not hand contributors a script. They chose tasks from their natural routines, then recorded them continuously while standing or walking.
The settings covered six segments:
- Household: laundry, kitchen tidy-up, pet and plant care, gardening, home maintenance
- Warehouse and logistics: packing, sorting, inventory handling, inspection
- Commercial and hospitality: janitorial work, housekeeping, retail stocking, back-of-house food service
- Industrial: light manufacturing, assembly, equipment maintenance
- Agriculture and landscaping: grounds care, farm work, outdoor material handling
- Skilled trades and field services: construction, automotive service, repairs, installation
Contributors were spread across the United States and Latin America. Each could contribute up to 100 approved hours, which kept the dataset from leaning too heavily on any one person’s habits.
Standardize the camera, not the behavior
Freedom in what people recorded only works if how they recorded is tightly defined. We allowed three device families: iPhone 11 and newer, Google Pixel 6 and newer, and Samsung Galaxy S21 and newer. Phones were mounted at forehead level and angled about 45 degrees down. That kept the hands, the objects and the workspace in frame.
Every contributor was onboarded before recording. Training covered task selection, device setup, camera placement, consent and what must never be filmed. A dedicated mobile app handled recording and submission. It also verified location, so footage came only from approved regions.
The lesson for any operator: consistency in the capture setup is what lets you accept variety in the content.
Make quality review fast, specific and fair
At this volume, quality control is the business. Every recording was checked for camera stability, lighting, focus, hand-object visibility, task continuity, posture, environment and authorization.
The rejection rules were concrete, not subjective. A video was rejected for:
- Obstructed or shaky views, or poor light and focus
- Long idle stretches
- Staged or robotic movement
- Excessive head, hand or body movement
- Hands out of frame for more than 10% of active grab-and-release work
- Unauthorized locations or missing consent
Speed mattered as much as the rules. Submissions were reviewed within one day, with a clear reason for every rejection. Contributors knew exactly what to fix and could recapture while the task was still fresh. That loop improved quality over time and kept good contributors engaged instead of frustrated.
Build governance into the workflow
Compliance cannot be a checklist at the end. In this program, consent, location authorization, secure submission, location validation and intellectual-property assignment were part of every step.
That design choice matters commercially. AI buyers increasingly ask where their training data came from and whether they truly own it. A dataset with clean provenance is easier to use, easier to defend and easier to sell to the next stakeholder.
The results
The program delivered an estimated 30,000 hours of first-person video and more than 24,000 hours annotated and approved for training. It spanned household, industrial and commercial settings across two regions and three device families, with one-day review throughout.
The client’s AI data program lead described the review workflow and capture standards as “a reliable foundation” for their activity-recognition and robotics models.
Five lessons for leaders building real-world AI
Whether you are a robotics startup or an enterprise team exploring automation, these lessons carry over.
- Define the perspective before the volume: Ten thousand hours from the wrong viewpoint are worth less than one thousand from the right one. Decide what your model must see first.
- Script the setup, not the task: Lock down devices, mounting and framing. Then let people behave naturally. Real behavior is the signal you are paying for.
- Write rejection rules people can act on: “Low quality” is not feedback. “Hands out of frame for more than 10% of the task” is.
- Treat review speed as a growth lever: A one-day feedback loop turns rejections into training. Slow reviews quietly shrink your contributor base.
- Design for provenance from day one: Consent and IP assignment cost little to build in early. Retrofitting them later can cost you the dataset.
The bottom line
Physical AI will not be won by model architecture alone. It will be won by the teams that can collect real-world data at scale, without giving up quality or trust. The playbook is not exotic: a clear point of view, disciplined capture, fast and specific review, and governance built in from the start. The companies that master it will have something rare in AI right now, a data advantage that is hard to copy.
Author bio
Hardik Parikh is SVP and Business Head at Shaip, where he leads go-to-market strategy for AI training data. Shaip helps technology companies collect, annotate and license high-quality data for conversational AI, computer vision, healthcare and physical AI. Hardik works with AI teams worldwide on scaling real-world data programs with built-in quality, consent and governance. Learn more at shaip.com.
