Streaming data

Note

Make sure that you have installed the necessary dependencies for streaming. If not, you can still go through large parts of this guide by streaming from a recording included in the package instead of an actual Pupil Core device.

Video devices

The Pupil Core cameras can be accessed via the VideoDeviceUVC class:

>>> import pupil_recording_interface as pri
>>> world_cam = pri.VideoDeviceUVC("Pupil Cam2 ID2", (1280, 720), 60)
>>> world_cam 
<pupil_recording_interface.device.video.VideoDeviceUVC object at ...>

The first argument is the name of the video device ("Cam1", "Cam2" or "Cam3" for different generations of the Pupil hardware; "ID0", "ID1" and "ID2" for left and right eye or world camera, respectively). If you don’t have a Pupil Core device or are missing the necessary dependencies, you can use a dummy device that streams from a recording instead:

>>> world_cam = pri.VideoFileDevice(pri.get_test_recording(), "world", timestamps="file")

A device needs to be started before streaming any data and stopped afterwards in order to release the resource. To facilitate this, using the device as a context manager automatically calls its start and stop methods upon entering and exiting, respectively.

You can grab a video frame and its timestamp from the device with the get_frame_and_timestamp() method:

>>> with world_cam:
...     frame, timestamp = world_cam.get_frame_and_timestamp()
>>> frame.shape
(720, 1280, 3)
>>> timestamp
1570725800.2383718

Video streams

The VideoStream class is a wrapper for video devices that handles functionality such as polling for new video frames and processing (pupil detection, recording, …):

>>> stream = pri.VideoStream(world_cam, name="world")
>>> stream 
<pupil_recording_interface.stream.VideoStream object at ...>

The get_packet() returns a Packet that bundles the data retrieved from the device. We use a context manager again to handle starting and stopping of the stream:

>>> with stream:
...     packet = stream.get_packet()
>>> packet 
pupil_recording_interface.Packet with data:
* stream_name: world
* device_uid: world
* timestamp: 1570725800.2383718
>>> packet.frame.shape
(720, 1280, 3)

Multiple streams

For simultaneous streaming from multiple devices, the StreamManager is used. The manager dispatches each stream to a separate process and handles communication between those processes. Instead of constructing VideoStream instances, we use a list of VideoStream.Config() instances:

>>> configs = [
...     pri.VideoStream.Config(
...         device_type="uvc",
...         device_uid="Pupil Cam2 ID2",
...         name="world",
...         resolution=(1280, 720),
...         fps=60,
...         pipeline=[pri.VideoDisplay.Config()],
...     ),
...     pri.VideoStream.Config(
...         device_type="uvc",
...         device_uid="Pupil Cam2 ID0",
...         name="eye0",
...         resolution=(192, 192),
...         fps=120,
...         pipeline=[pri.VideoDisplay.Config()],
...     ),
...     pri.VideoStream.Config(
...         device_type="uvc",
...         device_uid="Pupil Cam2 ID1",
...         name="eye1",
...         resolution=(192, 192),
...         fps=120,
...         pipeline=[pri.VideoDisplay.Config()],
...     ),
... ]

The manager then constructs the proper streams and devices from this list. With duration=30, the manager will stop streaming after 30 seconds.

Note

The concept of pipelines and processes such as VideoDisplay is explained in detail in Processing and recording data and the config mechanism in Custom devices, streams and processes.

>>> manager = pri.StreamManager(configs, duration=30)
>>> manager.streams 
{'world': <...>, 'eye0': <...>, 'eye1': <...>}

Alternatively, use this dummy configuration:

>>> configs = [
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="world",
...         loop=False,
...         pipeline=[pri.VideoDisplay.Config()],
...     ),
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="eye0",
...         loop=False,
...         pipeline=[pri.VideoDisplay.Config()],
...     ),
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="eye1",
...         loop=False,
...         pipeline=[pri.VideoDisplay.Config()],
...     ),
... ]
>>> manager = pri.StreamManager(
...     configs, duration=10, folder=pri.get_test_recording(), policy="read"
... )

Now we can run the manager to start streaming. You should see three windows opening with the eye and world video streams.

>>> manager.run()

The manager will automatically stop after the specified duration and can also be stopped with a keyboard interrupt. When no duration is set, the manager will run indefinitely.

It is also possible to run the manager in a non-blocking fashion by using it as a context manager. This allows us for example to print the current frame rates for each stream to the command line:

>>> with manager:
...     while not manager.stopped:
...         if manager.all_streams_running:
...             print("\r" + manager.format_status("fps", sleep=0.1), end="") 
eye0: ..., eye1: ..., world: ...

Self-contained scripts and Jupyter notebooks for streaming that you can download and modify can be found in the online examples section.

UVC camera settings

The Pupil Core cameras implement the USB Video Class (UVC) protocol and Pupil Labs provides a Python wrapper for accessing the cameras called pyuvc. In turn, pupil_recording_interface provides a high-level interface to this via the VideoDeviceUVC class.

Possible combinations of resolutions and FPS can be queried via the available_modes attribute, returning a list of (horizontal_res, vertical_res, fps) tuples:

>>> from pprint import pprint
>>> pprint(world_cam.available_modes) 
[(1920, 1080, 30),
 (640, 480, 120),
 (640, 480, 90),
 (640, 480, 60),
 (640, 480, 30),
 (1280, 720, 60),
 (1280, 720, 30),
 (1024, 768, 30),
 (800, 600, 60),
 (1280, 1024, 30),
 (320, 240, 120)]

Other settings (called controls) together with valid ranges of values can be obtained via available_controls.

>>> pprint(world_cam.available_controls) 
{'Absolute Exposure Time': range(1, 500),
 'Auto Exposure Mode': {'aperture priority mode': 8,
                        'auto mode': 2,
                        'manual mode': 1,
                        'shutter priority mode': 4},
 'Auto Exposure Priority': (0, 1),
 'Backlight Compensation': range(0, 2),
 'Brightness': range(-64, 64),
 'Contrast': range(0, 64),
 'Gain': range(0, 100),
 'Gamma': range(72, 500),
 'Hue': range(-40, 40),
 'Power Line frequency': {'50Hz': 1, '60Hz': 2, 'Disabled': 0},
 'Saturation': range(0, 128),
 'Sharpness': range(0, 6),
 'White Balance temperature': range(2800, 6500),
 'White Balance temperature,Auto': (0, 1)}

The current settings of a running device are stored in the controls attribute.

>>> with world_cam: 
...     pprint(world_cam.controls)
{'Absolute Exposure Time': 32,
 'Auto Exposure Mode': 8,
 'Auto Exposure Priority': 1,
 'Backlight Compensation': 1,
 'Brightness': 0,
 'Contrast': 32,
 'Gain': 0,
 'Gamma': 100,
 'Hue': 0,
 'Power Line frequency': 1,
 'Saturation': 60,
 'Sharpness': 2,
 'White Balance temperature': 4600,
 'White Balance temperature,Auto': 1}

You can also assign controls by passing a dictionary as the controls constructor argument…

>>> world_cam = pri.VideoDeviceUVC(
...     "Pupil Cam2 ID2", (1280, 720), 30, controls={"Gamma": 200}
... )
... with world_cam: 
...     print(world_cam.controls["Gamma"])
200

…or to a running device in the same manner:

>>> with world_cam: 
...     world_cam.controls = {"Gamma": 120}
...     print(world_cam.controls["Gamma"])
120