Processing and recording data

pupil_recording_interface uses the concept of processes to handle data produced by devices and streams. This tutorial will present the pupil detector as an example process, then introduce the concept of pipelines and notifications and finally list other available processes such as the gaze mapper and recorders.

Note

The data processing described in the following is aimed at the online (real-time) use case needed for recording as well as online pupil detection, gaze mapping and calibration. For offline (post-hoc) analysis of recorded data, refer to the reading and analysis pages.

Pupil detection

Note

Make sure that you have installed the necessary dependencies for pupil detection.

Pupil detection is implemented in the PupilDetector class. We create a stream from an eye video and a detector:

>>> import pupil_recording_interface as pri
>>> eye0_video = pri.VideoFileDevice(pri.get_test_recording(), "eye0")
>>> eye0_stream = pri.VideoStream(eye0_video, color_format="gray")
>>> detector = pri.PupilDetector()

Each process has a process_packet method that takes a Packet produced by the stream as an input and returns the same packet, possibly attaching additional attributes. Note that the detector also needs to be started and stopped, which we conveniently achieve with the context manager syntax:

>>> with eye0_stream, detector:
...     packet = eye0_stream.get_packet()
...     packet = detector.process_packet(packet)
>>> packet.pupil 
{'ellipse':
    {'center': (96..., 130...),
     'axes': (39..., 44...),
     'angle': 77...},
 'diameter': 44...,
 'location': (96..., 130...),
 'confidence': 0.99,
 'internal_2d_raw_data': ...,
 'norm_pos': (0.5..., 0.6...),
 'timestamp': 1570725800...,
 'method': '2d c++',
 'id': None,
 'topic': 'pupil'}

As you can see, the detector has detected a pupil and added the pupil attribute to the packet containing the location of the pupil among others.

Note

So far, only the standard 2D pupil detector is available. We are working on supporting more pupil detection methods.

Pipelines

Multiple processes can be chained with a Pipeline, e.g. a pupil detector and a display that shows the eye camera image with an overlay of the detected pupil:

>>> pipeline = pri.Pipeline(
...     [pri.PupilDetector(), pri.VideoDisplay("eye")]
... )
>>> pipeline.steps 
[<pupil_recording_interface.process.pupil_detector.PupilDetector ...>,
 <pupil_recording_interface.process.display.VideoDisplay ...>]

Starting/stopping the pipeline also starts/stop all of its processes. The process method pipes a packet through all of the steps:

>>> with eye0_stream, pipeline:
...     pipeline.process(eye0_stream.get_packet())
...     input("Press enter to close") 

Note

The input() call is necessary here because the pipeline is stopped upon exiting the context manager, which includes closing the window of the video display.

Pipelines can easily be attached to streams created with the config mechanism for use with a stream manager:

>>> configs = [
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="eye0",
...         loop=False,
...         pipeline=[
...             pri.PupilDetector.Config(),
...             pri.VideoDisplay.Config(),
...         ],
...     ),
... ]
>>> manager = pri.StreamManager(
...     configs, duration=10, folder=pri.get_test_recording(), policy="read"
... )

>>> manager.run()

Notifications

In addition to packets which are produced every time a stream provides new data (e.g. a new video frame from a camera), streams also need to deal with asynchronous data from other sources.

This data falls into two categories:

1. Events emitted from the StreamManager, e.g. instructing the Calibration process to start collecting calibration data.

2. Data from other processes, e.g. information about detected pupils from an eye camera stream that the GazeMapper process - attached to the world camera stream - uses to calculate the current gaze position in the world video frame.

For this purpose, processes have a process_notifications method that handles this kind of data. Notifications are passed a list of dictionaries; for events from the manager the dictionary key denotes the type of event and the value contains the notification’s payload.

Internally, each process filters the notification list and responds only to certain pre-defined types. One class of notifications that all processes understand are "pause_process" and "resume_process" that will temporarily pause the process:

>>> detector.paused
False
>>> detector.process_notifications([{"pause_process": "PupilDetector"}])
>>> detector.paused
True
>>> detector.process_notifications([{"resume_process": "PupilDetector"}])
>>> detector.paused
False

Note that the notification’s payload must match detector.process_name in order for this to work.

Gaze mapping

The GazeMapper collects data from one or two PupilDetector s and maps it to a gaze position according to a previously defined calibration.

For the next example, we need to create a stream for the second eye:

>>> eye1_video = pri.VideoFileDevice(pri.get_test_recording(), "eye1")
>>> eye1_stream = pri.VideoStream(eye1_video, color_format="gray")

We also create two PupilDetector s which we assign a camera_id because the mapper needs to know which eye the detected pupil came from:

>>> eye0_detector = pri.PupilDetector(camera_id=0)
>>> eye1_detector = pri.PupilDetector(camera_id=1)
>>> mapper = pri.GazeMapper()

Now we can read one frame from each eye camera, detect pupils and pass them as notifications to the gaze mapper. Note that for data from other streams, the key of the notification is the name of the stream that produced the data. Calling get_mapped_gaze returns a list of newly mapped gaze data since the last call.

>>> with eye0_stream, eye0_detector, eye1_stream, eye1_detector, mapper:
...     packet = eye0_detector.process_packet(eye0_stream.get_packet())
...     mapper.process_notifications([{"eye0": {"pupil": packet.pupil}}])
...     packet = eye1_detector.process_packet(eye1_stream.get_packet())
...     mapper.process_notifications([{"eye1": {"pupil": packet.pupil}}])
...     mapper.get_mapped_gaze() 
[{'topic': 'gaze.2d.01.',
  'norm_pos': (0.32..., 0.67...),
  'confidence': 0.97...,
  'timestamp': 1570725800.2788825,
  'base_data': [...]}]

When using the GazeMapper in a pipeline with the config mechanism, the stream manager takes care of forwarding the necessary notifications from the pupil detectors to the mapper. The left and right constructor arguments of the GazeMapper specify the names of the left and right eye camera stream and are set to "eye1" and "eye0" by default. The config below sets up all three video streams and overlays the mapped gaze onto the world camera image.

>>> configs = [
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="world",
...         loop=False,
...         pipeline=[pri.GazeMapper.Config(), pri.VideoDisplay.Config()],
...     ),
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="eye0",
...         name="eye0",
...         loop=False,
...         pipeline=[pri.PupilDetector.Config(), pri.VideoDisplay.Config()],
...     ),
...     pri.VideoStream.Config(
...         device_type="video_file",
...         device_uid="eye1",
...         name="eye1",
...         loop=False,
...         pipeline=[pri.PupilDetector.Config(), pri.VideoDisplay.Config()],
...     ),
... ]
>>> manager = pri.StreamManager(
...     configs, duration=20, folder=pri.get_test_recording(), policy="read"
... )

When running the manager you should now see detected pupils and mapped gaze position overlaid on the respective camera images:

>>> manager.run()

Note

So far, only the standard 2D gaze mapping is available. We are working on supporting more gaze mapping methods.

Calibration

For calibrating the gaze mapping, pupil_recording_interface provides two processes:

• The CircleDetector process that detects the circular calibration marker.

• The Calibration process that collects detected calibration markers and pupils and calculates and stores the calibration.

For more details, please refer to the calibration example.

Note

So far, only the standard 2D calibration is available. We are working on supporting more calibration methods.

Recording

Note

Make sure that you have installed the necessary dependencies for recording.

Video streams can be recorded to disk along with the timestamps with the VideoRecorder process.

For more details, please refer to the recording example.

Camera parameter estimation

For estimating camera parameters, pupil_recording_interface provides two processes:

• The CircleGridDetector process that detects the asymmetric circle grid.

• CamParamEstimator that collects detected circle grids and calculates and stores the camera parameters.

For more details, please refer to the camera parameter estimation example.