Personalisation and contextualisation

LinkedTV deals with user tracking and profiling, with respect to activity and behaviour processing, in order to provide end-users with effective content filtering based on their preferences and interests. It will exploit state-of-the-art tools and approaches, and extend them to allow for design, development and testing of intelligent methodologies and technologies for user tracking and the subsequent personalization of multimedia content, in terms of both object selection and content presentation. Intelligent social and content-based personalization methodologies will be implemented based on domain-independent personalization strategies that will elaborate recommendations based on the semantic relationships and context between the user’s preferences and the available content items.One part of this work deals with the processing and extraction methods involved in the modelling and extraction of profiling information. It will enable mining of profile elements through user interaction and exploitation of such personal information to provide more personalised access to the content.

The other part addresses the issues of modeling a strategy to anticipate user actions and provide focused interaction, by monitoring the user’s physical interaction with the video. A set of behavioral features will be extracted from sensors such as cameras, 3D depth cameras (time-of-flight), or microphones on user behavior including the focus of attention (gaze detection, quantity of motion, contraction index, symmetry index, voice signals, etc…), as well as collective reactions. Collective reactions may include collective laughter, questions that children ask their parents following a popular science program, or lively arguments in front of controversial messages. Those innovative techniques around user voice, eye and body tracking will be used to enhance the personal profiles and preferences.

User Modelling

We need user models to represent the different kinds of interests people have in multimedia content. We investigated existing ontological models on the Web (largely from Linked Open Data – LOD) for representing user interests but none is ideal for a user model, as a consequence we created the LinkedTV User Model Ontology (LUMO). It combines two main issues:

(1) it allows us to represent the mental model of the user as needed in a widespread multimedia domain, and
(2) it provides an ontological bridge to existing LOD ontologies including those used in the semantic annotation work (see ‘Linking video to Web content’).

A LinkedTV User Model Editor (LUME) supports the initial manual creation of sample profiles. The profiles consist of primitive and complex concepts from the reference LUMO ontology and a degree of preference (weighting) for each concept (sample below).

Implicit user profiling

Since users will typically not take much time and effort to create their own profile, and even less so to continue to modify that profile constantly over time, it is important to consider how implicit approaches can be included to derive and evolve a user interests model. In LinkedTV, we consider two distinct forms of implicit profiling:

Based on user attention tracking (when and where the user looks at the screen). We will use analysis of Kinect video streams to determine a viewer’s face direction with respect to the TV.
Based on user interaction in the LinkedTV player (which objects are selected, which content is browsed to). We will use the GAIN tracking component for this.

Preference learning methods are applied to modify the user model based on the collected implicit actions of the user, contributing to the development of the EasyMiner system. The overall workflow of the user model is shown below.

Semantic filtering

On the basis of the user model and the semantic annotations of the media content derived in the LinkedTV workflow (see ‘Linking video to Web content’) we filter the video enrichments according to the users’ preferences and interests.
We have defined two initial approaches to concept and content filtering, the LinkedTV Semantic Filter (LSF) and f-PocketKRHyper (see the LinkedTV document (D4.3)). LSF is based on weighted semantic matching (path distance) while f-PocketKRHyper takes advantage of more complex user preferences to post-process the initial LSF filtering results by means of fuzzy logical reasoning.

Television linked to the Web