An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Noebels JL, Avoli M, Rogawski MA, et al., editors. Jasper's Basic Mechanisms of the Epilepsies. 5th edition. New York: Oxford University Press; 2024. doi: 10.1093/med/9780197549469.003.0060
A major impediment to progress in basic epilepsy research is the fact that evidence-based, rigorous translational research is not only prohibitively time and labor-intensive, such as 24/7 video-electroencephalogram recordings, but rests on inherently subjective scoring by human observers, as exemplified by the Racine scale. Recent technical progress in machine learning and computer vision highlighted a variety of novel possibilities for quantifying behavior in animal models of epilepsies. This chapter briefly reviews the latest advances in artificial intelligence (AI)-guided animal motion tracking and segmentation of pose dynamics that bear great potential of revolutionizing behavioral phenotyping in basic epilepsy research. As an emerging field fueled by the recent successes of deep learning, AI-guided behavioral phenotyping will be discussed primarily in order to provide insights into the fundamentals of the field and at the same time raise awareness of potential pitfalls of the underlying technology. By concisely surveying the diverse and rapidly growing landscape of the relevant methods and toolboxes available in neuroscience research, this chapter aims to spark interest in AI-aided behavioral phenotyping in the epilepsy community.
For both research and diagnosis of epilepsy, studying the manifestation of the disease most often starts by assessing the signs and symptoms that are expressed in behavior. As such, behavior is a functional readout of brain activity, and in epilepsy, it is specifically shaped by ictal and interictal activity. The spectrum of behavior associated with epilepsies ranges from uncontrolled movements (e.g., jerks or twitches), emotional (e.g., anxiety) and cognitive symptoms (e.g., memory impairments), to subtle changes in responsiveness to the environment (e.g., temporary confusion or loss of awareness). Analyzing behavior during (ictal) and between (interictal) seizures is central for classifying seizures (Fisher et al., 2017) and epilepsies (Scheffer et al., 2017), as well as uncovering comorbidities (e.g., autism [Abrahams et al., 2011; Tsai et al., 2012] or memory impairments [Bui et al., 2018]), assessing epilepsy progression and treatment efficacy in both clinical and lab settings. Together with its role (e.g., as a functional readout) in helping to uncover the causes of the epilepsies, the need for evaluating behavior can be found across the shared research priorities of the epilepsy community (Binder et al., 2020; Chang et al., 2020; Jones et al., 2020; Poduri and Whittemore, 2020; Traynelis et al., 2020).
A behavioral readout can serve different objectives. As a task-specific metric, behavior is used to assess known impairments by quantifying predefined labels. For example, a gold standard to evaluate memory deficit in a mouse model of temporal lobe epilepsy (TLE) is to determine and compare the exploration time (e.g., time spent at predefined distance to object) between moved and unmoved objects in the so-called object location memory task (Vogel-Ciernia and Wood, 2014; Bui et al., 2018; Kim et al., 2020). In contrast to such task-defined measurements, renouncing any a priori assumptions about relevant behavioral features, an observation-based deconstruction of naturally occurring behavior (e.g., observation of pathognomonic head movements) can give rise to a new perspective by suggesting testable hypotheses about the potential brain networks involved (e.g., the proposed [Noebels et al., 1990] and later confirmed [Khan et al., 2004] vestibular damage in stargazer mice). Historically, these seemingly dichotomous strategies to evaluate behavior, that is, assessing task-related response behavior and exposing the structure of behavior, arose from two distinct traditions, comparative psychology and ethology (Gomez-Marin and Ghazanfar, 2019). However, recent initiatives in systems neuroscience argue that these disciplines have to come together to disentangle the relationship between brain and behavior (Datta et al., 2019; Dennis et al., 2021).
For preclinical epilepsy research as well, the necessity of coalescing different methodologies to capture unique yet comparable behavioral phenotypes in animal models of epilepsies through an unbiased and data-driven manner becomes increasingly evident with more tools automating behavioral measurements at hand (Fig. 60–1). While recent technological advances make it more accessible to continuously track behavior at progressively higher resolution (Mathis and Mathis, 2020; Mathis et al., 2020; Pereira, Shaevitz, and Murthy, 2020), traditional approaches persist and still dictate how behavioral readouts are compiled and behavioral phenotypes are tessellated to fit their methodological framework. For example, while continuous video-electroencephalogram (EEG) can capture seizure burden across various timescales (e.g., ultradian and circadian dynamics; Quigg et al., 1998; Kim et al., 2020) and disease stages (e.g., epileptogenesis; Williams et al., 2009), the analysis of behavior is usually at best restricted to the ictal periods, timestamped by the EEG, while interictal periods in these datasets are largely ignored (Fig. 60–1A). Instead, in order to assess any additional, interictal impairments and characterize comorbidities, behavioral assays are separately chosen and deployed at selected time points, usually without any neural recordings (e.g., EEG). We argue that an integrative system is required that takes advantage of the recent technological advances in automated machine vision to capture the rich behavioral repertoire in animal models of epilepsy (Fig. 60–1B), combining multimodal measurements (such as video and EEG) to a holistic, functional readout capable of testing for known deficits and screening for novel impairments and drugs without observer bias.
The potential of artificial intelligence (AI)-guided phenotyping in basic epilepsy research. A. In classical approaches, behavioral phenotypes in animal models of epilepsies rely on separate measures of behavior, often segregating the acquisition of (more...)
As with data collection in animal models of epilepsy, the availability of new analysis tools, in particular for animal behavior and tracking, has rapidly grown and is likely to continue to do so over the next decades, thanks to the tremendous success of artificial intelligence (AI). AI has a great potential to facilitate and accelerate several aspects of basic epilepsy research, especially, as discussed below, for behavioral tracking and analysis. However, the shift from manual analysis of small datasets acquired by an individual research lab toward an AI-automated, large-scale computation of “big-data” with shared data structures (Teeters et al., 2015; Rübel et al., 2019) and datasets (Abbott et al., 2017; Bonacchi et al., 2019) requires the acquisition of new skill sets by the research community. With a growing landscape of tools with variable degrees of manual human contribution and intelligible insight into the algorithms used, it remains the burden of the end user to acquire a basic knowledge of the underlying technology, if only to gain awareness of its pitfalls. However, we argue, due to the growing presence of AI and a large body of literature about animal tracking, a basic knowledge about AI (including terms and functionalities) and an overview of current state-of-the-art behavioral tracking approaches in neuroscience will help both researchers and future tool developers to either use or adapt the technology for basic epilepsy research. Therefore, in this chapter, we will discuss the idiosyncrasies of AI with a focus on behavioral tracking approaches that bear potential to significantly benefit basic epilepsy research. Since other chapters in this book discuss more traditional behavioral studies in various animal models of epilepsies, here we will focus on pre–machine learning and machine learning–guided approaches for animal tracking and behavioral analysis in laboratory settings, while highlighting their potential for AI-guided behavioral phenotyping in epilepsy.
In epilepsy research and for neuroscience in general, studying the functions and dysfunctions of the brain, many have argued, requires a careful decomposition of behavior (Krakauer et al., 2017; Datta et al., 2019; Dennis et al., 2021). Describing an animal’s behavior in its environment usually starts by tracking individual body parts and analyzing the pose dynamics through time. Hence, it is important to remember that the features that are tracked directly define the granularity of behavioral description (Fig. 60–2). A large body of research in computer vision and more recently in neuroscience has focused on accurately and efficiently tracking key points up to entire poses of animals in videos (e.g., reviewed in Mathis et al., 2020; J. Wang et al., 2021). In parallel, another branch of research has focused on developing better tools to quantitatively assess behavior by segmenting extracted features (e.g., location and acceleration of the animal’s paws) into stereotypic, recurring units (e.g., grooming; reviewed in Datta et al., 2019; Pereira, Shaevitz, and Murthy, 2020). Both fields have benefited from several recent advances in machine learning (ML) and have started to tackle more complex scenarios (e.g., social interactions using multianimal tracking; Pereira et al., 2020; Lauer et al., 2021). Together with recent efforts to integrate multimodal measurements (e.g., ultrasonic vocalizations [USVs]; Karigo et al., 2021), progress in different disciplines deploying ML techniques (including computer vision or speech recognition) will help capture an animal’s interaction with its environment in greater detail and ultimately help to understand how the brain generates behavior or fails to do so. In the following paragraphs, we briefly discuss the origins of animal tracking algorithms and their advances toward estimating poses, methods used to segment pose dynamics into stereotypical behavioral modules, and other, less-known approaches with great potential that try to capture and model entire scenes.
Animal tracking—from point tracking to 3D surface reconstruction. A. Behavior involves coordinated movements; thus, the features that are tracked defined the granularity of behavioral description. Experimenters need to decide what tracking mode (more...)
Tracking animals has a long history and is used in several settings (e.g., neuroscience research, animal husbandry, industrial farming, or animal conservation). Besides being driven by a particular question—scientific or other—choosing a tracking approach often depends on the circumstances (i.e., on the particularities of the animal and the environment), in which the data has to be acquired (e.g., livestock tracking on a farm or tracking movements of an octopus in an aquarium). Thus, different approaches can be broadly categorized by the type of sensors that are used, such as videography, microphone, radio-frequency identification (RFID) tags, and inertial measurement unit (IMU) sensors. For neuroscience research, which largely takes place in controlled laboratory settings, video-based approaches have been the most fruitful for studying the behavior in model organisms and have been accompanied by several technical breakthroughs over the last two decades in particular regarding invasiveness (e.g., from marker-based to marker-less tracking) and richness of the readout extracted (e.g., from 2D position to 3D pose tracking).
Early computer-assisted approaches to track animals in videos focused on simple readouts like position and speed, which only required a single point to represent the animal (e.g., the animal’s centroid) (Fig. 60–2B). Such approaches often rely on background subtraction, removing pixels in an image belonging to the environment in order to identify the animal. In laboratory settings with static environments, one of the most common and fastest approaches, and thus also suitable for real-time applications, is to take advantage of contrast or difference in color hues (i.e., the chroma range) between animal and background by building behavioral arenas with a light background (Voigts, Sakmann, and Celike, 2008) or including a green screen (Maghsoudi et al., 2018; Haji Maghsoudi, Vahedipour, and Spence, 2019; Bala et al., 2020), while applying image intensity thresholding or chroma keying, respectively, to separate foreground (i.e., the animal) from the background. While such approaches do not constrain an animal’s behavior per se, they impose experimental settings with limited natural context and are prone to fail in dynamic scenes (e.g., with multiple animals) without the use of any additional markers or sensors (e.g., RFID tags).
With a growing research interest in quantifying an animal’s interaction with the environment (e.g., exploration of objects [Vogel-Ciernia and Wood, 2014] or hunting prey [Johnson et al., 2020]) and thus reasoning about an animal’s sensory perception about what lies ahead (e.g., through smell and vision), tracking the orientation of an animal became more important, especially since virtually all model organisms in neuroscience (from Caenorhabditis elegans to Mus musculus) share a bilaterally symmetric body plan with an anterior-posterior (head-to-tail) and a ventral-dorsal (belly-to-back) axis. From a technical point of view, however, distinguishing individual body parts to decipher the orientation and eventually the pose of an animal brought several new challenges of which some are solved and others remain to this day. Tracking body parts reliably despite obstruction (e.g., by objects in the scene or during grooming) or the lack of unique features (e.g., head vs. tail in C. elegans) was traditionally addressed by combining classical video-based approaches with markers (e.g., coloring body parts [Spink et al., 2001; Inayat et al., 2020] or affixing retro-reflective markers [Roy et al., 2011; Mimica et al., 2018; Marshall et al., 2021]) or special sensors (e.g., RFID [Peleh et al., 2019] or IMU [Pasquet et al., 2016; Vanzella et al., 2019]). While such marker-based approaches are highly accurate (e.g., as shown for tracking whisker and measuring head rotations) and usually do not need any further postprocessing of the data (e.g., manual annotation), they are widely deemed impractical and at worst interfere with the natural behavior of an animal. More recently, like other classical computer vision approaches (e.g., optical flow [Horn and Schunck, 1981; Hur and Roth, 2020]), marker-based methods are mostly replaced with a deep learning (DL) approach. Nevertheless, marker-based methods still serve an important role as valuable ground truth data, providing labels for training ML–based methods (see Dunn et al., 2021, for an excellent example of such a transition from marker-based to marker-less, ML–based motion capture in rodents).
AI as an academic discipline is rather young (1950s), but nonetheless it has shown its transformative potential in several areas, including basic neuroscience research, where it reduces manual labor for behavioral analysis in animals, while increasing reliability and accessibility of accurate pose estimation in different experimental settings (e.g., see Mathis et al., 2020; Pereira, Shaevitz, and Murthy, 2020). AI comprises different disciplines, including ML, the study of algorithms designed to improve over time by extracting patterns from raw data (Goodfellow, Bengio, and Courville, 2016). Computer vision, like other scientific fields in and around computer science (e.g., natural language processing), adopted ML approaches early and used hand-designed features in combination with a (machine) learning subsystem (e.g., classifiers) for identifying and classifying image features. For example, image features, such as histogram of gradients (HOG), which describes an image based on the distribution of pixel intensity gradients and edge directions, are extracted and vectorized into a feature vector to train ML classifiers like a support vector machine (SVM) in order to classify images into different categories (e.g., photos with and without humans) (Dalal and Triggs, 2005). Although such classical ML pipelines had great success in tasks like image classification and gave rise to great tools for animal tracking that simplified, for example, orientation tracking (Fig. 60–2C) (Branson et al., 2009; Dankert et al., 2009), these classical techniques required a fair amount of feature engineering (e.g., HOG) and domain knowledge (e.g., in image preprocessing such as image filtering) to design suitable representations (i.e., feature vectors describing the raw data in compressed form) for the downstream ML algorithm. Other preprocessing algorithms in computer vision such as simple linear iterative clustering (SLIC) (Achanta et al., 2010), which clusters pixels based on their color similarity and proximity in the image into superpixels (Ren and Malik, 2003), tried to reduce redundancies in the raw image itself instead of extracting specific hand-crafted features. Combined with an algorithm like conditional random fields (CRFs), SLIC had great success in tasks like image segmentation, which is useful for background subtraction or (body) part segmentation (i.e., semantic segmentation). Although image preprocessing algorithms such as SLIC and others are used for animal tracking in laboratory settings (Kyme et al., 2014; Machado et al., 2015; Maghsoudi et al., 2018; Haji Maghsoudi, Vahedipour, and Spence, 2019), nowadays their main utility lies more in their processing speed (e.g., for real-time applications) rather than providing the best features for a downstream ML algorithm.
The quest for good feature extractors that transform raw data into suitable representations largely ended with the emergence of efficient artificial neural networks (ANNs) that are able to learn directly from the raw data what representation is needed, for example, for detection or classification (Fig. 60–3). While DL comprises a large family of ML approaches that are based on ANNs with multiple layers (hence the term “deep”), DL architectures such as deep neural networks (DNNs) became particularly popular in tasks such as image classification and segmentation. Structurally, DL methods such as DNNs are composed of multiple processing layers with simple, nonlinear units that transform the raw input data into multiple representations with gradually higher levels of abstraction (LeCun, Bengio, and Hinton, 2015; Goodfellow, Bengio, and Courville, 2016). One of the main reasons for the great success of DL approaches came with their capability of being trained “end to end” by using a single multilayer neural network to learn the mapping of raw input data (e.g., array of pixels in an image) to an output (e.g., scores for categories like “cat” or “dog”). Although it is not the objective to provide a comprehensive introduction to DL here, in the following paragraph we will introduce a few general concepts and useful terms that hopefully will help researchers without DL experience navigate through the growing body of literature in animal tracking.
Deep learning basics. A. An example of a multilayer neural network with an input, two hidden, and an output layer. Such feedforward neural network can be trained with stochastic gradient descent in three main steps: forward pass, loss computation, and (more...)
While there are many different categories of DL algorithms, those for supervised learning are probably the most common. Supervised learning algorithms require training with large datasets that consist of several thousand or more of such input-output pairs with the objective (represented by an “objective function”) to reduce the error between predicted and desired output by iteratively adjusting internal parameters, the so-called weights. In practice, training a feedforward neural network in a procedure called stochastic gradient descent (SGD) can be divided into three main steps (Fig. 60–3A): forward pass, loss computation, backward pass. First, the output scores (“prediction”) are computed by (forward) passing the raw data from the input layer through the hidden layers to the output layer, computing for each unit in a layer the total input from the units of the layer below as a weighted sum (“weights”) and then passing the result through a nonlinear activation function (such as a rectified linear unit [ReLU]). Second, the error (or loss) between predicted value and ground truth is calculated by an objective function (also known as loss and cost function), which “can be seen as a kind of hilly landscape in the high-dimensional space of weight values” (LeCun, Bengio, and Hinton, 2015). Finally, the negative gradient, “the direction of steepest descent in this landscape” (LeCun, Bengio, and Hinton, 2015), is calculated by backpropagation, which can be used to adjust the weights and is merely a practical application of the chain rule for derivatives from calculus, a technique to find the derivative of composite functions. While the use of backpropagation to compute gradients was discovered in the 1970s and 1980s, neural networks were unpopular in the 1990s until the early 2000s, when they experienced a renaissance with deep feedforward networks and the emergence of fast graphics processing units (GPUs) (e.g., see review by LeCun, Bengio, and Hinton, 2015, or Bottou, Curtis, and Nocedal, 2018).
A subtype of deep feedforward networks that became particularly successful during the early 2010s were convolutional neural networks (ConvNets [CNNs]), especially in tasks requiring image recognition, segmentation, and detection (Krizhevsky, Sutskever, and Hinton, 2012; Farabet et al., 2013; Girshick et al., 2014; Sermanet et al., 2014). Key properties of ConvNets include local connections to extract highly correlated features, shared weights that can detect location-invariant features, and pooling to merge semantically similar features together (Fig. 60–3B). With the alternation of convolutional and pooling layers, ConvNets are capable of decomposing image features hierarchically, and thus, resembling the computation in the ventral visual stream of the mammalian brain (Hubel and Wiesel, 1962; Felleman and Van Essen, 1991). Interestingly, in computational neuroscience, models based on ConvNets were even shown to predict single-unit and population responses in higher visual areas (e.g., in V4 and inferior temporal cortex) to naturalistic images and thus ConvNet models turned out to be useful for studying high-level human abilities such as visual object recognition (Yamins et al., 2014). More generally, ConvNets can be used to study sensory cortical processing, modeling the encoding process that transforms external stimuli into neuronal activity; in contrast to the decoding process where neuronal activity generates behavior (Yamins and DiCarlo, 2016). From a practical perspective, an encoding-decoding architecture with ConvNets (Fig. 60–3B) is particularly useful in combination with transfer learning (Caruana, 1994; Bengio, 2011; Bengio et al., 2011; Yosinski et al., 2014). With architectures such as DeepLab (Chen et al., 2018), for example, this allows a neural network to be extensively pretrained on a large dataset such as ImageNet (Deng et al., 2009), and then the trained encoder (“backbone”) can be transferred to a custom network, where only higher-level portions (e.g., the decoder) of the new network have to be fine-tuned, requiring less labeled training data. ConvNets, transfer learning, and many other DL concepts have become building blocks of state-of-the-art animal tracking to address the specific needs in research settings.
As for so many fields, DL revolutionized tracking of animals in various ways, from robustly tracking single (Mathis et al., 2018; Pereira et al., 2019) and multiple animals (Romero-Ferrero et al., 2019) and their poses (Pereira et al., 2020; Lauer et al., 2021) to 3D pose estimation in different species (Arac et al., 2019; Dickinson et al., 2020; Dunn et al., 2021). Generally, most current pose estimation frameworks, which are mainly in 2D, use ConvNets and an encoder-decoder architecture (see above), where the representation of input image in the encoder and key points (landmarks) of the pose in the decoder are jointly learned (Fig. 60–2D). Instead of landmarks, it is common practice to learn landmarks as as heat maps, so-called confidence maps (Carreira et al., 2016; Pereira et al., 2019), different density functions in the image where the brightest pixel of a particular confidence map represents the most likely location of a particular landmark.
Directly learning to extract relevant features from images with DL usually requires large, labeled datasets for training, which is a problem in particular in laboratory settings with limited acquisition and labeling capacities. There are currently five ways how this labeling problem is addressed: transfer learning, efficient neural network design, active learning, marker-based ground truth, and synthetic datasets.
One of the most widely used animal pose estimation software package is DeepLabCut (Mathis et al., 2018), which takes advantage of transfer learning (see above) to lower requirements of labeled data and shorten training times. DeepLabCut pretrain their models on ImageNet, factually training an encoder for general-purpose visual feature detection (Mathis et al., 2018; Nath et al., 2019). However, to boost robustness on out-of-domain data (e.g., held out species during pretraining), DeepLabCut recently switched from this task-agnostic pretraining to a more domain-specific framework, pretraining their encoders directly for animal-pose-specific feature detection on a purposefully created SuperAnimal dataset, which provides uniform key points across species (Ye et al., 2021). Also capable of transfer learning is DeepBehavior (Arac et al., 2019), a tool box that offers different ConvNet architectures, which can be pretrained on datasets like CoCo (Lin et al., 2014) or ImageNet (Deng et al., 2009; Russakovsky et al., 2015) that offer bounding boxes annotation (i.e., imaginary rectangles outlining the object of interest in an image). One of the architectures in DeepBehavior is an improved version of the well-known model YOLO (“You Only Look Once”) (Redmon et al., 2016; Redmon and Farhadi, 2018), a fast ConvNet architecture that simultaneously predicts spatially separated bounding boxes and associated class probabilities to localize and categorize objects in images. Although YOLO models are elegantly designed for real-time object detection and are great in laboratory settings to quantify simple objects (e.g., in a three-chamber test) and social interactions (e.g., in the case of two mice with miniscopes) (Arac et al., 2019), their focus on predicting bounding boxes makes such models less applicable for accurately estimating more complex poses.
Another popular toolkit is LEAP (Pereira et al., 2019) and its successor DeepPoseKit (Graving et al., 2019), which designed efficient neural network with smaller architectures (e.g., less-deep—i.e., fewer layers—ConvNets) and thus need fewer parameters to be tuned when training on small labeled datasets. While neural networks like LEAP are primarily used on images of animals in constrained laboratory settings with more uniform imaging conditions, neural networks like DeepLabCut try to be more multipurpose, allowing pose estimation in both laboratory settings and “in the wild.”
Apart from such architectural choices, other approaches such as active learning and the related human-in-the-loop training seek to improve labels on smaller datasets by starting with a small number of images that capture the diversity of representative features in the dataset and then they iteratively let the network predict and the human observer correct labels on new data over several training loops (Collins et al., 2008; Branson et al., 2010). Others (Dunn et al., 2021) removed the burden of manual labeling entirely by combining different camera systems and synchronously acquiring training data (i.e., color images) with regular color cameras and corresponding ground truth labels with infrared cameras for marker-based motion capture (e.g., by using retroreflective piercings). While these datasets with marker-based motion capture itself can be used to study animal behavior (e.g., idiosyncratic perseverative grooming sequences in a rodent model of fragile X syndrome; Marshall et al., 2021), methods requiring markers to create ground truth datasets may not be a feasible option in a number of paradigms (e.g., for species like zebrafish—Danio rerio—or octopus). Hence, animating photorealistic 3D animal models in different poses with exact body part labels are a valuable alternative, with an opportunity to synthetically create a potentially infinite ground truth dataset for animal pose estimation (Bolaños et al., 2021). While all these different methods approach the general labeling problem in different ways, they show different strength and weaknesses, often depending on the recording conditions (e.g., pose estimation in 3D or for multiple animals).
Behavior, the temporal dynamics of poses (including postures and facial expressions), can be decomposed into movements of individual body parts and thus inherently takes place in three dimensions (3D). For 3D pose estimation in animals, most approaches combine two-dimensional (2D) ConvNets and traditional triangulation to retrieve the 3D positions of the detected 2D key points, including DeepLabCut (Nath et al., 2019), DeepBehavior (Arac et al., 2019), and Anipose (Dickinson et al., 2020) (Fig. 60–2E). Triangulation requires the cameras to be robustly calibrated beforehand, usually by deploying a calibration board (e.g., a checkerboard or AprilTags; Olson, 2011), and some use triangulation to reduce the labeling effort by projecting key points from one view to another (e.g., as used in the 62-camera setup of OpenMonkeyStudio; Bala et al., 2020). Calibration, the process of estimating camera internal (“intrinsics” like the focal length) and external (“extrinsics,” including rotation and translation) parameters, has a long history in computer vision with several highly accurate algorithms (e.g., bundle adjustment [Triggs et al., 2000] or direct linear transformation [Hartley and Zisserman, 2004]). Some 3D animal pose estimation frameworks (Zhang and Park, 2020) take advantage of the calibration and train ConvNets with supervision from the epipolar geometry, enforcing between camera views constraints on the 2D key point detection. In contrast, DeepFly3D (Günel et al., 2019) calibrates a multicamera setup without a calibration board and instead uses pictorial structures (Felzenszwalb and Huttenlocher, 2005) of detected key points in images, specifically line segments representing the legs of a tethered Drosophila, to improve through active learning iteratively the 3D pose accuracy of the fly’s legs. Leveraging the rigidness of rodent skeleton and the consistency of joint angles between similar postures, GIMBAL (Zhang et al., 2020) introduces spatiotemporal priors to their model to improve 3D key point detection. DAANCE (Dunn et al., 2021), on the other hand, uses a coarse estimate of each camera position and orientation to project features from 2D ConvNets into 3D space, where a 3D ConvNet fuses features from different views to predict accurate landmarks in 3D. While most approaches still require some form of precalculated triangulation for new data, LiftPose3D (Gosztolai et al., 2020) aimed to learn the nonlinear mapping between 2D and 3D poses directly in order to achieve high-quality 3D pose estimation in the absence of large camera arrays and calibration. As hopefully illustrated by this list of selected algorithms depicting the diversity of sophisticated architectural designs, the computer vision and animal tracking community has started to tackle several key limitations of DL-based 3D skeleton tracking in different laboratory settings, including the demand for specialized hardware (e.g., large arrays of calibrated cameras) and the robustness of DL-based methods compared to maker-based approaches.
While 3D pose estimation frameworks gain popularity in the neuroscience community, not the least due to their relevance in assessing motor impairments (Machado et al., 2015), there is a natural desire in neuroscience research community to describe the behavioral state of an animal beyond its pose, including somatosensory sensations like tactile information, and extend skeleton-based approaches toward a holistic 3D surface representation of the animal (Fig. 60–2F). Capturing the 3D surface of a body is a difficult task that receives growing research interest in computer vision. Although most research focuses on human bodies, there are a few attempts to apply the same techniques to animals; however, so far none of them has found its way into basic neuroscience research yet. Generally, there are two different approaches to capture the 3D surface of a body and enabling a dense pose tracking: model-based and reconstruction-based methods. Most model-based approaches leverage the success of key point and feature detection methods to establish correspondence between input data (e.g., an image depicting a person) and a 3D model (e.g., a generic human in a specific pose). Models are usually created using 3D scans, and there is a body of work in building large datasets of 3D models of humans in different poses and interactions with the environment, which includes the datasets and models of SCAPE (Anguelov et al., 2005), FAUST (Bogo et al., 2014), SMPL (Loper et al., 2015), Frank/Adam (Joo, Simon, and Sheikh, 2018), and GRAB (Taheri et al., 2020). Since such datasets build the foundation of almost all model-based approaches, similar datasets would need to be acquired for a variety of animals. Although these 3D setups are scaled and optimized to scan bodies of humans, there have been attempts to acquire animal datasets within similar setups, as shown for dogs (Kearney et al., 2020). To address the challenge of acquiring large datasets in a variety of animals, some of which being less cooperative (e.g., tigers and birds), 3D scans of animal figurines in different poses (e.g., SMAL [Zuffi et al., 2017]) or articulated 3D mesh models created by an artist (Badger et al., 2020) have shown to be great alternatives. Similar to pipelines of model-based approaches in humans, different DL approaches are deployed to then learn deforming these 3D animal-shaped models, so they fit the pose and shape of an animal in an image. Such model-based approach have already shown great success for dense pose estimation in a variety of animal image datasets and some smaller video datasets that include animals such as dogs (Biggs et al., 2019), birds (Badger et al., 2020; Y. Wang et al., 2021), tigers (Zuffi, Kanazawa, and Black, 2018), and zebras (Zuffi et al., 2019). Other methods that are in a broader sense model-based comprise approaches that try to learn the 3D shape of a particular class of animals directly from large image datasets of that class of animal (as shown in birds [Kanazawa et al., 2018] and dogs [Biggs et al., 2020]) or approaches based on DensePose (Güler, Neverova and Kokkinos, 2018) that directly learn the correspondence between key points on an image and a 3D surface model (as shown in chimpanzees [Sanakoyeu et al., 2020] and a variety of quadrupeds [Neverova et al., 2021]). However, mostly due to the lack of large 3D video datasets of real animals (as available for humans; see list of datasets above), model-based approaches for dense animal pose tracking to date are still struggling to create realistic movement patterns in 3D, with some research groups starting to integrate additional motion capture data to address this problem (e.g., horse-specific dataset called hSMAL [Li et al., 2021]). In contrast to model-based approaches, reconstruction-based methods are agnostic to the identity of objects and can reconstruct entire scenes. There are several approaches for 3D reconstruction with a long history in computer vision (Fitzgibbon and Zisserman, 1998; Seitz et al., 2006; Newcombe et al., 2011; Niessner et al., 2013; Dou et al., 2016; Schönberger et al., 2016; Innmann et al., 2020). While an individual description of these techniques is beyond the scope of this chapter, most of them rely on large sets of images that capture a scene from different viewpoints either using several regular RGB (red-green-blue) cameras or a few specialized RGB-D cameras that provide additional depth (“D”) information like stereo cameras (e.g., Intel® RealSense™) or time-of-flight (ToF) cameras (e.g., Microsoft® Kinect™). While some studies use multiple RGB-D cameras to facilitate the tracking of 3D poses (Matsumoto et al., 2013; Ebbesen and Froemke, 2020), to date there are no approaches for true 3D surface reconstruction that are used in neuroscience research settings. However, promising results come from studies using single RGB-D cameras and thus depth information (sometimes refer to as 2.5D) for behavioral tracking, which highlights the relevance of information that goes beyond simple poses and the necessity of incorporating details about an animal’s surface for behavior analysis. Such approaches were successfully used for unsupervised phenotyping of rodent behavior (Wiltschko et al., 2015, 2020), the study of striatal (Markowitz et al., 2018) and cerebellar (Rudolph et al., 2020) circuits, as well as studies focusing on complex social behaviors (Hong et al., 2015; de Chaumont et al., 2019).
Multianimal tracking has attracted more attention recently, not the least because of initiatives in the neuroscience community to study animal behavior in more natural and complex environments (e.g., Dennis et al., 2021). However, due to close interactions between animals, pose estimation with multiple animals is challenging. It is a daunting task to detect key points (i.e., body parts) in video frames of interacting animals and assign them accurately to individuals across frames. As discussed and supported by the software package SLEAP (Pereira et al., 2020), there are two main strategies for pose estimation with multiple animals: top-down and bottom-up. In the top-down approach, every animal (“instance”) is first localized (“anchored”) in the image and then body parts are detected within cropped images of each instance. In the bottom-up approach, all body parts are first detected and then assigned to different instances. The top-down (“instances-then-parts”) approach uses two distinct neural networks, one for instance and one for body part detection, factually predicting separate confidence maps for each, and requiring running the second neural network multiple times (i.e., for each animal separately). In contrast, the bottom-up (“parts-then-instances”) approach uses only one neural network and predicts for all body parts confidence maps and so-called part affinity fields (Cao et al., 2017), 2D vector fields that encode the location and orientation of each body part, which can be used to connect body parts to directed graphs (“skeletons”) for each instance. DeepLabCut has also recently added multianimal pose estimation to their framework, testing their bottom-up approach in different species (mice, marmosets, and fish) and a dataset with parenting mice (Lauer et al., 2021). Together with recent successes in identifying and tracking multiple animals in large collectives of up to 100 animals (Romero-Ferrero et al., 2019), it seems just a question of time when pose dynamics of animals, their behavior, can be accurately measured in ethologically meaningful or stressful social settings (e.g., such as John Calhoun’s “behavioral sink”; Calhoun, 1962).
Behavior is composed of coordinated movements of individual body parts and, as such, can be described by the animal’s pose at different levels of detail (see above and Fig. 60–2). Beside the granularity of the pose description, behavior can be analyzed agnostic to the surrounding environment (“egocentric representation”) or with respect to the surrounding environment (“allocentric representation”) (Fig. 60–1B), with some behavioral analysis software exploiting the first (like MotionMapper [Berman et al., 2014] and MoSeq [Wiltschko et al., 2015]) and others the second (JAABA [Kabra et al., 2013], B-SOiD [Hsu and Yttri, 2019], LiveMouseTracker [de Chaumont et al., 2019], SimBA [Nilsson et al., 2020], and MARS [Segalin et al., 2020]) for their analysis. Analyzing pose dynamics independently of position and orientation in the environment, poses in 2D or 3D can be centered at new egocentric coordinates and aligned by using two (e.g., head and pelvis) or three (e.g., plus sternum) anatomical landmarks, respectively. In this egocentric coordinate space, the movement of individual body parts or the relation between them can be quantified based on the speed and direction of the coordinate transformation of different anatomical landmarks (Fig. 60–1B). An “egocentric representation” of behavior can be used, for example, as kinematic description of both voluntary (i.e., physiological) behavior such as grooming and involuntary (i.e., pathological or optogenetically induced) behaviors such as “neomorphic” waddling or spinning (Wiltschko et al., 2015). For basic epilepsy research specifically, the egocentric representation of pose dynamics can replace labor-intensive, manual assessment of seizure behavior using the Racine scale (Racine, 1972) with an unbiased description of ictal behavior that can be acquired automatically within a variety of environmental contexts (e.g., home cage or different behavioral assays) (Fig. 60–1B). In contrast to representing poses in an egocentric coordinate system, velocity, distance, and angle between anatomical and environmental landmarks in the original allocentric coordinate system describes an animal’s interaction with its environment. Hence, an “allocentric representation” proves useful, for example, to assess higher order functions or dysfunctions in epileptic animals, such as spatial discrimination and memory functions (e.g., in the object location memory task [Bui et al., 2018]), or to infer an animal’s state of mind with regard to a particular environmental context (e.g., anxiety-related avoidance behavior in the open arms of an elevated plus maze). In essence, selectively switching between egocentric and allocentric representations of pose dynamics provides a way to translate the features tracked with DL approaches into classical descriptions of epileptic phenotypes and enables future AI-guided findings to be put in the context of the large body of work analyzing behavior in animal models of epilepsies with traditional approaches (Fig. 60–1).
The temporal structure of behavior is often thought to be composed of a set of discrete, reoccurring stereotypic modules as shown in a variety of model organisms in neuroscience—Drosophila melanogaster (Berman et al., 2014; Calhoun, Pillow, and Murthy, 2019), Caenorhabditis elegans (Yemini et al., 2013; Linderman et al., 2019), Danio rerio (Marques et al., 2018; Johnson et al., 2020), Mus musculus (Wiltschko et al., 2015; Markowitz et al., 2018), and Rattus norvegicus (Marshall et al., 2021). Measures of behavior (e.g., video frames or tracked features) can be decomposed into (behavioral) modules by a variety of methods (see also reviews: Anderson and Perona, 2014; Brown and De Bivort, 2018; Datta et al., 2019; Pereira, Shaevitz and Murthy, 2020). While both raw data (e.g., video frames) or extracted behavioral features (e.g., animal’s pose) in either 2D or 3D can be used as input data (Fig. 60–4A), it is common to align them to egocentric coordinates first. Similar to the alignment of tracked poses (see above), raw video footage (or the 3D voxel-representation of a scene), which in addition captures nonrigid body movements (e.g., facial expressions [Stringer et al., 2019; Dolensek et al., 2020]), can be cropped to the size and aligned to the orientation of an animal (Fig. 60–4A), which than can be used as input data for behavioral segmentation (Fig. 60–4B).
Decomposition and quantification of behavior. A. Input data to decompose behavior into discrete behavioral modules can be either the raw video image data or extracted behavioral features from this video data. In 2D, a sequence of either individual video (more...)
Generally, techniques that decompose the temporal structure of behavioral data into stereotypic, reoccurring segments, which are sometimes referred to as behavioral modules, can be roughly divided into two main categories: supervised and unsupervised methods. While there are a variety of different supervised ML approaches to segment behavior (e.g., decision trees or random forest [Kabra et al., 2013; Hong et al., 2015; Nilsson et al., 2020]), all supervised methods rely on a large set of labeled data, with the possible exception of self-supervised techniques (not discussed here). Besides pure manual labeling, one common practice to facilitate the creation of human-provided labels is to use the tracked features and a set of criteria. These predefined rules range from simply thresholding different tracked features of a single animal (e.g., speed threshold for the label “running”) to complex combinatorial rules for social descriptions (including dyadic dynamics and group-building behavior) using pose annotations from multiple animals (de Chaumont et al., 2012, 2019; Kabra et al., 2013; Segalin et al., 2020). While such algorithms can drastically reduce the annotation time, variation in human annotation (e.g., inter-annotator and inter-lab differences) remains a problem (Szigeti, Stone, and Webb, 2016; Leng et al., 2020; Segalin et al., 2020), which some researchers try to address directly by unveiling the annotation style itself (e.g., highlighting the relevance of behavioral features for an annotator’s choice of a label) to improve annotator consensus and thus reproducibility of behavioral studies (Tjandrasuwita et al., 2021).
Unsupervised methods, on the other hand, elude the requirement for human-provided labels entirely and provide a data-driven alternative without a priori assumptions about the feature composition of different behavioral modules. Most of these unsupervised pipelines first deploy linear (e.g., principal component analysis [PCA] [Jolliffe, 2002]) or nonlinear dimensionality reduction technique (e.g., manifold embedding like t-stochastic neighbor embedding, t-SNE [van der Maaten and Hinton, 2008]) to remove redundant information in the data that arises, for example, from correlative movements between body parts. For example, it was found that projecting high-dimensional posture descriptions onto a low-dimensional basis (i.e., by applying PCA), which is formed from eigenvectors that each represent a fundamental movement and/or pose, 95% of the shape variance in Caenorhabditis elegans can be described by just four dimensions (eigenvectors) (Stephens et al., 2008). These dimensions (or modes) were termed “eigenworms” (Stephens et al., 2008; Ahamed, Costa, and Stephens, 2021), and since then, similar “postural eigenmodes” were used in other animals such as fruit flies (Berman et al., 2014; Werkhoven et al., 2019), zebrafish (Girdhar, Gruebele, and Chemla, 2015), and rodents (Wiltschko et al., 2015; Marshall et al., 2021). After dimensionality reduction, there are two ways to analyze the preprocessed data: discretize low-dimensional data into (behavioral) modules or analyze its continuous representation. For the first one, there are two main strategies: clustering or modeling. Clustering algorithms such as k-means (Lloyd, 1982), Gaussian mixture models (McLachlan and Peel, 2000), or density-based clustering algorithms (e.g., watershed-based heuristic [Meyer, 1994; Berman et al., 2014] or DBSCAN [Ester et al., 1996]) will group similar points together with different strategies, using similarity metrics such as Euclidean distance (Fig. 60–4B). A popular behavioral analysis pipeline is MotionMapper (Berman et al., 2014), which deploys many of these techniques and is used in different animal models and experimental settings (Berman et al., 2014; Wang et al., 2016; Klibaite et al., 2017; Liu et al., 2018; Merel et al., 2019; Pereira et al., 2019; Zimmermann et al., 2020; Marshall et al., 2021). Specifically, MotionMapper uses PCA on aligned data, followed by converting low-dimensional time series into a frequency domain representation using a Morlet wavelet transform (Goupillaud, Grossmann, and Morlet, 1984), which is mapped with t-SNE to 2D and clustered with watershed transform into regions on the 2D embedding, where peaks represent more and valleys represent less stereotypic behaviors. Others like B-SOID (Hsu and Yttri, 2019) or OpenMonkeyStudio (Bala et al., 2020) are conceptually similar, but replace PCA with state-of-the-art dimensional reduction techniques like UMAP (i.e., uniform manifold proximation and projection) (McInnes, Healy, and Melville, 2018) and t-SNE with hierarchical DBSCAN (Campello, Moulavi, and Sander, 2013). In contrast to such clustering approaches, modeling the behavioral dynamics bears the potential to gain more insight into the structure of behavioral dynamics by reducing their complexity to a few model parameters. Motion sequencing (MoSeq), another popular behavioral analysis pipeline, uses a probabilistic graphical model, an autoregressive hidden Markov model (HMM), to identify hidden states and their transitions in low-dimensional (i.e. PCA preprocessed) RGB-D data (Wiltschko et al., 2015, 2020; Johnson et al., 2016; Pisanello et al., 2017; Markowitz et al., 2018; Rudolph et al., 2020). Noteworthy, MoSeq models each behavioral module as continuous auto-regressive process in behavioral-state space and thus builds a hybrid between analysis approaches with discretized and continuous representations of behavior. Generally, continuous representations take account of the continuous patterns of motion (e.g., oscillator structure during locomotion) and are used to analyze a variety of complex behaviors, including hunting behaviors of zebrafish and other stereotyped behaviors [Stephens et al., 2008, 2011; Bolton et al., 2019; DeAngelis, Zavatone-Veth, and Clark, 2019; Mearns et al., 2020; Ahamed, Costa and Stephens, 2021]). More recently, DL approaches such as structured VAE (Johnson et al., 2016) and VAE-SNE (Graving and Couzin, 2020) showed their great potential for extracting such continuous dynamic representations from behavioral time series in a flexible and reliable manner, while simultaneously being able to discretize them into stereotypic modules. Although there are still several open challenges about how to best segment continuous behavior into behavioral components and recover the hierarchical organization of behavior (Berman, Bialek, and Shaevitz, 2016; Datta et al., 2019; Tao et al., 2019), many of the unsupervised techniques discussed here (such as hierarchical DBSCAN or hierarchical HMM) are capable of unraveling the hierarchical structure of a behavioral dataset (Fig. 60–4B).
A key obstacle to the faster development of new therapies for the epilepsies is that rigorous preclinical epilepsy research typically requires labor-intensive and expensive 24/7 video-EEG monitoring followed by subjective scoring of behavioral seizures as exemplified by the Racine scale (Racine, 1972). While automated electrographic seizure detection algorithms are improving, the critically important behavioral manifestations of epilepsy both during ictal and interictal periods remain poorly quantified and are subject to observer bias. As discussed in this chapter, recent technological advances in AI-guided quantification of behavior, including animal motion tracking and description of behavioral dynamics, highlight a promising future for behavioral phenotyping in basic epilepsy research. While the techniques discussed in this chapter are not yet adopted broadly by the epilepsy community, we would like to highlight two main areas where AI-guided phenotyping in epilepsy would be particularly impactful: for screening and mechanistic insights.
Behavioral phenotyping has a long history in epilepsy research and is well described in a variety of animal models, including fruit flies (Parker et al., 2011) and zebrafish (Baraban et al., 2005; Griffin et al., 2017, 2021). In rodent models of epilepsy, a large body of work showed that a spectrum of behavioral symptoms can be correlated with electrographic discharges, which was used to create a variety of different behavioral seizure scales (Racine, 1972; Jobe, Picchioni, and Chin, 1973; Pinel and Rovner, 1978a, 1978b; Pohl and Mares, 1987; Haas, Sperber, and Moshe, 1990; Veliskova et al., 1990). The symptoms observed range from behavioral arrest and automatisms to wild running and tonic-clonic movements of the limbs. While the changes with strong behavioral components are easier to identify, other, more subtle changes (e.g., staring, trembling of whiskers) require a trained eye and therefore are highly dependent on the human observer. As an objective alternative to the Racine and other scales, AI-guided phenotyping is an unbiased approach to partitioning and labeling complex behaviors, providing an opportunity to systematically characterize epileptic phenotypes beyond traditional behavioral classification terms (i.e., hand-labeling coarse behavioral classes, as in the Racine scale [Racine, 1972]), automatically quantify epileptic seizures, and search for stereotypic behavioral modules and transitions that are not a priori defined to capture potentially unrecognized epileptic phenotypes. For example, it was shown that AI-guided behavioral analysis can identify subtle behaviors that are normally not distinguished by a human observer as, for example, for the waddling gait in Ror1b mutant mice (Wiltschko et al., 2015). Moreover, AI-guided behavioral phenotyping has a great potential for anti-seizure drug (ASD) screening at scale. For example, Wiltschko et al. (2020) showed that AI-guided behavioral phenotyping with the analysis pipeline MoSeq is capable of automatically discriminate between mice injected with different neuroactive and psychoactive drugs and can even be used to identify specific on- and off-target effects of drugs in a mouse model of autism spectrum disorder. One could imagine that a similar approach—an out-of-the-box methodology for AI-guided behavioral phenotyping in epilepsy, offering a powerful, freely shared experimental and analytical tool for quantifying complex behavior with subsecond precision—would have great impact on basic epilepsy research. Such an approach would increase reliability of behavioral phenotyping and accelerate the assessment of both established and candidate therapeutics in a variety of acquired and genetic epilepsy models. With growing efforts to introduce data standards such as Neurodata Without Borders (NWB) (Teeters et al., 2015; Rübel et al., 2019), one could envision that behavioral datasets similar to those of the International Brain Laboratory (Abbott et al., 2017; Bonacchi et al., 2019) with standardized descriptions for behavioral phenotypes could be shared across labs (e.g., in a central database or through individually managed datasets [Sun et al., 2021]), cross-examined, and used for generating new hypotheses about the mechanistic basis and potential treatment options for different phenotypes. For example, a large, publicly available dataset would open opportunities to exploit techniques such as transfer learning, if combined with behavioral analysis, which would be especially useful for rarer types of epilepsy syndromes or generally any phenotype with epilepsy-like symptoms where there is less data available such as organophosphate-induced epileptic phenotypes (Enderlin et al., 2020; Guignet et al., 2020) or radiation-induced memory deficits that are in many ways similar to the cognitive comorbidities observed in experimental TLE (Klein et al., 2021).
Leveraging the recent advance made in both AI-guided behavioral phenotyping (discussed here) and neurotechnologies (Vázquez-Guardado et al., 2020) will likely enable studies aiming to provide better mechanistic insights into alterations in neuronal activity underlying different motor and cognitive symptoms in epilepsy. The necessity to study the “Brain In Action” (Mott, Gordon and Koroshetz, 2018), a research priority of the NIH BRAIN Initiative (Bargmann and Newsome, 2014), applies to both physiological and pathological function. As pointed out by Datta et al. (2019), some of the most exciting discoveries in neuroscience in the past 50 years, which includes place cells (O’Keefe and Dostrovsky, 1971; O’Keefe, 1976), grid cells (Hafting et al., 2005) and replay (Skaggs and McNaughton, 1996; Nádasdy et al., 1999), were made in studies using neural recordings in freely behaving animals. Hence, studying their dysfunction in the epilepsies will also require both neural recordings and behavioral measures in more naturalistic and complex environments. For example, recent findings in animal models of TLE that show altered CA1 place cell (Liu et al., 2003; Shuman et al., 2020) and mossy cell function (Bui et al., 2018) indicate that more sophisticated exploration and learning paradigms (e.g., a more naturalistic and complex maze environment [Rosenberg et al., 2021]) will be helpful for gaining a better mechanistic understanding of spatial memory impairments in TLE. Navigation, as most behaviors, is complex and requires the integration of multiple cues through multimodal (e.g., visual or auditory) perception. Analyzing and modeling these modalities (Fig. 60–5A,B) can be done with similar approaches as discussed above for behavior in general. For example, there is a rich literature of different unsupervised clustering and modeling techniques that can be used to analyze acoustic events in animals (Pearre et al., 2017; Coffey, Marx and Neumaier, 2019; Sainburg, Thielk and Gentner, 2020; Fonseca et al., 2021), and these insights have shown to be useful in particular for social assays, where ultrasonic vocalizations (USVs) seem to be associated with distinct behaviors (Sangiamo, Warren and Neunuebel, 2020). Interestingly, ancillary information such as acoustic data is sometimes necessary to assist behavioral analysis pipelines in distinguishing similar behaviors and inferring motivational states, as shown in mice for same- and opposite-sex mounting behavior that could only distinguish between sexual and aggressive motivation by the presence and absence of USVs, respectively (Karigo et al., 2021). Altogether, such multimodal measurements can be combined in so-called agent-based models, where an artificial neural network (“agent”) can be trained to react based on perceived stimuli in a virtual environment. These agent-based models have recently gained more attention in neuroscience, including building “playgrounds” for cognitive AI in form of classical animal cognition tasks (Beyret et al., 2019; Crosby, Beyret and Halina, 2019) or modeling behavioral strategies and kinematics in a virtual rodent in an neuroethologically meaningful manner (Merel et al., 2019). Interestingly, training such agent-based, abstract models like a recurrent neural network for path integration lead to the emergence of grid-like representations similar to those seen in the entorhinal cortex (Banino et al., 2018). Such computational approaches may prove to be particularly useful for modeling alterations in cognitive functions in basic epilepsy research (e.g., spatial memory deficits), as they can use in silico perturbation strategies similar to those performed in biophysical models for studying the neuropathophysiology of the epilepsies (Case et al., 2012). Beyond the field of navigation research, agent-based models may also become increasingly attractive for modeling sensorimotor systems in combination with marker-less motion capture and biomechanical modeling (Merel et al., 2019; Sandbrink et al., 2020; Hausmann et al., 2021). In animal models of epilepsy, such combined approaches of acquiring and modeling data of sensorymotor systems would be highly valuable for developing a better understanding of the semiologies that are associated with distinct seizure types.
A possible future of hypothesis-driven research in epilepsy. A. Behavioral experiments such as object location memory tasks usually consist of task-related environmental stimuli and include multimodal measurements (e.g., video and audio recordings) that (more...)
In summary, AI-guided phenotyping provides an excellent interface between experimental and computational epilepsy research. On the one hand, AI-guided tracking algorithms provide a more functionally relevant (i.e., behavioral) readout that can be correlated to or even jointly modeled (Batty et al., 2019) with neural activity (Fig. 60–5A,C). On the other hand, AI-guided behavioral phenotyping can provide training data (e.g., field of view, pose, and movement trajectory) for computational models such as agent-based models to run virtual experiments on the basis of behavioral data from epileptic animals (Fig. 60–5B). The latter has potential to inspire new hypotheses that can be tested experimentally (e.g., with real-time closed-loop optogenetics [Armstrong et al., 2013; Krook-Magnuson et al., 2013]) (Fig. 60–5C), and thus will likely be of increasingly greater benefit for quantitative, rigorous hypothesis-driven research in the epilepsies.
Grant support to IS from the National Institutes of Health (5R01NS114020-02) and to TG from the Swiss National Science Foundation (174811 and 186757) is gratefully acknowledged. TG would also like to thank Frances Cho for useful discussions and constructive criticism.
The authors declare no relevant conflicts.
This is an open access publication, available online and distributed under the terms of a Creative Commons Attribution-Non Commercial-No Derivatives 4.0 International licence (CC BY-NC-ND 4.0), a copy of which is available at https://creativecommons.org/licenses/by-nc-nd/4.0/. Subject to this license, all rights are reserved.
Your browsing activity is empty.
Activity recording is turned off.
See more...
