Graser-et-al_2023_Deep_Learning_From_Trajectory_Data_Review_fixed-fig-ref.pdf

Graser-et-al_2023_Deep_Learning_From_Trajectory_Data_Review_fixed-fig-ref.pdf

1. 

   Deep Learning From Trajectory Data: a Review of DeepNeural Networks and the Trajectory Data Representationsto Train ThemAnitaGraser1,AnahidJalali1,JasminLampert1,AxelWeißenfeld1andKrzysztofJanowicz21AIT Austrian Institute of Technology, 1210 Vienna, Austria2University of Vienna, Vienna, AustriaAbstractTrajectory data combines the complexities of time series, spatial data, and (sometimes irrational) movement behavior. As dataavailability and computing power have increased, so has the popularity of deep learning from trajectory data. This paper aimsto provide an overview of deep neural networks designed to learn from trajectory data, focusing on recent work publishedbetween 2020 and 2022. We take a data-centric approach and distinguish between deep learning models trained using densetrajectories (quasi-continuous tracking data), sparse trajectories (such as check-in data), and aggregated trajectories (crowdinformation).KeywordsDeep learning, data engineering, movement data, trajectories1. IntroductionDeep learning has become a popular approach for devel-oping data-driven prediction, classification, and anomalydetection solutions. Work on deep learning from trajec-tory data is spread out over many domains, including butnot limited to computer science, geography, geographicinformation science, urban planning, and ecology. Con-sequently, it covers many use cases and correspondingtrajectory dataset types.Trajectory datasets can be categorized according to thelevel of detail: from dense trajectories (quasi-continuoustracking data of individual movement) to sparse trajec-tories (such as check-in data of individuals), and finally,aggregated trajectories (crowd-level information, typi-cally aggregated to edges/nodes in a mobility graph, toa grid, or to a set of points of interest)[1,2,3]. In manycases, the titles and abstracts of papers are not sufficientto determine which type of trajectory data was used totrain the deep learning model. While many papers startwith dense trajectories, most convert them into sparsetrajectories [4,5,6,7,8,9,10] or even aggregate themto crowd-level [11,12,13]. Common approaches to turn-ing dense trajectories into sparse trajectories include:converting them into a sequence of stop locations [4,7]Proceedings of the Workshop on Big Mobility Data Analytics (BMDA)co-located with EDBT/ICDT 2023 Joint Conference (March 28-31,2023), Ioannina, [email protected](A. Graser);[email protected](A. Jalali);[email protected](J. Lampert);[email protected](A. Weißenfeld);[email protected](K. Janowicz)Orcid0000-0001-5361-2885(A. Graser)© 2021 Copyright for this paper by its authors. Use permitted under CreativeCommons License Attribution 4.0 International (CC BY 4.0).CEURWorkshopProceedingshttp://ceur-ws.orgISSN 1613-0073CEUR Workshop Proceedings (CEUR-WS.org)or a sequence of traversed regions (grid cells) [6,8], orconverting them to trajectory images [5,9,10].In a related review of location encoding methods forGeoAI [14], the authors stress the analogy between NLPword-to-sentence relations and location-to-trajectory re-lations. This analogy has led to Word2Vec-inspired ap-proaches encoding location into a location embeddingusing, for example, Location2Vec [15], Place2Vec [16],or POI2Vec [17]. Another recent review [18] is dedi-cated to deep learning for traffic flow prediction models,which are primarily trained on aggregated trajectory data.However, to the best of our knowledge, there is no reviewpaper that provides an overview of the different neuralnetwork architectures used to learn from trajectory data.The goal of this work is to provide a first overviewof the current state of neural networks / deep learningtrained with trajectory data, structured by1.Use casecategory (travel time/crowd flow/location predictions,location/trajectory classifications, anomaly detection),2.Neural network architecture (CNN, RNN, LSTM, GNN,...), and3.Trajectory data granularity (dense, sparse, ag-gregated) and representation. Therefore, this review doesnot include classic ML approaches and does not providean exhaustive historical analysis of the field. Due tothe page limit, this paper does not fit an exhaustive listof all relevant works published in recent years. How-ever, we provide at least one paper for each use caseand network combination we identified. We specificallyreviewed publications at recent events, including SIGSpa-tial 20221[6,7,19,8,20,12,21,22], Sussex-Huawei Lo-comotion (SHL) Challenge 20212at the ACM interna-1https://sigspatial2022.sigspatial.org/accepted-papers/2http://www.shl-dataset.org/activity-recognition-challenge-2021/
2. 

   Figure 1:Overview of use cases and neural networks for trajectory data included in this review.tional joint conference on pervasive and ubiquitous com-puting (UbiComp) [23], Traffic4cast challenge 20213atNeurIPS [24], and Big Movement Data Analytics work-shop BMDA 20214at EDBT [11,25,26].Even though we focus explicitly on deep learning, itis worth noting that deep learning may not always bethe best approach [27]. In particular, the SHL Challengesummary [23] shows that regular machine learning mod-els outperform deep learning models on all three metrics:F1 score, train time, and test time.This review does not attempt to compare the per-formance of different deep-learning approaches. Eventhough there are some commonly used open datasets,such as the Porto taxi data5, the T-Drive taxi dataset6andGeoLife dataset7, and the Gowalla check-in data8, cross-paper comparisons outside of dedicated data challengesare notoriously difficult. For example, “Despite the Portodataset’s original use as a standardized benchmark foropen competition, design choices in subsequent workmake cross-paper comparison difficult. Firstly, differentpapers often augment the dataset with their metadata notpresent in the original release, which may give some mod-els an advantage over others independent of architectureor training design.” [20]Due to the large range of domains working on tra-jectory data analysis, the terminology used in different3https://www.iarai.ac.at/traffic4cast/2021-competition/4https://www.datastories.org/bmda21/BMDA21Accepted.html5https://www.kaggle.com/c/pkdd-15-predict-taxi-service-trajectory-i/data6https://www.microsoft.com/en-us/research/publication/t-drive-trajectory-data-sample/7https://www.microsoft.com/en-us/research/publication/geolife-gps-trajectory-dataset-user-guide/8http://snap.stanford.edu/data/loc-Gowalla.htmlpublications is not necessarily consistent. We, therefore,define the most important terms and abbreviations in aglossary at the end of this paper.2. Representing Trajectory DataFor Deep Learning Use CasesThis review is structured around eight use case categoriesof deep learning from trajectory data, as shown in Fig-ure1. The following subsections describe the use cases,neural network designs used to address them, and trajec-tory data used to train these networks. Figure2providesan overview of the diversity of identified trajectory rep-resentations. More details on the trajectory datasets andthe data engineering steps applied to the trajectory databefore they are used as input to train the neural networksare summarized in Tables1-3. Some works (e.g. [28,29])use additional data sources in combination with trajec-tory data to train their models. These additional datasources have been omitted from our review in favor ofclarity and conciseness.2.1. Location classificationThis use case category covers the classification of loca-tions using patterns derived from movement data. Theclassification of regionally dominant movement patternsmay be of interest in and of itself [9] or help with theclassification of POIs (e.g. ports [4]) or the classificationof trip destinations [19].To detect regionally dominant movement patterns,Yang et al. [9] use direction information and density mapsto generate directional flow images. They convert thetrajectories into images where each pixel contains the
3. 

   Rasterized trajectories(trajectory images &directional flow images)Yang et al. (2018)Chen et al. (2020)Yang et al. (2022)CNNTemporal graphsAltan et al. (2022)Derrow-Pinion et al. (2021)Lippert et al. (2022)(T)GNNDecomposed Space-Activity MatrixLyu et al. (2022)Memory networkRaw trajectoriesWang et al. (2018)Buijse et al. (2021)GEO-Conv CRNNTime series on streetnetwork edges / POIsBuroni et al. (2021)LSTMGeneralized trajectoriesMehri et al. (2021)Liatsikou et al. (2021)LSTMDiscretized trajectoriesFan et al. (2022)GRUTraffic moviesLu (2021)Wang et al. (2022)CNNLi et al. (2021)GCN+LSTMGao et al. (2022)GCN+GRUZhang et al. (2020)GANCarroll et al. (2022)Musleh et al. (2022)TransformersNguyen et al. (2022)RNNSingh et al. (2022)RNNSparserIndividual levelDenserCrowd level / AggregatedTrajectory data representationLocation (sequence)embeddingsFeng et al. (2018)Gao et al. (2019)GRU+attentionLi et al. (2020)SANNatural languagesentencesXue et al. (2022)TransformersLiao et al. (2018)RNNNN designResampled sequences ofspace & time deltasTritsarolis et al. (2021)RNNHong et al. (2022)TransformersTenzer et al. (2022)Hypernetworks(LSTMs)Resampled trajectoriesLiatsikou et al. (2021)LSTM-basedautoencoderRao et al. (2020)LSTM+GANZhang et al. (2022)VAE-likedeepgenerativemodelsOD matricesSimini et al. (2021)MLPFigure 2:Overview of trajectory data representations usedto train neural networksdirectional flows. They use a CNN to classify the inputimage patterns and detect the dominant regional move-ments.An approach that makes more use of temporal informa-tion is presented by Altan et al. [4]. They use a temporalGNN (TGNN) to distinguish gateway ports from actualports using AIS vessel movement data. After extractingthe ports (nodes) from the raw AIS messages using DB-SCAN, they extract trips between consecutive ports andbuild a graph for each time step to generate the time-ordered daily graph sequence for the TGNN.Lyu et al. [19] train a plug-in memory network to pre-dict trip purposes based on destination locations. Theirmodel is trained using activity, origin, and destinationmatrices derived from OD data using a non-negativeTucker decomposition scheme.2.2. Arrival time predictionThis use case category covers the prediction of traveltimes or arrival times, such as arrival time prediction intrain networks [30] and street networks [31,32].Since travel time often depends on historical traveltimes at a given time of day, recurrent mechanisms arecommonly used [31,32,30]. For example, Derrow-Pinionet al. [31] train GNNs on aggregated trajectories to pro-vide travel time predictions in Google Maps. The GNNgraph consists of segment and supersegment-level em-bedding vectors. Nodes store street segment-level data(average real-time and historical segment travel speedsand times, segment length, and road class), while edgesstore supersegment-level data (real-time supersegmenttravel times).In contrast, Wang et al. [32] introduces the GEO-convolutional network layer (GEO-Conv, also used byBuijse et la. [30]), which is trained on dense trajectoriesstating that “directly mapping the GPS coordinates intogrid cells is not accurate enough to represent the originalspatial information in the data”. The proposedGEO-Convlayertakes dense trajectories as input and applies a non-linear mapping of each trajectory (latitude and longitude)point, followed by a GEO-Conv step with multiple ker-nels. The resulting feature map of local paths is appendedwith a final column of distances of the local paths.2.3. Traffic volume predictionThis use case category covers traffic or crowd predictionsof volumes or flows, e.g., predicting traffic volume onstreet segments [11,18], human activity at specific POIs[28] and metropolitan areas [33,24], or predicting animalmovement dynamics [34].Aggregated trajectory data in the form of traffic moviesis provided in theTraffic4Cast 2021competition whichchallenged participants to predict traffic under conditionsof temporal domain shift (Covid-19 pandemic) and spatialshift (transfer to entirely new cities). Lu [24] won thischallenge using CNN (U-Net) and multi-task learning.Their multi-task learning approach randomly samplesfrom all available cities and trains the U-Net model tojointly predict the future traffic states for different cities.Wang et al. [12] follow this traffic movie approach as wellby aggregating individual-level trajectories into a gridwith inflow referring to the total number of incomingtraffic entering this region from other regions during agiven time interval and outflow representing the total
4. 

   number of traffic leaving the region. Zhang et al. [13]also follow the traffic movie approach, creating temporalgrids of average traffic speed and taxi inflow per cell.Li et al. [33] build a graph for their GCN by aggregat-ing CDR data and representing spatial statistical units asnodes and their relationship (physical distance, physicalmovement, phone calls) as edges. Similarly, Lippert etal. [34] build temporal graphs from bird migration datawhere nodes represent radar locations, and edges repre-sent the flows between the Voronoi tessellation cells ofthe radar locations.Finally, Buroni et al. [11] provide a tutorial using vehi-cle counts derived from GPS tracks to build and train a Di-rect LSTM encoder-decoder model. The model is trainedto predict counts of vehicles per network edge per timestep for the Belgian motorway network. Similarly, Gaoet al. [28] use their GPS tracks to count vehicles per POIper time step (hourly) to train a GCN+GRU model thatpredicts these visit counts. And Xue et al. [21] propose atranslator calledmobility promptingwhich converts dailyPOI visit counts into natural language sentences so theycan use (and fine-tune) pre-trained NLP models such asBert, RoBERTa, GPT-2, and XLNet to predict these visitcounts.2.4. Trajectory prediction/imputationThis use case category covers the prediction of trajecto-ries in artificial [35], urban [6], and maritime environ-ments [36,26], as well as imputation of trajectories [8].Mehri et al. [36] generalize AIS trajectories usingcontext-aware piecewise linear segmentation before feed-ing them into their LSTM three vertices at a time. Thisenables their model to perform short-term trajectory pre-dictions with high spatial detail. Tritsarolis et al. [26], onthe other hand, represent trajectories by their composi-tion of differences in spaceΔ������,Δ������, and timeΔ������for theinput of their RNN-based models to predict the vessel’sposition at timeΔ������ + 1.Fan et al. [6] discretize mobile phone GPS trajectoriesusing the H3 hexagonal grid and use the grid cell se-quences to train their GRU. The resulting model is usedto predict cell sequences which are afterward used tosearch for similar high-resolution trajectories, which arereturned as the final trajectory prediction.Carroll et al. [35] use synthetic discrete movement se-quences in a minimalistic grid world environment. Theirtransformers are trained on trajectories as sequences ofstates, actions, and return-to-go tokens to predict tra-jectories. Another work using transformers to imputetrajectories is Musleh et al. [8]. They propose TrajBERT,a model trained using H3-discretized (tokenized) GPStracks. They “down-sample the trajectories by droppingthree-quarters of the points of each trajectory and thenrun TrajBERT to fill the gaps by imputing the missingpoints”.2.5. (Sub)trajectory classificationThis use case category considers the classification of com-plete trajectories [10] or sub-trajectories [5,23] in orderto learn more about different vessel and human move-ment patterns.By splitting trajectories into sub-trajectories, morefine-grained analyses are possible. Typical applicationsinclude the detection of movement types, such as shipmaneuvers [5] or the detection of transportation andlocomotion modes of smartphone users [23]. Chen etal. [5] generate colour-coded trajectory images from shipAIS data, where each pixel is assigned one of three col-ors according to the movement type (static, normal, ma-neuvering). These trajectory images are used to train aCNN-based ship maneuver classifier. To identify differentmovement modes (i.e. still, walk, run, bike, car, bus, train,and subway) from smartphone data, the SHL Challengewinner uses an AdaNet algorithm, a Tensorflow-basedframework for learning NN models and ensembling mod-els to obtain even better models [46,23].An example of the classification of complete trajecto-ries is the recognition of ship types introduced by Yanget al. [10]. It relies on the same technique as [5] for trans-forming the raw AIS trajectory data into colour-codedtrajectory images. The resulting images show character-istic trajectory patterns, which can be used to identifythe ship vessel type with a CNN classifier.2.6. Next location / final destinationpredictionThis use case category covers the prediction of the nextlocations or final destinations of trips [38,37,7,39,40,41,21,20]. Besides GPS tracks, a commonly used datasource in this category are social media check-ins (e.g.,from Foursquare). The task then becomes to predict thenext check-in location (e.g., a POI).Attention mechanisms have proven to be a popularapproach for next location prediction. Gao et al. [37]trainVANext, a semi-supervised network trajectory con-volutional network, on check-in data. They convert eachindividual user’s trajectory (check-in/POI sequence) into9https://www.kaggle.com/datasets/giobbu/belgium-obu11https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2017/index.html12https://zenodo.org/record/449841013https://sites.google.com/site/yangdingqi/home/foursquare-dataset?pli=114http://www.start.umd.edu/gtd/15https://github.com/bigdata-ufsc/petry-2020-marc/tree/master/data/foursquare_nyc16https://drive.google.com/file/d/1rLJz5E0igbrmAnmnDmazdBl97UuQ0sch/view?usp=sharing
5. 

   Table 1Trajectory datasets and data engineering; public datasets are printed in boldRefNN de-signTrajectory data Data engineeringLocation classificationYang etal. (2018)[9]CNNSynthetic data (manually drawn trajectories,rotated in a data augmentation step)Trajectories are converted todirectionalflow images (DFI)(resolution: 10×10)Altan etal. (2022)[4]TGNNShip AIS tracks in Halifax, Canada, covering15 ports, 10 vessels, for 4 months (total: 513kAIS records)Trajectories are converted totemporal(daily) graphsof ports (nodes with associ-ated port visit frequencies, waiting times, andspeed statistics) and trips between consecu-tive ports (edges)Lyu etal. (2022)[19]MemoryOD data from Tokyo Metro travel survey in-cluding travel mode, time, and purposeODs are converted into aSpace-Activity Ma-trixwhich is then decomposed into an activ-ity, an origin, and a destination matrixArrival time predictionWang etal. (2018)[32]CRNNTaxi GPS tracks in Chengdu and Beijing witha density of 2.6 to 5.5 GPS records per kmTrajectoriesare fed into aGEO-ConvlayerBuijse etla. (2021)[30]CRNN+GRU Train GPS tracks in the Netherlands covering350k train tripsTrajectoriesare fed into aGEO-ConvlayerDerrow-Pinion etal. (2021)[31]GNNGoogle Maps road segments with traveltimes/speedsRoad network graph edges are subdividedinto shorter segments (modeled asGNNgraphnodes) with associated aggregatedtravel time/speed information; segments arecombined into supersegments (GNN graphedges)Traffic volume predictionLu (2021)[24]CNN(U-Net)Traffic movies for 10 cities in 2019+2020 with8 dynamic channels encoding traffic speedand volume per direction and 9 static chan-nels encoding the properties of the road mapsThe multi-task learning randomly samplesfrom all available citytraffic movies(reso-lution: 495×436)Wang etal. (2022)[12]CNNTaxi trajectories from TaxiBJ in Beijing for 17months and bike trajectories from BikeNYCin New York City for 6 monthsTrajectories are converted toflow/traf-fic movies(32×32 for TaxiBJ & 16×8 forBikeNYC) with two dynamic channels encod-ing inflows and outflowsLi et al.(2021)[33]GCN+LSTM Sparse CDR trajectories in Senegal of 100,000individuals for one yearTrajectories are converted into amovementgraphwith edges representing the numberof transitions from one cell phone tower nodeto the nextBuroniet al.(2021)[11]LSTMLorry GPS tracks9in Belgium with 30s re-porting interval, anonymous IDs (reset daily),timestamp, latitude, longitude, speed, and di-rectionTrajectories are matched to street networktocount vehicles per network edge pertime stepGao etal. (2022)[28]GCN+GRU Taxi GPS tracks in Xi’an, China covering 7.7ktaxis for 3 months with a sampling intervalof 5–30 sTrajectories are used tocount vehicles perPOI (n=100) per time step (hourly)Lippertet al.(2022)[34]RecurrentGNNBird migration data in the form of simulatedtrajectories and measurements from the Eu-ropean weather radar networkTrajectories are convertedtemporal graphswith to flows (edges) between Voronoi tessel-lation cells of radar locations (nodes).Xue etal. (2022)[21]Trans-formersSafeGraph daily POI visit counts in NYC, Dal-las and MiamiHistorical visitation data are translated intonatural language sentences to fine-tune pre-trained NLP modelsZhang etal. (2020)[13]GANTaxi GPS tracks in Shenzhen, China for 6monthsTrajectories are converted totraffic movies(resolution: 40×50, hourly) with average traf-fic speed and taxi inflow
6. 

   Table 2Trajectory datasets and data engineering; public datasets are printed in bold – continuedRefNN de-signTrajectory data Data engineeringTrajectory prediction/imputationMehri etal. (2021)[36]LSTMAIS data from NOAA10for the US EastCoast, containing 58.5mio messages from10.7k vessels over 2 monthsTrajectory is generalized using context-awarepiecewise linear segmentation. The LSTM istrained on three vertices at a timeTritsaro-lis et al.(2021)[26]RNNShip AIS tracks in Piraeus11containing138k records from 246 fishing vessels andGe-oLife7Trajectories are resampled to regular1 minute intervals and converted intosequences of differences in spaceΔ������,Δ������, andtimeΔ������to predict the vessel’s position in thenext future stepFan etal. (2022)[6]GRUMobile phone GPS tracks in the Kanto area ofTokyo covering 220k users with a minimumreporting period of 5 minutes for 2 monthsTrajectories arediscretized using the H3hexagonal gridCarrollet al.(2022)[35]Trans-formersSynthetic data in a minimalistic grid worldenvironmentTransformers are trained on trajectories assequences of states, actions, and return-to-gotokensMuslehet al.(2022)[8]Trans-formersGPS tracks in San Francisco from the GIS-CUP’17 dataset with 5M recordsTrajectories arediscretized using the H3hexagonal grid (tokenization) followed by thecreation of spatial embeddings(Sub)trajectory classificationChen etal. (2020)[5]CNNShip AIS tracks in Tianjin, Chinacovering23k tripsTrajectories are converted intotrajectoryimageswith pixels coloured according tomovement type (static, maneuvering, normal)Yang etal. (2022)[10]CNNShip AIS tracks in Northern Americaofthe U.S. National Oceanic and AtmosphericAdministration’s Office of Coastal Manage-ment with 259k records after pre-processingTrajectories are converted intotrajectoryimageswith the same method as describedin [5].Wang etal. (2021)[23](C)RNN,LSTM,Trans-formers,AdaNetSHL dataset12: asynchronously sampled ra-dio data of smartphones with up to 2,812hours of labeled data, e.g. GPS reception andlocation, Wifi reception and GSM cell towerscans[SHL challenge summary paper] Mostlyhand-crafted feature engineering as input,but also two teams using raw trajectory dataNext location / final destination predictionGao etal. (2019)[37]GRU &CNNFoursquare check-ins in New York and Singa-pore and Gowalla data in Houston and Cali-fornia with an average of 229k records of 3kusersLocation (<id, lon, lat>) sequences are con-verted intosequences embeddingsusing acausal embedding methodFeng etal. (2018)[38]Attention-GRUFoursquare check-ins13with 294k recordscovering 15k users, other mobile applicationlocation records (search & check-ins) with15mio records covering 5k users, and CDRwith 491k records covering 1k usersLocation sequences are converted intoem-beddingsusing two independent attentionmechanisms which are then fed intoDeep-Move’sGRUs and a historical attention mod-uleLi et al.(2020)[39]SANFoursquare check-ins, Tweets, Yelp andNYC dataLocation sequences are converted intoem-beddingsLiao etal. (2018)[40]RNNFoursquare check-insin NYC and Tokyowith an average of 400k records of 1.7k usersLocation sequences and activity/locationgraphs are converted intoembeddingsforMCARNNLiu et al.(2016)[41]RNNGowalla check-ins8andGlobal TerrorismDatabase (GTD) incidents14Location sequences are converted into time-specific and distance-specific transition ma-trices forST-RNNHong etal. (2022)[7]Trans-formersMobile phone GPS tracks in Switzerland fromthe Green Class (GC) study covering 139 par-ticipants for a year and from the yumuv studycovering 498 participants for 3 monthsLocation sequences are generated from GPStracks by first filtering stay locations with astay duration >25min and then spatially ag-gregating stays into locations. These locationsequences are converted into location, time,day, and mode embeddings which are fedinto the transformersTenzer etal. (2022)[20]Hypernetwork(LSTMs)Porto taxi tracks5covering 1.7mio trips by442 taxis in Porto, Portugal for 12 monthsSequences of trajectory points are convertedtosequences of spatial embeddingsvia ageospatial encoding mechanism
7. 

   Table 3Trajectory datasets and data engineering; public datasets are printed in bold – continuedRefNN de-signTrajectory data Data engineeringAnomaly detectionLiat-sikou etal. (2021)[25]LSTM-based AEPorto taxi tracks5Trajectories are down-sampled to 60s and rep-resented as asequence of vectors(lat, lon)and clipped to the first nine points to fit theautoencoder requirementsNguyenet al.(2022)[42]Proba-bilisticRNNShip AIS tracks from a single receiver inUshant, FranceTrajectories are down-sampled to 600s andconverted to afour hot vectors(lat, lon,SOG, COG) with the resolutions of 0.01° forlat/lon, 1 knot for SOG, and 5° for COG.Singh etal. (2022)[43]RNN re-gressionmodelShip AIS tracks in the Baltic sea region andnear Bremerhaven, Germany for two months(including a comparison between satellite-based and coastal AIS)Trajectories are resampled and interpolatedat the 60s and converted into agraphwithnodes representing turning points for the ves-sel trajectories and edges representing thesea lanes traveled by vesselsSynthetic data generationRao etal. (2020)[44]LSTM-GANFoursquare check-ins15in NYC covering193 users with 3k trajectories and 67k recordsTrajectories are processed by a trajectory en-coding model covering trajectory point en-codings (location, temporal and categoricalattributes) and trajectory padding (to ensurethat all trajectories have the same length)Zhang etal. (2022)[22]VAE-likedeep gen-erativemodelsPorto taxi tracks5;T-Drive6data consistingof 10.3k taxis for one week; andGowallacheck-ins8A coordinate encoding MLP converts two-dimensional points into a high-dimensionalrepresentation. Then, a Bidirectional LSTMis used to encode all representations withforward and backward information for a timestepSimini etal. (2021)[45]MLPEngland & Italy commuting flows,NY Stateflows16including origin & destination geo-graphic unit and estimated population flowsbetween two geographic unitsInput to the model is of the origin & destina-tion location as well as the distance betweenorigin and destination. The output of themodel is the probability to observe a trip be-tween two locations.sequence embeddings using a causal embedding method(similar to a high-order Markov Process). The result-ing embeddings are the input for their GRU to learnthe trajectory patterns. They further apply attentionto the embeddings for predicting the user’s next POI.Feng et al. [38] tailor two attention mechanisms to gen-erate independent latent vectors from large and sparsetrajectories. These embeddings are then fed into theirDeepMoveGRUs and a historical attention module. Thelearned attention weights can intuitively explain theprediction based on the user’s history of movement be-havior. Li et al. [39] introduce a spatio-temporal self-attention network (STSAN). They generate trajectoryembeddings by concatenating the temporal (activity se-quence), spatial (distance matrix of locations), and loca-tion attentions (location sequence and their categories).They feed these embeddings through a softmax layer andpredict the user’s next POI. They use a federated learn-ing setting to tackle the heterogeneity problem. Liao etal. [40] generate embeddings from location sequencesas well as graph embeddings from location-location andactivity-location graphs and train theirMCARNNmulti-task context-aware recurrent neural network to solveboth activity and location prediction tasks.Other works use neural networks for dimensionalityreduction and for creating embeddings. Liu et al. [41] in-corporate time and distance-specific transition matricesas temporal and spatial embeddings generated by RNNs.Hong et al. [7] reduce the dimensions of trajectories usinga multilayered embedding approach for transformers topredict next location and travel mode. Tenzer et al. [20]generate two geospatial and temporal embeddings by1.combining the random picking and the nearest neigh-bor to create sequences of spatial embeddings and2.us-ing a sinusoidal embedding to convert the timesteps totemporal vectors. They train a hyper network to learnto change its weights in response to these embeddings.
8. 

   2.7. Anomaly detectionThis use case category covers anomalous trajectory de-tection. Since the definition of anomalies is often context-dependent, ground truth labeled data is rare. Therefore,anomaly detection approaches often resort to trying toidentify trajectories that are different compared to previ-ously observed trajectories based on some spatial, spa-tiotemporal, or other metrics. Alternatively, researchersresort to using synthetically generated anomalies [25].Liatsikou et al. [25] developed an LSTM-based networkfor the automatic detection of movement anomalies, suchas the detection of synthetic anomalies in taxi trajectories.Since the autoencoder requires inputs of a certain fixedlength, all trajectories are clipped to nine points (andshorter ones discarded).The GeoTrackNet [42] is a model for maritime trajec-tory anomaly detection, which consists of a probabilisticRNN-based (Recurrent Neural Network) representationof AIS tracks and a contrario detection [47]. Detectedanomalies were evaluated by AIS experts.Singh et al. [43] present an anomaly detection systembased on RNN regression models to detect anomaloustrajectories, on-off switching, and unusual turns. Again,a quantitative accuracy analysis is not feasible due to thelack of ground truth data.2.8. Synthetic data generationThis category covers the generation of synthetic move-ment data, such as synthetic trajectories [22,44] andsynthetic flows [45].Rao et al. [44] focus on GeoAI-trajectory privacy pro-tection. For this, they develop an end-to-end deep LSTM-TrajGAN model to generate privacy-preserving synthetictrajectory data for data sharing and publication.Simini et al. [45] developed an MLP model (denotedDeep Gravity) to generate mobility flow probabilities.They evaluated Deep Gravity on mobility flows in Eng-land, Italy, and New York State and achieved a goodperformance even for regions with no data available fortraining.Zhang et al. [22] propose an end-to-end trajectory gen-eration model for generating synthetic trajectories. Thedesign of the model is VAE-like encoders (e.g., Global-semantics encoder: MLPs & Bidirectional LSTM) anddecoders (e.g., a prior generator based on variational re-current structure generates noise at time������by consideringthe noise at the previous time step).3. Conclusion and outlookIn this work, we reviewed deep learning-based researchfocussing on mobility data. In most cases, even if trajec-tory data is used in the process, it is not ingested directlyfor training the neural networks. Instead, data engineer-ing steps are applied that convert trajectories into morecompact representations of individual trajectories (sparsetrajectories) or aggregations of multiple trajectories. Thisaggregated trajectory data is commonly presented as timeseries of vectors, graphs, or images (movies).On the deep learning side, we expect the popularity ofGNNs to increase. For example, the Traffic4cast challenge(in its 4th year, 2022) is moving from (image/video-based)traffic forecasting to graph-based representations. Addi-tionally, AutoML methods (e.g., AdaNet used by the SHLChallenge winner [23]) will allow users with limited DLexpertise to build competitive DL models.As far as data engineering and development is con-cerned, we expect further uptake of trajectory analysislibraries, such asTrackintel17(e.g., used by [7]),Moving-Pandas18(e.g., used by [36]) andscikit-mobility19(e.g.,used by [45]) since these libraries implement many com-mon trajectory generalization, aggregation, and analysismethods and aim at a long(er) term availability. This is animportant next step, as the implementations summarizedin Figure3have not been substantially updated/main-tained since being published. This does not only reducethe likelihood of reuse but also will lead to security issuesdown the road.Future research should address the issues of modeltransferability, benchmark availability, and model ex-plainability. Current work rarely addresses the issueof model transferability. Since most existing global MLmodels “cannot perform well locally, or be transferred tostudy similar problems in other regions”[48], transferabil-ity should be considered when evaluating or comparingmodels. Additionally, developed models, even for thesame application and trajectory type, are difficult to eval-uate (e.g., due to the lack of ground truth for anomaly de-tection) and to compare due to different datasets and ap-plied metrics. Therefore, more open datasets are needed.Finally, to better understand the why and how of usingneural networks for a specific application, explainabilityshould play a more crucial role in model development.References[1]G. Andrienko, N. Andrienko, P. Bak, D. Keim,S. Kisilevich, S. Wrobel, A conceptual frame-work and taxonomy of techniques for an-alyzing movement, Journal of Visual Lan-guages & Computing 22 (2011) 213–232.URL:https://www.sciencedirect.com/science/article/pii/S1045926X11000139. doi:https://doi.org/10.1016/j.jvlc.2011.02.003.17https://github.com/mie-lab/trackintel18http://movingpandas.org19https://scikit-mobility.github.io/scikit-mobility/
9. 

   Figure 3:Neural network implementations published together with their reviewed papers, ordered by ML library and stars.[2]M. P. Konzack, Trajectory analysis: Bridging algo-rithms and visualization, Ph.D. thesis, TechnischeUniversiteit Eindhoven, 2018.[3]Y. Zheng, Trajectory Data Mining: An Overview,ACM Transactions on Intelligent Systems andTechnology 6 (2015) 1–41. URL:http://dl.acm.org/citation.cfm?doid=2764959.2743025. doi:10.1145/2743025.[4]D. Altan, M. Etemad, D. Marijan, T. Kholodna, Dis-covering Gateway Ports in Maritime Using Tempo-ral Graph Neural Network Port Classification, 2022.URL:http://arxiv.org/abs/2204.11855.[5]X. Chen, Y. Liu, K. Achuthan, X. Zhang, Aship movement classification based on AutomaticIdentification System (AIS) data using Convo-lutional Neural Network, Ocean Engineering218 (2020) 108182. URL:https://linkinghub.elsevier.com/retrieve/pii/S0029801820311124. doi:10.1016/j.oceaneng.2020.108182.[6]Z. Fan, X. Yang, W. Yuan, R. Jiang, Q. Chen, X. Song,R. Shibasaki, Online trajectory prediction formetropolitan scale mobility digital twin, in: Proc. ofthe 30th Intern. Conf. on Advances in GeographicInformation Systems, ACM, Seattle Washington,2022, pp. 1–12. URL:https://dl.acm.org/doi/10.1145/3557915.3561040. doi:10.1145/3557915.3561040.[7]Y. Hong, H. Martin, M. Raubal, How do you gowhere?: improving next location prediction bylearning travel mode information using transform-ers, in: Proc. of the 30th Intern. Conf. on Ad-vances in Geographic Information Systems, ACM,Seattle Washington, 2022, pp. 1–10. URL:https://dl.acm.org/doi/10.1145/3557915.3560996. doi:10.1145/3557915.3560996.[8]M. Musleh, Towards a unified deep model for tra-jectory analysis, in: Proc. of the 30th Intern. Conf.on Advances in Geographic Information Systems,ACM, Seattle Washington, 2022, pp. 1–2. URL:https:
10. 

    //dl.acm.org/doi/10.1145/3557915.3565529. doi:10.1145/3557915.3565529.[9]C. Yang, G. Gidofavi, Classification of regionaldominant movement patterns in trajectories with aconvolutional neural network, in: Martin Raubal,Shaowen Wang, Mengyu Guo, David Jonietz, PeterKiefer (Eds.), Spatial Big Data and Machine Learn-ing in GIScience, Melbourne, Australia, 2018.[10]T. Yang, X. Wang, Z. Liu, Ship Type RecognitionBased on Ship Navigating Trajectory and Convo-lutional Neural Network, Journal of Marine Sci-ence and Engineering 10 (2022) 84. URL:https://www.mdpi.com/2077-1312/10/1/84. doi:10.3390/jmse10010084.[11]G. Buroni, G. Bontempi, K. Determe, A tutorialon network-wide multi-horizon traffic forecast-ing with deep learning., in: Workshop Proc. ofthe EDBT/ICDT 2021 Joint Conference, Nicosia,Cyprus, 2021. URL:https://ceur-ws.org/Vol-2841/BMDA_6.pdf.[12]C. Wang, Y. Liang, G. Tan, Periodic residual learn-ing for crowd flow forecasting, in: Proc. of the 30thIntern. Conf. on Advances in Geographic Informa-tion Systems, ACM, Seattle Washington, 2022, pp.1–10. URL:https://dl.acm.org/doi/10.1145/3557915.3560947. doi:10.1145/3557915.3560947.[13]Y. Zhang, Y. Li, X. Zhou, X. Kong, J. Luo, Curb-GAN:Conditional Urban Traffic Estimation throughSpatio-Temporal Generative Adversarial Networks,in: Proc. of the 26th ACM SIGKDD Intern. Conf. onKnowledge Discovery & Data Mining, ACM, Vir-tual Event CA USA, 2020, pp. 842–852. URL:https://dl.acm.org/doi/10.1145/3394486.3403127. doi:10.1145/3394486.3403127.[14]G. Mai, K. Janowicz, Y. Hu, S. Gao, B. Yan, R. Zhu,L. Cai, N. Lao, A review of location encoding forGeoAI: methods and applications, Intern. Jour-nal of Geographical Information Science 36 (2022)639–673. URL:https://www.tandfonline.com/doi/full/10.1080/13658816.2021.2004602. doi:10.1080/13658816.2021.2004602.[15]M. Zhu, W. Chen, J. Xia, Y. Ma, Y. Zhang, Y. Luo,Z. Huang, L. Liu, Location2vec: A Situation-AwareRepresentation for Visual Exploration of Urban Lo-cations, IEEE Transactions on Intelligent Trans-portation Systems 20 (2019) 3981–3990. URL:https://ieeexplore.ieee.org/document/8664648/. doi:10.1109/TITS.2019.2901117.[16]B. Yan, K. Janowicz, G. Mai, S. Gao, From ITDLto Place2Vec: Reasoning About Place Type Sim-ilarity and Relatedness by Learning EmbeddingsFrom Augmented Spatial Contexts, in: Proc. ofthe 25th ACM SIGSPATIAL Intern. Conf. on Ad-vances in Geographic Information Systems, ACM,Redondo Beach CA USA, 2017, pp. 1–10. URL:https://dl.acm.org/doi/10.1145/3139958.3140054. doi:10.1145/3139958.3140054.[17]S. Feng, G. Cong, B. An, Y. M. Chee, POI2Vec:geographical latent representation for predictingfuture visitors, in: Proc. of the Thirty-First AAAIConf. on Artificial Intelligence, 2017, pp. 102–108.[18]A. A. Kashyap, S. Raviraj, A. Devarakonda, S. R.Nayak K, S. K V, S. J. Bhat, Traffic flowprediction models – A review of deep learn-ing techniques, Cogent Engineering 9 (2022)2010510. URL:https://www.tandfonline.com/doi/full/10.1080/23311916.2021.2010510. doi:10.1080/23311916.2021.2010510.[19]S. Lyu, T. Han, Y. Nishiyama, K. Sezaki, T. Kusakabe,A plug-in memory network for trip purpose classi-fication, in: Proc. of the 30th Intern. Conf. on Ad-vances in Geographic Information Systems, ACM,Seattle Washington, 2022, pp. 1–12. URL:https://dl.acm.org/doi/10.1145/3557915.3560969. doi:10.1145/3557915.3560969.[20]M. Tenzer, Z. Rasheed, K. Shafique, N. Vasconcelos,Meta-learning over time for destination predictiontasks, in: Proc. of the 30th Intern. Conf. on Ad-vances in Geographic Information Systems, ACM,Seattle Washington, 2022, pp. 1–10. URL:https://dl.acm.org/doi/10.1145/3557915.3560980. doi:10.1145/3557915.3560980.[21]H. Xue, B. P. Voutharoja, F. D. Salim, Leverag-ing language foundation models for human mo-bility forecasting, in: Proc. of the 30th Intern. Conf.on Advances in Geographic Information Systems,ACM, Seattle Washington, 2022, pp. 1–9. URL:https://dl.acm.org/doi/10.1145/3557915.3561026. doi:10.1145/3557915.3561026.[22]L. Zhang, L. Zhao, D. Pfoser, Factorized deep gener-ative models for end-to-end trajectory generationwith spatiotemporal validity constraints, in: Proc.of the 30th Intern. Conf. on Advances in GeographicInformation Systems, ACM, Seattle Washington,2022, pp. 1–12. URL:https://dl.acm.org/doi/10.1145/3557915.3560994. doi:10.1145/3557915.3560994.[23]L. Wang, M. Ciliberto, H. Gjoreski, P. Lago, K. Mu-rao, T. Okita, D. Roggen, Locomotion and Trans-portation Mode Recognition from GPS and Ra-dio Signals: Summary of SHL Challenge 2021,in: Adjunct Proc. of the 2021 ACM Intern. JointConf. on Pervasive and Ubiquitous Computing,ACM, Virtual USA, 2021, pp. 412–422. URL:https://dl.acm.org/doi/10.1145/3460418.3479373. doi:10.1145/3460418.3479373.[24]Y. Lu, Learning to Transfer for Traffic Forecast-ing via Multi-task Learning (2021). URL:https://arxiv.org/abs/2111.15542. doi:10.48550/ARXIV.2111.15542.[25]M. Liatsikou, S. Papadopoulos, L. Apostolidis,
11. 

    I. Kompatsiaris, A Denoising Hybrid Model forAnomaly Detection in Trajectory Sequences., in:Workshop Proc. of the EDBT/ICDT 2021 JointConference, Nicosia, Cyprus, 2021. URL:https://ceur-ws.org/Vol-2841/BMDA_4.pdf.[26]A. Tritsarolis, E. Chondrodima, P. Tampakis,A. Pikrakis, Online Co-movement Pattern Pre-diction in Mobility Data, in: Workshop Proc. ofthe EDBT/ICDT 2021 Joint Conference, Nicosia,Cyprus, 2021. URL:https://ceur-ws.org/Vol-2841/BMDA_15.pdf.[27]L. Grinsztajn, E. Oyallon, G. Varoquaux, Why dotree-based models still outperform deep learningon tabular data?, 2022. URL:http://arxiv.org/abs/2207.08815.[28]Y. Gao, Y. Chiang, X. Zhang, M. Zhang, Trafficvolume prediction for scenic spots based onmulti‐source and heterogeneous data, Trans-actions in GIS 26 (2022) 2415–2439. URL:https://onlinelibrary.wiley.com/doi/10.1111/tgis.12975.doi:10.1111/tgis.12975.[29]J. Rao, S. Gao, M. Li, Q. Huang, A privacy‐pre-serving framework for location recommendationusing decentralized collaborative machine learn-ing, Transactions in GIS 25 (2021) 1153–1175.URL:https://onlinelibrary.wiley.com/doi/10.1111/tgis.12769. doi:10.1111/tgis.12769.[30]B. J. Buijse, V. Reshadat, O. W. Enzing, ADeep Learning-Based Approach for Train ArrivalTime Prediction, in: H. Y. et al. (Ed.), Intel-ligent Data Engineering and Automated Learn-ing – IDEAL 2021, volume 13113, Springer,Cham, 2021, pp. 213–222. URL:https://link.springer.com/10.1007/978-3-030-91608-4_22. doi:10.1007/978- 3- 030- 91608- 4_22, series Title: LectureNotes in Computer Science.[31]A. Derrow-Pinion, J. She, D. Wong et al., ETA Pre-diction with Graph Neural Networks in GoogleMaps, in: Proc. of the 30th ACM Intern. Conf.on Information & Knowledge Management, ACM,Virtual Event Queensland Australia, 2021, pp.3767–3776. URL:https://dl.acm.org/doi/10.1145/3459637.3481916. doi:10.1145/3459637.3481916.[32]D. Wang, J. Zhang, W. Cao, J. Li, Y. Zheng, WhenWill You Arrive? Estimating Travel Time Basedon Deep Neural Networks, Proc. of the AAAIConference on Artificial Intelligence 32 (2018).URL:https://ojs.aaai.org/index.php/AAAI/article/view/11877. doi:10.1609/aaai.v32i1.11877.[33]M. Li, S. Gao, F. Lu, K. Liu, H. Zhang, W. Tu, Pre-diction of human activity intensity using the in-teractions in physical and social spaces throughgraph convolutional networks, Intern. Journalof Geographical Information Science 35 (2021)2489–2516. URL:https://www.tandfonline.com/doi/full/10.1080/13658816.2021.1912347. doi:10.1080/13658816.2021.1912347.[34]F. Lippert, B. Kranstauber, P. D. Forré, E. E.van Loon, Learning to predict spatiotempo-ral movement dynamics from weather radar net-works, Methods in Ecology and Evolution (2022)2041–210X.14007. URL:https://onlinelibrary.wiley.com/doi/10.1111/2041-210X.14007. doi:10.1111/2041- 210X.14007.[35]M. Carroll, J. Lin, O. Paradise, R. Georgescu, M. Sun,D. Bignell, S. Milani, K. Hofmann, M. Hausknecht,A. Dragan, S. Devlin, Towards Flexible Inferencein Sequential Decision Problems via BidirectionalTransformers, 2022. URL:http://arxiv.org/abs/2204.13326.[36]S. Mehri, A. A. Alesheikh, A. Basiri, A Contex-tual Hybrid Model for Vessel Movement Predic-tion, IEEE Access 9 (2021) 45600–45613. URL:https://ieeexplore.ieee.org/document/9380635/. doi:10.1109/ACCESS.2021.3066463.[37]Q. Gao, F. Zhou, G. Trajcevski, K. Zhang, T. Zhong,F. Zhang, Predicting Human Mobility via Varia-tional Attention, in: The World Wide Web Conf.on - WWW ’19, ACM Press, San Francisco, CA,USA, 2019, pp. 2750–2756. URL:http://dl.acm.org/citation.cfm?doid=3308558.3313610. doi:10.1145/3308558.3313610.[38]J. Feng, Y. Li, C. Zhang, F. Sun, F. Meng, A. Guo,D. Jin, DeepMove: Predicting Human Mobil-ity with Attentional Recurrent Networks, in:Proc. of the 2018 World Wide Web Conf. onWorld Wide Web - WWW ’18, ACM Press, Lyon,France, 2018, pp. 1459–1468. URL:http://dl.acm.org/citation.cfm?doid=3178876.3186058. doi:10.1145/3178876.3186058.[39]A. Li, S. Wang, W. Li, S. Liu, S. Zhang, PredictingHuman Mobility with Federated Learning, in: Proc.of the 28th Intern. Conf. on Advances in GeographicInformation Systems, ACM, Seattle WA USA, 2020,pp. 441–444. URL:https://dl.acm.org/doi/10.1145/3397536.3422270. doi:10.1145/3397536.3422270.[40]D. Liao, W. Liu, Y. Zhong, J. Li, G. Wang, Predictingactivity and location with multi-task context awarerecurrent neural network, in: Proc. of the 27thIntern. Joint Conf. on Artificial Intelligence, 2018,pp. 3435–3441.[41]Q. Liu, S. Wu, L. Wang, T. Tan, Predicting thenext location: A recurrent model with spatial andtemporal contexts, in: Thirtieth AAAI conferenceon artificial intelligence, 2016.[42]D. Nguyen, R. Vadaine, G. Hajduch, R. Garello, R. Fa-blet,GeoTrackNet—A Maritime Anomaly DetectorUsing Probabilistic Neural Network Representationof AIS Tracks andA ContrarioDetection, IEEETransactions on Intelligent Transportation Systems
12. 

    23 (2022) 5655–5667. URL:https://ieeexplore.ieee.org/document/9353410/. doi:10.1109/TITS.2021.3055614.[43]S. K. Singh, J. S. Fowdur, J. Gawlikowski, D. Medina,Leveraging Graph and Deep Learning Uncertaintiesto Detect Anomalous Maritime Trajectories, IEEETransactions on Intelligent Transportation Systems23 (2022) 23488–23502. URL:https://ieeexplore.ieee.org/document/9839418/. doi:10.1109/TITS.2022.3190834.[44]J. Rao, S. Gao, Y. Kang, Q. Huang, LSTM-TrajGAN:A Deep Learning Approach to Trajectory PrivacyProtection, in: 11th Intern. Conf. on GeographicInformation Science (GIScience 2021)-Part I, Poz-nan/online, 2020, pp. 12:1–12:17. URL:http://arxiv.org/abs/2006.10521.[45]F. Simini, G. Barlacchi, M. Luca, L. Pappalardo, ADeep Gravity model for mobility flows generation,Nature Communications 12 (2021) 6576. URL:https://www.nature.com/articles/s41467-021-26752-4.doi:10.1038/s41467- 021- 26752- 4.[46]C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri,S. Yang, AdaNet: Adaptive Structural Learning ofArtificial Neural Networks, in: D. Precup, Y. Teh(Eds.), Proc. of the 34th Intern. Conf. on MachineLearning, volume 70 ofProc. of Machine LearningResearch, PMLR, 2017, pp. 874–883. URL:https://proceedings.mlr.press/v70/cortes17a.html.[47]A. Desolneux, L. Moisan, J.-M. Morel, From gestalttheory to image analysis: a probabilistic approach,volume 34, Springer Science & Business Media,2007.[48]K. Janowicz, R. Zhu, J. Verstegen, G. McKenzie,B. Martins, L. Cai, Six GIScience Ideas ThatMust Die, AGILE: GIScience Series 3 (2022) 1–8.URL:https://agile-giss.copernicus.org/articles/3/7/2022/. doi:10.5194/agile- giss- 3- 7- 2022.AcknowledgmentsThis work is mainly funded by the EU’s Horizon Eu-rope research and innovation program under Grant No.101070279 MobiSpaces and No. 101021797 STARLIGHT.Glossary•AE – Autoencoder•AIS – Automatic Identification System•CDR – Call Detail Records•CNN – Convolutional Neural Network•COG – Course over ground•DNN – Deep Neural Network•DL – Deep Learning•GAN – Generative Adversarial Network•GeoAI – Geospatial Artificial Intelligence•GIS – Geographic Information Science•GNN – Graph Neural Network•GPS – Global Positioning System, often used synonymouslyfor all GNSS (incl. Galileo, GLONASS, and Beidou)•LSTM – Long Short-Term Memory•MLP – Multilayer Perceptron•NLP – Natural Language Processing•OBU – On-board Unit•OD – Origin-Destination•POI – Point of Interest•RNN – Recurrent Neural Network•SAE – Stacked Autoencoder•SAN – Self-Attention Network•SOG – Speed over ground•VAE – Variational AutoEncoder