I read much these days about aerial video. Taken from drones, small aircraft or satellites. And used for surveillance, tracking wildfires, monitoring deforestation; to name but a few. So let me ask a question. Suppose I am monitoring a wildfire. I fly the area three times a day, flying the same route, and recording both video and the GPS location of the aircraft. I now want to analyse the video, maybe share with others. We can replay each video, pause where needed and share our thoughts on changes in size, direction. Maybe make a few notes. Much of our decision making is based on these video and eye witnesses on the ground. So my question. This approach is tried and tested and I am sure is successful. But can it be improved?
Let me make a suggestion; video annotation. What do i mean by that? Well, maybe we could actually draw on the video itself. Show the details of the fire; the current extent, areas of concern, mark threatened properties etc. Possibly make the process of video annotation collaborative. Maybe done over the Internet. Search functionality could be added. We could search for annotations done at certain locations, or done by individuals. Maybe we are monitoring the changes in the fire at a certain sensitive location. Each time we get a new video, we pause it at that location; mark on the video the new fire extent and current wind direction. At the end of the day we can search and show each of these annotated video frames and note the changes. Not only could this be used as an ongoing decision tool, but can be used later, maybe as part of an operational and decision review or for training.
Build into this tool conferencing abilities, where users can view and annotate the video from different locations over the Internet, and you have a decision support tool. Include GIS in the mix and you have a very powerful decision support tool.
We thought it an interesting challenge. So we pulled the pieces together and started building a demo.
Figure 1 – Fly Over Video Annotation with Interactive Map
Flex Video Annotation – Interactive Maps
We realised early in the process that a map would be a prerequisite. The video would need some point of reference. We turned to OpenScales, the open source GIS Flex tool. In the demo we only included a basic interactive map, with zoom and pan capabilities and streets basemap. But OpenScales can provide GIS functionality. So if we wanted spatial analysis or buffering, measure or other tools, they could be added. High resolution satellite imagery can also be used for the basemap. Figure 2 shows the mapping interface. The path flown by the aircraft or drone is provided by KML. In the demo we have two flight paths marked in blue.
Figure 2 – OpenScales Flex Map
Figure 3 below shows the flight path. There are two markers on the path. The first marks the aircraft or drones position and is synced with the video. The green marker is a camera, this marks the position of an annotated frame. Selecting this brings up the image viewer, which we will discuss in due course.
Figure 3 – Flight Path and Annotated Frame Marker
The right pane of the application window contains the video, annotation tool and image viewer. Two buttons at the top of the pane allow users to switch between the video and the annotation tool. Figure 4 shows the video player. It has pause, play, replay and volume controls.
Figure 4 – Flex Video Player
Flex Video Annotation Tool
When a user selects the ‘Annotation Image Screen’ button the video is paused and a snapshot of the paused video frame is displayed in the tool window. Users can now annotate this frame of the video by holding down left mouse button and dragging across the image. In the bottom left is a colour picker tool. This changes the colour of the line drawn. Two additional buttons allow users to erase all annotations and save the annotated image to the server.
Originally we had planned to annotate the video itself. It struck us that by annotating individual frames as images, this allowed easier comparison between frames annotated at different times. Allowing annotated frames of specific sensitive areas to be compared and more easily found in a search. So this snapshot approach we thought more powerful.
Figure 5 – Fly Over video annotation with interactive map
Flex Image Viewer
In the top right corner of the application header is a search tool. This allows users to search for specific images. Currently this searches by image name; in our demos case if the title contains the word ‘frame’, these images will be loaded in the image viewer. But if this were done by GPS location, for example, then a search could be done for all images annotated at the specific sensitive area.
Figure 6 – Image Search Tool
The image viewer is shown below in Figure 7. It displays images based on an image search. At the moment all annotated videos contain the word ‘frame’. So the image viewer shows all currently annotated images. But, as described above, a powerful search can be implemented which searches against date, location, annotator or any other stored metadata attached to individual images. The image viewer displays thumbnails in the lower screen. Clicking a thumbnail displays this images enlarged in the upper area. We imagine in the future images could be displayed side by side in the viewer to aid comparison.
Figure 7 – Image Viewer
Though done largely as a proof of concept, we think this could be a very powerful decision support tool. We did not add conferencing ability in the demo. But this would allow for distributed collaboration. With the advent of tablet computers like the IPad, this would allow interaction with eyewitnesses on the ground. In addition, the mapping components offers the potential for additional spatial analysis.
Contact us to learn more about this application.