The HistoryMakers
Carnegie Mellon Informedia
Digital Media Library

Produced in July, 2006 • Mike Christel
The Informedia Research Group • Carnegie Mellon University • http://www.informedia.cs.cmu.edu

Index

Introduction: Informedia and The HistoryMakers Starting with IDVL: Text Queries and Preview Browser Tabs in the Interface Map Search Text SearchHow to Take Action from Thumbnails How to Magnify a Shot - Magnify One Shot - Magnify All Shots (using the Shot Thumbnail View) The Tabs (Video Sets) The Map Saving Video Sets For Future Use

The Idea of “Active Video Subset” Advanced Search Techniques to Focus Your Search Overview of Other Views in the View Controller - Shot Thumbnails (all shots in the top stories) - Filtering with Shot Thumbnails - Nested Lists (grouped by interview - Timeline View (plotted against a timeline) - VIBE Plot (plotted against query terms) - Map View (plotted against a map) - Common Text (text lists of common phrases) - Named Entity View (people/place/organization)

Introduction: Informedia and The HistoryMakers

Collaboration between The HistoryMakers and the Informedia research group made use of speech alignment, image processing, and language understanding technologies to promote multiple levels of access and fuel the viewing of the actual video recordings in a large oral history corpus.  Details on Informedia technologies can be researched at http://www.informedia.cs.cmu.edu and more information on The HistoryMakers can be found at http://thehistorymakers.com.

The HistoryMakers is the world’s largest African American oral history archive of thousands of video interviews with accomplished African Americans across a variety of disciplines, including those who have played a role in African American led movements and organizations.  The purpose of The HistoryMakers is to educate and to show the breadth and depth of this important American history as told by the first person, to highlight the accomplishments of individual African Americans across a variety of disciplines, and to preserve this material for years and generations to come. The HistoryMakers is committed to creating and exposing its archival collection to the widest audience possible, making use of new technologies as appropriate.

This document describes the Informedia Digital Video Library (IDVL) search and browsing interface into a digitized, annotated subset of The HistoryMakers interviews.  Specifically, The HistoryMakers archivists provided a transcript for each of the interviews fed into Informedia processing, identifying 18254 interview story segments across 399 interviewees.  Automatic speech alignment time-tagged each transcript word to video time, enabling quick access to the video sections of interest.  The synchronized metadata allows results for a text query like “war protest Nixon” to be presented with the terms color-coded in the matching transcripts and color-coded time markers to be shown on video timelines. The user can click on a marker and seek the video immediately to that point, e.g., skipping over the first three-fourths of the clip shown to the right in order to view the last 30 seconds where “Nixon” gets discussed.>

Details on the searching and browsing mechanisms provided in IDVL based on the synchronized metadata are overviewed in the following pages.  Our goal is for the IDVL interface features discussed here to expand the utility of oral history archives like The HistoryMakers by exploiting their digital nature to deliver synchronized video quickly and provide interactive visualizations dynamically.  These interfaces were designed to enable multiple investigative paths into the archive for witnessing, appreciating, and reflecting on prior historical times and experiences.

When you first launch IDVL you will see a screen similar to this one.  The shot collector area docked to the right may or may not be shown, plus you will see a splash screen with some version information.  Three starting points for exploration are:

• Browse The HistoryMakers text tree down to a HistoryMakers individual you are interested in, select an interview tape (i.e., a movie), and a story segment within that movie, and click the “Video” button to play that video segment.
• Check subject headings, “…Maker” categories, and/or time references you are interested in, and click “Load Selected Segments in New Tab” button, to see details on segments matching your checked criteria.
• Type in a text query (see the Advanced Search link for help), and type the "Enter" key or click the “Search” button to issue the text search, whose results are loaded in a new tab.

Each “tab” holds a video set, i.e., a set of interview segments that are returned from your interactions.  The “All data” tab is always listed first and contains all 18,254 interview segments.  The following screen shows tabs for a text query on “war protest Nixon”, a map search on Oklahoma or Texas, and a browsing action to the subject heading “Aeronautics” with only 6 segments tagged with the Aeronautics subject heading.  To get to this “Aeronautics” tab, in the prior view of “All Data” check “Clear All” in the Preview Browser, then check “Aeronautics” in subject headings, then “Load Selected Segments….”

You can search for a selected list of countries, of U.S. states or Canadian provinces, or a geographic rectangular region using a map interface, discussed later.  Here we overview the view you get in a video set produced by a map search.  The matched map terms are color-coded in the View Controller, with the same color-coding used to indicate the match in the video player, “Shot Thumbnails” view, and in the relevance bar thermometer shown to the left of every thumbnail image in the “Segment Grid” view.

You can search for text in the video transcripts, in the titles, or in the subject headings (notes).  For example, you might search for:

      war protest Nixon

By default, text from all three fields (transcripts, titles, notes) is searched.  The Advanced Search link presents more refined searching methods, to just specific fields, with adjacency, and’ing all terms, instead of the default or’ing, and other text search filtering criteria.  

How to Take Action from Thumbnails

You can toggle the Shot Collector (right pane shown on page 3) off and on with the pull-down menu View-Shot Collector.  When you see a relevant shot, i.e., a segment you are interested in bookmarking or referencing later, you place the mouse over the thumbnail image or video playback area and press the F1 key.
That shot will be copied into the “yes” section of the Shot Collector.

 

While less useful perhaps in an interview corpus than a news or documentary corpus, you can also issue color-based searches from the thumbnail imagery.  When you see a shot from which you wish to launch a color-based image search, place the mouse over the thumbnail imagery or video and press the F3 key.  The results of the color search will be posted as a video set in a new tab (labeled cryptically with “color, ID #”).

Other shortcut actions you can take when mousing over imagery:  F6 key to play the video, and the F12 key to show the segment and movie in the “All data” tree browser view (as shown on Page 3).  Your keyboard may be labeled with these options as a memory aid.  Also, by right-clicking the mouse over thumbnails or the playing video area, you will see the available options, including one specific to The HistoryMakers: you can select “Bio from HistoryMakers web site…” and get up-to-date biographical information on the interviewee.

There are two ways to make a shot appear larger.  You can also play the video in a resizable window.

Magnify One Shot

To magnify a single shot, place the mouse over it, and press the Shift key.

Magnify All Shots (using the Shot Thumbnail View)

To magnify all the shots in a video set, first check the “Shot Thumbnails” view on.  This presents all the thumbnails (along with triangle notches showing where matches occurred – double-clicking notches plays video from that point in the segment).  You can then display thumbnails from 1/8 to full size by clicking on the various toolbar icons on the top of the shot thumbnails view.

The tabs hold video sets, results of your prior search, browse, and filter actions.  Up to the last 7 will be held in memory for your convenience (after that, the oldest is replaced with the newest search).

The Map

You can issue map searches by first viewing the map window, which is accessible through the “Search” menu, “Map…” item.  The map window has a number of toolbar icons along its top row.  The first two are used to initiate a map search selection.  You can either click the rectangle icon, and then rubber-band a selection box on the map by pressing and holding the left mouse button down as you draw a rectangle enclosing the desired search area, or you can click the pointer icon, and then pick up countries (or states, with selection controlled by the Advanced Search – Map options).  Hence, the map interface is set up to allow you to search countries and/or states. 

The other icons, from left to right, are:

• “Binoculars” – used to issue the search (the binoculars icon will be disabled as shown at right until you create a selection through either the rectangle bounding box or by selecting countries with the up-arrow pointer cursor)
• “Eraser” – used to erase map search selection
• “Zoom in”, “Zoom out”, “Pan” – used to manipulate the map display
• “World – used to return to the full world view
• “Synchronize with playing video” – an “advanced” feature used to synchronize the map with a playing video and display text on the map corresponding to locations mentioned in the video.

Two different ways of selecting a Mediterranean area are shown.  The rectangle selection will match the North Africa countries but also matches Greece, Spain, and Gibraltar.  The map picker selection will match just the 5 selected countries (Morocco, Algeria, Tunisia, Libya, and Egypt).

Two ways of saving your work for future sessions are in progress of being better documented and made more accessible in the interface.  Until then, the two ways are dependent on either the Shot Collector, or the commands available in the “File” pull-down menu for the application which are tied to the shortcut keys Ctrl-S and Ctrl-Shift-S.

When you capture shots to the Shot Collector, you see a count of the number of shots at the upper right.  You can right-click the empty area at the top of the Shot Collector and see an option to “Save All Shots…”.  By selecting this option you can save the shots to a file, let’s say “MilitaryCareer.txt”.  You can then later load this video set into a tab by selecting the pull-down menu File-Load Shot ID List and selecting MilitaryCareer.txt as the file to load.  The advantage of this method of bookmarking is that it is easy and fast.  The disadvantage is that it only preserves shots (and for The HistoryMakers interviews, each interview segment holds one shot and so preserving shots hence preserves segments).  Match information is lost, because the Shot Collector allows you to grab shots from many different tabs produced in different ways.

If you wish to preserve the contents of a tab, along with the color-coded match terms and match synchronization, you can do so by selecting either “Save Video Set” (which is tied to the shortcut key chord Ctrl-S) or “Save Active Video Subset” (Ctrl-Shift-S).  Suppose you searched on Prohibition “roaring 20s” and get 43 story segments.  You want to save these with their match information.  By typing Ctrl-S (or selecting the menu), you will get a “Save 43 segments to XML File” dialog box, as shown.  If you name this saved set “SocialScene20s.xml”, you can then later retrieve this set into a tab by using the “File-Load Video Set…” menu item and loading SocialScene20s.xml, which brings back the 43 segments along with their match information.  

The Idea of “Active Video Subset”

You may be overwhelmed with too much information in a tab’s video set when you first issue a text query, map query, color query, or browsing action.  For example, perhaps even the 43 segments for Prohibition “roaring 20s” is too much to look through or save.  You have many ways of filtering down this set of segments to a smaller subset.  The most visible are the sliders at the bottom of the View Controller.  You can click the triangle next to “Score”, “Interview Date” and “Duration” to open the slider and see a histogram of how the segments in the video set are distributed.  You can slide either endpoint of the display, or grab the middle of the histogram plot and slide it as well, to move a window of what should stay active for the video set.  The figure shows the Interview Date slider adjusted to keep only interviews from Sept. 7, 2003 or earlier.  If you then wished to save just the 20 remaining active segments meeting this filter, you could do so with the File-Save Active Video Subset menu item.

Advanced Search Techniques to Focus Your Search

The Advanced Search link, in the Search area, enables you to focus your text search on specific dates, event ranges, categories, or database fields, by using the filters in the “Text” tab of this link.

You can check on and off various filters for your query, which will reduce the number of results returned.  For example, in one version of the corpus, a query on “baseball” returns 427 segments.  If you click “Advanced Search” and then check 1940s as shown below, and then click “Search”, you will get a smaller subset of stories for that text match and event range (e.g., from 427 down to 72 segments).  You could limit “baseball” to just the transcript (interview dialogue and transcriber notes) by checking “Transcript” which would return 403 stories instead of 427.  A check of “ArtMaker” would return 36 segments with ArtMakers discussing baseball.  Finally, a check of just a subject heading like “Rural areas” and all its subtopics (by checking “Auto-Select Subheadings”) would return 2 baseball stories.

A “Map” tab is also included in the “Advanced Search” link showing numerous map search options.  Experiment to see how these options change the precision, recall, and reported match details for your map searches.

For example, if “Just countries” are to be reported, and you select with the pointer four countries in southern Africa to search for interview stories, you will get the following report in the View Controller:

If you wanted detail on matching cities as well as countries, you would select “Countries, States/Provinces or Cities” and repost your map query against the same 4 selected countries by clicking the binoculars icon to “Execute the search” in the map toolbar.  You would then get this report in the View Controller:

Overview of Other Views in the View Controller

The View Controller lists a number of views, with earlier pages of this document only discussing the “Segment Grid” view.  Other views can be checked on to display additional windows into the video data held in the tab, which will be henceforth referred to as the video set.  These views each specialize in highlighting particular attributes of the video set, and many let you filter down to a subset in specialized ways, much like the slider filtering discussed on page 12.  The advantage of the different views is to let you explore the set of video data in varied ways, rather than restrict you to just a list of video segments as shown in the Segment Grid. 

NOTE:  The examples from this point forward are from an international news corpus, rather than The HistoryMakers oral history corpus.  For The HistoryMakers, the Shot Thumbnails view will be less relevant, because most shots are of a single interviewee, while the Timeline view will be more relevant, because the timeline shows times of events being discussed, rather than just the broadcast date for a news story.  Future User Guides may tailor these ending pages for the oral history corpus rather than using news video examples.

Shot Thumbnails (all shots in the top stories)

The Shot Thumbnail view emphasizes “shots” where each shot is a contiguous video sequence from a camera.  Shots typically last a few seconds in news footage, whereas story segments may last a few minutes.  The Shot Thumbnails view can quickly overwhelm you if it would show all the shots for all the segments in the video set were represented, and so two forms of collapse are employed.  First, the Shot Thumbnails only shows the “top”, i.e., “best”, N segments, were N is a “Maximum” set in the “Maximums” tab of the Preferences available through View...Preferences… menu if advanced options are left accessible.  The specific item within Maximums is the “Segments in storyboard” value.  For example, if this value is 200, and the video set holds 243 segments, only the first 200 (the top 200 ordered by the search service relevance score) are represented in the storyboard of Shot Thumbnails.  The second form of collapse is to show only the matching shots and their temporal neighbors in the segment.  By default, only the matching shots and neighbors are shown.  If the toolbar item is unselected by clicking the mouse on it, then all the shots for the top N segments are shown in the Shot Thumbnails view (also referred to as the video set’s storyboard).

Filtering with Shot Thumbnails

Shot filtering can be used in addition to trimming down to just the top segments and matching neighborhoods, for example to keep just the outdoor shots.  By sliding the slider to the left, more and more “possible outdoor” shots can be kept, increasing recall at the expense of precision.  Sliding the “outdoor slider indicator” to the right drops out more and more candidate outdoor shots, increasing precision at the expense of recall.  When you are happy with the overall count (38 remaining as shown in figure), you can click on the “collapse blank space” icon to produce a packed view of the thumbnails.

 

If you are happy with a subset, e.g., say with these 38 outdoor shots shown in their packed view to the right, and want to view them as a video set in its own tab, you can view the subset in one of two ways with the Shot Thumbnails window as the active, focused window: either choose “View..Active Segments in New Tab” from the application pull-down menus, or use the keyboard shortcut “Ctrl+Shift+T” to immediately post these 38 shots from 22 segments into its own video set and tab.

Nested Lists (grouped by interview)

For The HistoryMakers, this view is named “Nested Lists”, for news, it is the Explorer View.  This view is a quick way to see a distribution of the video set across different interviews (or for news, broadcasters and broadcasts).  It uses the traditional Microsoft Windows tree view, organizing the video set into libraries, collections, movies, and segments.  For example, for the same video set of 127 segments returned by a geographic query against Afghanistan-Pakistan-Uzbekistan-Tajikistan shown earlier, clicking the Explorer View open shows the following view.  Clicking open movies like CNN Sunday for February 6, 2005 and Mandarin News China for January 4, 2005 shows segments, with the Jan. 4 China segment clicked on to highlight it in blue and show additional information in the bottom segment information panel.  For now, this panel, and the “Annotation…” button too, are used primarily for debugging.  Of greater interest is the information shown in the tree.  For each library, collection, and movie, there is a count in parentheses of the number of segments in the video set, so that of the 127 segments, 54 come from CNN Lou Dobbs Moneyline, 20 from Aaron Brown, 31 from MCC News, etc.  As the mouse moves over the segments in the view, the segment is highlighted yellow in this view and also brushed in other views so that you can see where the segment titled “Time correspondent … and pakistani…” falls in other views like the timeline or Shot Thumbnails.  Likewise, brushing segments in other views will light up the segment in a yellow background highlight if it is currently shown in this Explorer tree view.  Finally, the count after the “#” is the order of the segment in the video set, where #1 is the top-scoring segment with “score” given by the search service.  The yellow-highlighted item is the fourth-highest scoring item in the video set.

 

Timeline View (plotted against a timeline)

The Timeline view emphasizes time, with the vertical axis by default score but also supporting other attributes like Duration if selected from the combo box shown in the upper left of the view.  Each green box (plot-box) represents at least one but possibly many segments.  If you right-click on a plot-box that represents a single segment, you can play its video, show its storyboard, or show its movie info, just like you can do with thumbnail representations.  If the plot-box represents more than one video, you can post the set of segments to a new tab with the “Show set in new tab…” menu item.  You can drill down in the timeline by rubber-band drawing a selection rectangle in the plot area, or by clicking on an x-axis button, like clicking on “February” to zoom into the view shown further below.

The plot-box items in the timeline can also be color-coded by checking the “Color-code” box for a slider in the View Controller, e.g., here “Duration” is checked to color longer stories red, shorter ones blue.

VIBE Plot (plotted against query terms)

VIBE stands for “Visualization By Example” and is a presentation technique first developed at the University of Pittsburgh School of Library and Information Science.  It uses the query terms (words for text queries, matching cities/states/countries for geographic searches) as anchors for the plot, and then distributes the video segments on the plot based on their relative score from each of the anchors.  For example, using the running example with 127 result segments produces this VIBE Plot:

Mousing over the anchor points displays tooltip text indicating how many segments match each anchor.  For example, there are 2 segments that match “Uzbekistan”:  one only matches that location, with the other segment matching about equally “Uzbekistan” and “Pakistan” and hence being plotted halfway between these two anchors.  A look at this plot shows most segments match either Afghanistan, Pakistan, or just these two countries and hence are plotted on the line between the Afghanistan and Pakistan anchors. Unchecking an anchor from the list removes it from the VIBE plot. 

As with the timeline, you can brush over the plot-box points and get tooltip titles for the segments, right click on them to get a menu of actions, and can draw a rubber band zoom box on the plot area.  If you wanted only the subset of segments matching both Afghanistan and Pakistan, you could rubber-band that portion of the VIBE plot as shown at right and get an “active” VIBE subset of 5 segments as listed in the VIBE window caption.  You could post these 5 segments into their own tab by either selecting “View..Active Segments in New Tab” from the application pull-down menus, or use the keyboard shortcut “Ctrl+Shift+T” to immediately post these 5 segments into its own video set and tab. 

Map View (plotted against a map)

The Map View emphasizes geographic distribution of the locations mentioned by the segments in the video set.  If you consider the 73 shots returned from an Arabic query concerning the Iraq elections from an early 2005 news corpus, “العراق   إنتخابات”, and then open up the Map View, you will see the distribution as shown below.  Gray countries are not mentioned, and by default the mentioned countries are colored green, just like the default plot color in VIBE and timeline plots is green.  If a View Controller slider “Color-code” is checked, you can see that slider’s relationship to plotted countries, e.g., the highest scoring segments include mention of Iraq, France, Russia, the U.S., etc. (colored red), and lowest scoring segments are the only ones that mention Argentina, Angola, and Algeria (colored blue).

The map view also lists the matching countries, here sorted alphabetically by Location Name but you can click on Matched Segments to sort by increasing or decreasing match count as well.  A prototype “filter” funnel toolbox icon is provided, initially disabled until you rubber-band box a “filter” area with the box toolbar icon.  The next figure shows the same Map View for the 73 segments, after you zoom into the Middle East region with the zoom toolbar and mouse clicks.  Then a using the box-filter icon, a rectangle was drawn intersecting Egypt, Libya, and Sudan.  This is shown in the next figure.

Upon clicking the filter, you would be left with the following display, indicating that there are now 31 active segments remaining, those segments mentioning one or more of Libya, Sudan, and Egypt (the filter area).  By typing the key chord Ctrl-Shift-T, or selecting View..Active Segments in New Tab from the application pull-down menu, you can post these 31 segments into their own video set presented in their own tab.

Common Text (text lists of common phrases)

The Common Text view is often used in conjunction with another view to show how a common phrase or theme might be distributed differently in time or place or visuals.  For example, consider the 562 segments returned from a text query on “economic impact of tsunami devastation.”  The Timeline view and Common Text view together allow you to mouse over a phrase and see it get brushed in the other view, so that for example you can see that the segments mentioning “sri lanka” (of which there are 61 from the video set of 562) are from mostly early January 2005 with a second pocket in mid-February 2005 (the plot points brush yellow and sri lanka is colored yellow to indicate a “brush” association, which is quickly conveyed and changes with mouse movement to a different phrase).

The Common Text view lets you order the phrases by “commonness” (most common first, i.e., appearing in the most segments in the video set) or alphabetically by phrase.  You can trim membership in the list by setting a minimum number of matching segments needed before a phrase is listed, set here at 23.  You can also set the number of words per phrase: usually a video set matches fewer longer phrases, but the ideal phrase length for browsing and filtering depends on task and corpus.  You can Ctrl-Click to select multiple items in the list (denoted with a triangle next to selected items), with the caption indicating how the active subset changes with selection.  Then, as with other views, you can use Ctrl-Shift-T or View..Active Segments in New Tab from the pull-down menu to post the active subset to a new tab.  In this example, the active subset would be the 64 segments from the original set of 562 that mention either or both of “illegal aliens” and “trade deficit.”

Named Entity View (people/place/organization)

The Named Entity View is the latest view created for IDVL.  While still in prototype form, it offers two unique presentation styles: 

(1) boxes-and-lines views of named entity relationships (entities as box shapes, connected with a line if they have a temporal relationship to each other through news broadcasts (i.e., are mentioned in the same news stories); and

(2) contextual skims: the video snippets for two related named entities can be played together, ordered by broadcast date, for a playable overview of the relationship. 

The boxes-and-lines named entity connections is illustrated below.  Contextual skims are available by right-clicking on a box and selecting one of the connections.  The yellow centroid node, here shown as Iran, can be changed by selecting from the list of entities in the upper right.  By default the first tier of connections will be plotted, for all of People, Place, and Organization, for the most common 15 co-references.  If the “Second Tier” option is checked, and if there is space available with the given “Nodes” ceiling count, then additional nodes will be displayed in white rather than blue showing connections between those entities shown in blue but not connected to the centroid node. 

A zoomed in view is shown below, with Germany right-clicked with the mouse to display the menu. 

The graph can become complicated quickly, with your option to either zoom into local areas of interest or reduce the scope by lowering the node count or only selecting a subset of People (shown as rectangles), Place (ovals), and Organization (hexagons).  If “Iran   6” is selected, the snippets from six video segments connecting “Iran” and “Germany” will be presented in a contextual skim, as shown at the right.  The six snippets are alternately striped in the video play bar to give an indication of relative length of each snippet, and ordered by time, with an overall length of 2 minutes 46 seconds.