Graphs are successful means to present data, in both tasks that of analyzing and that of communicating data. In addition to text-graph documents, in many professional communication and instruction settings, graphs, language, and often gestures accompany each other forming multimodal communication. In the present study, we investigate referring expressions used in a collaborative joint activity (Clark, 1996), namely in acquiring knowledge from graphs and language. Our long term goal is to realize a cooperative system that provides blind graph readers with verbal assistance.