data sparsity / data-starved

there is so much data in the world, and so much of it is useless to me.

there are only so many studies i can imagine in the still of the night. we need more annotated data or an improvement in unsupervised learning techniques. in the meantime, there are simplistic nlp techniques out there that include/provide visualizations to help us discover and measure linguistic phenomena.

see http://www.sketchengine.co.uk/ for interesting views into various corpora. For an example of a tool that allows you to expore past state of the union addresses, see this nytimes link. i particularly like the visualization. dang, i need to tinker with flash amd ajax more, so i can provide similar views of situation entities, dialogue acts, discourse relations, named entities, coreference chains, etc in documents. what the heck, these sort of tools might bring more attention to the applications of computational linguistics in the hands of everyday people. that could really push the field forward in ways that researchers could really benefit, financially and intellectually.

in my recent reading, there are so many evaluation metrics for unsupervised learning, some of which provide pleasant results, and others of which that do not. it seems researchers like to choose their evaluation measures to optimize their results. there need to be better eval metrics. there simply must be.