Data scales in applied statistics – are nominal data poor in information?

An earlier blog post by Jennifer Edmond on „Voluptuousness: The Fourth „V“ of Big Data?” focused on cultural data steeped in meaning; proverbs, haikus or poetic language are amongst the best examples for this kind of data.

But computers are not (yet) good in understanding human languages. Nor is statistics – it simply conceives of words as nominal variables. Quite in contrast to an understanding of cultural data as described by Jennifer Edmond, applied statistics regards nominal variables as the ones with the LEAST density in information. This becomes obvious when these data are classified amongst other variables in scales of measurement. The scale (or level) of measurement refers to assigning numbers to a characteristic according to a defined rule. The particular scale of measurement a researcher uses will determine what type of statistical procedures (or algorithms) are appropriate. It is important to understand the nominal, ordinal, interval, and ratio scales; for a short introduction into the terminology, follow this link. Seen in this context, nominal variables belong to qualitative data and are classified into distinctively different categories; in comparison to ordinal variables, these categories cannot be quantified or even ranked. The functions of the different scales are shown in the following graph:



Here it becomes visible that words are classified as nominal variables; they belong to some distinctively different categories, but those categories cannot be quantified or even be ranked; there is no meaningful order in choice.

This has the consequence that in order to be able to compute with words, numeric values are being attributed to them. E.g. in Sentiment Analysis, the word “good” can receive the value +1, while the word “bad” will receive a -1. Now they have become computable; words are thus transformed into values, and it is exactly this process which reduces their “voluptuousness” and robs them of their polysemy; just binary values remain.

To my knowledge, the most significant endeavor to provide for a more complex measurement of linguistic data has been undertaken in the field of psychology: In their book “The Measurement of Meaning”, Charles E. Osgood, George J. Suci, and Percy H. Tannenbaum developed the “semantic differential” as a scale used for measuring the meaning of things and concepts in a multitude of dimensions (see for a contemporary approach f.ex. But imagine each word of a single language measured – with all its connotations and denotations, each in its different functions, in the context of the other words around it … not to speak of figurative language such as metaphors, irony, and sarcasm.

This is why words – and cultural, voluptuous data in a broader sense – are so difficult to compute; and why they are the next big computational challenge.


“Voluptuousness:” The Fourth “V” of Big Data?

Dr Jennifer Edmond, TCD

The KPLEX project is founded upon a recognition that definitions of the word ‘data’ tend to vary according to the perspective of the person using the word.  It is therefore useful, at least, to have a standard definition of ‘big’ data.

Big Data is commonly understood as meeting 3 criteria, each conveniently able to be described with a word beginning with the letter V: Volume, Velocity and Variety.

Volume is perhaps the easiest to understand, but a lot of data really means a lot.  Facebook, for example, stores more than 250 billion image .  ‘Nuf said.

Velocity is how fast the data grows.  That >250 billion images figure is estimated to be growing by a further 300-900 million per day (depending on what source you look at).  Yeah.

Variety refers to the different formats, structures etc. you have in any data set.

Now, from a humanities data point of view, these vectors are interesting.  Very few humanities data sets would be recognised as meeting criteria 1 or 2, though some (like the Shoah Foundation Video History Archive) come close.  But the comparatively low number of high volume digital cultural datasets is related to the question of velocity: the fact that so many of these information sources have existed for hundreds of years or longer in analogue formats means that recasting them as digital is a highly expensive process, and born digital data is only just proving its value for the humanist researcher.

But Variety?  Now you are talking.  If nothing else, we do have huge heterogeneity in the data, even before we consider the analogue as well as the digital forms.

Cultural data makes us consider another vector as well, however: if it must start with V, I will call it “voluptuousness.”  Cultural data can be steeped in meaning, referring to a massive further body of cultural information stored outside of the dataset itself.  This interconnectedness means that some data can be exceptionally dense, or exceptionally rich, without being large.  Think “to be or not to be;” think the Mona Lisa; think of a Bashō haiku. These are the ultimate big data, which, while tiny in terms of their footprint of 1s and 0s, sit at the centre of huge nets of referents, referents we can easily trace through the resonance of the words and images across people, cultures, and time.

Will the voluptuousness of data be the next big computational challenge?

Tim Hitchcock on ‘Big Data, Small Data and Meaning’

“I end up is feeling that in the rush to new tools and ‘Big Data’ Humanist scholars are forgetting what they spent much of the second half of the twentieth century discovering – that language and art, cultural construction, human experience, and representation are hugely complex – but can be made to yield remarkable insight through close analysis. In other words, while the Humanities and ‘Big Data’ absolutely need to have a conversation; the subject of that conversation needs to change, and to encompass close reading and small data. ”

KPLEX Kick-Off: 1 February 2017, Dublin Ireland

The KPLEX Project held its official kick-off meeting on 1 February 2017 in Dublin, Ireland.  The project team took this opportunity for some structured discussion and knowledge sharing on our 4 key themes and set out the plans for the work programme in the months ahead:

Toward a New Conceptualisation of Data,

Hidden Data and the Historical Record

Data, Knowledge Organisation and Epistemics

Culture and Representations of System Limitations

We are particularly grateful to our EU project officer, Pierre-Paul Sondag, who joined us in Dublin for this meeting.

Data wealth and data poverty

Jörg Lehmann, FUB

Beyond the ‘long tail’ metaphor, the distribution of data within the scientific field has been described in terms of ‘data wealth’ and ‘data poverty’. Steve Sawyer has sketched a political economy of data in a short essay (a slightly modified version of this paper is freely accessible here). According to him, in data-poor fields data are a prized possession; access to data drives methods; and there are many theoretical camps. By contrast, data-rich fields can be identified by three characteristics: pooling and sharing of data is expected; method choices are limited since forms drive methods; and only a small number of theoretical camps have been validated. This opposition leads to an unequal distribution of grants received, since data wealth provides for legitimacy to claims of insight as well as access to additional resources.

While Sawyer describes a polarity within the scientific field with respect to funding and cyberinfrastructures, which he sees as a means to overcome obstacles in data-poor fields, the KPLEX Project will take a look into how contents and characteristics of data relate to methodologies and epistemologies, integration structures and aggregation systems.

Response to “How can data simultaneously mean what is fabricated and what is not?”

Pic Paul Sharp/SHARPPIX
Dr Jennifer Edmond (TCD)

As discussed in the earlier post this week “How can data simultaneously mean what is fabricated and what is not fabricated?”, the term ‘raw data’ points toward an illusion, to something that is naturally or even divinely given, untouched by human hand or interpretive mind.  The very organicism of the metaphor belies the work it has to do to hide the fact that progression form data to knowledge is not a clean one: knowledge also produces data, or, perhaps better said, knowledge also produces other knowledge.  We need to get away from the idea of data as somehow ‘given’ – Johanna Drucker’s preference for the term ‘capta’ over ‘data’ is hugely useful in this respect.

Raw data is, however, an indicator for where we begin.  Human beings have limited lifespans and limited capacities for experimentation and research, sensitive as we are to what John Guillory referred to as the ‘clock time’ of scholarship.  Therefore to make scientific progress, we start on the basis of the work of others.  Sometimes this is recognised overtly, as in citation of secondary sources.  Sometimes it is perhaps only indirectly visible, as in the use of certain instruments, datasets, infrastructures or libraries, that themselves (to the most informed and observant) belie the conditions in which particular knowledge was born.  In this way, ‘raw data’ merely refers to the place where our organisation started, where we removed what we found from the black box and began to create order from the relative chaos we found there.

Do big data environments make these edges apparent?  Usually not.  In the attempt to gain an advantage over ‘clock time,’ systems hide the nature, perhaps not of the chaos they began with, but of how that chaos came to be, whose process of creating ‘capta’ gave origin to it in the first place.  Like the children’s rhyme about the ‘House that Jack Built’, knowledge viewed backwards toward its starting point can perhaps be seen recursing into data, but going further, that data itself emerges from a process or instrument that someone else created through a process that began elsewhere, with something else, at yet another, far earlier point.  At our best, humanists cling to the provenance of their ideas and knowledge.  Post-modernism has been decried by many as the founding philosophy of the age of ‘alternative facts’, as an abandonment of any possibility for one position to be right or wrong, true or untrue.  In the post-truth world, any position can offend, and attempt at defence result in being labelled a ‘snowflake.’  But Lyotard’s work on The Post-Modern Condition wasn’t about this: it was about recognising the bias that creeps in to our assumptions once we start to generalise at the scale of a grand narrative, a truth.  In this sense, we perhaps need to move our thinking from ‘raw data,’ to ‘minimally biased data,’ and from the grand narratives of ‘big data’ to the nuances of ‘big enough data.’  If we can begin speaking with clarity about that place where we begin, we will have a much better chance of not losing track of our knowledge along the way.

How can data simultaneously mean what is fabricated and what is not fabricated?

Jörg Lehmann, FUB

Common wisdom holds it that data are recorded facts and figures; as an example see Kroenke et Auer, Database Processing, slide 8. This is not astonishing if one considers the meaning of the term “raw data”. Most often it refers to unprocessed instrument data at full resolution; the output of a machine which has differentiated between signal and noise – as if the machine has not been conceived by human beings and designed according to needs defined by them. But the entanglement of data and facts refers to a mechanical understanding of objectivity, where instruments record signals which can easily be identified by humans as something that has been counted, measured, sensed. “Raw data” is seen as the output of that operation, as facts gained by experiment, experience or collection. Data products derived from these machine outputs seem to inherit their ‘factual’ status, even if processed to a high degree in order to enable modelling and analysis. Turning against this distinction between “raw” and “cooked” data and underlining data as a result of human intervention, Geoffrey Bowker has termed the phrase “’Raw data’ is an oxymoron”, which provided the title of a book edited by Lisa Gitelman (find the introduction here). But common sense is not this precise with differentiations; it keeps the aura of data as something always being pre-processed.

Data and facts are not the same. In a contribution to the anthology edited by Gitelman, Daniel Rosenberg explains the etymology of both terms as well as their differences. Facts are ontological, they depend on the logical operation true/false. When they are proven false, they cease to be termed “facts”. Data can have something in common with facts, namely their “out there-ness”, a reference apparently beyond the arbitrary relation between signifier and signified. “False data is data nonetheless”, Rosenberg puts it, and he points out that “the term ‘data’ servers a different rhetorical and conceptual function than do sister terms as ‘facts’ and ‘evidence’.” But what exactly is the rhetorical function of the term “data”? Rosenberg’s answer is that “data” designates something which is given prior to argument. Again, this brings the term “data” close to the term “fact”: In argumentation, both terms assume the task of a proof, of something that substantiates. In which settings is this rhetoric particularly relevant?

As Steve Woolgar and Bruno Latour have pointed out a generation ago, facts are social constructions, purified from the remainders of the process in which they were created: “the process of construction involves the use of certain devices whereby all traces of production are made extremely difficult to detect”, they wrote in “Laboratory Life”. There is a process at work which can be compared to ‘datafication’: In a laboratory, the term “fact” can simultaneously assume two meanings. At the frontier of science, the scientists themselves know about the constructed nature of statements and are aware of their relation to subjectivity and artificiality; and at the same time these statements are referred to as a thing “out there”, i.e. to objectivity and facts. And it is in the very interest of science, aiming at “truth effects”, to make the artificiality of factual statements forgotten, and, as a consequence, to have facts taken for granted. The analogy of the nature of these statements to “raw” and “cooked” data is obvious; “facts” and “data”. With respect to the latter, the process of division and classification of phenomena into numbers obscures ambiguity, conflict, and contradiction; the left-over of this process, “data”, are completely cleansed of irritating factors; and no trace remains of the purifying process itself. Therefore “data” deny their fabricated-ness; and in struggles for legitimacy of insight or the application for resources, they serve their purpose in the very same way as “facts” do.

Do Physicists Talk About Whether The Right To Read Is The Right To Mine?

In Europe, copyright laws continue to vary from country to country, in spite of many years’ pressure towards harmonisation.

One of the biggest questions this raises is whether having the right to read a work gives you to right to data mine it. In terms of literature, you might think the right to mine could be the less restrictive one, if anything, as the use and market for mining is so different from a linear reading mode. I don’t know of anyone who takes a nice big set of unstructured data (rather than a novel) with them on a beach holiday for fun, though I am sure such people exist (albeit with sand in their laptops). But no one is sure whether or if the rights to these modes of use will be harmonised, largely because the original rules and customs date to the era of print, and the tension between the unknown benefit of relaxing them and an unknowable future profit model that could emerge from data mining is unresolved.

This is a problem for literary scholars, but also for historians and others working with cultural data. Which leads me to the observation that the very beauty and utility of humanities data creates a two tier system of science, in which legal hurdles hinder research in some disciplines (those that co-own their data) or for some methodologies, but others. There are surely some medical or other data sets that are viewed as having equivalent market value to a best-seller, but even there, the mining model would be the only reading paradigm.

So how would data-driven research in other disciplines be different if the raw materials of their research were sold in airports, hung on the walls of galleries, or revered as the founding documents of a nation?

Data’s Long Tail

Jörg Lehmann, FUB

The ‘long tail’ is a metaphor for a statistical distribution, where about 20 percent of the distribution is in the head of the curve, and the remaining 80 percent are distributed along the tail. This well-known distribution – a power law – was popularized by Chris Anderson, former editor-in-chief of the Wired Magazine. In an article published in 2004 [], Anderson described the market for goods in physical stores versus online stores by means of this distribution: In the head of the curve, there are the hits: blockbusters, bestsellers, top-ten chart tracks, all sold by retailers with stores. But in the long tail, there are the much-loved film classics, the back catalog, older music albums. Combined, the non-hits make for a market bigger than the hits, which opens the door for online stores. As Anderson explains, successful businesses in the digital economy are about aggregating the long tail, focusing on a million of niches rather than onto the mass market.

The Long Tail distribution. File Source: By User:Husky (Own work) [Public domain], via Wikimedia Commons

But the metaphor of the ‘long tail’ can also be applied to scholarly research: Only a fraction of research teams work on large volumes of data, say, about 20%; in comparison to them, a much bigger number of researchers work with ‘little data’. These small datasets are harder to handle and to exchange, as Christine Borgman et al have recently described in their research paper “Data Management in the Long Tail: Science, Software and Service”(1)  [].

The attention of the public is towards ‘big data’; in contrast, the smaller datasets used by the humanities as well as cultural data are ‘hidden’, not standardized, and they lack metadata. Even more, they are unstructured, polyvalent, messy; and this is what the KPLEX-Project focuses on – long tail data.


  1. Christine L. Borgman, Milena S. Golshan, Ashley E. Sands, Jillian C. Wallis, Rebekah L. Cummings, Peter T. Darch, Data Management in the Long Tail: Science, Software and Service. In: International Journal of Digital Curation 11(1), (2016) 128–149. DOI: 10.2218/ijdc.v11i1.428




Can you describe the characteristics of a well-formed dataset?

Jörg Lehmann, FUB

If not, don’t worry – you are in good company. The majority of people working with data have not been training in this respect: It is surprising that amongst those who establish and collect data, only rarely a statistician or data scientist can be found. We know that datasets consist of variables and observations; but it is surprisingly difficult to precisely define variables and observations in general. Tidy datasets provide a standardized way to link the structure of a dataset with its meaning; they form the basis for effective computation, modeling and visualization. In his contribution “Tidy data” Hadley Wickham, Chief Scientist at RStudio and Adjunct Professor of Statistics, explains what accounts for the quality of datasets, and how they can effectively be cleaned and prepared for analysis. This article from the Journal of Statistical Software is freely accessible.