Too much data not enough information: a survival guide to the information age

Saw this: https://informedinsport.com/new-blog/special-post-the-illogic-of-being-data-driven The numbers have no way of speaking for themselves. We speak for them. We imbue them with meaning — Nate Silver So, does data have a way to speak for itself, is all interpretation and meaning imbued solely by the people analyzing the data? The problem with this question is that it misses a level of analysis. True, meaning is only conferred through humans, outside the system, mapping external systems to the data.  Semiotic definition of meaning holds. However, since there are multiple potential interpretations we need a step to maximize the Information in the data before communicating its message to the people. True it requires that people impose a metric, a type of a prior assumption, yet it is not an assumption on interpretation rather an assumption of the field of analysis.  My favorite metric is that coincidence is informational.  Thats it.  Hence two obse...

VII) appendix https://ashlag-cause-and-kook-affect.blogspot.com/2018/10/the-natural-scale-of-thing.html https://ashlag-cause-and-kook-affect.blogspot.com/2018/07/information-flow-in-complex-systems-how.html https://ashlag-cause-and-kook-affect.blogspot.com/2018/03/scale-dependent-truths-in-big-data.html https://ashlag-cause-and-kook-affect.blogspot.com/2017/07/autonomous-cars-trolleys-and-scale-of.html https://ashlag-cause-and-kook-affect.blogspot.com/2018/12/being-judgmental-is-necessary-for.html

V) How do we know we made a reasonable judgement? I was by my brother in NY, on my way to the airport, and I spotted a book by Umberto Eco on information and open systems.  I borrowed the book (and still have it -- sorry Jacob),  just on the whim that I would enjoy more Eco in my life.  I discovered much more, the book is Eco's earlier writing, semiotics mixed with art and science, and has had a profound affect on me.  Eco makes the argument that Shannon's description of information, a measure of the communicability of a message, provides for a measure of art. If it helps think about 'On Interpretation' by Susan Sontag, experience art without interpreting it.  There is no message not even one that we the viewer creates.   There is no meaning to be had, just an experience.  The flip side of this argument is that when there is interpretation there is meaning.  This view, proposed by Semiotics, states that when two closed systems meet and are ...

Listening to Dr. Stephen C. Meyer, here is a quote from the web: “But it is also necessary to distinguish Shannon information from information that performs a function or conveys a meaning. We must distinguish sequences of characters that are (a) merely improbable from sequences that are (b) improbable and also specifically arranged so as to perform a function. That is, we must distinguish information-carrying capacity from functional information.”  ―  Stephen C. Meyer,  Signature in the Cell: DNA and the Evidence for Intelligent Design Meyers speaks of 'information that performs a function or conveys a meaning', I think we are employing different definitions of those key words.  Let me try and define his message with my terms. Data: a recorded observation.  Data is a thing, an object, as perceived. Information: a metric of data.  Just like we measure the weight of things in pounds or grams, we can measure that quantity of data in Information. Me...

IV) Abstractions and judgements When the third thing is introduced an abstraction is made. That moment when a continuous function takes on a discrete shape.  The moment when we utilize a metric, so that instead of saying something is better or worse we say it is in a different category.  That is the moment when we create the idea of the category.  The category provides a framework to describe the abstraction we just made by saying these two things are more similar to each other than to a third thing. It is also the moment when we lose something, we go from an infinite potential to a specific realization.  We make a judgement call. https://ashlag-cause-and-kook-affect.blogspot.com/2018/12/being-judgmental-is-necessary-for.html Abstractions are both the key element in learning and the realization of a judgement, a phase transition.   I want to define learning, as opposed to memorization and prediction, as a method for creating a model.  A model i...

III) Metrics One of these things is not like the other -- but two of these things are distant from a third. I grew up with Brisk Torah, more specifically my father was a Talmid of Rabbi Joseph Soloveichik and dialectic thinking was part and parcel of our discussions.  Two things, two dinim, the rhythm in the flow between two things.  Dialectics not dichotomies.  The idea espoused by the Rambam in his description of Love and Awe, mutually exclusive, we travel between them. Why create duality?  Dialectics or dichotomies provide a powerful tool, but what is it that tool? What is the challenge? I think the Rabbinic language might be נתת דברך לשיעורים, 'your words are given to degrees', the idea being that without clear definitions we are left with vague language, something is more than something else, ok, but how much more? This I think is the reasoning for the first of the twenty one questions I was taught by my father's mother, 'is it bigger than a breadbox?',...

II) Too much data Way back when, in the late 90's, before the advent of Big Data, we had little computing.  I was working with the Volcani institute (the ministry of agriculture's research institute) on the problem of quality sorting of tomatoes.  The project, multi-sensor fusion for tomato quality control, required us to analyze tomatoes with vision, smell and touch sensors.  I was responsible for the vision sensor, setting up the lighting and the new digital camera, image processing was an art, very different than the art of Deep Learning today.  I took pictures of several hundred tomatoes stored on CD-ROMs (I think) to analyze.  The basic image processing technique was to break the image up into small windows of 16x16 pixels, create a signature (feature vector) based on texture and color and then cluster those signatures into groups.  With the idea being that sections of the tomato that look alike must also share similar qualities (bruised/sweet/healthy)...