Skip to main content

III) Metrics

III) Metrics

One of these things is not like the other -- but two of these things are distant from a third.

I grew up with Brisk Torah, more specifically my father was a Talmid of Rabbi Joseph Soloveichik and dialectic thinking was part and parcel of our discussions.  Two things, two dinim, the rhythm in the flow between two things.  Dialectics not dichotomies.  The idea espoused by the Rambam in his description of Love and Awe, mutually exclusive, we travel between them.

Why create duality?  Dialectics or dichotomies provide a powerful tool, but what is it that tool? What is the challenge?

I think the Rabbinic language might be נתת דברך לשיעורים, 'your words are given to degrees', the idea being that without clear definitions we are left with vague language, something is more than something else, ok, but how much more?

This I think is the reasoning for the first of the twenty one questions I was taught by my father's mother, 'is it bigger than a breadbox?', by referencing an object with fixed dimensions we can now convert the unknown object's size to one of two categories, larger or smaller than a breadbox, a duality.  In data science we would call that; creating a categorical feature.

Data scientist like to divide the world into two types of data, numerical and categorical.  Numerical data has an advantage that typically it is fairly easy to create a metric of distance.  If I observe the temperature twice, I can ask what is the difference between the observations.  A simple number provides a tool to differentiate between the two observations.

This simplicity is misleading.  Numeric metrics are absolute because they rely on a prior assumptions.  Why should we measure the distance between two temperature observations with the difference between them?  Perhaps a cosine similarity measure, perhaps a euclidean distance, perhaps a Manhattan block distance....

Categorical data seems much more difficult to deal with, what is the distance between two categories?  Is blue very distant from yellow?  Are cows very different than sheep?  Yet within this difficulty lays an elegant solution.  We need a third thing!  So if I ask now are cows more similar to sheep than to an apple?  That is a question that can be answered.

To speak of metrics in the numeric world two observations are necessary, however, metrics in the categorical world requires three.

Two are better than one, for if one falls the other can raise him.  But three now that is another story, three are an eternal braid.  Not sure if Solomon was thinking about metrics or Heidegger's philosophy but the idea is similar.

So what to do.  Pay attention to messages that speak of something being better than something else.  Here is a pareve example: the trolley problem has one of two options, action vs. inaction, flip the switch and kill one person, or leave the switch as is and kill five.  Which is the better option?

This question does not have a well defined metric.  Better by what metric?  Should we measure utility or action?

The choice of metric defines the scale.

A less pareve example: Vaccination, should the government impose public vaccinations?  Which is better actively vaccinating hundreds of millions of children with known side effects to individuals or not vaccinating and passively placing thousands at risk?

Again what is the metric of 'better', our a prior acceptance of a metric defines the question.  But why should we choose one metric over another.  This is not a scientific question, rather a political/religious one.

So listen to the tell, words that assume a metric, better than, not good, less, more...in a conversation without a well defined metric.







Comments

Popular posts from this blog

0.0 Introduction to advanced concepts in AI and Machine Learning

Introduction to advanced concepts in AI and Machine Learning I created a set of short videos and blog posts to introduce some advanced ideas in AI and Machine Learning.  It is easier for me to think about them as I met them, chronologically in my life, but I may revisit the ideas later from a different perspective. I also noticed that one of things I am doing is utilising slightly off-centre tools to describe an idea.  So for example, I employ Kohonen Feature Maps to describe embeddings.  I think I gain a couple of things this way, first it is a different perspective than most people are used to.  In addition, well you will see :-) I recommend first opening the blog entry (as per the links below), then concurrently watching the linked video. Hope you enjoy these as much as I did putting them together, David Here are links: https://data-information-meaning.blogspot.com/2020/12/memorization-learning-and-classification.html https://data-information-meaning.blogspot.com/...

No a penguin is not an ashcan; how to evolve from supervised to semi-supervised learning

Recently Yann LeCun complemented the authors of 'A ConvNet for the 2020s' https://mobile.twitter.com/ylecun/status/1481194969830498308?s=20 https://github.com/facebookresearch/ConvNeXt These statements imply that continued improvements in the metrics of success are indicators that learning is improving.  Furthermore says LeCun, common sense reinforces this idea that 'helpful tricks' are successful in increasing the learning that occurs in these models.  But are these models learning? Are they learning better? Or perhaps they have succeeded at overfitting and scoring better but have not learnt anything new. We took a look at the what the model learned, not just how it scored on its own metric.  To this end we created a graph with links between each image and its top 5 classifications, the weights of the links are in proportion to the score of the class.   Here are the data files: https://github.com/DaliaSmirnov/imagenet_research/blob/main/prediction_df_resnet50.p...