Division Divided by Division

The debate that rages on about the COVID-19 quarantine suffers from the classic complexity of trade-offs that are non-resolvable. Shutting down saved lives but it also destroyed [socioeconomic] lives. Opening up too soon or too quickly will cost lives, but remaining closed will destroy more lives. Saving lives means delaying herd immunity, but achieving herd immunity means losing lives. The list of trade-offs goes on, and the real socioeconomic costs of quarantine measures can only be justified by estimated benefits of undertaking them, making such measures difficult to defend in concrete terms. In these kinds of situations, no amount of data or science can build adequate consensus, and this is especially true when the data comes into question (e.g. the variance in death rate based on the true infection rate, which we do not know) and the science comes into question (e.g. the general population being unable to grasp that the very basis of the scientific method). The only thing that can build consensus in these types of situations is trust in government, as in: "I know that they do not have all the answers, I know that they might get it wrong from time to time, but I also know that they are taking advantage of the best data and science that the country has to offer and are making the most informed decision they can on my behalf." I need not say why we do not have this level of trust in government today, but I do need to say that the breakdown along party lines is a sad state of affairs. If we had stronger leadership, we would be more unified, especially in troubled times. I think we can all agree on that, but maybe not.

Salesforce Einstein Analytics Dataflow to Deduplicate a Dataset

Let's say you have the following data:

Key,Value,Timestamp
K1,V1,2020-05-01
K1,V2,2020-05-02
K1,V3,2020-05-03
K1,V4,2020-05-04
K2,V1,2020-05-04
K2,V2,2020-05-04
K2,V3,2020-05-02
K2,V4,2020-05-01
K3,V4,2020-05-01
K3,V3,2020-05-02
K3,V2,2020-05-03
K3,V1,2020-05-04
K4,V4,2020-05-04
K4,V3,2020-05-03
K4,V2,2020-05-02
K4,V1,2020-05-01
K5,V1,2020-05-01
K5,V2,2020-05-01
K5,V3,2020-05-01
K5,V4,2020-05-01

Further, let's say that you want to remove duplicate rows (based on the Key) and keep the latest row (based on the Timestamp), with the additional complexity of there being duplicate Timestamps.

Your target result would consist of the following, reduced data:

Key,Value,Timestamp
K1,V4,2020-05-04
K2,V2,2020-05-04
K3,V1,2020-05-04
K4,V4,2020-05-04
K5,V4,2020-05-01

To achieve this result, you can generalize the following approach, in reference to the sample dataflow included in its entirety at the bottom of this post.

1. Get the dataset ;-)
2. computeRelative (computePartitionCounter)
• Partition by the Key
• Order by the Timestamp (ascending)
• Compute a partitionCounter column (number, default = 1)
• saqlExpression: previous(partitionCounter) + 1
3. computeRelative (computeKeepRow)
• Partition by the Key
• Order by the partitionCounter (descending)
• Note that weather you sort by descending timestamp and then ascending partitionCounter, or vice versa (as I have done here) is immaterial
• Compute a keepRow column (text)
• saqlExpresion: case when current(partitionCounter) == first(partitionCounter) then "true" else "false" end
• Note that in the case of K2 and K5, since some or all of the rows have the same Timestamp, keepRow will be "true" for one of them, arbitrarily, more than likely based on any internal ordering with Einstein Analytics. 
4. Filter by keepRow == "true"
5. Slice off parititionCounter and keepRow
6. Register the de-duplicated dataset

In my example, I went from this...

 

...to this...

Cheers! 

Dataflow:

{
  "getDataset": {
    "action": "edgemart",
    "parameters": {
      "alias": "KeyValueTimestamp"
    }
  },
  "computePartitionCounter": {
    "action": "computeRelative",
    "parameters": {
      "source": "getDataset",
      "computedFields": [
        {
          "name": "partitionCounter",
          "label": "partitionCounter",
          "expression": {
            "saqlExpression": "previous(partitionCounter) + 1",
            "type": "Numeric",
            "scale": 2,
            "default": "1"
          }
        }
      ],
      "orderBy": [
        {
          "name": "Timestamp",
          "direction": "asc"
        }
      ],
      "partitionBy": [
        "Key"
      ]
    }
  },
  "computeKeepRow": {
    "action": "computeRelative",
    "parameters": {
      "source": "computePartitionCounter",
      "computedFields": [
        {
          "name": "keepRow",
          "label": "keepRow",
          "expression": {
            "saqlExpression": "case when current(partitionCounter) == first(partitionCounter) then \"true\" else \"false\" end",
            "type": "Text"
          }
        }
      ],
      "orderBy": [
        {
          "name": "partitionCounter",
          "direction": "desc"
        }
      ],
      "partitionBy": [
        "Key"
      ]
    }
  },
  "filterDataset": {
    "action": "filter",
    "parameters": {
      "source": "computeKeepRow",
      "saqlFilter": "keepRow == \"true\""
    }
  },
  "sliceDataset": {
    "action": "sliceDataset",
    "parameters": {
      "mode": "drop",
      "fields": [
        {
          "name": "keepRow"
        },
        {
          "name": "partitionCounter"
        }
      ],
      "source": "filterDataset"
    }
  },
  "registerDataset": {
    "action": "sfdcRegister",
    "parameters": {
      "alias": "KeyValueTimestampDeduplicated",
      "name": "KeyValueTimestampDeduplicated",
      "source": "sliceDataset"
    }
  }
}

Conscientious Courtesy

I am seeing a lot of posts about ‘government control’, ‘mask Nazis’, ‘media fear mongers’, etc, as it pertains to the practice of wearing masks in public, on flights, etc. To me, it isn’t about any of those mechanisms or motivators. It is about courtesy. Not just the courtesy of protecting the health of others, but courtesy in kind. For me, this morning was a perfect example. I finally got an appointment at the groomer for Maggie (my dog), who has been shedding balls of wool lately. I had no notion of wearing my mask to drop her off, but when I saw the person receiving dogs wearing a mask, I put mine on as well. This is the new social grace, and, as surely as there is no mind control at play when I respond “good morning” in kind, there is no mind control at play here either. Of course, I would hate to be fined for not saying “good morning”, but the lack of a greeting in kind cannot possibly make someone sick, whereas not wearing a mask presents that risk, however small, so a different level of conscientious courtesy is required. Remember that.

Salesforce OAuth 2.0 Web Server Flow Using cURL

If you are in need of validating a Salesforce API from start to finish using cURL - i.e. from authorization code to access token to a successful API call using the access token - try this:

Create a Connected App in your Salesforce Org:

Check the Salesforce documentation for Create a Connected App with OAuth. Include https://localhost as one of the Callback URLs to follow along with this example, or use whatever URL you like, but just be sure to modify the example below accordingly.

Once you have the Connected App, you can copy-paste the following URL in browser:

Note: If you are attempting to do this in Production, then use 'login' instead of 'test' in the URL.

https://test.salesforce.com/services/oauth2/authorize?client_id=[Consumer Key from Connected App]&response_type=code&scope=api&access_type=offline&redirect_uri=https://localhost

You will be presented with a login screen.

After successfully logging in, your browser will be redirected to https://localhost, which, obviously, will not render a working page, but the authorization code will be available in the URL window:

Copy the value of the code (i.e. code=[authorization_code]) into a text editor, and then URL decode any characters that are URL encoded. In my experience, this is typically two %3D characters at the end of the code, which equates to two equal signs, but I make no claim as to the standard.

Use the Authorization Code to get the Access Token via cURL

Note: If you are attempting to do this in Production, then use 'login' instead of 'test' in the URL.

curl --request POST 'https://test.salesforce.com/services/oauth2/token' --data 'client_id=[Consumer Key from Connected App]' --data 'client_secret=[Consumer Secret from Connected App]' --data 'code=[Authorization Code from Previous Call]' --data 'redirect_uri=https://localhost' --data 'grant_type=authorization_code'

This will return the following response (in non-pretty print format):

{"access_token":"[access_token]","signature":"[signature_redacted]","scope":"api","instance_url":"https://[instance].my.salesforce.com","id":"https://test.salesforce.com/id/[id_redacted]/[id_redacted]","token_type":"Bearer","issued_at":"1588342208615"}

Use the access_token to make API calls:

Note: The following example is highly specific (e.g. POST, json, etc), so adjust accordingly.

curl --request POST '[REST API URI]' --header 'Authorization: Bearer [access_token]' --header 'Content-Type: application/json' --data '{"key_1" : "value_1","key_2" : "value_2"}'

Special Note: The examples worked using Git Bash on Windows, and may need to be modified to work in the Windows Command Window (i.e. you may need to enclose arguments in double quotes and escape double quotes in JSON, etc). Just beware of this nuance.

The Restoration of Action

During some of the more labored exchanges I have on Facebook, there are times when it would be legitimate for any rational person to wonder: "Why bother? You think your way. Others think their way. So why argue?" This is where knowing a little something about Hannah Arendt (1906-1975) is useful. Arendt divided the human experience into three categories: Labor, Work and Action. Labor includes what we do every day to maintain our biological existence (drinking, eating, sleeping). Work includes what we do to build the world we live in (carpenter, bus driver, doctor or, the ultimate profession, programmer - wink). Action is the interaction between people as equals in a public forum to debate and determine the meaning things in the world they share. In short, it is the realm of the political, but not the post-Enlightenment version as a means to an end for economic systems and legal rights. Arendt is referring to the pre-Enlightenment version, where people engaged with one another to establish their identities, their cultures and a sense of meaning and belonging that does not come from Labor, Work or economic and legal systems that are based on the notion of rational self interest. In today's modern society, we cycle between Labor and Work, and we relegate Action to the margins. It is no wonder that we suffer from a collective identity crisis. Our point of confusion around WHO we are versus WHAT we do is what Arendt dubbed 'Economic Man'. The ancient Greeks had a very different notion of life. For them, Labor and Work were a means to an end for Action as the most important aspect of life, where engaging with their fellow man (woman) to discuss, debate and arrive at the meaning of things, including the meaning of life and their position in it, took precedence. Today's version of this engagement is rooted in polarized media outlets, hash-tag campaigns, political memes and echo chambers, which do not foster real discourse and do not create real meaning. We have ceded our role as political actors responsible for shaping the world around us, to a role of political casualties, just trying to survive through Labor and Work, and arranging ourselves along black and white political lines that serve interests that are not our own. It is little wonder that one of Arendt's most famous books is The Origins of Totalitarianism. Whatever your opinion about the evils of Social Media, up to and including the amplification of all of the evils I just mentioned, one undeniable fact about it is that it has restored a public realm for people to engage in, which has liberated us from the isolated cycle of Labor and Work. There are certain exchanges that I have had with certain people that I would have never had outside of a heated Facebook thread. This form of engagement has re-established vital lines of communication. This is why I bother engaging in those belabored exchanges. I am not trying to achieve some kind of end, I am engaging in the restoration of action to find meaning.

The Brave New World

We will need to begin a phased reopening of the country prior to a coronavirus vaccine being available, if there ever is one, and no matter what phasing and coordination, there will be an uptick in COVID-19 cases. This so-called 'M curve' can only be minimized, not avoided, because we have to live our lives, and there is no such thing as a risk-free life or, in some cases, a risk-free livelihood. This will probably create an awareness in us that we should already have, i.e. clean hands, clean phone, sensitivity to symptoms, self-isolation, proactive medical care, etc. It will also create a minimalist mindset with benefits to society, e.g. minimizing activities that contribute to global warming, and detriments, e.g. minimizing activities that contribute to heart warming :-) I believe that one of the most valuable upsides will be the fact that when someone shakes your hand or gives you a hug or attends your party or eats at your restaurant, they will be conveying to you that you are worth the risk. That is a good thing.

The Choice

In The Republic (380 BC), Plato wonders if the average citizen of a society is the right person to be making the decisions about who should be making the decisions for a society. And this has been a question we (at the very least, Western Civilizations) have grappled with since then. In Walter Lippmann's 1922 book Public Opinion, he states that, in the face the ever-increasing complexity of the world, especially relative to the democracy that our Founding Fathers created, average citizens navigate the world through stereotypes (about everything, including the government, environment, medicine, etc), which are fed, in large part, by the media. (This is back in 1922!). If you look at the ever increasing polarization of stereotypes fed by the ever-widening gap between media outlets, you can see how perfectly rational people can be led to vastly different conclusions about the world. I think that "our" job is to close that gap. That means that I actually read posts (and shared articles) of people that I do not agree with, and I actually expect them to consider my counterpoints. I am not sure if this expectation is realistic, but what choice do I have?