I've got the byte on my side

Poestmortem

Wed, 22 Aug 2018 00:00:00 +0000

Midnight strikes, I lose the wager, woken by a screeching pager,
Scarlet hue means issue’s major, major OOMing on the node.
SSH into the victim, tail and grep per playbook’s dictum,
‘Til revealed is cache eviction as what crumbled ‘neath the load.
Run a dump of heap and threading, start the steps for shedding load.
Who hath merged this wretched code?

Bleary eyes see dashboards healing, weary mind’s still looping, reeling,
Ired at some dev who’s stealing hours from my restful mode.
Fueled by spiteful indignation, trace the clues to bug’s location,
Spot the shabby computation, dismal waster of the node.
Desperate now to find the root of this nocturnal episode —
Who hath merged this wretched code?

git blame’s verdict fast unspools, indictments of the cowboy fools
Whose feeble floats and clumsy bools charred to ash beneath the load.
The term’nal casts a pallid glare on bloodshot eyes that, squinting, stare
At guilty method now laid bare and the author who is showed.
A curse escapes my lips as it’s plain to whom this crash is owed —

It was I who wrote the code.

Originally posted on Twitter (with more emoji)

🌔 A Poestmortem 📟
— Lincoln Atkinson (@LincolnAtkinson) August 21, 2018

When the Scala compiler doesn't help

Tue, 02 May 2017 00:00:00 +0000

Much of Scala’s power comes from its flexibility and generality as a language. You can mold it quite extensively to suit your particular problem domain or coding style.

The downside of this is that Scala can sometimes be permissive and accommodating to a fault. You often hear Haskell or F# users attest to a sense of “if it compiles, it works” – in my experience this is not generally the case with Scala.

To illustrate this point, let’s walk through a few examples, all of which are distilled from honest-to-goodness I-swear-I’m-not-making-this-up bugs my team or I have encountered.

Example 1

Setup

The following nonsensical code compiles with no warnings, no errors:

val (a, b, c) =
    if (foo) {
        "bar"
    } else { 
        Some(10)
    }

Bug

This will crash with a MatchError at runtime. Always. It’s super broken. Why does the compiler let it through?

Because when an expression has multiple return branches, Scala tries to be helpful, by picking the first common ancestor type of all the branches as the type of the whole expression.

In this case, one branch has type String and the other has type Option[Int], so the compiler decides that what the developer really wants is for the whole if/else expression to have type Serializable, since that’s the most specific type to claim both String and Option as descendants.

And guess what, Tuple3[A, B, C] is also Serializable, so as far as the compiler is concerned, the assignment of the whole mess to (a, b, c) can’t be proven invalid. So it gets through with nary a warning, destined to fail at runtime.

Example 2

Setup

I had dashed off some tests that more or less boiled down to something like this, which compiled with no errors or warnings:

"foobar".toList == List('f','o','o','b','a,'r')

Bug

This comparison was, surprisingly, returning false, and I couldn’t figure out why. Can you?

With syntax highlighting, the problem is clearer:

"foobar".toList == List('f','o','o','b','a,'r')

You see the character a in the list? There is a typo – I meant to type 'a' but I missed a quotation mark and typed 'a.

As it turns out, the syntax 'blah is actually valid in Scala, and represents a symbol literal.

Now recall the “helpful” compiler behavior mentioned in Example 1. Instead of yelling at me for putting a Symbol in the middle of my Char list, the compiler generalizes and assumes what I really wanted from the start was a List[Any], as Any is the first common ancestor of Char and Symbol.

And comparing a List[Char] with a List[Any] also raises no objections, because in theory it’s not impossible for it to work.

Example 3

Setup

Here’s a simple method definition on a class. Works fine.

class Widget {
    def doTheThing() = 
        synchronized {
            ...
        }
}

Somebody makes a small change - adds a log line at the start of the method.

class Widget {
    def doTheThing() = 
        info("Doin' that sweet thing")
        synchronized {
            ...
        }
}

Jolly good, doesn’t get much simpler than that. I think most code reviewers would glaze over it in the middle of a big PR. The compiler (the best code reviewer) is also 100% happy with this change and raises no objections.

Bug

Three of Scala’s features come into play:

Method bodies without { } brackets can only consist of a single expression
Any statements or expressions in a class definition, outside of member definitions, become part of the primary constructor.
Member definitions and primary constructor content can intermingle with no ordering restrictions.

So the resulting code really acts like this:

class Widget {
    // executes within Widget constructor
    synchronized {
        ...
    }

    def doTheThing() = {
        info("Doin' that sweet thing")
    }
}

What was previously the body of the doTheThing method has been silently bumped into the body of the primary constructor. Oops!

Example 4

Setup

Here’s another example of “I’ll just add one thing” gone awry.

Quick background: Scala has a shorthand syntax _ to represent an anonymous argument in a lambda function. A number of languages (Perl, Mathematica, Powershell to name a few) have a similar capability.

This allows one to use the following shorthand, which is actually quite nice:

val lst = List(1,2,3)

// these lines can be re-written
lst.map { i => i + 1 }
lst.foreach { i => println(i) }

// like this
lst.map { _ + 1 }
lst.foreach { println(_) }

So let’s say you come across some code like this, and you want to add one little thing: a counter in the loop.

val lst = List(1,2,3)

// let me just add one thing
lst.foreach { println(_) }

// yay
var j = 0
lst.foreach { j += 1; println(_) }
println(s"Looped $j times")

Compiler is happy, looks good right?

Bug

Not so fast. This code will print the following:

1
2
3
Looped 1 times

Why is the counter only incremented once if we are looping over a 3-element list?

Because { j += 1; println(_) } is not a lambda, it is an expression that increments j then yields a lambda, one which does nothing but invoke println. So the function argument passed to foreach is just the lambda that invokes println, the j += 1 part is only executed once along the way to obtaining this argument.

In very explicit pseudo-code, this is the equivalent of

var j = 0
val f = {
   j += 1
   return { i => println(i) }
}
lst.foreach(f)

Note that everything would have worked fine if the shortcut syntax was avoided, because the “full-fledged” lambda syntax does capture the whole of the bracketed code into the lambda.

// works as expected
var j = 0
lst.foreach { i => j += 1; println(i) }
println(s"Looped $j times")

Example 5

Setup

This one’s a twofer. Consider the following two test cases, executed with ScalaTest.

def bar() {
  "bar"
}

// test 1
("foo" + bar() + "baz") should be("foobarbaz")

// test 2
"short string" should be
    ("unequal long string that was moved to its own line because it's long")
}

For those not familiar with ScalaTest, using should be matchers just means “LHS should be equal to RHS, fail if not.” So given the code above I would think test 1 would pass and test 2 would fail.

Bug

If you compile and run this, you will find that exactly the opposite happens: test 1 fails while test 2 passes.

The cause of both unexpected results is the same – stray expressions in statement position.

In test 1, bar() actually returns Unit, not String as you might assume from quick inspection. When you declare a Scala method without a = (i.e. def bar() { ... like in the example, as opposed to def bar() = { ...) that indicates to the compiler that the method has Unit return type (i.e. it’s a “void” method). So the "bar" string is simply ignored, Unit is returned, and the LHS string of the test is in fact "foo()baz". Test fails.

In test 2, we see another manifestation of the same problem. Due to the implementation details of ScalaTest and its matchers DSL, the code "short string" should be actually represents a complete expression, whose type is some kind of curried lambda function that expects a RHS value. Nonetheless, it’s a complete expression in statement position, so its value just gets thrown away. Similarly, ("unequal long string ...") is a complete, valid string expression, which also gets ignored. Thus this “test” is really just two expressions that do nothing, and that’s considered a “pass.” (Also worth noting that sometimes Scala expressions can flow across line breaks, so it’s not too unreasonable to see this during a review and assume it works)

To be fair – in this example, both the bar method and the first line of test 2 trigger warnings from the compiler. It’s certainly good that warnings are issued, but I’ll explain why that’s not enough in the next section.

Root causes

I see 3 basic themes in these examples, each representing an aspect of Scala I don’t personally care for.

The type system feels very loose

Due to the kind of automatic generalization demonstrated above, I never feel fully confident that the compiler has my back. I’m always wondering what I screwed up, type-wise, that the compiler isn’t telling me about. Call me old-fashioned, but isn’t type safety supposed to be one of a compiler’s strong suits?

I readily acknowledge that some of the more powerful type system capabilities Scala offers hinge on this behavior, and that some devs fully rely on it for more advanced usages. I just personally prefer the safety of something stricter.

Too much syntax is optional

On the one hand, Scala’s syntax is “dynamic” and “flexible.” Scala is great for crafting DSLs, and it accommodates a wide range of coding styles.

That sounds nice in theory, but I’ve found that in practice it leads to headaches. In a team setting, everybody ends up writing their own personal brand of Scala code, and it takes constant policing to maintain a uniform style. With so much variation in syntax, code reviews become more difficult since the visual patterns you are accustomed to from your own syntactic style don’t neccessarily carry over to what you’re reviewing. And the rules are such that mistakes don’t always result in errors - they might just shift you into another supported form.

Example 3 would have been prevented if methods were simply required to have { }, or if primary constructor content was required to come before method definitions.

Example 4 would have been prevented if there was a single syntax for lambda functions, or if the rules for _ , { }, and ; weren’t so subtle and overloaded.

The first part of Example 5 would have been easier to catch if return type annotations and/or the = in method definitions were required.

The second part of Example 5 would have been prevented if methods required ( ) around their arguments, or if ScalaTest wasn’t inviting people to use a magical DSL that nobody can actually reason about.

There is no scoped “nowarn” pragma

Like most languages, Scala has a “fatal warnings” flag which will promote warnings into errors. But it doesn’t have any way to suppress individual warnings.

The result is an “abstinence only” kind of situation – you either have to commit to never introducing a single warning of any kind, or else you can’t benefit from fatal warnings at all.

Here in the real world, it’s to be expected that a big project will pick up a warning here, a deprecation there. These useful warnings that indeed should be fixed, but for whatever reason the team decides they can’t or won’t be fixed yet.

Ideally one could suppress those particular issues while maintaining the protection that fatal warnings provide against new problems. Instead, we are left with a situation where the build log is already sullied with a bunch of (known) warnings, and nobody notices when new ones (e.g. Example 5) are introduced.

[Update]

Interesting discussion on reddit.

There are a number of comments along the lines of “just don’t rely on type inference,” “just use a linter,” “nobody uses that syntax anyway,” etc. I fully agree! We are indeed looking to implement these kind of things in our CI system and style guide, and I expect those efforts to help.

These comments are making my broader point for me, though – you get a remarkably flexible and powerful language in return, but using Scala safely and confidently means avoiding flagship language features, opting in to experimental compilation flags, and maintaining a small constellation of 3rd-party plugins.

1Poshword - PowerShell client for 1Password

Mon, 12 Sep 2016 00:00:00 +0000

I’m happy to publicize a little project I’ve been working on recently: 1Poshword, a PowerShell client for the 1Password password manager. Code is available at https://github.com/latkin/1poshword.

You can read the full details in the project README, but I’ve included the demo screencast and basic bullets below.

Cross-platform (Windows/OSX/Linux, PowerShell v3.0+)
agilekeychain and opvault support
Login, Password, Secure Note, and Generic Account decryption
Metadata for all entries
Tab completion (agilekeychain only)
Output formats
- PSCredential
- SecureString
- Plaintext
- Clipboard
Complete Get-Help documentation

Try it out and let me know what you think!

Better TeX math typesetting in Hugo

Sun, 07 Aug 2016 00:00:00 +0000

There is a page in the Hugo documentation that describes how to use MathJax to embed nicely-typeset mathematics in one’s Hugo-generated site.

For my own site, I took this as a starting point and made a few improvements. Here’s how I do the math typesetting in this blog.

Authoring

I author my posts in Markdown, where the bulk of the content is plain text. When I want to include specially-typeset mathematical expressions, I use a custom shortcode to convert TeX-style equation definition into a snippet of HTML that looks nice when rendered in the browser.

So when I author a post, I’ll write something like this:

Here's sum inline math: {{< tex "\sum_{n=1}^{\infty} 2^{-n} = 1" >}}.

Display mode math looks like
    {{< tex display="\int \frac{1}{x} dx = \ln |x|" >}}

Hugo will process this, apply my shortcode, and generate HTML that ultimately renders like this:

Here’s sum inline math: .

Display mode math looks like

Shortcode and supporting markup

Here’s the definition of the shortcode:

<!-- Hugo TeX shortcode
     usage:
         inline eqn:       {{< tex "eqn here" >}}
         display mode eqn: {{< tex display="eqn here" >}}
-->
<span class="jsonly">
    {{ if .IsNamedParams }} <!-- display mode, wrap eqn with $$ $$-->
        $${{ .Get "display" }}$$
    {{ else }}              <!-- inline mode, wrap eqn with \(  \)-->
        \({{ .Get 0 }}\)
    {{ end }}
</span>
<noscript>
    {{ if .IsNamedParams }} <!-- display mode -->
        <div style="text-align:center;">
            <img src="https://latex.codecogs.com/gif.latex?{{ .Get "display" }}" title="{{ .Get "display" }}" />
        </div>
    {{ else }}              <!-- inline mode -->
        <img style="display:inline;vertical-align:middle;" src="https://latex.codecogs.com/gif.latex?\inline&space;{{ .Get 0 }}" title="{{ .Get 0 }}" />
    {{ end }}
</noscript>

This takes care of inserting the required content and tags for each particular expression, but that alone isn’t enough. Some additional Javascript and CSS are required in the header and footer of the page, as well.

In my Hugo page template I conditionally include this extra markup, which will pull down the CSS and Javascript libraries that do the real typesetting work when someone loads the page:

<!-- top of page -->
{{ if .GetParam "hasMath"}}
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.6.0/katex.min.css">
  <script src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.6.0/katex.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.6.0/contrib/auto-render.min.js"></script>
{{ end }}


<!-- bottom of page -->
{{ if .GetParam "hasMath"}}
  <script>renderMathInElement(document.body);</script>
{{ end }}

I simply add hasMath: true to the front matter of any post that has equations, and everything is taken care of. If a post doesn’t contain any mathematics, no superfluous scripts or CSS are included.

As you can see above, I am using KaTeX, a MathJax competitor from Khan Academy, as my backend typesetting library. KaTeX renders noticeably faster than MathJax, but has a more limited feature set. Given the meager sophistication of the equations appearing in this blog, I don’t need the more exotic features anyway, so KaTeX a pretty clear win.

Handling noscript

I’ve also taken care to include a <noscript> block for each equation, which will activate when Javascript is disabled.

In such a case, KaTeX or MathJax won’t work, so I fall back to using flat images. Images don’t render as crisply, and won’t resize or zoom quite as well as proper HTML + CSS, but they will do in a pinch.

There are a few free cloud services available for this, which dynamically generate math typesetting images. You just embed the equation you want in the URL query parameters, and the server will create the image on the fly.

Google Charts is one such service. I’m currently using CodeCogs, as I find that its images look better than Google’s, and they support both inline and display-mode image rendering.

With Javascript disabled, the earlier example will render like this:

Here’s sum inline math: .

Display mode math looks like

To prevent the Javascript-only equation markup from appearing “raw” next to the images, I include a small bit of CSS in every page to hide such elements:

  <noscript>
    <style>.jsonly { display: none }</style>
  </noscript>

RSS readers

Some people consume my blog through an RSS reader, rather than visiting my site directly. It would be nice if equations came through in a reasonable way for for these people, too.

Javascript is not exposed to RSS at all, so KaTeX and MathJax are out the window, and I am limited to HTML and inline CSS for formatting. Thankfully, this is exactly how the noscript fallback operates, so most of the work is already done.

After running Hugo to generate all of my site’s HTML and RSS content, I run a post-processing script to fix up the RSS XML slightly. The script does 2 things:

Removes the jsonly spans entirely, since they aren’t useful for RSS and will be rendered in their ugly raw form.
Removes the noscript opening and closing tags, leaving just the inner image content. Some RSS readers will not render content inside of noscript tags, so it’s important to remove these.

Here’s a snippet of PowerShell that does the trick:

$content = [io.file]::ReadAllText('index.xml')

# delete jsonly spans completely
$content = $content -replace '&lt;span class=&#34;jsonly&#34;&gt;\s*.+?\s*&lt;/span&gt;',''

# remove <noscript> tags, leaving just the inner content
$content = $content -replace '&lt;noscript&gt;\s*((?:.|\s)+?)\s*&lt;/noscript&gt;', '$1'

[io.file]::WriteAllText('index.xml', $content)

What’s left should look fairly decent in most RSS readers.

Curious behavior when de-duplicating a collection in PowerShell

Tue, 02 Aug 2016 00:00:00 +0000

This is a bug/curiosity in PowerShell that I stumbled upon a few years ago, but never wrote up. The behavior hasn’t changed significantly in the intervening verions, so now I’m finally getting around to a quick blog post.

Here’s a chart detailing the runtime of 4 different PowerShell approaches to de-duplicate a collection - i.e. filter an input collection to just its unique elements. Code used for the benchmark can be found here.

Same chart, with log scale so it’s a little easier to distinguish the trend lines:

The solutions being compared are:

$data | select -unique
- Built-in, does nothing but de-duplicate the input collection.
$data | group |% Name
- Composition of built-ins group (which groups together duplicate items) and % (to read off the key from each group)
$data | sort -unique
- Built-in, sorts the input collection and additionally de-duplicates it.
$data | hashunique
- Custom cmdlet using simple hashset-based approach. 12 lines, available alongside benchmark code.

I was surprised to see how poorly select -unique performs. It’s part of the core PowerShell library (presumably authored by experts), implemented as compiled C# code (not script), and its sole purpose is to solve this exact problem.

So why is it the slowest option?!

Specific questions I’m still wondering about -

Why do “select” and “group” have quadratic runtime?

Filtering out duplicates from a collection, or grouping together equivalent items, can be done in time by utilizing a hash set or hash table, respectively. The trendline above hints, and a quick ILSpy session confirms, that select and group are instead relying on an algorithm that doesn’t use hashing at all.

My best guess is that somehow PowerShell’s object and equality system preclude efficient, reliable hashing for all scriptable objects, so they decide not to do it for any objects. Perhaps everything just relies on equality.

But this doesn’t quite hold up. Here’s a simple example where two items are considered non-unique by select, sort, and group yet aren’t considered equal:

PS> 'dude' > sweet.txt
PS> $f1 = dir .\sweet.txt
PS> $f2 = dir .\sweet.txt
PS> ($f1, $f2 | select -Unique).Length
1
PS> ($f1, $f2 | sort -Unique).Length
1
PS> ($f1, $f2 | group).Length
1
PS> $f1 -eq $f2
False

Practically speaking, when I’m at the terminal and need to de-duplicate a collection, 99% of the time it’s just going to be strings or numbers. It’s a shame this core use case is left completely unoptimized. Going forward I will just keep hashunique in my profile and use that.

Why is “group” faster than “select -unique”?

Setting aside asymtotic complexity, how can select -unique be slower by some constant factor than piping together group |% Name? The latter has to carry around a ton of extra data, and also incurs the cost of an additional pipeline command. This makes no sense to me.

Git for Windows accidentally creates NTFS alternate data streams

Wed, 20 Jul 2016 00:00:00 +0000

As part of the small minority of devs at my company who primarily run Windows, I’m accustomed to working around occasional Unix-specific behaviors in our build and deployment systems. Cygwin makes most stuff just work, I can fix simple incompatibilities myself, and as a last resort I can always boot into OSX for a while if needed.

One oddity that took me quite some time to diagnose, though, was Git’s strange behavior when dealing with files in our repo whose names contained a colon.

What happens when you sync a file with a colon in the filename?

Besides the inital drive prefix (e.g. C:\), Windows does not permit the colon character in file or directory paths. Unix has no such restriction. So what happens if a Git repo of Unix origin contains a file with a colon in the name, and that repo is cloned on a Windows machine?

I’ve created a sample repo that contains a single file foo:bar with the content hello. Cloning the repo with a default installation of Git for Windows you get no errors or warnings:

C:\src
> git clone https://github.com/latkin/filetest.git
Cloning into 'filetest'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Unpacking objects: 100% (3/3), done.
Checking connectivity... done.

Instead of a file named foo:bar, though, you get a file named foo, with nothing in it:

C:\src
> cd .\filetest\
C:\src\filetest
> dir -force

    Directory: C:\src\filetest

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--h--        7/17/2016   5:53 PM                .git
-a----        7/17/2016   5:47 PM              0 foo

That’s kind of strange on its own, but even more peculiar is that Git has a different opinion of what things look like:

C:\src\filetest
> git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        foo

nothing added to commit but untracked files present (use "git add" to track)

Git notices the untracked file foo, but seems to think foo:bar is both present and contains the expected content. How strange…

Confusing matters further is that when you enable the Git config option core.fscache (which is enabled by default in version 2.8.2 and later), the working set suddenly changes - now foo:bar is reported as missing:

C:\src\filetest
> git config core.fscache true
C:\src\filetest
> git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    foo:bar

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        foo

no changes added to commit (use "git add" and/or "git commit -a")

That’s what we originally would have expected given that only the foo file was created when we cloned the repository, but why is it different from core.fscache = false? And why was this empty file foo created in the first place?

Alternate data streams

The root cause of all this is a relatively obscure NTFS feature called alternate data streams. Some good summary links here and here.

Briefly, files in NTFS are not simple buckets of data, but rather a collection of 1 or more data streams. What we normally think of as a file’s contents is really the contents of the primary, unnamed stream. One can also create and add data to alternate, named streams. These streams are directly addressable by appending :streamname to the normal file path. e.g. the stream MyStream in file qwerty.txt can be accessed via the path qwerty.txt:MyStream.

So although foo:bar is not a legal Windows file name, Windows file APIs are nonetheless happy to accept it for read and write operations because it is indeed legal as a path to something in the filesystem, namely the bar alternate stream of the file foo.

What Git does

Once you are aware of alternate data streams, Git’s behavior starts to make sense.

When cloning, Git naively blasts content into the path foo:bar. That is a 100% legal path, so no errors are raised by the OS. The result is a file foo with no content in the primary data stream (hence reported as length 0), but 6 bytes in an alternate stream bar:

C:\src\filetest
> Get-Item .\foo -Stream * | ft Stream,Length

Stream Length
------ ------
:$DATA      0
bar         6

C:\src\filetest
> cat .\foo:bar
hello

When checking the status of the working set, Git uses different algorithms depending on whether core.fscache is enabled.

When core.fscache is false, file metadata checks are done one at a time, ultimately invoking GetFileAttributesEx for each path. Git has no clue it’s even dealing with an alternate stream, because these file APIs behave exactly the same as they would with a normal file path. Does foo:bar exist? Yep! Does the last modified time on foo:bar match what Git expects? Yep! Is the content of foo:bar what Git expects? Yep! Well alright, that file must be unchanged.

When core.fscache is true, Git pre-caches file metadata per directory, then reads it from the cache instead of invoking file APIs directly. This leads to a different view of the world - when enumerating files in the containing directory, Windows only mentions foo, since that’s the only file present. Thus the cache, when asked for the metadata of foo:bar, believes this file does not exist.

Conclusion

In my opinion, this is all rather silly and should never have been allowed to happen in the first place. Git should simply detect the bogus filename, issue an error, and never even attempt to write the file to disk. This is how other valid-in-Unix-but-invalid-in-Windows filenames are handled already (e.g. a file named \Windows\System32\crypt32.dll will be blocked). Such files would then (correctly) be reported as missing from the working set, regardless of core.fscache setting.

I opened a bug against Git for Windows to track this issue, and provided a PR with a fix, but these have sat dormant with no feedback for the past 4 months. This week I’m making noise again on the PR, hopefully that will spur some action by the maintainers.

Moving to a static site generator

Mon, 13 Jun 2016 00:00:00 +0000

This blog started on wordpress.com back in February of 2012, then in November 2013 I moved it to a hosted WordPress.org site here at latkin.org. WordPress is quite nice, but it seemed like it was a bit heavyweight given my very basic needs. I’ve wanted to slim down the site and get more hands-on for a while, now.

Over the past few weeks, I’ve been migrating the entire blog to the Hugo static site generator. I’m pleased to announce that the migration is complete!

Why Hugo?

No particular reason. I happened to be reading a blog with a “created by Hugo” footer one evening when inspiration struck. Skimming through comparisons of Jekyll, Hugo, Octopress, Middleman, etc left me with the opinion that they are all capable tools, especially for a simple personal blog.

Some nice aspects of Hugo that I have noticed, though, include simplicity, performance, and ease of use. The entire Hugo platform is just 1 command-line executable, which works great out of the box with 0 config. It’s written in Go so overall it feels pretty snappy.

Leaner, meaner kinder

I wanted the site to be much lighter-weight, simple, and portable. Not quite as barebones as, say, http://danluu.com/, but something closer to that.

Where I’m at now:

Statically-generated pages
No custom fonts
Simple, tame CSS
Very minimal usage of javascript
Switched from MathJax to KaTeX for math typesetting
Dropped embedded GitHub gists for code samples in favor of static HTML (no more JS)

The blog landing page weighs in at about 54KB, which is quite trim. Despite my efforts to avoid gratuitous javascript, about 90% of that 54K is is external JS (Google Analytics, Disqus comment counts, KaTeX).

Disqus comment threads are unfortunately pretty heavyweight, making the posts themselves chubbier than I’d like. My most recent post about benchmarking IEnumerables is ~300KB to load, 85% of which is Disqus. That’s a real bummer, but I don’t know of any significantly better alternatives. I do enjoy the (infrequent) comments here, so I’d prefer to keep some kind of commenting system.

I’ve had fun reworking the site. I am no kind of designer or web dev - certainly there is room for improvement/optimization - but it’s been fun to learn along the way.

Benchmarking IEnumerables in F# - Seq.timed

Mon, 08 Feb 2016 00:00:00 +0000

It’s pretty straightforward to do basic benchmarking of a single, self-contained piece of code in .NET. You just make a Stopwatch sandwich (let sw = Stopwatch.StartNew(); <code goes here>; sw.Stop()), then read off the elapsed time from the Stopwatch.

What about measuring the throughput of a data pipeline? In this case one is less interested in timing a single block of code from start to finish, and more interested in bulk metrics like computations/sec or milliseconds/item. Oftentimes such pipelines are persistent or very long-running, so a useful benchmark would not be a one-time measurement, but rather something that samples repeatedly.

Furthermore, it’s sometimes difficult to determine where the bottleneck in a chain of computations lies. Is the root data source the culprit? Or is it perhaps an intermediate transformation that’s slow, or even the final consumer?

This kind of problem came up for me recently, so I put together a timing function Seq.timed. The complete code is at the bottom of the post.

Seq.timed

val timed : blockSize:int option -> f:(int -> TimeSpan -> float -> unit) -> source:seq<'t> -> seq<'t>

blockSize is the sample size - how many sequence elements should flow in between timing callbacks? Or specify None to only make a single callback when the source is exhausted.
f is the timing callback - it is invoked once for every blockSize items enumerated, and/or when the source sequence is exhausted. It is passed the count of items since the last callback, a TimeSpan indicating how long it took for that many items to be enumerated, and a float indicating the proportion of that time spent upstream in the source sequence (as opposed to downstream in consumers of the resulting sequence).
source is the input sequence.

The idea is that you can simply slip this into an existing F#-style collection-processing pipeline, or tag it on the end of any existing seq/IEnumerable. The resulting sequence yields exactly the same values as the source, but also keeps track of timing each block of items as they are enumerated, invoking the provided callback for each block.

Examples

Here’s a small example that benchmarks a toy sequence, showing measurement of overall throughput along with detection of upstream/downstream relative speed:

open System.Threading

seq{ 1 .. 100 }
|> Seq.timed None Scale.s                // add timing here
|> Seq.map (fun i -> Thread.Sleep(10); i)
|> Seq.map (fun i -> Thread.Sleep(25); i)
|> Seq.iter (fun i -> Thread.Sleep(15))
// output
// 100 items       5.15s      19.43 items/s       0.05 s/item ( 0% upstream | 100% downstream)

seq{ 1 .. 100 }
|> Seq.map (fun i -> Thread.Sleep(10); i)
|> Seq.timed None Scale.s                // move timing to here
|> Seq.map (fun i -> Thread.Sleep(25); i)
|> Seq.iter (fun i -> Thread.Sleep(15))
// output
// 100 items       5.13s      19.51 items/s       0.05 s/item (20% upstream | 80% downstream)

seq{ 1 .. 100 }
|> Seq.map (fun i -> Thread.Sleep(10); i)
|> Seq.map (fun i -> Thread.Sleep(25); i)
|> Seq.timed None Scale.s                // move timing to here
|> Seq.iter (fun i -> Thread.Sleep(15))
// output
// 100 items       5.15s      19.43 items/s       0.05 s/item (70% upstream | 30% downstream)

Here’s something a little more realistic - a tiny grep routine. Where is the time spent here - reading content from the disk or doing the regular expression matching?

open System.IO
open System.Text.RegularExpressions

let grep dir files patt =
    Directory.EnumerateFiles(dir, files, SearchOption.AllDirectories)
    |> Seq.collect (File.ReadAllLines)
    |> Seq.timed (Some 100000) Scale.ms
    |> Seq.filter (fun line -> Regex.IsMatch(line, patt))

// a cold search only reads ~150K lines/sec and large
// majority of time is spent reading from disk
grep @"C:\src\visualfsharp" "*.fs*" @"\blet\b" |> Seq.length
// 100000 items     552.14ms     181.11 items/ms       0.01 ms/item (89% upstream | 11% downstream)
// 100000 items     559.99ms     178.58 items/ms       0.01 ms/item (88% upstream | 12% downstream)
// 100000 items     777.57ms     128.61 items/ms       0.01 ms/item (91% upstream |  9% downstream)
// 100000 items     708.15ms     141.21 items/ms       0.01 ms/item (92% upstream |  8% downstream)
// 100000 items    3372.02ms      29.66 items/ms       0.03 ms/item (98% upstream |  2% downstream)
// 100000 items    3788.68ms      26.39 items/ms       0.04 ms/item (98% upstream |  2% downstream)
// 6740 items      75.73ms      89.00 items/ms       0.01 ms/item (95% upstream |  5% downstream)

// but for a second invocation the disk is warm and results are from the cache,
// so the time is more evenly split (and overall throughput is much higher)
grep @"C:\src\visualfsharp" "*.fs*" @"\blet\b" |> Seq.length
// 100000 items      99.67ms    1003.26 items/ms       0.00 ms/item (44% upstream | 56% downstream)
// 100000 items     175.45ms     569.95 items/ms       0.00 ms/item (59% upstream | 41% downstream)
// 100000 items     109.65ms     911.98 items/ms       0.00 ms/item (39% upstream | 61% downstream)
// 100000 items      92.56ms    1080.36 items/ms       0.00 ms/item (35% upstream | 65% downstream)
// 100000 items     193.58ms     516.59 items/ms       0.00 ms/item (70% upstream | 30% downstream)
// 100000 items     217.47ms     459.84 items/ms       0.00 ms/item (73% upstream | 27% downstream)
// 6740 items       8.71ms     774.07 items/ms       0.00 ms/item (49% upstream | 51% downstream)

How much overhead does Seq.timed add? That’s the real test - can Seq.timed benchmark itself?

// 1000000 items 80442000.00ns       0.01 items/ns      80.44 ns/item (46% upstream | 54% downstream)
Enumerable.Range(1, 1000000)
|> Seq.timed None Scale.ns
|> Seq.length

// 1000000 items 170637400.00ns       0.01 items/ns     170.64 ns/item (81% upstream | 19% downstream)
Enumerable.Range(1, 1000000)
|> Seq.timed None (fun _ _ _ -> ())
|> Seq.timed None Scale.ns
|> Seq.length

// 1000000 items 242433000.00ns       0.00 items/ns     242.43 ns/item (88% upstream | 12% downstream)
Enumerable.Range(1, 1000000)
|> Seq.timed None (fun _ _ _ -> ())
|> Seq.timed None (fun _ _ _ -> ())
|> Seq.timed None Scale.ns
|> Seq.length

Our baseline is about 80ns/item. Adding in one layer of Seq.timed slows throughput to 170ns/item, and two layers slows throughput to about 240ns/item. Looking at these deltas and rounding up, we make a rough estimate that Seq.timed adds a cost of ~100ns per item to a sequence (in my environment). So for data pipelines that are processing fewer than 1M items/sec, Seq.timed adds less than 1% overhead. For anything faster than that, it starts to stick out a bit.

Most of the work being done within Seq.timed is the stopping and starting of timers used to calculate the upstream/downstream ratio. We can speed things up significantly by removing that feature and measuring only the total throughput. I’ve called that version Seq.timedSlim.

// 1000000 items 8791000.00ns       0.11 items/ns       8.79 ns/item
Enumerable.Range(1, 1000000)
|> Seq.timedSlim None Scale.ns
|> Seq.length

// 1000000 items 13693300.00ns       0.07 items/ns      13.69 ns/item
Enumerable.Range(1, 1000000)
|> Seq.timedSlim None (fun _ _ _ -> ())
|> Seq.timedSlim None Scale.ns
|> Seq.length

// 1000000 items 18784500.00ns       0.05 items/ns      18.78 ns/item
Enumerable.Range(1, 1000000)
|> Seq.timedSlim None (fun _ _ _ -> ())
|> Seq.timedSlim None (fun _ _ _ -> ())
|> Seq.timedSlim None Scale.ns
|> Seq.length

Once again considering the deltas and rounding up, it appears this version is about 10x faster, slowing throughput by only <10ns/item. One needs to be processing 10M items/sec before Seq.timedSlim makes more than 1% impact.

Code

Full implementation is below. A few notes/gotchas:

The final callback won’t be invoked if the downstream consumer stops before the upstream source is finished. e.g. when the sequence is shortened downstream by Seq.take.
Beware of transformations like Seq.sort which slurp in the entire upstream sequence at full speed before doing any work to contribute downstream.
Beware of sequences that do their work not in MoveNext() but in Current. Whether to assign the cost of Current to the upstream or downstream sequence is debatable, but I’ve left it assigned to downstream.

open System
open System.Collections
open System.Collections.Generic
open System.Diagnostics

module Seq =
    type private TimedEnumerator<'t>(source : IEnumerator<'t>, blockSize, f) =
        let swUpstream = Stopwatch()
        let swDownstream = Stopwatch()
        let swTotal = Stopwatch()
        let mutable count = 0
        interface IEnumerator<'t> with
            member __.Reset() = failwith "not implemented"
            member __.Dispose() = source.Dispose()
            member __.Current = source.Current
            member __.Current = (source :> IEnumerator).Current
            member __.MoveNext() =
                // downstream consumer has finished processing the previous item
                swDownstream.Stop()

                // start the total throughput timer if this is the first item
                if not swTotal.IsRunning then
                    swTotal.Start()

                // measure upstream MoveNext() call
                swUpstream.Start()
                let result = source.MoveNext()
                swUpstream.Stop()

                if result then
                    count <- count + 1

                // blockSize reached or upstream is complete - invoke callback and reset timers
                if (result && count = blockSize) || (not result && count > 0) then
                    swTotal.Stop()
                    let upDownTotal = swUpstream.Elapsed + swDownstream.Elapsed
                    f count swTotal.Elapsed (swUpstream.Elapsed.TotalMilliseconds / upDownTotal.TotalMilliseconds)
                    count <- 0
                    swUpstream.Reset()
                    swDownstream.Reset()
                    swTotal.Restart()

                // clock starts for downstream consumer
                swDownstream.Start()
                result

    type private SlimTimedEnumerator<'t>(source : IEnumerator<'t>, blockSize, f) =
        let swTotal = Stopwatch()
        let mutable count = 0
        interface IEnumerator<'t> with
            member __.Current = source.Current
            member __.Current = (source :> IEnumerator).Current
            member __.Reset() = failwith "not implemented"
            member __.Dispose() = source.Dispose()
            member __.MoveNext() =
                // start the timer on the first MoveNext call
                if not swTotal.IsRunning then
                    swTotal.Start()

                let result = source.MoveNext()
                if result then
                    count <- count + 1

                // blockSize reached or upstream is complete - invoke callback and reset timer
                if (result && count = blockSize) || (not result && count > 0) then
                    swTotal.Stop()
                    f count swTotal.Elapsed Double.NaN
                    count <- 0
                    swTotal.Restart()
                result

    let mkTimed blockSize getEnum =
        let block =
            match blockSize with
            | Some(bs) when bs <= 0 -> invalidArg "blockSize" "blockSize must be positive"
            | Some(bs) -> bs
            | None -> -1
        { new IEnumerable<'a> with
              member __.GetEnumerator() = upcast (getEnum block)
          interface IEnumerable with
              member __.GetEnumerator() = upcast (getEnum block) }

    let timed blockSize f (source : seq<'t>) =
        mkTimed blockSize (fun block -> new TimedEnumerator<'t>(source.GetEnumerator(), block, f))

    let timedSlim blockSize f (source : seq<'t>) =
        mkTimed blockSize (fun block -> new SlimTimedEnumerator<'t>(source.GetEnumerator(), block, f))

// sample callbacks that scale time units
module Scale =
    let private stats unitString getUnit blockSize elapsedTotal upstreamRatio =
        let totalScaled = getUnit (elapsedTotal : TimeSpan)
        let itemsPerTime = (float blockSize) / totalScaled
        let timePerItem = totalScaled / (float blockSize)
        let upDownPercentStr =
            if Double.IsNaN(upstreamRatio) then ""
            else sprintf " (%2.0f%% upstream | %2.0f%% downstream)" (100. * upstreamRatio) (100. * (1. - upstreamRatio))

        Console.WriteLine(
            sprintf "%d items %10.2f{0} %10.2f items/{0} %10.2f {0}/item%s"
                blockSize totalScaled itemsPerTime timePerItem upDownPercentStr,
            (unitString : string)
        )

    let s = stats "s" (fun ts -> ts.TotalSeconds)
    let ms = stats "ms" (fun ts -> ts.TotalMilliseconds)
    let μs = stats "μs" (fun ts -> ts.TotalMilliseconds * 1e3)
    let ns = stats "ns" (fun ts -> ts.TotalMilliseconds * 1e6)

JNI object lifetimes - quick reference

Mon, 01 Feb 2016 00:00:00 +0000

I’ve recently had reason to do a bit of work with JNI . Throughout the course of this work I had to do quite a lot of Googling in order to figure out how to properly manage the caching of various JNI objects used by my C++ code. Some JNI objects can be safely cached and re-used at any point, while others have limited lifetimes and require special handling. Obtaining JNI objects through JNI APIs is, broadly speaking, fairly expensive, so it’s smart to persist those objects which will be re-used in multiple places. You just need to be careful.

I expected to find a big table somewhere that documented the lifetime restrictions (or lack thereof) for each of the JNI object types, but sadly I was unable to locate one. Instead, I wound up trawling through numerous Stack Overflow replies, blogs, forums, and other documentation to obtain this information.

This post is my effort to provide to others the missing quick reference I wish I’d found. I recommend this Android documentation as further reading if you want more details.

JNI Object	Use across JNI calls?	Use on different thread?	Notes
`JavaVM`	Yes	Yes	To obtain an instance of `JavaVM`, either: Expose a native export `jint JNI_OnLoad(JavaVM* vm, void* reserved)`, which will be invoked when `LoadLibrary` is called from Java Call `pJNIEnv->GetJavaVM(&pJavaVM)` on a valid `JNIEnv` You don't need to clean up a `JavaVM`.
`JNIEnv`	Yes	No	A `JNIEnv` is valid across JNI calls, but only on its original thread. If you are going to cache a `JNIEnv`, consider using a `thread_local` or similar. If a `JNIEnv` is not available for the current thread (e.g. within a native callback), one can be obtained by calling `pJavaVM->AttachCurrentThread(&pJNIEnv, null)`. In this case you are responsible for cleaning up with `pJavaVM->DetachCurrentThread()`. Otherwise, you don't need to clean up a `JNIEnv`.
Primitive value types `jboolean`, `jchar`, `jshort`, `jfloat`, `jdouble`, `jsize`, `jint`, `jlong`, `jbyte`	Yes	Yes	These are just aliases for normal native value types, so there are no special lifetime or cleanup considerations.
Reference types `jobject`, `jclass`, `jthrowable`, `jstring`, `jarray`, `jbooleanArray`, `jbyteArray`, `jcharArray`, `jshortArray`, `jintArray`, `jlongArray`, `jfloatArray`, `jdoubleArray`, `jobjectarray`	No (local refs)	No (local refs)	If one of these is passed in as an argument to a JNI call, or returned from another JNI API, it is typically a local reference and is only valid for the duration of that JNI call, on that same thread. These do not require any cleanup. A durable global reference can be created from any local reference by calling `globalJThing = (jthing) pJNIEnv->NewGlobalRef(localJThing)`. Global references are valid across JNI calls and on any thread. Global references must be cleaned up with `pJNIEnv->DeleteGlobalRef(globalJThing)`.
Yes (global refs)	Yes (global refs)
ID types `jfieldID`, `jmethodID`	Yes	Yes	These are durable and can be used without restriction. No cleanup is required.

Pentago

Thu, 02 Jul 2015 00:00:00 +0000

Pentago is a favorite board game of mine, which I used to play regularly with coworkers during lunch (and occasionally during not-lunch). The rules are very simple, and casual games can be played in just a few minutes, but it’s deep enough to still be satisfying if you’re willing to put some thought into your strategy.

In 2009 I wrote a computer player for Pentago in C#, and even managed to cobble together Silverlight and Windows Phone UIs for it that aren’t terrible. The engine uses a negamax algorithm with alpha-beta pruning.

I put the engine up on Codeplex a long time ago, but never thought to publish the UI projects. This week I moved the whole thing to GitHub (UI projects included) and made sure it builds and runs with VS 2015.

Things I hope to find time/motivation for in the not-too-distant future:

Re-implement the engine in F#, with various performance and design improvements
Publish the phone version
Make an HTML/JS front-end

In the meantime, here’s the Silverlight version to play with! [Note that if you’re using Chrome you’ll need to take extra steps to enable Silverlight.]

Null-checking considerations in F# - it's harder than you think

Mon, 18 May 2015 00:00:00 +0000

The near-complete obviation of nulls is perhaps the most frequently- (and hilariously-) cited benefit of working in F#, as compared to C#. Nulls certainly still exist in F#, but as a practical matter it really is quite rare that they need to be considered explicitly within an all-F# codebase.

It turns out this cuts both ways. On those infrequent occasions where one does need to check for nulls, F# actually makes it surprisingly difficult to do so safely and efficiently.

In this post I’ve tried to aggregate some best practices and pitfalls, in the form of DOs and DON’Ts, for F# null-checking.

Recall that there are a handful of ways that nulls can be introduced into F# code, even for “non-nullable” F# types:

Affecting all reference types
- C# or other CLI language consumers passing null
- .NET framework APIs returning null (e.g. LINQ FirstOrDefault<T>())
- F# Unchecked.defaultof
- F# Array.zeroCreate
Affecting only “nullable types”: non-F# types or F# types marked with [<AllowNullLiteral>]
- null literals

DON’Ts

Don’t use ‘x = null’ or ‘x <> null’

It’s unfortunate that the most obvious approach is not the best. This form is perfectly functional for nullable types, but the generated IL is unoptimized and ends up being quite slow. Standard F# structural/generic comparison is used here, which is a well-known drag on performance.

F#:

let nullCheck01 x = (x = null)

Codegen (C# equivalent):

public static bool nullCheck01<a>(a x) where a : class
{
    return LanguagePrimitives.HashCompare.GenericEqualityIntrinsic<a>(x, default(a));
}

// --- F# core library ---

public static bool GenericEqualityIntrinsic<T>(T x, T y)
{
    return LanguagePrimitives.HashCompare.GenericEqualityObj(false, LanguagePrimitives.HashCompare.fsEqualityComparerNoHashingPER, );
}

internal static bool GenericEqualityObj(bool er, IEqualityComparer iec, object xobj, object yobj)
{
    if(xobj == null) return yobj == null;
    if(yobj != null) { /* ... core logic, fairly big, prevents JIT inlining ... */ }
    return false;
}

By design, this form is not permitted for non-nullable types, as the compiler will not allow null to represent an instance of such a type. One workaround is to convert to System.Object first, i.e. (box x) = null. This is perfectly functional, even for non-nullable types, but it is still slow.

Don’t use x = Unchecked.defaultof<_>

When the need arises to null-check a non-nullable type, one might be tempted to compare directly against Unchecked.defaultof<_>, rather that converting to System.Object and comparing against a null literal.

Unfortunately, this approach falls on its face when used with functional-style record or discriminated union types. Due to its baked-in notion of immutability and non-nullability for such types, the compiler assumes that it is always safe to codegen their equality comparisons as simply x.Equals(y). That’s fine until x is null, in which case the null-check itself blows up with a NullReferenceException!

F#:

type Cat = { Name : string; Lives : int }

let nullCheck02 (x : Cat) = (x = Unchecked.defaultof<_>)

Codegen (C# equivalent):

public static bool nullCheck02(Cat x)
{
    Cat obj = null;

    // nullref
    return x.Equals(obj, LanguagePrimitives.GenericEqualityComparer);
}

Don’t expect to catch NullReferenceException

Let’s say you decide that you’ll mostly ignore the possibility of nulls, and but harden your code by catching NullReferenceException.

F#:

type Cat = { Name : string; Lives : int }

let ohNoes x =
    try
        if x.Lives > 1 then Some({ x with Lives = x.Lives - 1})
        else None
    with
    | :? NullReferenceException -> None

ohNoes (Unchecked.defaultof<_>)

Most F# developers will be surprised to find that this still fails with an unhandled NullReferenceException.

How is that possible? NullReferenceException is clearly handled!

Due to Cat’s immutability/non-nullability, the compiler believes the contents of the try block cannot fail. Thus it performs an “optimization” by removing the try/catch altogether. Indeed, if you inspect the generated IL, you will find that the try/catch block simply isn’t generated:

Codegen (C# equivalent):

public static FSharpOption<Cat> ohNoes(Cat x)
{
    if (x.Lives > 1)
    {
        return FSharpOption<Cat>.Some(new Cat(x.Name, x.Lives - 1));
    }
    return FSharpOption<Cat>.None;
}

FWIW this is an altogether bogus optimization anyway, and has been removed in F# 4.0.

DOs

Do pattern match against null

Pattern matching against null results in much better generated IL than comparison with = or <>. Nullable types can be matched directly, non-nullable types must be converted to System.Object first by using box.

F#:

type Cat = { Name : string; Lives : int }

let nullCheck03 x = match x with null -> true | _ -> false
let nullCheck04 (x : Cat) = match box x with null -> true | _ -> false

Codegen:

// nullCheck04 is identical except generic "a" becomes concrete "Cat"
.method public static bool nullCheck03<class a> (!!a x) cil managed 
{
    .maxstack 8

    IL_0000: ldarg.0
    IL_0001: box !!a
    IL_0006: brfalse.s IL_000a

    IL_0008: ldc.i4.0
    IL_0009: ret

    IL_000a: ldc.i4.1
    IL_000b: ret
}

The JIT produces exactly the same native code for both of these functions:

          ; if x is null, go to 00D50503
00D504FB  test        ecx,ecx
00D504FD  je          00D50503  

          ; set return value to 0 (false)
00D504FF  xor         eax,eax  
          ; return 

          ; set return value to 1 (true)
00D50503  mov         eax,1  
          ; return

Do use Object.ReferenceEquals(x, null)

Another way to get efficient null-checking is to call the BCL method Object.ReferenceEquals.

F#:

let nullCheck05 (x : 't when 't : null) = System.Object.ReferenceEquals(x, null)

Codegen:

.method public static bool nullCheck05<class t> (!!t x) cil managed
{
    .maxstack 8

    IL_0000: nop
    IL_0001: ldarg.0
    IL_0002: box !!t
    IL_0007: ldnull
    IL_0008: call bool [mscorlib]System.Object::ReferenceEquals(object, object)
    IL_000d: ret
}

 // --- BCL ---

.method public hidebysig static bool ReferenceEquals (object objA, object objB) cil managed 
{
    .maxstack 8

    IL_0000: ldarg.0
    IL_0001: ldarg.1
    IL_0002: ceq
    IL_0004: ret
}

At the IL level this appears to suffer from the overhead of an additional method call, but ReferenceEquals is small enough that it will be inlined by the JIT. The resulting native code winds up being at least as good as our earlier efforts, perhaps even better:

          ; if x is null, set the lowest byte of return value to 1
          ; otherwise set it to 0
012404A3  test        ecx,ecx  
012404A5  sete        al  

          ; fill out the rest of return value with 0s
012404A8  movzx       eax,al
          ; return

By some measurements, Object.ReferenceEquals (or pattern-matching) is 7-8x faster than comparison with = or <>.

Do use isNull

In F# 4.0, a new helper function isNull has been added, which has the same pattern matching implementation as nullCheck03 above. It only applies to nullable types, so you will need to box non-nullable types first. box and isNull are both inline functions, so either way the final JITed code is the same as shown above.

Do use Option.ofObj

In F# 4.0, a new helper function Option.ofObj has been added, which will convert a nullable object into an option instance, based on whether it’s null or not. This adds a bit of perf overhead, but is very handy for when you’d prefer to lean on the type system rather than worry about nulls.

A handy Powershell filter for converting plain text to objects

Fri, 15 May 2015 00:00:00 +0000

Working on the command line with Powershell, much of the time I have the luxury of dealing directly with rich .NET objects. If I need to sort, filter, or otherwise process cmdlet output, I have easy access to typed properties and methods right at the prompt.

Often, though, I’ll need to wrangle plain text, perhaps from a log file or the output of an executable. In these cases an intermediate step is required in order to extract the typed information (timestamps, substrings, numerical fields, etc) from the plain strings.

This comes up often enough that I whipped up a handy Powershell filter, ‘ro’ (for ‘regex object’), to make it easy:

# converts text to objects via regex,
#  with properties corresponding to capture groups
filter ro
{
    param($pattern)
    
    if($_ -match $pattern)
    {
        $result = @{}
        $matches.Keys |?{ $_ } |%{
            $raw = $matches[$_]
            $asInt = 0
            $asFloat = 0.0
            $asDate = [datetime]::Now
            if([int]::TryParse($raw, [ref] $asInt)){ $result[$_] = $asInt }
            elseif([double]::TryParse($raw, [ref] $asFloat)){ $result[$_] = $asFloat }
            elseif([datetime]::TryParse($raw, [ref] $asDate)){ $result[$_] = $asDate }
            else{ $result[$_] = $raw }
        }
    [pscustomobject]$result
    }
}

This takes each line of text piped to it and attempts to match it with the provided regex pattern. If it matches, an object is created with properties corresponding to capture groups of the match (ignoring group 0, which represents the full match). It will even check if the capture group can be parsed as an int, float, or timestamp, in which case that strongly-typed value is used in favor of the flat string value.

I get a lot of mileage out of this guy. As an example, when producing the Visual F# and Roslyn contributor mugs, we wanted to etch the username and SHA of each contributor’s first commit.

Using just the basic git tools and ‘ro’, this info can be produced in a 1-liner.

In this case I got to pick convenient delimiters, so I could be very sloppy with my regex! 🙂

Bug with Powershell Get-Date

Wed, 01 Apr 2015 00:00:00 +0000

5 years ago today, I sent the following bug report to the Powershell team.

From: Lincoln Atkinson Sent: Thursday, April 01, 2010 10:33 PM To: PowerShell Discussions Subject: Problem with get-date Hey experts, maybe you can help me with this issue. Get-Date seems totally broken for me. I've tried it multiple times with all different parameters, but it's just not working in my environment. I'm still single and I haven't gotten a date in months :-( Get-Help really wasn't useful this time. Funny, since many of the ladies I wanted to Get-Date with specifically suggested I “get-help.” A few told me to try “get-life” and “get-bent,” as well, but I couldn't find those cmdlets. They must be running some V3 beta. I've debugged this a bit, and have eliminated some possibilities: Localization – I live in Seattle, seems there should be a pretty good social scene there. Build – I'm average height/weight, don't see this as holding me back. Concurrency – In our modern era of tolerance I'd like to think this isn't a race condition. Hardware – Not the issue. Low Resources – Get-Job worked ok for me so I have at least a little money saved. Any thoughts? I just want to allocate some memories with someone and maybe attend a few advanced functions. -Lincoln

The report was forwarded to the Powershell MVPs, who weighed in with reasoned opinions and sage advice.

Tibor Soós – And if there is a parent object, he should also use the –confirm switch.
Hal Rottenberg – I do NOT recommend concurrent job scenarios with this cmdlet. I have heard of it being done, but it never ends well.
Hal Rottenberg – I just hope he didn’t try the Force parameter. That one has some illegal page fault regression bugs in v2.
Susan Bradley – Not to mention you really want to run with antivirus or have some protection in this scenario.
Oisín Grehan – His deduction on localization might be flawed: Get-Date typically only works at places where Get-Culture returns a non-null value. [ouch!]
Tobias Weltner – You also want to make sure not to use the undocumented –recurse switch which leads to unexpected results.

Non-transitive Grime Dice, via Mathematica

Fri, 16 Jan 2015 00:00:00 +0000

For Christmas this year, I got myself a fun mathematical gift: a set of 10 non-transitive dice, namely Grime Dice! You can get your own set here. Behold their dicey splendor:

These dice possess the fascinating property that their winning relationships (in the sense of “winning” = “rolls a higher number > 50% of the time”) are non-transitive. i.e. if die A wins against die B, and die B wins against die C, it actually does *not* hold, in general, that die A wins against die C. In fact, die C might win against die A!

If we label the 5 Grime Dice colors Red, Blue, Yellow, Olive, and Magenta, there are 2 primary non-transitive winning cycles

By word length: Red beats Blue beats Olive beats Yellow beats Magenta beats Red
Alphabetically: Blue beats Magenta beats Olive beats Red beats Yellow beats Blue

That’s pretty neat and non-intuitive by itself, but things get weirder when you roll two dice of the same color together: the word length cycle reverses, while the alphabetical cycle (almost) stays intact.

By word length (doubles): Red/Red loses to Blue/Blue loses to Olive/Olive loses to Yellow/Yellow loses to Magenta/Magenta loses to Red/Red
Alphabetically (doubles): Blue/Blue beats Magenta/Magenta beats Olive/Olive loses to Red/Red beats Yellow/Yellow beats Blue/Blue

Using Mathematica, we can calculate the winning probabilities, and visualize the cycles:

These 4 cycles are the only ones which are advertised, but it turns out there are many more that you can form using just the 10 dice included in the set.

In fact, there are 298 such cycles! Here are the plots of all 1-die cycles, all 2-dice cycles, and all 3-die cycles.

In this post, we’ll walk through the Mathematica code used to create these plots and to compute the complete set of possible cycles.

Modeling dice

The first step is to provide a simple representation of the different dice colors, each of which has a unique pip configuration:

(* represent the dice, their names, and their face values *)
red = dice["Red"] = {{"Red"}, {4, 4, 4, 4, 4, 9}};
blue = dice["Blue"] = {{"Blue"}, {2, 2, 2, 7, 7, 7}};
olive = dice["Olive"] = {{"Olive"}, {0, 5, 5, 5, 5, 5}};
yellow = dice["Yellow"] = {{"Yellow"}, {3, 3, 3, 3, 8, 8}};
magenta = dice["Magenta"] = {{"Magenta"}, {1, 1, 6, 6, 6, 6}};

To compute which of two dice beats the other, and the odds of that win, we generate every possible roll between the two dice, and see which one comes out on top more often:

(* compute which of two dice would win, including the odds *)
(* returns {winner -> loser, odds} *)
compareDice[{lName_, lVals_}, {rName_, rVals_}] := (
  rolls = Tuples[{lVals, rVals}];
  winDiff = Total[rolls /. {l_, r_} -> Sign[r - l]];
  odds = 1/2 +  Abs[winDiff]/(2*Length[rolls]);
  {If[winDiff > 0, rName -> lName, lName -> rName], N[odds, 3]}
);

Thus we can see that, for example, Red beats Blue 58% of the time, and Yellow beats Magenta 56% of the time.

compareDice[red, blue]
compareDice[magenta, yellow]
 
(* output:
    {{"Red"} -> {"Blue"}, 0.583}
    {{"Yellow"} -> {"Magenta"}, 0.556}
*)

Non-transitive cycles

We are already in a position to verify and quantify the primary word-length and alphabetical cycles with single dice:

byWordLength = {{red, blue}, {blue, olive}, {olive, yellow}, {yellow, magenta}, {magenta, red}};
byAlpha = {{blue, magenta}, {magenta, olive}, {olive, red}, {red, yellow}, {yellow, blue}};

compareDice @@@ byWordLength
compareDice @@@ byAlpha

(* output:
  {{{"Red"} -> {"Blue"}, 0.583}, {{"Blue"} -> {"Olive"}, 0.583}, {{"Olive"} -> {"Yellow"}, 0.556}, {{"Yellow"} -> {"Magenta"}, 0.556}, {{"Magenta"} -> {"Red"}, 0.556}}
  {{{"Blue"} -> {"Magenta"}, 0.667}, {{"Magenta"} -> {"Olive"}, 0.722}, {{"Olive"} -> {"Red"}, 0.694}, {{"Red"} -> {"Yellow"}, 0.722}, {{"Yellow"} -> {"Blue"}, 0.667}}
 *)

We can represent and compare pairs of dice as if they were each one 36-sided die, with each face corresponding to the total from a possible roll of the constituent dice. This lets us compute the odds of the word-length and alphabetical double dice cycles, too:

(* create a new "die" by combining two dice *)
combine[{name1_, vals1_}, {name2_, vals2_}] := {Join[name1, name2], Plus @@@ Tuples[{vals1, vals2}]};
double[die_] := combine[die, die];

compareDice @@@ Map[double, byWordLength, {2}]
compareDice @@@ Map[double, byAlpha, {2}]

(* output
  {{{"Blue", "Blue"} -> {"Red", "Red"}, 0.590}, {{"Olive", "Olive"} -> {"Blue", "Blue"}, 0.590}, {{"Yellow", "Yellow"} -> {"Olive", "Olive"}, 0.691}, {{"Magenta", "Magenta"} -> {"Yellow", "Yellow"}, 0.593}, {{"Red", "Red"} -> {"Magenta", "Magenta"}, 0.691}}
  {{{"Blue", "Blue"} -> {"Magenta", "Magenta"}, 0.556}, {{"Magenta", "Magenta"} -> {"Olive", "Olive"}, 0.583}, {{"Red", "Red"} -> {"Olive", "Olive"}, 0.518}, {{"Red", "Red"} -> {"Yellow", "Yellow"}, 0.583}, {{"Yellow", "Yellow"} -> {"Blue", "Blue"}, 0.556}}
*)

Plotting

It’s much nicer to visualize the winning relationships between the dice, rather than just printing out the data. Mathematica has excellent plotting and visualization capabilities, so this is certainly possible.

GraphPlot is a good choice here. Its default visual output isn’t very well-suited to this problem, though, so we will need to do some customization. We can take advantage of the various hooks which are exposed by the function, enabling us to specify custom graphical objects to represent the vertices and edges of the relationship graph.

The below code will create nice graph plots where the vertices are represented by appropriately-colored dice icons, and the edges point from winner -> loser and are labeled with the probability of that win.

(* keep track of which colors should be used in plots *)
colors["Red"] = Red;
colors["Blue"] = Blue;
colors["Olive"] = Green;
colors["Yellow"] = Yellow;
colors["Magenta"] = Purple;

(* plot colored rectangles to represent the dice at a graph vertex *)
getVertex[center_, names_] := (
  numDice = Length@names;
  positions = {-0.08 + #, 0.08 + #} & /@ 
    Range[-0.04*(numDice - 1)/2, 0.04*(numDice - 1)/2, 0.04];
  Transpose[{colors /@ names, Rectangle[center + #1, center + #2, RoundingRadius -> 0.02] & @@@ positions}]
  );

(* plot a nicely-formatted labeled arrow for graph edges *)
getEdge =
  ({Gray, If[#3 == 0.5, Line[#1], Arrow[#1, 0.15]], Black, 
     Inset[#3, Mean[#1], Background -> White]} &);

(* given a list of dice pairs, creates a nicely-formatted plot of
winning relationships and odds *)
plotDice[pairs_] :=
 GraphPlot[compareDice @@@ pairs,
  VertexRenderingFunction -> getVertex,
  EdgeRenderingFunction -> getEdge];

Let’s take a look at the single and double cycles visually:

Pretty neat! Besides various oddities with orientation and ordering, these plots are quite appealing. Exact placement of the vertices can be specified by the VertexCoordinateRules parameter to GraphPlot, but the default layout works well enough for our purposes.

More cycles

We have looked at the primary 5-color cycles using both single dice and doubles of the same color. That’s just the beginning, though. For example, besides the 5-color cycles, various smaller cycles also exist:

How many of these smaller cycles exist? What about bigger cycles? And what about cycles involving doubles composed of 2 different colors? Or even cycles consisting of sets of 3 dice? We want to compute every possible cycle that can be created using the 10 dice from the set.

Our overall approach to solving this will be to generate directed graphs which encode all of the winning relationships between unique dice sets of a certain size (single dice, pairs, or triples), then search for cycles within those graphs.

It should be noted (we won’t prove it here) that for Grime Dice, any pair of dice beats any single die, and any triple of dice beats any pair or single die. Thus it is indeed acceptable to split this computation up into separate buckets for single dice, pairs, and triples. There are no heterogeneous cycles with respect to number of competing dice.

Single dice

One might assume that the cycles of single dice would be the easiest to compute. In fact, single dice pose a couple of unique challenges that pairs and triples do not. Specifically, in a set of 10 dice, we could potentially find cycles up to size 10. But since we only have 5 colors, once a cycle becomes length-6 or longer we must necessarily have 2 same-color nodes in the cycle. We need to make sure to differentiate between the two copies of each color.

In order to capture the fact that we have 2 copies of each color on hand, we will use a bit of a hack. The “second” copy of each color will be represented as a combination with a special “white” die which has 1 face and always rolls 0.

(* dummy "white" die used to differentiate between 
two instances of the same color die *)
white = dice["White"] = {{"White"}, {0}};

(* when plotting, just make the white die invisible *)
colors["White"] = Transparent;

(* all distinct single dice from set of 10 *)
allDice[1] = 
  Join[allColors, combine @@@ Tuples[{allColors, {white}}]];

To start building the actual relationship graph, we will define a couple of helper functions. The first is used to create the DirectedEdge values Mathematica consumes when Graph is called in the second function. The edges are directed from “winning di©e” to “losing di©e”.

(* Note that we don't return an edge here if the 2 dice are equally matched *)
getGraphEdge[left_, right_] := (
  {relationship, odds} = compareDice[left, right];
  If[odds != 1/2, relationship /. Rule -> DirectedEdge]
);

(* builds the graph of winning relationships for
n-tuples of dice *)
makeGraph[n_] := 
 Graph[Cases[getGraphEdge @@@ Subsets[allDice[n], {2}], DirectedEdge[__]]];

We can now generate the full graph of single dice relationships, and have Mathematica compute all cycles up to maximum size of 10. In the last step, note that we need to deduplicate the cycle list to eliminate those cycles which are unique only due to inclusion of the dummy “white” die.

diceGraph[1] = makeGraph[1];

(* built-in function DeleteDuplicatesBy is present only
in Mathematica 10+ *)
deDupeBy[expr_, f_] := Values[GroupBy[expr, f, First]];

(* compute all cycles of single dice that can
be made from the 10 included dice *)
cycles[1] = 
  deDupeBy[FindCycle[diceGraph[1], 10, All], 
   Sort[(# /. {e_, "White"} -> {e})] &];
   
CountsBy[cycles[1], Length]
cycles[1] // Length

(* output:
  <|3 -> 5, 4 -> 5, 5 -> 2, 6 -> 15, 7 -> 20, 8 -> 20, 9 -> 10, 10 -> 3|>
  80
*)

We see that there are a total of 80 unique single-die cycles, with sizes ranging from 3 to 10.

Pairs of dice

Dice pairs turn out to be the simplest case.

With pairs (and above), we do not need to consider the possibility of distinct-yet-identical nodes in the cycle. Proof: Between any two identical nodes, there must be at least 2 other nodes (if there was only one node, it would be simultaneously beating and losing to identical nodes on either side), so a full cycle with identical nodes must have length at least 6 (the 2 identical nodes + 2 separating nodes on each side). When each node consists of a pair of dice, this requires at least 12 dice. Since we only have 10 dice, this is impossible.

This eliminates the need for the dummy die, as well as the de-duplication at the end.

The only additional wrinkle we need to consider is the possibility for a computed cycle to contain more than 2 dice of a particular color. Such cycles are invalid in our scenario, since we are only utilizing the 10 dice in the set. We will update our helpers and add some additional filtering to eliminate such cycles.

Finally, for pairs, we only need to search for cycles up to length 5.

(* all unique dice pairs *)
allDice[2] = Flatten[Table[combine @@ allColors[[{i, j}]],
    {i, 1, Length[allColors]},
    {j, i, Length[allColors]}], 1];

(* updated to avoid creating edges between nodes 
that combine to use more than 2 of any color *)
getGraphEdge[left_, right_] :=
 If[FreeQ[Tally[Join[left[[1]], right[[1]]]], {_, count_} /; count > 2],
  {relationship, odds} = compareDice[left, right];
  If[odds != 1/2, relationship /. Rule -> DirectedEdge]
 ];

(* check if a given full cycle uses more than 2 of
any particular color *)
isValidCycle[cyc_] :=
  FreeQ[Tally[Flatten[cyc /. DirectedEdge[a_, _] :> a]], {_, count_} /; count > 2];

(* compute all cycles of pairs of dice that can
be made from the 10 included dice *)
diceGraph[2] = makeGraph[2];
cycles[2] = Select[FindCycle[diceGraph[2], 5, All], isValidCycle];

CountsBy[cycles[2], Length]
cycles[2] // Length
(* output:
  <|3 -> 55, 4 -> 89, 5 -> 25|>
  169
*)

There are 169 unique cycles using pairs of dice.

Triples of Dice

Triples are the largest sets we need to consider. At least 3 nodes are required to form a cycle, and if those nodes consist of 4 or more dice each, our set of 10 dice will not be sufficient.

Similarly, we need only search for cycles of length 3 here – a cycle of 4 triples requires more dice than we have.

We start by extending our dice-combining function to handle triples, building up all such unique triples, and generating their winning relationship graph. Note that any triples where all 3 dice are the same color are invalid, and should be filtered.

(* extend to handle triples *)
combine[die1_, die2_, die3_] := combine[die1, combine[die2, die3]];

(* all unique dice triples *)
allDice[3] = Select[
   Flatten[Table[combine @@ allColors[[{i, j, k}]],
     {i, 1, Length[allColors]},
     {j, i, Length[allColors]},
     {k, j, Length[allColors]}], 2],
   Length@Union@#[[1]] != 1 &];

diceGraph[3] = makeGraph[3];

From here, computing the triple cycles should be as simple as calling FindCycle again. Unfortunately, Mathematica spins (seemingly) indefinitely when one tries this. The relationship graph for triples is 30 nodes and 208 edges - not trivial, but not really that big. I’m not sure why FindCycle has trouble with it. Oddly enough, FindCycle immediately finds 1 cycle if that’s all you ask for, but exhibits the hang if you ask for even just 2 cycles, let alone all of them.

So we will need to search for the 3-cycles in this graph manually. The below code does the trick.

(* for each edge in the graph, collect potential second edges
e.g. for edge A -> B, find all pairs {{A -> B, B -> X},{A -> B, B -> Y}, ...} *)
edgePairs = 
  Flatten[EdgeList[diceGraph[3]] /. 
    DirectedEdge[a_, b_] :> ({DirectedEdge[a, b], #} & /@ 
       EdgeList[diceGraph[3], DirectedEdge[b, _]]), 1];

(* find and validate the 3rd and final edge of a 3-cycle.
e.g. given {A -> B, B -> C}, check that C -> A exists, and 
the cycle A -> B -> C -> A is valid *)
completeCycle[DirectedEdge[a_, b_], DirectedEdge[c_, d_]] := (
   lastEdge = DirectedEdge[d, a];
   If[MemberQ[EdgeList[diceGraph[3]], lastEdge], (
     cycle = {DirectedEdge[a, b], DirectedEdge[c, d], lastEdge};
     If[isValidCycle[cycle],
      Sow[cycle]
     ])
   ]
);

This enables us to compute the cycles, though we do need to deduplicate them (unlike FindCycle, our manual code is not smart enough to realize the cycle A -> B -> C ->A is the same as the cycle B -> C -> A -> B).

cycles[3] = deDupeBy[Reap[Scan[completeCycle @@ # &, edgePairs]][[2, 1]], Sort];

CountsBy[cycles[3], Length]

(* output:
  <|3 -> 49|>
*)

There are 49 triple-dice cycles. This brings the grand total to 298 unique non-transitive cycles in a set of 10 Grime Dice.

Plot all the cycles!

Finally, the fun part - making giant plots of every possible cycle!

To plot a single cycle, we just need to massage the data a little bit so that it works with plotDice from earlier.

(* 'combining' a single die is a no-op *)
combine[{name1_, vals1_}] := {name1, vals1};

(* plot a single non-transitive dice cycle *)
plotCycle[cyc_] :=
  plotDice[cyc /. DirectedEdge[l_, r_] :> {combine @@ (dice /@ l), combine @@ (dice /@ r)}];

A couple of example plots:

To generate the full plots, as linked at the top of the post, all we need is

(* plot everything! *)
plotCycle /@ cycles[1]
plotCycle /@ cycles[2]
plotCycle /@ cycles[3]

Today is Phi Day -- at least, it ought to be

Tue, 06 Jan 2015 00:00:00 +0000

By now, most everyone is aware of Pi Day, celebrating the famous mathematical constant on ³⁄₁₄. On this day each year, students and math enthusiasts eat pie and engage in light-hearted -related activities. Then there is e Day, a day for commemorating the equally-important-if-somewhat-less-famous constant on ²⁄₇. The activities are similar, though it’s less clear-cut what food one should eat (a high school teacher of mine insisted that the proper e Day food is waffles, evocative of the Cartesian coordinate system). Even events like Pi Approximation Day (²²⁄₇ in day/month format), and Mole Day (6:02 ¹⁰⁄₂₃) have gained enough momentum to warrant dedicated results from both Google and Bing.

So when is Phi Day?

, the Golden Ratio, is arguably the 3rd-most “pop-famous” mathematical constant after and . It has enough geek-cred status that you can buy shirts with hundreds of its digits on the front, arranged into the shape of the character. So you’d think Phi Day, whenever it is, must surely be marked on numberphiles’ calendars worldwide.

As it turns out, there are major disagreements over the date on which Phi Day should be held, and this lack on consensus has kind of killed the whole thing before it ever got started. The surprisingly long list of candidate dates has fractured the community such that a critical mass of participation on a single day has never been reached.

What are these different Phi Day camps?

The digits division

Following the precedent set by Pi Day on ³⁄₁₄ and e Day on ²⁄₇, the most straightforward choice for Phi Day is ¹⁄₆, based on the leading decimal digits of the constant. And indeed, there is some support for this date: see here, here, and here.

Interestingly, the more popular digit-based candidate seems to be ⁶⁄₁₈. Some basic web searches turn up a number of results, e.g. here, here, here, here, here, and here. The standard explanation for this choice is that it matches the first digits after the decimal point, or that it matches the leading digits of the reciprocal form .

The fraction faction

There is another group which argues that picking the date of Phi Day based on decimal digits is passé and should be avoided. Maybe that’s good enough for Pi Day and e Day, they say, but with Phi Day we have an opportunity for something more meaningful, man!!

The idea uniting this faction is to pick Phi Day as the date which divides the year into two parts, where the parts are in proportion matching the Golden Ratio.

This is actually a neat idea, but sadly even this camp shatters into further subdivisions.

Taking the year to be 365 days, one gets closest to a division in the ratio of by taking the 226th day of the year (226/(365 - 226) is about 1.626), which is ⁸⁄₁₄. See here and here for advocacy of this date.

But what about leap years? In those years there are 366 days, of which 8/14 is the 227th. In this case the previous day, ⁸⁄₁₃, is actually a better pick, as 226/(366 - 226) = 1.614 comes closer to the desired ratio than 227/(366 - 227) = 1.633.

Thus ⁸⁄₁₄ is rejected by some in favor of the more robust alternative where the shorter division is put first. ⁵⁄₁₉ is the 139th day of standard years, and the 140th day of leap years. Happily, this ends up being the ideal pick for both cases: (365 - 139)/139 = 1.626 and (366 - 140)/140 = 1.614. See here for advocacy of this date.

Finally, this prominent site catapults the navel-gazing to stunning new heights by condemning the Gregorian calendar itself as too mainstream, declaring that the Phi Day year-division should be measured starting from a more “natural” delimitation of the annum, namely the vernal equinox.

But wait, doesn’t the Spring equinox occur on different dates in the Northern and Southern Hemisphere? Indeed, this site goes so far as to advocate for two separate Phi Days: ¹⁰⁄₃₁ in the Northern Hemisphere, ⁵⁄₆ in the Southern Hemisphere (where the dates are computed the same way as ⁸⁄₁₄, but starting from the respective equinoxes).

Keep it simple

My stance is that Phi Day belongs on the obvious choice: ¹⁄₆. There is strong precedent in Pi Day and e Day, it’s easy to remember, and there are no worries about leap years or hemispheres. It’s accessible and easy to explain to anyone.

⁶⁄₁₈ strikes me as an arbitrary choice to ignore digits, or to prefer the secondary, reciprocal form. I just don’t see what’s gained by that choice. A further disadvantage is that many schools (in the US, at least) are already out for summer break by this time, so this date can’t really be used as a fun day for students.

And all of the year-fraction dates can be dismissed outright, if you ask me. Yes, explaining their position within the year provides a potential teaching opportunity, but that computation (let alone the overall explanation) is just totally inaccessible to most people. I had to break out a bit of code and lean on a date/time library just to verify these dates, and I assume most others would need to do the same. When there is such a high hurdle to even figure out what date the event is on, there’s no way it will ever gain traction outside of fanatics.

The ultimate goal of these silly “math holidays” is to provide a fun excuse for enthusiasts to celebrate, but also to create a friendly and inviting opportunity for non-enthusiasts to learn a little bit and join the fun without feeling intimidated. ¹⁄₆ seems like the clear choice.

Happy Phi Day!