Functional Design in Clojure

View Top 200 Chart

Back to Rankings Page

Technology
**Functional Design in Clojure**

Each week, we discuss a software design problem and how we might solve it using functional principles and the Clojure programming language.

Latest Episodes:

Episode 009: Channeling Re-Search Dec 28, 2018

Show notes

Nate can't decide what to watch on Twitter, and the app restarts are killing him.

The Twitterthon continues. "It's an infinite stream of Twitter."
Nate wants to watch #clojurescript too.
Just change the search string to "#clojure OR #clojurescript"?
Should avoid hardcoding the value so we don't have to recompile every time.
Command line argument still requires a restart.
Let's use a curl UI (like Ep 004)!
Wait, what?! Why curl?
- Can send something into the running process
- Can separate out the client
- Builds on top of HTTP, so there are lots of tools for testing (like curl itself)
Use a URL query string: http://localhost:8000/search?The+search+string
"It's a true query string in the truest sense of the term 'query.'"
"It is actually using the thing for the thing it was meant to be."
How do we get the query string from the webserver thread to the polling loop thread?
"This is a perfect case for a global." Oh wait, that's mutation.
How should we structure the app? The main function must:
- start the thread for the web server
- start the thread for the polling loop
Specific problem: the /search route handler function needs a way to send the new query string to the polling loop.
With mutation: share an atom and give the handler and the polling loop a reference to it.
No mutation? Use a core.async channel.
A channel allows a thread to pass data to another thread.
With no channel buffer, it will synchronize two threads ("rendezvous").
(We lie. There is no peek function.)
Problem: polling thread is stuck waiting on the channel, so it stops polling.
Solution: Use alt!! to simultaneously listen to the "new search" channel and a timeout channel.
What is a timeout channel? A channel that closes after n msecs have passed.
New problem: the cache (for de-duplicating) has stale content after changing the query.
Solution: same logic that adopts the new search term will reset the cache. (See Ep 007.)
Polling loop structure:
1. fetch
2. process results
3. block while listening to the channels
4. if new search string, recur with new cache and new search string
5. otherwise, recur with same cache and search string
Only want the fetch in one part of the loop.
Don't even need curl. Just type in the URL for the backend on your phone.
core.async lets threads coordinate without having to know about each other!

Clojure in this episode:

atom
loop, recur
core.async/
- chan
- <!, <<!, >!, >>!, put!
- alt!, alts!
- timeout

Related projects:

Episode 008: Twitter, Plated Dec 21, 2018

Show notes

Christoph tries to figure out why Twitter stopped talking about Clojure.

"Are you twitterpated?"
Building on where we left off last episode.
Runs and just stops working.
"I was pretty sure it stopped working because people on Twitter just stopped talking about Clojure. After about a day of that, I realized people were talking about Clojure, and I just wasn't seeing it."
The auth token expired! What should we do?
Why should the main loop have to deal with getting a new auth token?
"The Twitter wrapper should be concerned with all of the warts and complexities of dealing with Twitter."
"What problems should bubble up, and which ones shouldn't?"
The wrapper should handle the retry.
It's like a kitchen in a restaurant. What are the steps of fulfilling an order? The customer doesn't care.
"There's a side-effect: the freezer mutates."
The wrapper gets to worry about all the steps:
- turning the order into the specific request for the kitchen
- do the I/O to fetch and fulfill the request
- the "input transform" takes the mass of data and picks out the relevant parts
- the "internal" version is returned
"Like all good metaphors, they stretch to the point where they break, like a rubber band."
Maybe avoid expired tokens by authenticating every time? Too much overhead.
If the handle is mutable, then retry can just update the handle with the new token.
A mutable handle does allow the wrapper to control the concern.
The "handle" is the state of the wrapper. The term "handle" comes from I/O libraries.
Instead of mutation, have the search function return [updated-handle, result].
search can catch an auth exception, retry, and return a new auth handle.
Instead of search retrying, the fetcher can do it! Then it works for all kinds of requests.
Better yet, leave fetch simple, and have a fetch-with-retry function that uses fetch.
Can have even more policy functions like, fetch-with-retry-forever.
"Keep calm, and assoc on."
"I'm never going to miss another Clojure tweet. I'm going to read them all!"

Clojure in this episode:

loop, recur
try, catch
atom
assoc

Episode 007: Input Overflow Dec 14, 2018

Show notes

Nate just wants to print the tweets, but the input wants to take over.

Got tweet fetching working last time. Now we want to print out the tweets.
API returns a lot information
"We should probably not use pprint as our output mechanism."
Using the uniqueness filtering from last time
The goal is NOT to write a generic Twitter wrapper.
Goal is to write a wrapper that just gives us what our application needs.
"I don't want to spread the complexity of Twitter all over my app. I want to hide it inside of the wrapper."
Clojure data structures are generic enough already!
We pick out just what we need and put it in our own data structure.
We use our "internal" structure throughout the application code.
Our internal structure protects our application code from changes in Twitter's format.
Keep the structure minimal and grow it when the application code needs more fields.
Where should we put the internal model? Make it a part of the wrapper.
A wrapper should expose a data model
A "sequencing" function just threads steps defined by other functions. Pure steps are easy to test, so make the I/O steps as minimal as possible.
Technique is called "pushing I/O to the edges".
Let any exceptions fly back to the main loop. It can sleep a little and retry.
Separate out formatting from println, so you can unit test output.
Put the cache behind its own data model
The cache data model provides logical operations like (defn filter-new [cache tweets] ...)
The filter-new function would return [cache new-tweets]. Cache could have changed.
How do you know what's in the data model exposed by the wrapper?
"What is in this thing that we're passing around? You don't just want to read the code and figure it out?"
"You rapidly get to the point where you don't know what's in your data structure."
Use schema.
Put the schemas the wrapper exposes in its own namespace.

Parts of a data model

the data structure: what that model needs to store or represent
a schema (or spec) to define and validate the data structure
"constructor" functions that take args and return a data structure
"update" functions that take the data structure and return a modified one
"output" functions that take the data structure and return some other representation

Parts of a wrapper

Application-facing functions. Sole job is to sequence internal wrapper functions.
Data model for a Twitter API "request"
Functions that execute a request and return raw data
Data model exposed by the wrapper

Clojure in this episode:

println
get-in
->
loop, recur, reduce
lists as "tuples"
schema/defschema, schema/Int, schema/String, ...

Related projects:

Episode 006: All Wrapped Up in Twitter Dec 07, 2018

Show notes

Christoph tries to get a handle on his #clojure tweet-stream habit.

NOT tic-tac-toe
Follow #clojure tweet stream and see it print out in the terminal
"We like reinventing things."
"The terminal is the best UI out there."
Does Twitter have an API?
Websocket? Nope! Requires a big "E" plan: "enterprise".
PubSub? Nope! Not from Twitter.
Alas, we must poll the /search/tweets.json endpoint
Problem: we're going to keep getting results we've already seen
Avoid duplicates? Let's use core cache.
Once again, we use loop and recur for our main loop
Time for an API wrapper, but what does the wrapper do?
HTTP POST form-encoded parameters
Ack! 401 Access Denied
"An important step in any API exploration is your first 401 response."
"OAuth?" "Oh....auth..."
Meet OAuth, the API bouncer.
Make an auth function to call the OAuth endpoint and get an auth token
Have auth return a "handle" with the auth token. Other wrapper functions will need handle.
Need to keep handle around. Put that in the app state too.
Let the exceptions fly!
"Exceptions are an exceptionally accepted way of handling exceptional circumstances."
"I caught what you meant."
Make a fetch function that does the I/O work.
Create a search function that takes handle and query
Look for pure logic and move it into its own function, then it's easy to test.
Transform args to search into a "request description" and have fetch operate on that.
"Twitch, I mean Twitter. You know, that Internet thing that starts with t-w-i-t."
Different layers of the wrapper:
- Top-level functions used by the application. Sole job: sequence internal functions
- Pure transforms from args to "request description"
- A fetch function that follows the "orders" of the "request description"
"The point of testing this code is not to test if Aleph works or Twitter works. The logic is in the translation function, so that's why we test it."
"Our hero, the Twitch wrapper, is poised and ready to fetch data on our behalf. What will happen next?"

Clojure in this episode:

loop, recur
Thread.sleep
get
try, catch
->

Related projects:

Episode 005: Tracking, for the Win Nov 30, 2018

Show notes

Nate tries to figure out who actually won this never-ending game of tic-tac-toe.

Tic-tac-toe is "just boring enough, just interesting enough."
How do we know who won the game? Inspect the board.
If you can track progress toward the win, you check for the win quickly
"Tic-tac-toe at scale!"
Tracer bullet: go simple, just examine the 8 options for winning
"In that case, nil has won...which is nobody."
Keep detection logic out of the high-level winner function--should read like a process description of steps.
Make new "verbs" like row-winner and column-winner and use those.
"You're just adding new verbs to raise Clojure up to the level of your problem. You can speak about your problem using those verbs."
Let's make it faster! Need incremental detection to be efficient.
Tracking structure with win condition totals
- keys like: [player case index]
- value is a counter for that case
- eg. { [:x :row 0] 1, [:y :row 0] 0, [:x :diag] 2, ...}
The win tracker is a "nested model" of the game state
Put the tracker it its own namespace app.game.tracker
Use [:x 1 0] as the play
Nested updates: (update game-state :tracking tracker/update [:x 1 0])
How do we handle diagonals? Not all moves will increment those totals.
Make helpers for diag? and rdiag? to use in cond-> (see code below)
High-level functions describe the process. Low-level functions describe the steps.
"You can see the animal, not the intestines."
"If you see a word that's a higher level concept, it allows you to stay at that higher level and be able to view the algorithm instead of viewing the implementation. That's the point of lifting up all these little functions."
Bonus: the tracker tells us all the ways a player won.

Clojure in this episode:

nil punning streak: 3 episodes
get-in
update, update-in
or short circuits, = does not
->, cond->, some->
frequencies
lists as "tuples" and "triples"

Code sample from this episode:

(ns app.game.tracker)

(defn update
  [tracking [player row column]]
  (cond-> tracking
    true (record row column player)
    (diag? row column) (record-diag player)
    (rdiag? row column) (record-rdiag player)))

Episode 004: Atomic Curls Nov 23, 2018

Show notes

Christoph tries to make tic-tac-toe work over the Internet and discovers the power of the atom.

Let's get a web framework, build a UI, hook that up to an HTTP API, throw in some websockets for notifications!
"We'd end up with our first 12 hour podcast episode."
A "tracer bullet": get something functioning to see the important parts of the solution.
Can choose when to replace important parts by something more "production worthy".
"Let's just ditch all the complexity of having a UI and just use curl."
curl will print the text response. We have a terminal UI!
"We're extending the command line out to the web!"
"I just keep curling every day or every other day until it's my turn."
"It's your morning curls!"
How do we handle web requests? What is a route?
Routes: /new, /show, /play?row=0&col=1
"Super simple URL API. Who needs REST?"
Let's run this on port 1337 to hide it from the Internet.
One shared game state stored in an atom. It's the only game in town!
Use an atom instead of a database to cut complexity while problem solving.
The ! ("bang") in swap! and reset! indicates you're causing a side effect.
Handler's sole job: take web requests and alter the game state using the game model.
Function called by swap! should be pure. Don't throw exceptions!
Dilemma: How much do you put inside the function called by "swap!"?
For errors, the transaction function can:
- Return unchanged reference. Use "triage" function to diagnose.
- Have an "error" attribute in the state and set an "outcome" code like :success,:invalid-coordinate, etc.
Tracer bullet shows lots of ways to complex-ify this into "Tic-Tac-Toe, Enterprise Edition"

Clojure in this episode:

defonce
atom
deref
swap! and reset!
aleph and http-kit
compojure

Episode 003: Tic-Tac-REPL Nov 16, 2018

Show notes

Nate tries to turn the tic-tac-toe "game engine" into a real application he can play with a friend.

Let's play the game!
How do you keep track of the game state as it changes?
Bucket brigade the reference to the future loop using recur
The game loop: read input, evaluate, print out new board, loop.
"It has occurred to me that we are basically writing a REPL."
"We have the tic-tac-REPL"
How do you get input from the user? How to you make sure it's right?
"It keeps harassing the non-compliment user until they type the right thing in"
Input loop: read, validate, loop on error, return on success
Keep the logic pure! Separate out the parsing and validation functions.
"Much better to tuck it away in a function!"
Sequence the pure parts with a minimalist function that does the I/O.
"I don't like having ovens present because they're hard to put in my test cases."
Unit test those pure parts. (No one likes to be mocked.)
I/O is a side effect!
"Every time I redefine one of those things I feel like I'm reaching down into the bowels of Clojure and doing something moderately illegal."
Using keywords as error codes is nifty
"You don't have any different kinds of nil. You just have one. It's the nuh-uh."
Can use a tuple with the first element always being a keyword and the second being data for the "details"