Functional Design in Clojure

View Top 200 Chart

Back to Rankings Page

Technology
**Functional Design in Clojure**

Each week, we discuss a software design problem and how we might solve it using functional principles and the Clojure programming language.

Latest Episodes:

Ep 019: Dazed by Weak Weeks Mar 08, 2019

Show notes

Nate wants to see more than data structures in a REPL.

Goal: see a text-based calendar of hours worked by day and week.
"With the power of the predicate, I summon thee from the bag of data!"
If data items share structure, you only need one predicate function, not separate functions per "type".
Common pattern: filter then map, then reduce
Output:
- Looks like a calendar with one line per week
- Show daily total for each day
- Show weekly total
Problem: how do we split the days into weeks?
"I don't remember split-weeks in the Clojure cheat sheet."
"What day does your week start on? My week starts on Tuesday."
"I always assume that when a programmer says 'most', they mean 'most of the people around me'." "I actually think it just means 'me'."
"Anecdata: when you have two anecdotes, it makes data."
Create higher-level summaries: entry → day → week
Sift through data at the level of your question.
Problem: A date identifies a "day", but what identifies a week? A "week ID", so to speak.
It would be nice if that week ID was ordered, so we could sort on it.
Idea: Use the date of the first day of the week
- uniq, sortable, spans years
- weeks that start on Tuesday will never have the same ID as weeks that start on Sunday
Might as well have the starting day of the week in the map too: :sunday, :tuesday, etc.
Convenient to have two different views for the same data in the same data structure
- immutable data won't change and get inconsistent.
- it's pre-calculated if you need to use it a lot vs just using a predicate "on the fly".
Want to group-by the week:
- need a predicate from "date ID" → "week ID": (week-id starting-day-of-week day)
- need to take a "date" and produce "date of first day in week"
- Eg. (week-id :sunday {:date "2019-03-08" ...}) => "2019-03-03"
"You're teaching Clojure core how to describe your data by giving it vocabulary."
Write something that converts data to data, not data to println.
Principle: move the I/O to the edges
- make the println a trivial step at the end
- even making the strings is a data transform
- everything prior to the println could be tested, even the formatting!
"Test it to the level of detail you can tolerate."
Much easier to reason about pure functions.

Related episodes:

Clojure in this episode:

filter, map, reduce
group-by
sort-by with partition-by
take-while, drop-while, and split-with

Code sample from this episode:

(ns time.week-05
  (:require
    [time.week-04 :as week-04]
    [java-time :as jt]
    ))


; Date helpers

(defn start-of-week
  "Get the the starting date of the week containing local-date. The week will
  start on the named day. Eg. :sunday"
  [starting-day-of-week local-date]
  (jt/adjust local-date :previous-or-same-day-of-week (jt/day-of-week starting-day-of-week)))

(defn week-dates
  "Create a week's worth of dates starting from the given date."
  [starting-date]
  (->> (range 0 7)
       (map #(jt/plus starting-date (jt/days %)))))


; Aggregates

(defn sum-minutes
  "Sums the minutes for all kinds: entry, day, week"
  [entries]
  (->> entries
       (map :minutes)
       (reduce +)))


; Conversions

(defn entries->days
  "Convert a seq of entries to days."
  [entries]
  (->> (group-by :date entries)
       (map (fn [[date xs]] {:date date :minutes (sum-minutes xs)}))))

(defn days->week
  "Convert a seq of days into a week. Week is picked by first date."
  [starting-day-of-week days]
  (let [lookup (into {} (map (juxt :date identity) days))
        starting-date (start-of-week starting-day-of-week (:date (first days)))
        all-days (->> (week-dates starting-date)
                      (map #(or (get lookup %)
                                {:date %
                                 :minutes 0})))]
    {:starting-day-of-week starting-day-of-week
     :date starting-date
     :days (vec all-days)
     :minutes (sum-minutes all-days)}))

(defn partition-weeks
  [starting-day-of-week days]
  (->> days
       (sort-by :date)
       (partition-by #(start-of-week starting-day-of-week (:date %)))))

(defn days->weeks
  "Convert a seq of days into an ordered seq of weeks."
  [starting-day-of-week days]
  (->> (partition-weeks starting-day-of-week days)
       (map (partial days->week starting-day-of-week))))


(comment
  (->> (week-04/log-times "time-log.txt")
       (entries->days))

  (->> (week-04/log-times "time-log.txt")
       (entries->days)
       (days->weeks :sunday))
  )

Ep 018: Did I Work Late on Tuesday? Mar 01, 2019

Show notes

Christoph wants to teach filter some vocabulary.

Continuing our discussion of rummaging through the big bag of data.
Mental shift for solving the problem:
- Prior thinking: one function that has to look at all entries
- New thinking: filter out irrelevant entries then reduce on just those
New mentality emphasizes the problem of "picking" over "combining". Once you have the right set of entries, the reduce becomes trivial.
Idea: build up a "vocabulary" of picking.
"You build up the language to the level you want to use it at."
A "predicate" is simply a function that gives you a truth value.
We want to create a set of predicates to use with filter.
By convention, predicates in Clojure end with ?. Eg. some?, contains?, every?
First predicate to create: (spans-midnight? start-timestamp end-timestamp)
Problem: using it is verbose: (filter #(spans-midnight? (:start %) (:end %)) entries)
Better idea: have the predicate take an entry.
The predicate should speak at the level of the items for filter.
- Just take entry: (spans-midnight? entry)
- Usage: (filter spans-midnight? entries)
New question: how many minutes did I work on the weekend?
- New predicate: (weekend? entry)
- Usage: (filter weekend? entries)
"My time in Clojure makes me look at big, long functions and wonder if they should be broken into smaller pieces."
Simplify implementation of weekend? with a simpler predicate: (day-of-week? weekday entry)
Order matters: put weekday first for partial.
Now the weekend? function is a simple or of calls to day-of-week?
Even better: use an "extractor" function (day-of-week entry) that returns the day.
Useful for day-of-week? but also for any other logic that needs to pull out the day.
An "extractor" provides a useful view of the data.
Now a weekday? predicate becomes trivial: (not (weekend? entry))
Key idea: the use of language mirrors how we talk about it.
Not just about decomposition, but about how it reads linguistically.
Can make a predicate for any day of the week with: (partial day-of-week? :sunday), etc.
Use like so: (filter (partial day-of-week? :sunday) entries)
"Partial to parameterize a predicate." (Say that three times fast.)
New question: did I work a long day on Tuesday?
- Won't work to write a predicate at the "entry" level
- Need a new "day" level
Once again, the language hints at the level of abstraction.
Idea: function that "uplevels" by taking a list of entries and producing a list of days
Predicates can work at both levels if entry and day have some consistent structure.
The "structure" (or "data shape") is a consistent use of keys and key paths between abstractions. It is not a "base class".
Eg.: both entry and day have a :date key, so the same day-of-week? predicate works on both.

Related episodes:

Clojure in this episode:

filter
reduce
partial
or

Code sample from this episode:

(ns time.week-04
  (:require
    [time.week-03 :as week-03]
    [java-time :as jt]))


; Helper for loading up time entries

(defn log-times
  [filename]
  (->> (week-03/lines filename)
       (week-03/times)))


; Extractors

(defn day-of-week
  [entry]
  (jt/day-of-week (-> entry :date)))


; Predicates

(defn spans-midnight?
  [entry]
  (not= (jt/local-date (:start entry)) (jt/local-date (:end entry))))

(defn day-of-week?
  [day entry]
  (= (day-of-week entry) (jt/day-of-week day)))

(defn weekend?
  [entry]
  (or (day-of-week? :saturday entry)
      (day-of-week? :sunday entry)))

(defn weekday?
  [entry]
  (not (weekend? entry)))


; Aggregations

(defn total-minutes
  [entries]
  (->> entries
       (map :minutes)
       (reduce +)))


(comment
  (->> (log-times "time-log.txt")
       (filter spans-midnight?))
  (->> (log-times "time-log.txt")
       (filter (partial day-of-week? :wednesday)))
  (->> (log-times "time-log.txt")
       (filter weekend?))
  (->> (log-times "time-log.txt")
       (filter weekday?)
       (total-minutes))
  )

Ep 017: Data, at Your Service Feb 22, 2019

Show notes

Nate finds it easier to get a broad view without a microscope.

After last week's diversion into time math, we are back to the core problem this week.
Now we want a total by date.
Need to refactor the function to return the date in addition to minutes.
"We're letting the code grow up into the problem."
"Let's let the problem pull the code out of us."
First attempt
- Use map to track running totals by day
- As each new entry is encountered, update the total for that day in the map
New complication: Now we want a total for all work on Sundays.
The loop + recur approach is getting complicated!
- More and more concerns all mixed together in one place
- Closely ties the traversal of the data to the processing of the data
Better idea: use reduce. Just write "reducer" functions.
Simplify by ensuring data passed to reduce is already filtered.
"In imperative land, let's take three different dimensions of consideration and shove them all together in this one zone."
Motivating question for a solution: "How is this composable?"
"In Clojure you end up with really small functions because you end up composing them at the end."
Ugly: the reducer for "work on Sundays" still has an if for throwing away data.
Better: add another filter to just pass through Sundays.
Best: minimal work in the reducer. Use map and filter to get the data in shape first.
Imperative thinking: what value do I need to operate on?
Functional thinking: how can I accurately represent the data present in the input?
After you have all the data at hand, you can summarize it however you want!
Why reducers? When you need to operate one step at a time: streaming data, game state, etc.
Clojure's sequence abstraction is powerful and unifying.
"All the functions in the core work on all the data."

Related episodes:

Clojure in this episode:

loop, recur
map, filter, reduce
group-by
if
->, ->>

Code sample from this episode:

(ns time.week-03
  (:require
    [clojure.java.io :as io]
    [clojure.string :as string]
    [java-time :as jt]))


; Functions for parsing out the time format: Fri Feb 08 2019 11:30-13:45

(def timestamp-re #"(\w+\s\w+\s\d+\s\d+)\s+(\d{2}:\d{2})-(\d{2}:\d{2})")

(defn localize [dt tm]
  (jt/zoned-date-time dt tm (jt/zone-id)))

(defn parse-time [time-str]
  (jt/local-time "HH:mm" time-str))

(defn parse-date [date-str]
  (jt/local-date "EEE MMM dd yyyy" date-str))

(defn adjust-for-midnight
  [start end]
  (if (jt/before? end start)
    (jt/plus end (jt/days 1))
    end))

(defn parse
  [line]
  (when-let [[whole dt start end] (re-matches timestamp-re line)]
    (let [date (parse-date dt)
          start (localize date (parse-time start))
          end (adjust-for-midnight start (localize date (parse-time end)))]
      {:date date
       :start start
       :end end
       :minutes (jt/time-between start end :minutes)})))


; How many minutes did I work on each day?

(defn daily-total-minutes
  [times]
  (->> times
       (group-by :date)
       (map (fn [[date entries]] (vector date (reduce + (map :minutes entries)))))
       (into {})))


; How many minutes total did I work on Sundays?

(defn on-sunday?
  [{:keys [date]}]
  (= (jt/day-of-week date) (jt/day-of-week :sunday)))

(defn sunday-minutes
  [times]
  (->> times
       (filter on-sunday?)
       (map :minutes)
       (reduce +)))


; Functions for turning the time log into a sequence of time entries

(defn lines
  [filename]
  (->> (slurp filename)
       (string/split-lines)))

(defn times
  [lines]
  (->> lines
       (map parse)
       (filter some?)))


; Process a time log with the desired summary calculation

(defn summarize
  [filename calc]
  (->> (lines filename)
       (times)
       (calc)))


(comment
  (summarize "time-log.txt" daily-total-minutes)
  (summarize "time-log.txt" sunday-minutes)
  )

Ep 016: When 8 - 1 = 6 Feb 15, 2019

Show notes

Christoph discovers that time creates its own alternate universe.

Continuing our exploration of "literate" time logs
We want a function to turn the timestamps into minutes.
Format: Fri Feb 08 2019 11:30-13:45
Keep it simple: extract times with a regex and use math
- minutes in day = hours * 60 + minutes
- total minutes = end minutes - starting minutes
Problem: What happens when we work past midnight? Negative minutes!
We decided to have only one date, so a time before the starting time must span midnight.
Format only allows for an activity to be <24 hours.
"If we end up doing any activity in our life longer than 24 hours, I think we should that we might have other problems."
Easy Solution: If the end time is before start time, add 24 hours.
"When I get past any sort of simpler math, I just type it into my connected editor REPL because I know Clojure can do it faster than I can."
Now we have a function to get minutes, want to add them all up.
Use loop and recur to iterate through the array and track the sum.
Oh wait, what about Daylight Savings Time?
"We all pretend that time is actually in the future."
If it involves dates and times, we can't just do simple math.
"If I do this the right way, I now have to open a whole new can of worms."
Easy way out: write "doesn't support DST" in the release notes and call it "user error"!
"Any time you have to be careful about something, you're probably doing it wrong."
Use a time API.
"The Clojure library is just a wrapper because nobody wants to reinvent this thing."
Java time is immutable, so that works nicely with functional languages. No cloneing!
Lots of functions in the time API. Which ones do we need?
Our workflow: try out different expressions in a comment block in our connected editor to figure out the specific time functions we need.
"Local" dates and times don't have time zones, but "zoned" dates and times do have them.
Need to create timestamps for accurate math: date + time + zone
In practice, when we use dates and times, they are implicitly connected to a time zone.
Your time zone is your alternate universe: it affects the meaning of your dates and times.
We added support for DST without changing the function signature.
But, how do we add up other totals? Looks like we're going to need to change even more.

Related episodes:

015: Finding the Time

Clojure in this episode:

nil
re-matches
loop, recur

Related projects:

Code sample from this episode:

(ns app.time
  (:require
    [clojure.java.io :as io]
    [java-time :as jt]))

(def timestamp-re #"(\w+\s\w+\s\d+\s\d+)\s+(\d{2}:\d{2})-(\d{2}:\d{2})")

(defn localize [dt tm]
  (jt/zoned-date-time dt tm (jt/zone-id "America/Los_Angeles")))

(defn parse-time [time-str]
  (jt/local-time "HH:mm" time-str))

(defn parse-date [date-str]
  (jt/local-date "EEE MMM dd yyyy" date-str))

(defn parse-for-minutes
  [line]
  (if-let [[whole dt start end] (re-matches timestamp-re line)]
    (let [date (parse-date dt)
          start (localize date (parse-time start))
          end (localize date (parse-time end))]
      (if (jt/before? start end)
        (jt/time-between start end :minutes)
        (jt/time-between start (jt/plus end (jt/days 1)) :minutes)))
    0))

(defn total-time
  [filename]
  (with-open [rdr (io/reader filename)]
    (loop [total 0
           lines (line-seq rdr)]
      (if lines
        (recur (+ total (or (some-> (first lines) parse-for-minutes) 0)) (next lines))
        total))))

Ep 015: Finding the Time Feb 08, 2019

Show notes

Nate spends some time figuring out how to track his time.

New problem: Want to track time using a text file.
Text files are great:
- Keyboard oriented: in terminal, in editor
- Something you can put under revision control
Need: date and time range
Format of the time stamp should be convenient for us to read and edit.
Let the computer do the work to convert the convenient notation to something usable!
Format: Fri Feb 08 2019 11:30-13:45
One timestamp per line. Oh wait, we need a description. Six minutes in and we're already changing the requirements!
What are "literate" text files? "You can mix human words in amongst data the computer will use to make it more understandable for the humans."
How to find the times? Attempt to match the time format within each line.
"It's kind of like an inverse comment. Instead of every line being valid and you have to comment out lines you don't want, if it's in a known format, those are the lines we want and everything else is a comment."
Can use line-seq with clojure.java.io/reader. That uses lazy I/O.
Potential Issue: lazy I/O can defer I/O errors to when you're in the middle of processing the data.
Lazy I/O is less of an issue with files and more of an issue with network sockets.
Another approach: slurp in all the data at once and call split-lines
"Trade a little memory for some safety."
Clojure uses the seq abstraction whichever way you choose. It's Clojure's unifying abstraction.
"Even maps get squeezed into a seq if you push them hard enough."
Next step: figure out which lines are time entry lines and which lines are commentary
Use a regex to match against each string using Clojure's built-in #"..." form.
In other situations, you can use instaparse for grammar-based parsing.
Put timestamps on their own line
- Easier to work with
- Allows for more error detection (in the future)
Use capture groups and re-matches detect the match and extract the parts.
Use when-let to destructure and do something only if it matches
Better yet, make a function time-info that does the match and returns either nil or the string that matched.
For now, just start by using doseq to just go through all the lines and test our matching.
Making sense of the timestamp string is a whole new problem for next time.

Clojure in this episode:

nil
seq
line-seq
clojure.java.io/reader
slurp
clojure.string/split-lines
#"", re-matches
let, when-let
doseq

Related projects:

Instaparse

Code sample from this episode:

(def timestamp-re #"(\w+)\s(\w+)\s(\d+)\s(\d+)\s+(\d{2}):(\d{2})-(\d{2}):(\d{2})")

(with-open [rdr (clojure.java.io/reader "time-log.txt")]
  (doseq [line (line-seq rdr)]
    (when-let [[whole d m dt yr hs ms he me] (re-matches timestamp-re line)]
      (println whole))))

Episode 014: Fiddle With the REPL Feb 01, 2019

Show notes

Christoph gets some work done by fiddling around.

Editor integration is a massive change in how to use the REPL.
Put a comment block right underneath a function you are working on, and invoke the function with test arguments.
Can quickly make changes to code and try it out--all in the same place!
Problem: comment blocks littering the code.
"A good rule of thumb I've found is: when I have to use the world 'careful', it's a big red flag. I'm doing something I should be able to prevent without having to be careful."
Code in comment blocks can be helpful in the future.
"'It's not done yet." What file is ever done? You're always going to come back to it in 6 months."
Easy to skip around between forms you've run before.
"It's like a random access REPL history."
Ah ha moment: make a separate file for the helpful comment blocks.
"[The comment blocks] are like the copilot of my development experience." "They're me helping me!" "You're your own copilot!"
Separate file needed a namespace: the fiddle namespace was born.
"I'm just fiddling around."
Put the "fiddle files" in the dev tree so it only gets loading in development.
"These fiddle files started multiplying, like little helpful rabbits."
Like pages in a notebook centered on a theme. Each "page" is a file:
- a file for remembering how to call certain core functions
- a file for experimenting with date and time conversions
- a file for exploring a new concept
- a file for interacting with new code being developed
Never have to leave to editor. REPL is working behind the scenes.
Unlike REPL history, you can share fiddle files.
Fiddles can help show the developer's mindset during development.
"It's like seeing into the back of your mind of the way you explore this on your own."
Example: exploring a data set
- work on code for parsing a data file
- write different expressions to sift through the data
- e.g. threaded expression to: open the data, parse it, filter, and map
- build up helpers for making sense of the data
- parser and helpers in fiddle are the start of the "real" application code
- write examples with commentary
Example: exploring the working state of the application
- set of utility functions for grabbing information from component
- that set of helpers can be share between developers
- code that pulls out state from specific components all at once
- "Where did this nil come from?"
Example: exploring external APIs
- write simple function for calling the HTTP endpoint
- start calling it with different examples
- can easily see and re-run calls that you've tried
- Slow API? Use (def response (...)) to save a result, and write expressions using response.
"A fiddle file like a scratch pad for new features."
Characteristics of fiddles
- a working set of expressions
- like a notebook with code around a theme
- like a scratch pad for exploring toward a solution
- can be shared
- written context from solving the problem
"The me of the future won't have as much access to the mental state of the me of the present."
Do your future self a favor and write down some of your current mental context.
Could make educational fiddles for new developers in a project.
"Fiddles are a way of capturing process: a way of capturing the craft of making it, not just the outcome."

Related episodes:

Clojure in this episode:

comment
def, defn
->, ->>
filter, map, reduce
select-keys
clojure.pprint/
- pprint
- print-table

Related projects:

Episode 013: Connect the REPL Jan 25, 2019

Show notes

Nate continues to explore the REPL by gluing it to his editor to see what happens.

We revisit Tic-Tac-Toe and change it using our new REPL superpowers.
Let's add request logging to the play endpoint: /play?row=0&col=1.
Need to add log/info calls in handler function for play endpoint: on-play.
Naive Christoph would add logging to the function, recompile, and run the jar.
Less naive Christoph would reload the namespace but still restart the web server.
Crazy idea: paste the function definition into the REPL!
1. Copy the whole defn for on-play
2. Go to the REPL window
3. Switch to the app.main namespace (where we put on-play)
4. Paste the function definition into the REPL
No restarting required! Each time we evalute the defn in the REPL, the new function replaces the old version immediately!
We can iterate on a single function quickly.
Clojure is Lisp, and Lisp is dynamic and isomorphic
- Dynamic: can change definition on the fly, so the new function exists immediately after executing the definition
- Isomorphic: everything is a "form". No special "declaration syntax" vs "expression syntax". A declaration is a form just like any other expression.
"Clojure is dynamic to the end. Every statement is modifying the running application, and because every statement has access to the entire application, all the way down to the definitions, you can redefine on the fly."
Declarations are just ordinary macros (def, defn, defmacro, etc), not special syntax only the compiler understands.
A declaration macro updates the enclosing namespace dynamically on the fly!
"You just replaced the wheels on the car while you were driving."
"I can replace the steering wheel and my hands don't even realize they're using a new steering wheel."
"Running the form is what brings that thing dynamically into existence or replaces the one that was there."
"It's the way you can speed your development up because you can replace these functions on the fly."
Problem: you have to copy and paste between the editor and the REPL.
Idea: use terminal automation to help with the copy + paste!
"Yet another example of solving a problem you probably shouldn't even have."
Pre-Lisp thinking: "The right way to do software is to compile it from grains of sand each and every time, and then run it from the beginning."
Better Idea: Connect the REPL directly to your editor. (See "More reading" for how.)
How it works:
- Move your cursor to the form to evaluate
- Do the "right" key combo (depends on your editor)
- Editor sends the form to the REPL
- REPL evaluates the form and sends the result back
- Editor shows the result
Use comment blocks as a nifty way to provide executable examples within the code.
Your editor is a first-class REPL.

Related episodes:

More reading:

Clojure in this episode:

ns, in-ns, use
*ns*, ns-map
def, defn
comment
let
println
clojure.tools.logging/debug

Related projects:

clojure.tools.logging for application logging
vim-fireplace for VIM connected REPL
Cider for Emacs connected REPL
Ring for HTTP
Compojure for HTTP request routing

Episode 012: Embrace the REPL Jan 18, 2019

Show notes

Christoph complicated development by misunderstanding the REPL.

We go back to the roots of Christoph's experience with Clojure...
How do I run Clojure code?
The REPL is fun for evaluation, but how do you run a "real" program?
Experience in other languages: edit the file, compile, rerun the program
Mentality: get through the edit, compile, run loop as fast as possible!
Autobuilder is the logical end.
"Where's my autobuilder in Clojure?!"
The REPL model is fundamentally different.
"[The REPL] is like cutting the Gordian Knot. You change the problem and that's how you solve it."
"The reason why the problem I wanted solved, wasn't 'solved', is because nobody has that problem because they do things differently here."
Tactic 1: edit, save, restart the REPL
"Restarting the REPL isn't super slow, but let's just say it's not instantaneous."
Tactic 2: edit, save, run (use 'the.namespace :reload) in the REPL
"Now I have a REPL history of use statements!"
Problems:
- forgetting to reload dependent namespaces
- loading dependencies in the wrong order
- old definitions built up
Big Problem: function renames left the old version around, so accidental calls using the old name produced no errors and ran old behavior!
Back to quitting the REPL to clean out the cruft. Ugh!
Discovered clojure.tools.namespace! Reloads everything and cleans out the cruft!
Tactic 3: edit, save, (use '[clojure.tools.namespace.repl :only (refresh)]), (refresh)
Problem: refresh would purge itself!
"I don't know why it took me so long to discover the magical user namespace!"
The REPL will automatically use the user namespace.
"I can put code into a user namespace and it will be there for me."
Christoph would switch namespaces in the REPL without even stopping to wonder what namespace he started in.
"It's like walking out of a door and not even thinking about the fact you're in a building. Oh wait! What room did I just leave?"
Tactic 4: make sure refresh is in the user namespace, edit, save, (refresh)
However, Christoph still didn't understand the REPL.
Just thought the REPL was for:
- reloading the code and restarting
- evaluating snippets of code
- inspecting stuff
Nate's big ah-ha moment: "Not only is my application inspectable, it's fungible. I can change it in place as it's flying along! I don't have to restart it!"
Christoph's hint that there was still more came from seeing comment blocks in code and reading about vim-fireplace.
"There's a new room you can explore. That room is editor integration with the REPL."

Clojure in this episode:

use
clojure.tools.namespace.repl/refresh
clojure.pprint/pprint

Related projects:

Episode 011: The Convention of Configuration Jan 11, 2019

Show notes

Nate is worried about the hardcoded credentials in the code.

It's episode 011 on 01/11. It's binary!
"As a developer, you quickly learn what's under your control and what's not. Very little is under your control."
Don't accidentally check in the credentials.
"We need a configuration management system. Oh wait! That's a totally different problem."
What about putting the configuration into an EDN file?
Let's call it config.edn
Why EDN? Isn't JSON the "one, true format for config information"?
Pros of EDN:
- native to Clojure
- can have comments
- is extensible
Why not environment variables?
Environment variables are great for production, but in development files are better.
Have to restart the REPL to change env variables.
Make a component that reads the config. That will reload config when you reset with component.repl.
Two main considerations:
1. Dynamically reloading configuration during development
2. Plumbing the configuration values through the app
Make a namespace for app.config
"Files are way more fungible than the environment."
We want both options: env for production and a file for dev.
Make two functions in app.config: from-env and from-file.
Use schema to make sure both functions return the same config map.
"A thing I have done before...you decide if it's clever."
Define a default configuration and then merge the config maps with that.
Better yet, from-env handles defaults and we merge the map from dev.edn into that.
Can use environ with Leiningen profiles. Still requires restarting the REPL.
Defaults should make sense for development, so you can just check out and run.
For bad config, we want helpful error messages, not stack traces.
What goes in configuration?
- Items under your control
- Items that vary
Twitter API URL does not vary.
Even if Twitter provided sandbox URLs, those URLs don't vary. The config option should be "production" and "sandbox", not the URL. The wrapper code will select the right URL.
Avoid the nonsense of trying to infer "sandbox" from reading the URL.
"The hallmark of good design is: the less thinking you have to do, the better."
"The more semantic the config, the less thinking you have to do."
It's important to think about the data first, where it comes from, and where it goes.

Clojure in this episode:

merge
try, catch
slurp
clojure.edn/read-string
environ.core/env
component.repl/reset

Related projects:

Code sample from this episode:

(ns app.config
  (:require
    [clojure.edn :as edn]
    [environ.core :refer [env]]))

(defn from-env []
  {:twitter-key (or (env :twitter-key) "")
   :twitter-secret (or (env :twitter-secret) "")
   :initial-tweets (or (env :initial-tweets) 20))

(defn config []
  (merge (from-env)
         (edn/read-string (try (slurp "dev.edn") (catch Throwable e "{}")))))

Episode 010: From Mud to Bricks Jan 04, 2019

Show notes

Christoph can't stand the spaghetti mess in main. Time to refactor.

Main function does too much. We want cohesive pieces that each have one job.
Two part problem:
- Too much in the main loop.
- Main starts and then never stops. Have to exit the repl to restart.
We need 1) separate parts that can be 2) started and stopped.
Main should just focus on the code needed for command line invocation.
Let's get the parts component-ized!
"It's one thing that I've become more sure of in my career is that no application is actually done." "Useful apps only get more complex."
Internal dependencies are different than external dependencies ("libraries").
Many internal dependencies create high coupling throughout the code.
"Once everything starts touching everything you have to understand everything to understand anything."
Like functions use parameters to limit scope, a component is the next level up and uses resource dependences to limit coupling.
Each component implements a clear behavior (interface) and can be a resource to other components.
Can understand component's behavior via its interface (and docs) without reading all the code--just like understanding a function through it's signature and docs.
We use the "Component" library to make components in Clojure--has REPL integration too.
Components to make:
1. web server
2. polling and fetching loop
3. the core.async channel used between them
The Lifecycle interface allows a component to be started and stopped.
- start must return a reference to the "started" state
- stop must return a reference to the "stopped" state
- Gotcha: don't return a nil!
To use Lifecycle, you'll need to make your component a "record".
Two main goals of Component:
1. allow stateful components
2. define dependencies between components
Surprise! Any reference can be a "component" as a non-lifecycle dependency.
Write a function new-system which returns the component "system map"
Mind your names. Make the system map key match a component's dependent field.
"Component is a convenient way of being able to specify all those dependencies in a concise map, so you know this is the intersection of all of my application together."
A component should be able to be understood alone.
"Component is like giving you application parts as function parameters. Just like when making a function, you don't worry about how something gets passed in as a parameter."
You still need to understand each of the parts, but you don't have to worry about where the part came from.
At the top level, you can see all the parts together and how they are connected.
Immutability gives you bulkheads between each of your components so you can safely reason about them separately.
Use component.repl to start and stop the whole system without restarting the REPL!
Need some tooling to keep the main thread from exit. Can use promise, deref and a shutdown handler (see below).
"We can keep each ball of mud it its own little basket so all the mud doesn't ooze together."

Clojure in this episode:

defrecord
promise, deliver
deref, @
component/
- Lifecycle
- system-map
- using
- start, stop
component.repl/
- go
- stop
- reset

Related projects:

Code sample from this episode:

(ns app.main
  (:require
    [clojure.core.async :as async]
    [com.stuartsierra.component :as component]
    [app.component
     [fetcher :as fetcher]
     [web :as web]])
  (:gen-class))

(defn new-system
  []
  (component/system-map
    :new-search-chan (async/chan)
    :web (component/using (web/new-component) [:new-search-chan])
    :fetcher (component/using (fetcher/new-component) [:new-search-chan])

(defn -main
  [& args]
  (let [system (component/start (new-system))
        lock (promise)
        stop (fn []
               (component/stop system)
               (deliver lock :release))]
    (.addShutdownHook (Runtime/getRuntime) (Thread. stop))
    @lock
    (System/exit 0)))

Our TOPPODCAST Picks

Follow Us

Stay Connected

Functional Design in Clojure

Related Podcasts

Links

Stay Connected