Prague Astronomical Clock

Source

The log is an interesting invention. It began with 16th century ships – surrounded with water, no land in sight, no sense of direction, and no sense of speed. What did you have to compare your speed with anyway?

The sailors were ingenious. They used a log - a real piece of wood - attached to a string. They’d throw it in the water and use it to measure speed1. Then they’d track this speed over time, in a logbook.

That’s the origin story of log books, now known as logs.

Over time, the logs grew to track even more information. The sailors felt that tracking their speed wasn’t enough. They filled gaps in their measurements - the weather, incidents, and the last time someone cleaned the toilet.

The idea? If you have a log, you can figure out what went wrong. You can figure out where you made mistakes. If the ship crashes, you know the last state it was in, thanks to the logs.

This concept has creeped into almost everything we build.

Airplanes have black boxes for the same purpose. Like every episode of Air Crash Investigation says - you’re lost without the blackbox. They’re more durable than traditional log books, since they have to survive the plane crash.

Sports-men and women have game recordings. This is possibly the best kind of log - you record everything that happened, or everything that’s visible. They go back to it, see the recordings, figure out where they went wrong. They also use their opponents logs to figure out a winning strategy.

Computer systems do this all the time. Every respectable software ever built has a log. Be it a nuclear reactor controller, your facebook account or a bank transaction management system.


Purpose

I think logs serve two main purposes. (1) Crash resolution and (2) Behaviour detection.

Crash resolution

When a system crashes, you get into the logs to figure out the state of the system. The logs, if they are designed well and tracking the right things, tell you what was going on with the system and where the system broke down. This helps you uncover bugs in your program, figure out what’s wrong with the Boeing 737 Max and make things better.

If you can’t figure out what went wrong, you can’t fix it. You can’t treat your problems like blackboxes. Throwing a solution at problems and hoping it sticks seldom works.

If you have to be on the plane to figure out why it crashed, you can’t fix it.

Behaviour detection

When you’ve fixed all the major bugs, when your airplanes are flying without crashing, you enter the sweet world of optimization. You’ve got a working system, and now you want to make it better.

Logs help figure out why your system is being used. They help figure out usage patterns. They help figure out why users leave your system.

For example, let’s take Facebook. Facebook tracks everything you do on Facebook2 (and maybe few external sites too). They figure out how you use the service, what makes you stick, what makes you leave - by logging all your actions and analysing them later. They know what you’re seeing on the feed. They know how fast you’re scrolling, so they can figure out what interests you and what doesn’t. You’ll slow down when you see something interesting.

Then, using all this data, they optimise their algorithms to show you more of what they think you’d like. It’s a beautiful process, all empowered by logging.

I’ll leave the moral and political debates out of the technological beauty of things.

The most important system of all

Every important system has logs since they’re necessary to improve the system over time.
That brings me to the most important system I know. One that we never designed - but use all the time.

The human body.

Why don’t we have a log for the most important system ever?

Before we get to that question, I think we need to explore more about what happens with the human body.

There is no stop button. There is no start button. There is not restart button.

There are only crashes and then recovery.3

I don’t mean accidents, those are the extreme crashes.

I mean the micro-crashes. Our body crashes and recovers on a weekly, if not daily basis. What do these look like?

Remember the headache from a while ago? That’s a micro-crash.

Remember not being able to sleep, no matter what you did? That’s a micro-crash.

Remember not being able to focus on anything - and ending up Netflixing all day? That’s a micro-crash.

Continuous micro-crashes can turn into bigger crashes. If you keep cracking the egg, the yolk would yellow your apron soon.

That’s the headaches turning into a migraine.

The sleep-irritation turning into insomnia.

And not being able to focus turning into chronic procrastination. ( I wouldn’t say ADD because we don’t know the cause, yet)

Sure, you might get out of the funk, and life continues like normal.

“They are just phases”, you brush them off and move on.

They way I see it, they are bugs brought out by system behaviour. We don’t brush them off in computer systems - we figure out exactly what went wrong.

True, figuring things out in the human system is much more difficult - we never designed this system, so we aren’t sure how it works. I don’t think that’s a good enough excuse to not solve the bugs we can solve.

Not picking the low hanging fruit, simply because the highest fruit is out of reach is a logical fallacy. But perhaps, humans aren’t rational.

Designing the human log

How do you design the human log?

With code, you can usually recreate the conditions for the crash. The computer logs give us enough idea - they log all the important variables.

The human body doesn’t work like that. It crashes, but you’ll have a terrible time trying to recreate the conditions - there are too many variables and we don’t know the important ones. Logging them all is prohibitive.

This is a difficult problem with logging - you need to figure out the important variables to log. Bonus points if they are easy to log.

Well, we didn’t design this system - so we probably don’t know what’s important.

In this case, I think the best way to proceed is an evolutionary algorithm. You begin logging something simple, like your weight everyday. As you keep growing, you crash. Then you go back to your logs, and see if you can figure out the cause of the crash. If yes, amazing. If no, you figure out what data is missing. You figure out if the crash is important to fix. If yes, you add the new data to your log and continue.

This might make you uneasy. “Come on, Neil! If I start logging now, that doesn’t help me fix the crash in the past! I didn’t have the data then!”

True, but you’re going to have the data the next time it happens. Remember, before this time - you were living all your life without logs. Also remember, it’s a long term game. You aren’t going to fix all your errors today - it’s a lifelong pursuit.

For example, let’s say headaches & migraines are constant trouble. I’d start with logging days on which I get headaches.

A headache on day 1, a more pronounced one on day 7, and a migraine on day 15. It can be pretty hard to figure out they were connected, if you’re not zoomed out enough, or not tracking your headaches.

When I write it down like this, it seems pretty obvious. “Day 1, Day 7, Day 15! Only an idiot could miss that!”. But do you remember the last two times you had a headache?

Structure

Here are some ideas to help you begin.

Your log can have:

  • Daily weight
  • Food intake
  • Bowel movements
  • Any persistent problems (like headaches above)
  • Workout details
  • Meditation tracking
  • What made you happy? - to figure out what you should do more of
  • What made you sad? - to figure out what you should do less of

You will change, you will evolve - and your logs will evolve with you.

Extending the log

There’s still one problem. Despite knowing you had headaches on Day 1, 7 and 15, you may not remember the conditions around you that led to those headaches. What can you do about it? Log, duh!

Except, it’s not that easy. Here’s where the tradeoff comes in - recording your entire day and your thoughts versus noting down the important things that happened.

One takes prohibitively too much effort to store and to search, and with the other you have no idea what’s important.

The compromise I find works best, again - is the evolutionary algorithm.

You track the symptoms and the environment both. The symptoms would be the headache, and the environment would be a short description of what you do everyday. I’m not afraid to log things that seem meaningless - because, I don’t know what’s important.

The next time a headache pattern emerges, and I can’t figure out what in my environment caused it - I know I can evolve my logging: Focus more on what happened before the crash. ( Hopefully, I’d be logging the time of headaches. If not, that’s step 1 of evolution! )

Jim Collins uses a Bug Book to track his day. He’d pretend he is a bug to be observed - so he observes everything he does, irrespective of what bits he thinks are important or not. He also rates his days between -2, -1, 0, 1 and 2. This way, he has a mapping between his best days and what he did those days.

Data Driven journaling

At this point, you might be thinking - “Eh, wait. How is this different from journaling?”. I think journaling can be a part of your log - it can be the environment log, with a few caveats.

“My aim is to put down on paper what I see and what I feel in the best and simplest way.” - Ernest Hemingway

It’s your journal, it’s your space, and you can use it however you want it.

There’s some bullshit advice on how to write a journal. Let’s get that out of the way first.

If it doesn’t feel useful - you’re going to stop journaling. Off the bat, most journaling prompts are like that.

  1. I couldn’t imagine living without…
  2. If I could talk to my teenage self, the one thing I would say is…
  3. I feel most energized when…
  4. Make a list of everything that inspires you — from books to websites to quotes to people to paintings to stores to the stars.
  5. Make a list of 30 things that make you smile.

These are great questions to answer. But do these “journaling experts” expect me to answer these questions … by writing them down? What the? That’s the best they got? “Just think hard about it for 10 minutes?”

This feels like bullshit to me. Sure, I can think about this today, make something up, write it down - and arguably it makes me more self aware. Doing this everyday is a painful exercise. I’m done.

I find this pretty useless because it’s missing necessary data.

Most journaling advice is like this, and the reason is short term thinking. People want to see the benefits now. If they don’t, they think it’s useless.

So, all advice is driven towards answering these interesting questions. All by thinking hard for 10 minutes. What it’s missing is the structure that comes from journaling consistently.

How would you figure these questions out? Apart from thinking hard? 4

You collect data. Every day is a data point. I call this data driven journaling.

Let’s say, the big question about my life that I want to answer is what I can’t live without. Not what I can’t imagine living without ( how does that help anyway? )

Thus, data collection every day morphs into writing in my journal everyday answering one question: What couldn’t I live without today?

Over time, I’d collect enough data to answer this question knowledgeably. That’s real self awareness.

You don’t become self aware just by thinking hard. You get there by matching your thinking to your reality, and the link is data.

And, like us, our journaling evolves too. It can be a powerful element of our log, helping us answer some of the toughest questions about ourselves.

Here’s a list of questions worth answering. Each one can take upto a year.

  • When do you feel most energized? This becomes logging your energy peaks during the day. Come back to it in a month, and you’ll see some patterns. If nothing, get more data. Come back to it in 3 months time.

Here’s something similar I’ve been doing with my weight.

Yes, that’s a red alert for me. Something’s going wrong. Or maybe it’s because I started lifting.

  • What makes you happy?

The previous question had a discrete log - you only had a number to track, which becomes easy to visualise.

A question like this morphs into asking, What made me happy today?

It’s going to be difficult to visualise it as a chart. So, with questions like these, we go back to our subjective lens.

Come back to your answers in a month, read through them, and that should give you some idea about what makes you happy.

Similarily, you can transform the below questions.

  • What makes you sad?

  • What inspires you?

  • What are you good at?

To end the journaling section, one disclaimer. This isn’t about bashing existing journaling methods, prompts and techniques. This is about figuring out that some questions are answered better by data, and some other questions are answered better by thinking.

It’s worth journaling about both. Above, I’ve focused on the questions that are easier to answer with data. This is what I feel has been missing from most journaling advice.

Here are some questions where I think daily data logs might not help enough. But it’s still worth thinking about them. On the flip side, I don’t think you’d find an answer in 10 minutes.

  • Who do you want to become?

  • Why do you wake up every morning?

  • How is your past shaping your present?

  • How can your past shape your future?

Your log is a platform. You can plug into it whatever plugins you want. You can extend it to include decision journals, cognitive journals, bucket lists, and data driven journaling.

Yes, I know it feels like a complex system. But remember, think long term. Start small. Do one thing right. And continue evolving.

That’s how all complex systems grow.

Effective tracking

The next, and final question is, How do I track all this effectively?

The answer depends on your goals, personal preferences and what you want to track.

Here’s a non exhaustive list of tracking mechanisms.

  • Pen and Paper
  • Desktop & Mobile

    • Excel. Easy to graph. Works well with numeric data points.

    • Evernote. Electronic version of the pen and paper log.

    • Trello. Works well for tracking non numeric items.

Workflow

Here’s how I do it. I’m sharing this to inspire you and give you some ideas. Again, warning: Don’t try copying this completely. Evolve your own - you’d appreciate it much more.

Every morning, I meditate. Right afterwards, I fill in my evolved version of the minimalistic journal - which tracks most of my numeric data points. I do this on the right side of my note-book. I’m filling in most of yesterdays and some of todays data - like what time I slept yesterday and what time I woke up today. The left side has other important stuff that comes up during meditation - things I want to think about. As well as things I want to remember everyday - like the mental model of the month.. I also track habits I’m developing (see bonus section below).

During the day, I log interesting ideas I come across in my Evernote daily log.

Every evening, I collate all the ideas in Evernote. Then I read yesterdays ideas. In an excel sheet, I do a short description of everyday (tracking environment variables). I log the number of hours spent on things that matter to me. And I give my days a rating. This comes from Jim Collins’ system. I loved it, so I adapted it to mine.

Whenever I want to analyse a data point, I go into excel and feed my pen and paper minimalistic journal into an excel sheet, and get out a chart like the one you saw above. I don’t directly input all the data into excel because I don’t want to open my laptop first thing in the morning. It creates distractions. My mind is fresh and high on energy in the morning. I want to use that energy to do something meaningful, and not waste it on routine tasks like bathing, cleaning, and cooking. You might be an evening person, so this will be reversed for you.

Bonus: Building habits

The log is great for behaviour detection (data driven journaling) and crash resolution (log things that matter and the environment in which they occur). But, it can also be used for something powerful - building habits.

Since you come to your log everyday, it acts as an accounting mechanism. It holds you accountable. If you were to start tracking a habit you want to form, like working out - you’d have to put a cross in your log every time you didn’t do it. That solves one reason people aren’t able to build habits, or keep resolutions: They simply forget about them.

This method feeds into Jerry Seinfields technique, “Don’t break the chain”

The beauty of it? You don’t need an extra device to track your chains. It’s already inbuilt into your log! It’s a new plugin your log can seamlessly handle.

It takes on average 60 days to make a habit out of something5. I have two whitespaces reserved in my daily log for habits. Not more, because I don’t want to be building more habits at the same time. Otherwise, I spread myself way too thin. Every 456 days (or whenever I decide this habit isn’t worth it) I can remove the habit from the log - it has become a habit (or crashed and burned) - and clear space for a new habit.

I keep it fun - instead of ticks and crosses for whether I did it, I sketch icons representing the habits. It usually ends up being initials, though.

I’ve been vague about how exactly I design my log. That’s deliberate. I’m sure you’ve already figured out why – you need to evolve your own system that matches your goals, to stick with it.

Drop a comment / email if you have questions about my system, or are just looking for more ideas.

Conclusion

It can seem overwhelming. Taken all together, it is. But remember, you don’t have to begin with everything. I don’t recommend that. Instead, begin with just one thing. Figure out what medium works for you. Figure out what time works for you. Once you get comfortable, you can keep going as is, or add more interesting things to your log. It’s not going to happen instantly but the time investment will be worth your while.

After all, it’s a lifelong habit.

You’ll become more self aware, more accountable and most importantly, with a system that can always bring you back to your efficient, productive and happy self - if that’s what you choose to optimise for.

  1. Distance travelled by string over time taken. See more: Logbook ↩

  2. But it’s anonymised so you’re a statistic not a person. This complys with regulations too. (I’m not an insider so I don’t know in Facebooks’ specific case, but this is the general industry practice) ↩

  3. Crash only thinking ↩

  4. I think even thinking hard is questionable. We don’t know what we want - and what we think we want may not make us as happy as we thought. See Miswanting ↩

  5. The study says it takes about 66 days on average to build a habit. It took people anywhere between 18 and 254 days to turn behaviour automatic enough to call it a habit. ↩

  6. When talking about just a single person, like you or me - averages are meaningless. You could be on either end of the spectrum. ↩