Crux New World Assignment: Await transactions

With special thanks to Daniel Mason for help with the story-line and wording.

earth pluto mercury neptune saturn jupiter comet planet a

Introduction

This is the await-tx installment of the Crux tutorial. We shall assume you have a Crux standalone node set up locally and are now comfortable with put operations and basic datalog queries. If this is not the case, you can find all you need to know on Earth, Pluto and Mercury. Click on a destination above to take you there. If it’s not fresh in your mind, make sure to follow the gravity option below to get a quick recap.

Recap

Last time you opted to leave the solar system on the secret space ship 'Oumuamua on an exciting new adventure. You have been traveling for 25 years at near light speed to reach the star system Gilese 667C. This star system is home to intelligent life far superior to our own. With you on the ship is the captain, Ilex, and your new friend Kaarlang. You were put into cryostasis so you could safely withstand the long journey.

Awakening

We begin again in the year 2140.

You open your eyes after what feels like only a moment, your 25 year slumber broken by a loud hiss from the opening of the cryogenic pod door.

You take a minute to acknowledge the cold surroundings. You realize the ship engines have fallen still and assume that you are now in geostationary orbit.

Captain Ilex’s voice comes over the intercom.

Good morning passengers. And hello new world. We’ve reached the planet Kepra-5 of the Gilese 667C star system.

If you could make your way to the space elevator which is waiting to take you to the planet surface, where you will need to go through customs and get a new passport.

We would like to wish you all the best in your future travels.

— Ilex

Finding your way to the space elevator dock, you notice that you feel a lot heavier than usual. You wonder if this is your body being weaker than usual from being in cryostasis for so long, or if this new planet has stronger gravity.

Choose your path:

You want to test this, and are interested to know how the gravity of the planet below compares to planets back in your home solar system. You think about testing this, but you’re in a rush and want to get your passport as fast as possible.

You see it as a good opportunity to refresh yourself with ingestion and queries.

Head over to passport control

Gravity comparison

You spin up a new Crux node and ingest the known data from the solar system.

(def crux
    (crux/start-node
       {:crux.node/topology :crux.standalone/topology
        :crux.node/kv-store "crux.kv.memdb/kv"
        :crux.standalone/event-log-dir "data/eventlog-1"
        :crux.kv/db-dir "data/db-dir"
        :crux.standalone/event-log-kv-store "crux.kv.memdb/kv"}))
(def stats
  [{:body "Sun"
    :type "Star"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 109.3
    :volume 1305700
    :mass 33000
    :gravity 27.9
    :crux.db/id :Sun}
   {:body "Jupiter"
    :type "Gas Giant"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 10.97
    :volume 1321
    :mass 317.83
    :gravity 2.52
    :crux.db/id :Jupiter}
   {:body "Saturn"
    :type "Gas Giant"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius :volume
    :mass :gravity
    :crux.db/id :Saturn}
   {:body "Saturn"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 9.14
    :volume 764
    :mass 95.162
    :gravity 1.065
    :type "planet"
    :crux.db/id :Saturn}
   {:body "Uranus"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 3.981
    :volume 63.1
    :mass 14.536
    :gravity 0.886
    :type "planet"
    :crux.db/id :Uranus}
   {:body "Neptune"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 3.865
    :volume 57.7
    :mass 17.147
    :gravity 1.137
    :type "planet"
    :crux.db/id :Neptune}
   {:body "Earth"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 1
    :volume 1
    :mass 1
    :gravity 1
    :type "planet"
    :crux.db/id :Earth}
   {:body "Venus"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 0.9499
    :volume 0.857
    :mass 0.815
    :gravity 0.905
    :type "planet"
    :crux.db/id :Venus}
   {:body "Mars"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 0.532
    :volume 0.151
    :mass 0.107
    :gravity 0.379
    :type "planet"
    :crux.db/id :Mars}
   {:body "Ganymede"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 0.4135
    :volume 0.0704
    :mass 0.0248
    :gravity 0.146
    :type "moon"
    :crux.db/id :Ganymede}
   {:body "Titan"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 0.4037
    :volume 0.0658
    :mass 0.0225
    :gravity 0.138
    :type "moon"
    :crux.db/id :Titan}
   {:body "Mercury"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 0.3829
    :volume 0.0562
    :mass 0.0553
    :gravity 0.377
    :type "planet"
    :crux.db/id :Mercury}])

(crux/submit-tx crux
                (mapv (fn [stat] [:crux.tx/put stat]) stats))
;=> #:crux.tx{:tx-id 0, :tx-time #inst "2140-03-24T09:01:04.472-00:00"}

The elevator arrives and you drag yourself aboard. As you are carried to the surface, you note the relief as the force of gravity on your body is canceled by the movement of the lift.

As soon as you reach the surface, you waste no time in taking the gravity reading from your iPhone CM. The reading is 1.4g - no wonder you feel sluggish.

You want to check against the data of the other planets on your node, to see how the gravity from this planet compares, so you write a function to add the new planetary data and query it against the other planets:

(crux/submit-tx
 crux
 [[:crux.tx/put
   {:body "Kepra-5"
    :units {:radius "Earth Radius"
            :volume "Earth Volume"
            :mass "Earth Mass"
            :gravity "Standard gravity (g)"}
    :radius 0.6729
    :volume 0.4562
    :mass 0.5653
    :gravity 1.4
    :type "planet"
    :crux.db/id :Kepra-5}]])

(sort
 (crux/q
  (crux/db crux)
  '{:find [g planet]
    :where [[planet :gravity g]]}))
;;=>([0.138 :Titan] [0.146 :Ganymede] [0.377 :Mercury] [0.379 :Mars] [0.886 :Uranus] [0.905 :Venus] [1 :Earth] [1.065 :Saturn] [1.137 :Neptune] [1.4 :Kepra-5] [2.52 :Jupiter] [27.9 :Sun])

Nice, you see that Kepler 5 has gravitational forces stronger than Neptune but weaker than Jupiter.

Now you’ve satisfied your curiosity, you head over to passport control.

Passport Control

You find yourself at passport control where you are told your Crux experience is needed.

Kaarlang has arrived there first and has been chatting to the manager here. As an advanced civilization, they are quite happily using Crux with no issues, but still have a problem with human error. Some employees have been handing out passports before putting the travelers information into Crux. When it gets particularly busy, it’s not uncommon for the employees to forget to go back and put the data in, resulting in unregistered travelers.

Kaarlang has told the manager that you have a background in solving problems using Crux, so has offered them your skills.

Your task is to make a function that ensures no passport is given before the travelers data is successfully ingested into Crux.

(defn ingest-and-query
  [traveler-doc]
  (crux/submit-tx crux [[:crux.tx/put traveler-doc]])
  (crux/q
   (crux/db crux)
   {:find '[n]
    :where '[[e :crux.db/id id]
             [e :passport-number n]]
    :args [{'id (:crux.db/id traveler-doc)}]}))

You test out your function.

(ingest-and-query
 {:crux.db/id :origin-planet/test-traveler
  :chosen-name "Test"
  :given-name "Test Traveler"
  :passport-number (java.util.UUID/randomUUID)
  :stamps []
  :penalties []})
;;=> #{}

This strikes you as peculiar - you received no errors from your Crux node upon submitting, but the ingested traveler doc has not returned a passport number.

You are sure your query and ingest syntax is correct, but to check you try running the query again. This time you get the expected result:

;;=>#{[#uuid "be78684f-559f-42b3-8783-2b78a9422420"]}
The plot thickens.

Confused, you open your trusty Crux manual, skimming through until you hit the page on await-tx:

Blocks until the node has indexed a transaction that is at or past the supplied tx. Will throw on timeout. Returns the most recent tx indexed by the node.

— Crux manual
Read More

Of course. Submit operations in Crux are asynchronous - your query did not return the new data as it had not yet been indexed into Crux. You decide to rewrite your function using await-tx:

(defn ingest-and-query
  "Ingests the given travelers document into Crux, returns the passport
  number once the transaction is complete."
  [traveler-doc]
  (crux/await-tx crux
                 (crux/submit-tx crux [[:crux.tx/put traveler-doc]]))
  (crux/q
   (crux/db crux)
   {:find '[n]
    :where '[[e :crux.db/id id]
             [e :passport-number n]]
    :args [{'id (:crux.db/id traveler-doc)}]}))

You run the function again, Changing the traveler-doc so you can see if it’s worked. This time you receive the following:

(ingest-and-query
 {:crux.db/id :origin-planet/new-test-traveler
  :chosen-name "Testy"
  :given-name "Test Traveler"
  :passport-number (java.util.UUID/randomUUID)
  :stamps []
  :penalties []})
;;=> #{[#uuid "9ef9514d-448b-44a5-a2c8-682afce49568"]}
Caution

Crux is fundamentally asynchronous: you submit a transaction to the central transaction log - then, later, each individual Crux node reads the transaction from the log and indexes it. If you submit a transaction and then run a query without explicitly waiting for the node to have indexed the transaction, you’re not guaranteed that your query will reflect your recent transaction. On a small use-case, you might get lucky - but, if you want to reliably read your writes, use await-tx. If you’re ingesting a large batch of data though, calling await-tx after every transaction will slow the process significantly - you only need to await the final transaction to know that all of the preceding transactions are available.

You show this to the manager at the passport control office. They are happy that this will work.

Thank you, this will save us so much time in dealing with missing traveler information. As a token of our gratitude, we would like to grant you with free entry to the planet.

— Passport control manager

You graciously accept. For the passport you must provide your origin planet, name and also a chosen name. Many people come to Kepra-5 in order to start fresh, so your chosen name can be anything you wish.

Once chosen you can not easily change it, so you think carefully.

(ingest-and-query
 {:crux.db/id :earth/ioelena
  :chosen-name "Ioelena"
  :given-name "Johanna"
  :passport-number (java.util.UUID/randomUUID)
  :stamps []
  :penalties []})
;;=> #{[#uuid "6486f9e1-aff2-444c-9eb8-b6f2a07c75ff"]}

New name, new you.

Now that you are in the Gilese 667C you must keep your passport up to date wherever you travel. You take a note of your passport number and put it somewhere safe.

Through customs, you are now free to explore the exciting new planet. Taking in your surroundings, you see a hostel nearby. You head over. Although you’ve been in cryostasis for such a long time, the new gravity has made you very tired.

Read more
aws   crux  
May 12, 2020
The Crux 'Document Store' Making it possible to store your documents in S3 + more
jms
by James Henderson
crux   clojure  
Mar 04, 2020
Metrics metrics and more metrics Know what your nodes are up to
tmt
by Tom Taylor
crux   clojure  
Jan 06, 2020
Crux Development Diary Shoring Up Foundations
jdt
by Jeremy Taylor