Metrics metrics and more metrics
Know what your nodes are up to

In the new 1.7 alpha release of Crux there is a LOT of content. Some bug fixes, some refactoring and some new features, including the addition of metrics!

Metrics give you information on how long your queries are taking, how fast/many documents are being ingested and how large your local store is (amongst other things).

In the first release of this feature we have included metrics for several key components:

  • query engine

  • indexer

  • RocksDB

Dropwizard’s metrics library is used to create a registry that can then be passed around to reporters that expose the inner metrics. Currently we provide 5 components to expose metrics:

We’ve made metrics as easy as possible to add to existing Crux nodes. Below is an example node with a RocksDB backend which displays prometheus metrics to the default port 8080. RocksDB metrics have also been included here with the kv-store item in the topology map.

 {:crux.node/topology ['crux.standalone/topology
  :crux.kv/db-dir "data/db-dir-1"
  :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv
  :crux.standalone/event-log-dir "data/eventlog-1"})

Besides pointing at the endpoint in your prometheus config, that’s all that’s required.

prometheus example

In 1.7 we’ve changed the way kv-store is specified. It is now done in the topology map, although the old method of defining it as an argument still works for the moment.

Cloudwatch is just as easy.

  {:crux.node/topology ['crux.standalone/topology
                        ;; this time without RocksDB metrics
   :crux.kv/db-dir "data/db-dir-1"
   :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv
   :crux.standalone/event-log-dir "data/eventlog-1"})

When run on ECS the AWS api is able to detect the relevant credentials to upload metrics to cloudwatch. If running locally we also provide options to specify the desired region to upload and also to provide valid credentials.

cloudwatch example

In the most recent Crux showcase I demonstrated this in action working with our new benchmark environment running on AWS (Skip to 11 mins for metrics).

For more configurations and options take a look at the docs.

If you need any more help, give us a shout!

Read more
aws   crux  
May 12, 2020
The Crux 'Document Store' Making it possible to store your documents in S3 + more
by James Henderson
crux   clojure  
Apr 06, 2020
by Johanna Antonelli
crux   clojure  
Jan 06, 2020
Crux Development Diary Shoring Up Foundations
by Jeremy Taylor